Variable compression, de-resolution and recovery of medical images based on diagnostic and therapeutic relevance

Abstract

An image processing platform obtains an input image of a sample and globally reduces resolution of the input image into a reduced resolution image. After global resolution reduction of the input image, the image processing platform globally increases the resolution of the reduced resolution image into an increased resolution image using a generative adversarial network (GAN) model, and classifies a plurality of regions of the reduced resolution image based on cellular morphology and / or diagnostic relevance, the GAN model is configured to reconstruct an image containing features corresponding to the sample. The image processing platform transmits the upresolution image to a communication network for delivery to a client device.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 377,005, filed on September 23, 2022, entitled "Variable Compression, De-Resolution, and Restoration of a Medical Image Based Upon Diagnostic and Therapeutic Relevance", which is hereby incorporated by reference in its entirety. Technical Field

[0003] This application relates to the field of digital pathology, namely systems and methods for acquiring and processing electronic pathology slide images for their acquisition, encoding, compression, storage, transmission, reconstruction, display, navigation, evaluation, diagnosis, and annotation to be included in a resulting pathology report. Background Art

[0004] In the field of pathology, a mounted section of a pathological specimen can be converted into a whole-slide digital image for subsequent use in a computer-based diagnostic workflow. Examples of systems for performing such conversion are included in Bacus (US 8,625,920) and Soenksen (US 6,711,283).

[0005] However, after two decades of technological progress and slow but gradual market adoption, most pathologists still use optical microscopes rather than scanned slide images as their primary diagnostic workflow. One reason is that whole-slide digital images are very large, up to 4.8 gigapixels. By analogy to a 4K computer monitor with a resolution of 3840x2160 pixels, a typical whole-slide digital image can represent the detailed visual content of approximately 500 full screens. Compared to being able to instantaneously move a slide with one's fingertips, such large image files typically load slowly and are cumbersome to navigate. Pathologists generally prefer to physically touch the specimen slide with their own hands. Given the importance of the diagnostic outcome, it has a direct and profound connection to the patient.

[0006] In addition to the problems associated with the capture and storage of whole-slide digital images, the diagnostic workflow involved in viewing such images can be inefficient and unintuitive when compared to the experience of observing a physical specimen slide through a microscope. Current software solutions for digital diagnostic workflows typically follow the conventions of computerized user interface design, such as click boxes, radio buttons, scroll bars, and text boxes.

[0007] To improve the value proposition, many of these software-based solutions now incorporate artificial intelligence (A.I.) to assist in the diagnostic process by identifying cells, patterns, or features. However, most pathologists have not yet chosen to entrust third-party software solutions to handle such extremely important issues. Additionally, depending on their characteristics and capacity, the cost of slide scanning instruments ranges from $20k to $300k. For most pathologists in the United States, this is a significant portion of their annual income. Senior pathologists nearing retirement (within a decade) would face diminishing returns if they were to make such a costly transformation to their established careers. Summary of the Invention

[0008] The present disclosure addresses the above needs by identifying, classifying, and spatially mapping known cellular, intracellular, and extracellular morphologies to assist in the subsequent reconstruction and super-resolution of high-resolution images. The methods described herein further evaluate, quantify, and spatially map and define regions of the original image to include two or more levels of diagnostic relevance. These regions are then each assigned an optimal compression level, typically inversely proportional to the diagnostic relevance. This takes advantage of the file size benefits of the inherent fact that normal cells and regular cells are generally healthy and have lower diagnostic relevance in the diagnosis of one or more pathologies. This same regularity and / or normality of healthy tissue regions and cell morphologies makes them more suitable for A.I.-assisted super-resolution circuits, such as generative adversarial networks (GANs). By specifying a priori dedicated GANs for each subtype of tissue or cell or state or morphology, the methods described herein can more accurately reproduce the original high-resolution image from its downsampled pixel-shifted progeny.

[0009] An advantage of the methods described herein is that the file size for storage and transmission is reduced because the increasingly powerful computing power of mobile phone processors, along with neural network software and firmware implementations, can be used to reconstruct high-resolution images in real time as needed. Additionally, this advantage exists beyond the compression achieved by file formats such as JPEG-2000 (whose lowest compression settings are close to lossless methods), despite the reduction in the total number of pixels to be stored. Brief Description of the Drawings

[0010] To better understand the various described embodiments, reference should be made to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, like reference numerals refer to corresponding parts.

[0011] Figure 1 A diagram of an image processing environment configured to apply correlation-based variable compression, de-resolution, and restoration of medical images according to some embodiments.

[0012] Figures 2A - 2C A diagram of a medical image acquisition process according to some embodiments.

[0013] Figure 3 Diagram of the α layer and metadata generation process according to some embodiments.

[0014] Figures 4A - 4C Diagram of the downsampling, compression, and pre-verification process according to some embodiments.

[0015] Figure 5 Diagram of the super-resolution, decompression, and display process according to some embodiments.

[0016] Figure 6 Diagram of the pixel shift process according to some embodiments.

[0017] Figure 7 Flowchart of the process for correlation-based variable compression, downsampling, and restoration of medical images according to some embodiments.

[0018] Figures 8A - 8B Diagram of a system for displaying and interacting with medical images according to some embodiments.

[0019] Figures 9A - 9D Diagram of a device configured to interact with medical images according to some embodiments.

[0020] Figures 10A - 10B Diagram of a device configured to interact with medical images according to some embodiments.

[0021] Figure 11 Diagram depicting multiple usage modes of a device configured to interact with medical images according to some embodiments.

[0022] Figures 12A - 12C Diagram depicting the usage mode of a device configured to interact with medical images according to some embodiments.

[0023] Figure 13 Diagram of a system for collaborative interaction with medical images according to some embodiments.

[0024] Figure 14 Diagram of a system for interacting with medical images using facial expressions according to some embodiments.

[0025] Figure 15 Diagram of a system for interacting with medical images using facial expressions according to some embodiments.

[0026] Figure 16 Diagram of a system for interacting with medical images using facial expressions according to some embodiments. Detailed embodiments

[0027] The present disclosure describes systems and methods for using machine vision and A.I. to better perform image compression, transmission, super-resolution rendering, and subsequent diagnostic workflows regarding medical images. By pre-identifying, mapping, and indexing diagnostic-related tissue features and image regions relative to a library of known cells, tissue types, and features, the systems and methods described herein are subsequently able to reliably reconstruct the original compressed images and navigate the images more quickly during the diagnostic workflow. Using one or more peripheral control devices, the systems and methods described herein use diagnostic feature indexing to enable a pathologist to capture each successive diagnostic-related tissue feature or image region.

[0028] The present disclosure aims to isolate and extract diagnostic-related (i.e., suspicious and / or potentially cancerous) cells, tissues, and image regions, thereby providing them with less lossy compression than the remaining diagnostically less-related cells, tissues, and image regions, which by definition are more normal and regular (individually and collectively) in their state, properties, and morphology. Thus, such normal and regular image content is more suitable for various types of higher compression levels, such as wavelets, token libraries such as LZW, color compression, pixel-shift super-resolution, run-length encoding, etc.

[0029] Generally, less-related cell and tissue features will be more normal or regular and thus more predictable and suitable for favorable down-resolution and compression, while more-related cells and tissues can be extracted and maintained at their original or near-original resolution, where minimal or no compression is applied to ensure the highest level of accuracy for those image regions that are most important for diagnosing medical problems and determining courses of treatment. By compressing the less-related image regions and maintaining the more-related image regions, the systems and methods described herein achieve an optimal balance between a more feasible image file size and pathologist confidence.

[0030] Figure 1 FIG. for an image processing environment 100 configured to apply correlation-based variable compression, down-resolution, and restoration of medical images according to some embodiments.

[0031] Specifically, environment 100 shows an electronic network 130 that can be connected to a server system 102, and the server system includes hosting partners such as hospitals, laboratories, and / or doctor's offices. For example, a physician server, a hospital server, a clinical trial server, a research laboratory server, and / or a laboratory information system can each be connected to the electronic network 130 through one or more computers, servers, and / or handheld mobile devices, such as the Internet. The server system 102 includes a processing device configured to implement an image processing platform 110, and the image processing platform includes an image acquisition module 112, an image mapping / classifier module 114, an image compression module 116, and a DICOM compliance engine 118, which are each discussed in more detail below.

[0032] The environment 100 facilitates efficient and trusted remote access to medical images between a pathologist device 150 and a collaborator device 160. The devices 150 / 160 are electronic devices associated with corresponding users, sometimes referred to as client devices. The devices 150 / 160 can include, but are not limited to, smartphones, tablet computers, laptop computers, desktop computers, smart cards, voice-assisted devices, or other technologies (e.g., hardware-software combinations) known or yet to be discovered that have a structure and / or capabilities similar to those of the mobile devices or computer peripherals described herein. In some embodiments, the devices 150 / 160 can include peripheral devices, such as a dial configured to navigate regions of a medical image, the features of which are disclosed in more detail below. The devices 150 / 160 are communicatively coupled to the server system 102 using communication capabilities (e.g., modems, transceivers, radios, etc.) for communication through the network 130. The AI partner device 140 can approximate the functions of the devices 150 / 160 or otherwise assist in certain aspects of image analysis, as described in more detail below.

[0033] Server system 102 is communicatively coupled to devices 140 - 160 via one or more communication networks 130. Communication network 130 is configured to convey communications (messages, signals, transmissions, etc.). The communications include various types of information and / or instructions, including but not limited to data, commands, bits, symbols, voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, and / or any combination thereof. Communication network 130 uses one or more communication protocols, such as any one of Wi-Fi, Bluetooth, Bluetooth Low Energy (BLE), Near Field Communication (NFC), Ultra-Wideband (UWB), Radio Frequency Identification (RFID), Infrared Wireless, Inductive Wireless, ZigBee, Z-Wave, 6LoWPAN, Thread, 4G, 5G, etc. Such protocols can be used to send and receive communications using one or more transmitters, receivers, or transceivers. For example, hardwired communications (e.g., wired serial communications) can use techniques suitable for hardwired communications, short-range communications (e.g., Bluetooth) can use techniques suitable for short-range communications, and long-range communications (e.g., GSM, CDMA, Wi-Fi, Wide Area Network (WAN), Local Area Network (LAN), etc.) can use techniques suitable for long-distance remote communications (e.g., via the Internet). Generally, communication network 130 can include or otherwise use any known or yet-to-be-discovered wired or wireless communication technology.

[0034] Server system 102 (e.g., including one or more hospital servers, clinical trial servers, research laboratory servers, and / or laboratory information systems) can create or otherwise obtain digital images of cytology samples, histopathology samples, sections of cytology samples, sections of histopathology samples, or any combination thereof for one or more patients. Server system 102 can also obtain patient-specific information, such as any combination of age, medical history, cancer treatment history, family history, past biopsy or cytology information, etc. Server system 102 processes the digital slide images and transmits the processed images to devices 150 / 160 via network 130. Server system 102 can include one or more storage devices 120 for storing the foregoing images and processed image data. Server system 102 can also include processing devices for processing the images and data stored in storage device 120, such as one or more processors, each processor including one or more processing cores. Server system 102 can further include one or more machine learning tools or capabilities, the features of which are described in more detail below. Additionally or alternatively, the present disclosure (or portions of the systems and methods of the present disclosure) can be executed on a local processing device (e.g., a laptop computer).

[0035] The server system 102 can be implemented on a distributed network of one or more independent data processing devices or computers. In some embodiments, the server system 102 also uses various virtual appliances and / or services of a third-party service provider (e.g., a third-party cloud service provider) to provide the underlying computing resources and / or infrastructure resources of the server system 102. In some embodiments, the server system 102 includes, but is not limited to, a handheld computer, a tablet computer, a laptop computer, a desktop computer, or a combination of any two or more of these data processing devices or other data processing devices.

[0036] The storage device 120 includes a non-transitory computer-readable storage medium, such as volatile memory (e.g., one or more random access memory devices) and / or non-volatile memory (e.g., one or more flash memory devices, disk storage devices, optical disk storage devices, or other non-volatile solid-state storage devices). The memory may include one or more storage devices located remote from the processor. The memory stores programs (described herein as modules and corresponding to instruction sets) that, when executed by the processor, cause the server system 102 to perform the functions described herein. The modules (e.g., 112 - 118) and data described herein need not be implemented as separate programs, processes, modules, or data structures. Thus, various subsets of these modules and data may be combined or otherwise rearranged in various embodiments.

[0037] Figure 2A FIG. for a medical image acquisition process 200 according to some embodiments. In some embodiments, one or more source images 202, 204 are scanned at a resolution corresponding to the capabilities of the scanning camera used to scan the source images (referred to as the native scanning resolution). Although the figure shows two images 202 and 204, other embodiments may include one image or more than two images (e.g., 3 images, 4 images, etc.). For embodiments in which multiple images are scanned, the multiple images may be pixel-shifted images. Specifically, each successive image may be offset by at least one pixel in at least one direction. This can significantly reduce the scanning time and can cut the cost of the scanner camera in half.

[0038] The source images 202 and 204 can be obtained by the image acquisition module 112 of the image processing platform 110( Figure 1 ). That is, the source images can be obtained locally at the hospital where the server system 102 is located, or remotely from the hospital where the server system 102 is located.

[0039] Each pixel-shifted source image 202, 204 is a component of the final source image 210. In other words, the pixel-shifted source images 202, 204 are combined to form the final source image 210. In some embodiments, the final source image 210 may not be incident on the image sensor of the scanning camera because it is a composite image composed of the best pixels within a non-planar volumetric z-stack or a series of tiles stitched together that exceed the resolution of the original image sensor.

[0040] The final source image 210 includes one or more regions (e.g., regions 220a - 220e), commonly referred to as regions of interest, or more specifically as diagnosis-related regions or treatment-related regions. These regions are diagnosis-related if they contain, for example, cytomorphology related to the diagnosis of the patient associated with the tissue in image 210. Similarly, these regions are treatment-related if they contain, for example, features related to the treatment outcome of the patient associated with the tissue in image 210. Throughout this disclosure, "diagnostic relevance" can refer to diagnostic relevance and / or treatment relevance.

[0041] In some embodiments, treatment relevance can include such factors as can reasonably be used to guide the selection of treatment, the applicability of various drug trials or research procedures, the optimal specific site or institution for treatment. In other words, treatment relevance corresponds to the matching of the sample / patient to a study, drug trial, and / or treatment protocol.

[0042] In some embodiments, diagnostic relevance is not limited to the diagnostic relevance indicating the decisive presence or likelihood of cancer, dysplasia, fibrosis, inflammation, or any other pathology, but also to any feature within the sample related to the positive or negative determination of pathogenesis, such as the presence or state of cancer, or the degree of remission or minimal residual disease, or the degree and location of dysplasia, or the degree and composition of fibrosis or inflammation, regardless of whether such an indicator is in the state or morphology of an organelle or cell or tissue or protein matrix or body fluid or organic polymer or anatomical structure, or in the secondary evidence generated by fixation, clearing, processing, sectioning, mounting, staining, and any other method practiced in a histology laboratory. An example list of relevant features that can be indexed by the image mapping / classifier module 114 in an AI scanner includes cells, cell nuclei, organelles, lacunae, glands, blood vessels, signet ring cells, atypical pleomorphism, and atypical collagen structures.

[0043] In some embodiments, the diagnostic and / or treatment relevance can be based on quantification of the number of instances of things or cells or states within a particular proximity or region. For example, the diagnostic and / or treatment relevance can be based on how many cells of a particular type are within a certain proximity of each other or are contained within a field of view of a defined size. For example, the diagnostic and / or treatment relevance can be based on the count of mitotic figures within a particular region. As another example, in breast cancer, one of the grading criteria is the number of mitotic figures in the tumor in ten high power (40X) fields of view. If the specifications of the microscope are known, the 40X field of view can also be considered a region. Melanoma uses the number of mitoses per square millimeter. Counting mitoses can be important in grading malignancies and helps to distinguish between benign and malignant tumors. Gastrointestinal stromal tumors and many other sarcoma / soft tissue tumors are stratified as benign, of uncertain malignant potential, and malignant based on the number of mitotic figures in a number of high power fields of view (usually 50 or more).

[0044] In some embodiments, a relevance region (also referred to as a zone, subset, or portion) can be based on the level of relevance, type of relevance, or context in multiple layers or any combination thereof, as discussed in more detail below (e.g., reference Figure 3 ).

[0045] Figure 2B FIG. for a medical image acquisition process 230 according to some embodiments. The acquisition and processing of source images 202 and 204 are similar to the acquisition and processing described above with reference to Figure 2A . However, in process 230, the final source image 212 includes multiple regions of interest levels, or diagnostic and / or treatment relevance levels. For example, regions 220a - 220e have a first diagnostic and / or treatment relevance level, regions 222a - 222c have a second diagnostic and / or treatment relevance level lower than the first level, and regions 224a - 224b have a third diagnostic and / or treatment relevance level lower than the second level. The regions of the final source image 212 not included in regions 220a - 220e, 222a - 222c, and 224a - 224b (e.g., region 226) have a diagnostic and / or treatment relevance level lower than the third level. The region with the highest relevance level (e.g., 220a) may be most beneficial to a pathologist in diagnosing a patient or determining a course of treatment action, while the region with the lowest relevance level (e.g., 226) may be least beneficial (or of no benefit) to a pathologist in diagnosing a patient or determining a course of treatment action.

[0046] Figure 2C FIG. for a medical image acquisition process 250 according to some embodiments. The acquisition and processing of source images 202 and 204 are similar to the acquisition and processing described above with reference to Figures 2A - 2BThe acquisition and processing described. However, in process 250, the final source image can be visualized in three-dimensional format, where regions are separated on the z-axis by their corresponding diagnostic and / or treatment relevance levels. Thus, Figure 2C the visualization of the final source image in Figure 2C can be similar to a topographical map, where the relevance levels of multiple regions of the image are represented as heights. Using such visualization, a three-dimensional flyover model of the samples in the final source image can be displayed, where labels, capture points, and / or relevance grades are shown as various z-heights. In some embodiments, the most relevant pixel at a given (x,y) location in the image can be identified in a stereoscopic z-stack.

[0047] In some embodiments, the image mapping / classifier module 114 uses a third-party input aggregator (TPI or TPIA) to identify and map relevant regions in the final source image. In some embodiments, the image mapping / classifier module 114 uses relevance ratings provided by an internal system or person to identify and map relevant regions in the final source image.

[0048] In some embodiments, the diagnostic and / or treatment relevance can be a composite of multiple diagnostic and / or treatment relevance ratings provided from several different sources.

[0049] In some embodiments, in the analysis of the composite z-stack, the image mapping / classifier module 114 assesses diagnostic and / or treatment relevance independent of focus sharpness. A feature or image region can be highly diagnostically and / or therapeutically relevant but out of focus or less sharply focused compared to features above or below it on the z-axis. Some systems that use a selective process within the z-stack typically select the best focus rather than diagnostic relevance, which is a completely different attribute. However, the disclosed system can, for example, create a composite image with out-of-focus features that can subsequently be sharpened by an algorithm or neural network, especially since they are first determined and selected based on diagnostic and / or treatment relevance.

[0050] In some embodiments, the relevance ratings of samples in the final source image features and / or image regions (e.g., regions 220, 222, and 224) can be organized into hierarchical classes, categories, or associated meta-tags.

[0051] In some embodiments, direct prior data (e.g., pre-specified point locations or regions of slices or features of samples that a pathologist or oncologist deems important) and metadata (e.g., health information about the patient, or the community / environment where the patient lives or works or conducts business) can be used as inputs to the machine vision / learning system in the image mapping / classifier module 114 to evaluate and index sample features and score the diagnostic and / or treatment relevance of individual regions in the final source image.

[0052] In some embodiments, the image mapping / classifier module 114 can accommodate revisions by a pathologist to the relevance ratings assigned to image regions. In some embodiments, such revisions can track an individual pathologist's relevance model. In some embodiments, such revisions can inform the ongoing improvement of a global model used to determine the relevance of specific cell morphologies or other features of a medical image. Multiple such profiles for modeling and continuously improving consistency at each level can be based on a pathologist's self-consistency, consistency within a multi-physician pathology practice, and / or consistency with board standards. In some embodiments, there can be a system shift towards any one or all of the above for better sensitivity and specificity (thus driving the advancement of the field).

[0053] In some embodiments, the image mapping / classifier module 114 generates a cellular index of samples in a source image and saves this cellular index as metadata corresponding to the image. The data in the cellular index can be mapped to a vector file associated with coordinate values of the image in which certain features of the cellular index exist. The metadata can be included in one or more discrete files contained within a file wrapper, or can be included in the file header of an image file, or can be included steganographically within the image file.

[0054] The cellular index is an accurate, complete, descriptive, and guiding, quantitative and qualitative assessment of a biopsy sample in a source image. The cellular index can include intracellular, extracellular, qualitative, and / or quantitative attributes of the sample. The type of sample can include any one of bone, blood, body fluid, tissue, etc. The cellular index itself can be any one of an index, census, profile, map, list, etc. The cellular index can be referred to as a Cellular Index and Compression Key (CICK). In some embodiments, the cellular index can include pre-annotations.

[0055] For a given medical image, a list of examples of cell index features (additive index features of a sample) contained in the metadata includes any of the following: abscess, absorptive cell (enterocyte), acid-fast bacilli / bacteria, alveolar (acinar) gland, acinus, adipocyte, adipose tissue, adventitia, alveolar space, alveolus, amacrine cell, ameloblast, apocrine cell, arrector pili muscle, arteriole, artery, astrocyte, atherosclerotic plaque, atypical mitosis, bacilli, bacteria, band, Barr body, basement membrane, basophil, basophilia, basophilic stippling, bile duct, acute change, blood vessel, bone, bone marrow, Bowman's capsule, Brunner's gland, brush border, Bunina body, Cabot ring, calcification, canaliculus, cancellous bone, capillary, capsule, myocardium, heart valve, cartilage, cell, cementum, central vein, centriole, chief cell, chondroblast, chondrocyte, chromatin, cilium, collagen, columnar cell, connective tissue, corpus albicans, cord tissue, Bihler's cord, cornea, corona radiata, corpus luteum, cortex, Cowdry body, crypt, Lieberkühn crypt, cyst, cytoplasmic vacuole, decidual cell, dentin, dermis, Descemet's membrane, duct, ductal epithelial cell, Dutcher body, dystrophic calcification, eccrine duct, exocrine gland, elastic fiber, enamel, endocardium, endocrine cell, endometrial gland, endometriosis, endomysium, endosteum, endothelial cell, vascular endothelial cell, enteroendocrine cell, eosinophil, eosinophilia, ependymal cell, epidermis, epineurium, epithelium, erythrocyte (red blood cell), exocrine cell, external elastic membrane, fascia, fascicle, fenestrated endothelium, fibrin, purulent exudate, fibroblast, fibroma, fibrosis, fibrovascular core, fimbria, follicle, fruiting body, fungal hyphae, fungal yeast, fungus, ganglion cell, gangrenous necrosis, gastric pit, germinal center, giant cell, gland, glial cell, glomerulus, goblet cell, Graafian follicle, granulosa, granulocyte, granuloma, granulosa cell, stroma, hair, hair follicle, halo cell, Hassall body, Haversian canal, Heinz body, Helicobacter pylori / H. pylori, helical bacilli, primitive blood cell, hemosiderin, hepatocyte, hilum, histiocyte, Hofbauer cell, Howell-Jolly body, hyaline cartilage, hydroxyapatite, chromatin hyperplasia, hypha, hypnosome, immunoblast, inclusion body, inner circular muscle, internal elastic membrane, interstitial cell of Cajal, interstitial cell, intranuclear inclusion, islet cell, islet, paraganglion cell, keratin, keratin pearl, keratinocyte, Kupffer cell (liver macrophage), lacteal, lacuna, lamina propria, Langerhans cell, lens, leukocyte (white blood cell), Lewy body, ligament, lipofuscin, loose connective tissue, lumen, lumen contour, lumen space, Luschka duct, lymphatic vessel, lymphoblast, lymphocyte, lymph follicle, lysosome, macrophage, desmosome, dense plaque, mast cell, megakaryocyte, meiosis, melanin, membrane, Merkel cell, mesangial cell, mesenchyme,Mesothelial cells, mesothelium, metamyelocytes, Michaelis-Gutmann bodies, microcalcifications, microorganisms, microvilli, mitosis, mitotic figures, molluscum bodies, monoblasts, monocytes, mucosa, mucous cells, muscle tissue, outer muscular layer, muscular mucosa layer, myeloblasts, myelocytes, myenteric (Auerbach's) plexus, Nabothian cysts, necrosis, nerves, nerve tissue, neurons, neutrophils, Nissl bodies, nuclear membrane, nucleolus, cell nucleus, oligodendrocytes, oocytes, ordinary connective tissue, organelles, osteoblasts, osteoclasts, osteocytes, osteoid cells, outer longitudinal muscle, ova, oxyntic cells, eosinophils, Pacinian corpuscles, pancreatic acinar cells, Paneth cells, papillae, papillary dermis, Babes-Ernst bodies, parasites, parasympathetic ganglion cells, parenchyma, parietal (oxyntic) cells, periarteriolar lymphatic sheath, pericytes, perikaryon, periosteum, Peyer's patches, phagocytes, phagocytosis, Pick bodies, pigments, plasma cells, plasma membrane, malaria parasite species, platelets, polymorphism, alveolar cells, podocytes, polychromatic erythrocytes, portal triad, pronormoblasts, promyelocytes, psammoma bodies, dental pulp, Purkinje cells, Purkinje fibers, pyramidal cells, red pulp, Reed-Sternberg cells, Reinke's crystalloids, respiratory epithelium, rete ovarii, rete pectiniforme, rete spinalis, rete testis, reticular dermis, reticulocytes, retinal pigment epithelium, Russell bodies, sarcolemma, schistocytes, sebaceous glands, secretory epithelium, seminal vesicles, seminiferous tubules, serosa, serous membrane, serous cells, Sertoli cells, signet ring cells, simple columnar epithelium, simple cuboidal epithelium, simple squamous epithelium, sinus / sinusoid, sinusoid, skeletal muscle, smooth muscle cells, stained cells, spermatids, spermatocytes, spermatogonia, spermatogonium, spermatozoa, spindle cells, spirochetes, sporozoites, stratified squamous epithelium, stroma, subcapsular space, subcutaneous adipose tissue, submucosa, submucous plexus (Meissner's plexus), submucous glands, surface mucous cells, sweat glands, synovium, tendons, terminal bars, thrombi, thymus, Treponema pallidum, tubular glands, tubules, urothelium, vacuoles, vascular plexuses, veins, venules, villi, viral inclusions, Wharton's bodies, Wharton's jelly, white pulp, Wolffian duct, woven bone, and yeasts. This list is provided for illustrative purposes and is not meant to be exhaustive or to limit the scope of the disclosure in any way.

[0056] In some embodiments, other data associated with the above features may also be additively indexable, such as attributes and / or parameters of any of the above features, including but not limited to width, height, thickness, diameter, optical density, color offset, opacity, polarization, dimensional distortion relative to a norm, angular deviation, rotational state, focus sharpness, etc.

[0057] In some embodiments, other types of features may also be indexable. For example, miscellaneous or aberrant features may be indexable, such as cracked sections, cracked coverslips, excess coverslip media, trapped air bubbles, smudges, fingerprints, protein spot inclusions, hair inclusions, folded tissue edges, tissue wrinkles, non-recurrent tissue thickness variations, recurrent tissue thickness variations, dry top residues, dry bottom residues, mislabeled tags, detached coverslips, extra slides. Although these features may be associated with a medical image, they are not associated with the sample itself. Thus, in some embodiments, these features may be recorded but not included in the cell index.

[0058] Any subset of the above features in the cell index may be generated as a vector file or spline, which can effectively describe the region enclosing multiple (e.g., thousands of) cells. Thus, in some embodiments, the example ratio of pixels to recorded features may be at least 1,000 to 1. In some embodiments, the cell index may include overlapping meta-tags, since some features may be more than one thing (e.g., a skin cell may be part of a gland wall). Since most normal / healthy tissue will contain dozens or hundreds of similar cells, the index may be run-length encoded such that the net ratio should still be far less than 1,000 to 1. And since at least half of the index may not be used for any given sample, the image mapping / classifier module 114 may use a serialized roster for each sample, which can potentially reduce the size of the entries in the index to eight-bit (one-byte) identifiers. Thus, each sample vocabulary may be limited to far less than 256 feature types.

[0059] Figure 3 Diagram of the alpha layer and metadata generation process 300 according to some embodiments. In some embodiments, the hierarchical levels of diagnostic relevance are established through machine vision analysis of the system (implemented by the image mapping / classifier module 114), each level including one or more regions of the total image and thus a fractional percentage of the total pixels. Then, such regions may be divided into discrete alpha layers, each layer including a diagnostic relevance rank. These alpha layer images may then be compressed using the best compression type for their composition, and wherein the compression level is negatively correlated with the diagnostic relevance.

[0060] For example, in process 300, medical image 302 (e.g., corresponding to image 214, Figure 2C)Obtained by the image acquisition module 112 and mapped by the image mapping / classifier module 114. As a result of the mapping, a metadata layer 303 corresponding to the image layer 302 is generated, containing cell indices, which include diagnostic relevance scores by region and feature. This metadata layer may also contain a recommended sequential diagnostic workflow (explained in more detail below), including pre-annotations, or such comprehensive recommendations aggregated from multiple diagnostic sources.

[0061] The image 302 and the cell index 303 are divided into three different α-layers 304, 306, and 308, each layer defining the image fidelity for the most preferred type, format, ratio, and / or compression level. For example, layer 304 (containing the image file or layer 304a and the metadata file or layer 304b) contains the most diagnostically relevant part of the image (e.g., 220a - 220e, Figure 2C )), layer 306 (containing the image file or layer 306a and the metadata file or layer 306b) contains the moderately diagnostically relevant part of the image (e.g., 222a - 222c, Figure 2C ), and layer 308 (containing the image file or layer 308a and the metadata file or layer 308b) contains the less diagnostically relevant part of the image (e.g., 224a - 224b, Figure 2C ).

[0062] For each layer, then dedicated reconstruction super-resolution GANs (layers 314, 316, 318) can be assigned to each region and feature from the palette of such dedicated GANs (described in more detail below) based on cell attributes, morphology, coloring, pathological status, etc. In this and other ways, the metadata is not only descriptive but also potentially indicative.

[0063] Figure 4A Figure for the downscaling, compression, and pre-verification process 400a according to some embodiments. In some embodiments, the process 400a is executed by the image processing platform 110 of the server system 102, including the image mapping / classifier module 114 and the image compression module 116.

[0064] The source image 402 (e.g., corresponding to the image 210, 212, or 214 in Figures 2A - 2C , or the image 302 in Figure 3 ) is analyzed by the mapping / classifier module 114 to determine the diagnostic and / or treatment-relevant regions as described above. One or more regions with the highest relevance level or reaching a threshold relevance level are optionally extracted into one or more α-layers 404 (e.g., corresponding to layer 304a in Figure 3 ).

[0065] The mapping / classifier module 114 generates metadata 406 for each layer. The image compression module 116 down-resolves and / or compresses each image layer that has a lower diagnostic and / or treatment relevance than layer 404, or does not meet a relevance threshold, thereby producing one or more down-resolved and / or compressed images 408. Each down-resolved and / or compressed image 408 corresponds to a metadata layer 406, which contains data indicating how the upsampling and / or decompression algorithm can restore the image. Thus, this upsampling and / or decompression process is pre-verified by upsampling and / or decompressing (reconstructing) one or more of the down-resolved and / or compressed images 408 using the corresponding metadata 406 to generate a reconstructed image 410.

[0066] The image processing platform 110 compares the reconstructed image 410 with the original image 402a without the extraction region 404, and determines the difference between the two images based on the comparison. Due to the nature of down-resolving, compressing, up-resolving, and / or decompressing the image file (e.g., using a lossy algorithm), some differences are expected. However, since the images 408 are from a lower diagnostic and / or treatment relevance level, some lost details (e.g., sharpness) can be tolerated due to the accuracy of the machine learning / vision models used to reconstruct the images at the client device. These machine learning / vision models are tested during the pre-verification phase, including during this comparison step. If the difference is above a threshold, the comparison step fails, and the process is repeated at the down-resolution / compression step.

[0067] Upon failure, one or more machine learning / vision models are updated, and the images are down-resolved and / or compressed again using the updated machine learning / vision models. One such example of a machine learning / vision model is a GAN. A GAN uses a generator circuit (e.g., a convolutional neural network (CNN)) to generate images, and a discriminator circuit (e.g., another CNN) to determine whether the generated images are real or fake. The reconstruction circuitry and / or algorithm 432 (also referred to as the upsampling and / or decompression circuitry and / or algorithm 432) for reconstructing the images 408 into the images 410 can be implemented as the generator network of the GAN, while the comparison circuitry and / or algorithm 434 for comparing the reconstructed image 410 with the original image 402a can be implemented as the discriminator network of the GAN. Thus, each time the comparison results in a failure, the reconstruction circuitry 432 learns according to the updated generator model and updates its generator model to provide a more realistic image 410.

[0068] When the comparison between the reconstructed image 410 and the original image 402a passes (e.g., the difference is less than a threshold), the image processing platform 110 packages the extraction layer 404, the downsampled and / or compressed image 408, and the latest metadata 406 containing the cell index and the pre-verified latest version of the machine vision / learning model (e.g., GAN model) for reconstructing the downsampled and / or compressed image 408. The images 404, 408, and metadata 406 are packaged into one or more files for transmission over the network 130 to one or more client devices 150 / 160.

[0069] In some embodiments, the comparison between the reconstructed image 410 and the original image 402a may be close to passing but not actually pass. In other words, the difference may be below the failure threshold but only higher than the passing threshold by a threshold amount. The difference data 412 itself may be included in the packaged file for transmission over the network 130 to one or more client devices 150 / 160, rather than continuing to improve the machine learning / vision model and spending more time on pre-verification.

[0070] In some embodiments, different algorithms may be used to downsample and / or compress each image layer (e.g., 404 and 408), depending on the algorithm optimized for that layer. For example, different compression ratios, compression methods, or compression times may be used to compress different layers. In other words, each layer may be compressed to a different degree and with different compression algorithms or types. Since each layer is created based on diagnostic and / or treatment relevance, the resolution reconstruction and compression of the image layers are based on diagnostic and / or treatment relevance. Specifically, more relevant image layers (containing more relevant regions) are downsampled and / or compressed to a greater extent and may even use completely different resolution reconstruction and / or compression algorithms compared to less relevant image layers (containing less relevant regions).

[0071] In some embodiments, the compression module 116 may create a region map of gradient-variable compression for each image 408 with respect to fidelity and diagnostic and / or treatment relevance. Thus, the image compression module 116 may combine gradient-variable downsampling with respect to diagnostic and / or treatment relevance, perform gradient-variable compression with respect to diagnostic and / or treatment relevance, and then perform super-resolution of the medical image with cell index using a predefined tissue-specific neural network library. These downsampled and compressed images are the result of several non-redundant complementary empirical measurements and evaluations of the original image.

[0072] In some embodiments, using a machine learning / vision model at reconstruction step 432 involves mapping the indexed features in the cell index to a library or palette of dedicated machine learning / vision models. For example, for embodiments using a GAN model, the indexed features are mapped to a library or palette of dedicated GANs. In other words, a GAN or any other dedicated tissue-specific, feature-type-specific, or morphology-specific machine learning model can be used to transform a parametric representation instance of cell morphology into super-resolution pixels or vector graphic elements, which can then be rasterized into pixels. In some embodiments, other machine learning / vision models can be used in addition to or as an alternative to the GAN model, such as Stable Diffusion or any other type of machine learning / vision model known or yet to be discovered.

[0073] In some embodiments, the aforementioned GAN palette can be implemented as any modular library of machine learning / vision super-resolution models, each modular library specialized according to cell and / or tissue type, state, or morphology. For example, Figure 2A regions 220a - 220e in can be associated with different models in the model palette, each model specific for reconstructing any cell and / or tissue type, state, or morphology present in the corresponding region.

[0074] Repeat the reconstruction process at step 432 in the pre-verification operation in Figure 4A using the same metadata and machine learning / vision models at the client devices 150 / 160; thus, these machine learning / vision models are packaged together with the image layers as described above at step 436. Thus, the machine learning / vision (e.g., GAN) palette-mapped images can be transmitted to the client devices for subsequent decompression and super-resolution.

[0075] In some embodiments, the pre-verification reconstruction (steps 432 - 434) performs a final image check to ensure that the image data transmitted to the client devices can be precisely reconstructed to the same fidelity as the original image 402. Referring to Figure 4A , if the check of the reconstructed image 410 does not have sufficient fidelity ("Fail"), then the first loop (following path "A") isolates the discrepant pixels and / or features in the image, tries different GANs or other machine learning / vision models for the specific discrepant pixels / features, and repeats the check (steps 432 - 434). This loop can be repeated multiple times until the check passes ("Pass") and the image file to be transmitted to the client devices is packaged.

[0076] In some embodiments, after the loop has repeated a threshold number of times, or once the difference at comparison step 434 is below a threshold, the raw (true) pixels contained in the difference (between image 410 and 40a2) can be isolated into a correction layer 412, which is also packaged into a file for transmission to the client device. Additionally or alternatively, the correction GAN can generate a fine correction alpha layer for a particular layer, region, or the entire image that is the subject of the difference (between image 410 and 402a).

[0077] In some embodiments, the diagnostic and / or treatment-related alpha layer 404, the downsampling and / or compression layer 408, the cell quantification index and other metadata layers 406, the GAN mapping (or other machine learning / vision mapping) layer, and the optional correction layer 412 are encapsulated within a single file wrapper and isolated, retained, and remotely hosted as a known image check key (KICK) file. In some embodiments, the KICK file can be approximately 30% of the size of the original file, which is a significant improvement for the purpose of optimizing limited storage resources by significantly reducing the storage burden at the server system 102 and the client devices 150 / 160. In some embodiments, the original image 402 can be deleted from the memory 120 after the KICK file is packaged, thereby being replaced by the KICK file itself and being available for future viewing requests.

[0078] In some embodiments, the diagnostic and / or treatment-related alpha layer 404, the cell quantification index and other metadata layers 406, the GAN mapping (or other machine learning / vision mapping) layer, and the optional correction layer 412 are packaged as a "key" file 414 separate from the downsampling and / or compression layer 408 of the main packaged file. In some embodiments, this key file can be approximately one quarter of the size of the original image 402, and the main file can also be one quarter of the original image 402, thus providing a more compelling reduction in the storage burden for the client device. Accordingly, the client device will receive the diagnostic and / or treatment-related key file for super-resolution, as well as the main file for combination with the key file to create a complete image (e.g., looking like the original image 402).

[0079] The following example illustrates the above with reference to Figure 4AThe described features. The AI engine (mapping / classifier 114) performs quantitative mapping of the input sliced image 402, classifying (categorizing) and mapping each feature, morphology, organelle, nucleus, cell orientation, cell state, etc. The most relevant features and / or regions are isolated into the alpha layer 404 and isolated as the original elements of the source image (e.g., less than 30% of the total pixels). Then, the remaining features and / or regions are also isolated into the alpha layer, and each layer is de-resolved and / or compressed based on its corresponding diagnostic / therapeutic relevance score. This takes advantage of the fact that healthy tissue is generally more normal and regular, and thus more predictable for a dedicated neural network. The final inspection step validates the restoration (comparison 434), thus adjusting the quality metric until the fidelity is perfect (or above a predetermined threshold). Finally, a corrected layer 412 with reduced bitrate is also created (but only as needed).

[0080] Figure 4B An alternative implementation of a diagram depicting the de-resolution, compression, and pre-verification process 400b according to some embodiments. Except for the placement and function of the mapping / classifier 114, the process 400b( Figure 4B ) is the same as the process 400a( Figure 4A ).

[0081] The process 400b supports a first method, where the input image 402 is first mapped by the mapping / classifier 114, which then guides the de-resolution or compression process 116, as well as the extraction of the more / most relevant features and / or regions. Specifically, based on the classification and mapping of the relevant features and / or regions, the module 114 instructs the de-resolution / compression module 116 which features and / or regions to process into the corresponding alpha layer, and instructs the feature extraction module which parts to extract in the most relevant alpha layer 404.

[0082] The process 400b supports a second method, where the input image 402 is globally de-resolved at the module 116, and the lower-resolution image is used for classifier mapping, which then guides feature extraction and selective variable compression or up / down resolution or other processing (e.g., color reduction). Specifically, based on the globally de-resolved input image 408, the mapping / classifier module 114 can more effectively determine the relevant features and / or regions for extraction and subsequent down / up resolution or compression / decompression, because the de-resolved image 408 has less data to process. This increased efficiency saves time, allows for faster processing of the input image 402, and has little impact on quality.

[0083] The process 400b supports any combination of the first and second methods discussed above, such as a first (simpler) classifier at full resolution (as in the first method), followed by a richer / more complete classifier at lower resolution (as in the second method).

[0084] Figure 4C Depicts another embodiment of a medical image processing scheme 400c according to some embodiments. Process 400c( Figure 4C ) with features identical to those in process 400a( Figure 4A ) and process 400b( Figure 4B ) are similarly labeled.

[0085] In process 400c, the input image 402 is divided into multiple tiles (unless the tiles are provided from an image scanner). The tile size of each image is based on the priority of speed, quality, and compression potential. After being processed by the mapping / classifier 114 (as described above with reference to processes 400a and 400b, the input image (each tile) is de - resolved and / or compressed. Since the entire image portion (e.g., the entire tile) is de - resolved and / or compressed, this step can be referred to as global down - resolution and / or global compression. Thus, the entire image (all tiles) is globally down - resolved and / or compressed. In one example, the resulting de - resolved image layer 408 can be 50% or less of the size of the input image 402 (or 50% or more in other examples).

[0086] In some embodiments, the full - resolution image can be reconstructed from the respective / all portions of the image that have undergone various levels of processing (de - resolution / compression, etc.) by managing the full - resolution image at the tile level. For example, highly processed tiles can be glued together with unchanged tiles. If this would produce noticeable visual artifacts, a dither mask can be used to mitigate the edges of one or more modified tiles.

[0087] Then, the de - resolved / compressed image data 408 is up - resolved and / or decompressed to return it to its original resolution / size, thereby generating an output image 410 having the same resolution and / or size / quality as the resolution and / or size / quality of the input image 402. Since the entire image (all tiles) is up - resolved and / or decompressed, this step can be referred to as global up - resolution and / or global decompression. As described above with reference to processes 400a and 400b, the up - resolution / decompression module 432 uses a GAN that predictively enhances the clarity of the image data 408, thereby producing an output image 410 that is at least as detailed as (and in some cases, more detailed than) the original input image 402.

[0088] Concurrent (in parallel) with up - resolving / decompressing the image data 408 using module 432, the mapping / classifier 114 analyzes( Figure 4CThe line G) in the reduced-resolution / compressed image data 408 indicates subsequent processing. Specifically, if at least a portion of the image data (and in some cases, all of the image data) undergoes feature categorization and correlation classification when the image data is reduced in resolution / compressed, the mapping and classification process is more efficient, thereby saving time in generating the fully mapped and classified output image 410 from the unmapped and unclassified input image 402. In other words, the analysis at the mapping / classifier module 114 may be much faster because it is performed using the lower-resolution image data. Based on the foregoing analysis of the classifier 114, one or more portions of the input image 402 can be manipulated to provide a higher-quality portion of the input image corresponding to the diagnostic / therapeutic relevant features.

[0089] Although the upscaled / decompressed output image 410 has the same resolution and quality as the input image 402, the output image 410 is more compressible because run-length encoding is more effective on the resulting (sharpened image). Additionally, in some embodiments, the re-upscaling is performed by filling in predictable pixels (using a GAN or other AI upscaling process), where the original pixels were deleted during the downscaling / compression process. This method provides a sharpening in the output image 410 that can be superior to the original input image 402. In other words, the pixels missing due to downscaling and / or compression are backfilled with predicted pixels during the upscaling / decompression process, thereby increasing the number of predictable pixels by simply overwriting the previously deleted pixels with the predicted pixels used by the free upscaling / decompression module 432.

[0090] The output image 410 is provided (in some embodiments, along with the metadata layer 406) to the client devices 150 / 160 via encapsulation 436 and the network 130 (as described above with reference to processes 400a and 400b).

[0091] Figure 5FIG. for the super-resolution, decompression, and display process 500 according to some embodiments. In response to receiving a KICK file or key and master file from the server system 102 via the network 130, the process 500 is executed at the client device 150 / 160. The diagnostic and / or treatment-related alpha layer 404, the downsampling and / or compression layer 408, the cell quantification index and other metadata layer 406, the GAN mapping (or other machine learning / vision mapping) layer, and an optional correction layer are unwrapped for separate processing. The downsampling and / or compression layer 408 is super-resolved (also known as upsampled) and / or decompressed using the GAN mapping data in combination with the metadata 406 (e.g., having cell indices), resulting in a reconstructed image 402a (corresponding to the final version of the image 402a in FIG. 4 at the server system 102 during the pre-verification process), and the reconstructed image 402a is combined with the diagnostic and / or treatment-related alpha layer 404 to restore the original image 402, which has the same fidelity level as the original image acquired by the image acquisition module 112 at the server system 102.

[0092] The upsampling / decompression function uses the cell index metadata and / or a dedicated GAN mapping (or other machine learning / vision mapping) received from the server system 102 to upsample and / or decompress the image 408. As an optional final upsampling process (e.g., after combining the images 404 and 402a), the pixel shift resolution change function as described herein (e.g., referring to Figure 6 ) can be used to further upsample the restored image 402 beyond the original sensor resolution.

[0093] In some embodiments, the metadata layer 406 contains diagnostic workflow instructions, including the order of displaying relevant regions (e.g., 220a, followed by 220b, followed by 220c, and so on ( Figure 2A ). In some embodiments, the diagnostic workflow is predicted by a TPI or an internal A.I. prediction algorithm. Thus, not only can the diagnostic result (e.g., in the form of diagnostic and / or treatment-related regions of a medical image) be predicted, but also the workflow for performing the diagnosis (the diagnostic workflow of a pathologist) can be predicted. In other words, the prediction algorithm determines which regions of the medical image the pathologist using the client device 150 / 160 will want to view first, second, etc.

[0094] By including diagnostic workflow instructions in the metadata layer 406, the image processing platform 110 not only encodes medical images (e.g., whole slide images), but can also encode the imaging or guided viewing of a pathologist's A.I. prediction workflow, which typically accounts for only 25% to 50% of the total images. In some embodiments, the prediction algorithm not only determines the order of regions, but also determines the zoom level, angle, which regions are shown adjacent to each other, etc.

[0095] In some embodiments, during the actual diagnostic workflow (when the pathologist is viewing various regions of the image), additions or revisions to the workflow can be recorded at the client device 150 / 160 and fed back to the prediction model at the server system 102. Such additions and revisions can be used to update the prediction workflow model used at the server system 102.

[0096] In some embodiments, the foregoing additions or revisions can be associated with a user-specific profile, thereby allowing each pathologist to personalize his or her prediction workflow. These user-specific profiles can track individual relevance models corresponding to individual pathologists. These user-specific profiles can additionally or alternatively inform the ongoing improvement of the global model used at the server system 102 for diagnostic workflow prediction for all pathologists. Thus, while some systems convert a pathology slide into an image or even rich data, the disclosed system converts the slide into an effective stand-alone diagnostic workflow that is concise and efficient enough to function on a user device (e.g., a smartphone) anywhere and anytime, thereby allowing the pathologist to view medical images (e.g., whole slide images) without having to go to an office or use a dedicated viewing device.

[0097] For embodiments in which the client device is provided with an image at the original resolution (e.g., as described in process 400c above Figure 4C ), there is no need to up-resolve the received image, unroll the additional alpha layer 404, and combine them with the up-resolved received image as shown in Figure 5 . In these scenarios, the image itself can simply be decompressed and / or provided directly to the client device.

[0098] Figure 6FIG. for the pixel shift process 600 according to some embodiments. The source image (e.g., 402) is pixel shifted and downsampled to multiple images in the retrosource proxy layer. For example, a group of 256 pixels can be converted into four pixel shift groups of 16 pixels each. For each group of pixels, each pixel is a combined version (e.g., average) of 16 pixels from the source image. The retrosource proxy layer can be the downsampled image 408 that is packed and transmitted to the client devices 150 / 160, as described herein with reference to FIGS. 4-5. To reconstruct the image, the retrosource proxy pixel groups can be upsampled and superimposed (e.g., stacked in a pixel shift manner) with the combined (e.g., average) overlapping pixel values to form the reconstructed image.

[0099] Figure 7 FIG. is a flowchart showing an example process 700 for compressing and transmitting, reconstructing, and presenting an image for diagnostic annotation according to some embodiments. The process can be managed by instructions stored in a computer memory or a non-transitory computer-readable storage medium (e.g., storage device 120). The instructions can be included in one or more programs stored in the non-transitory computer-readable storage medium. The instructions, when executed by one or more processors, cause the server system 102 to execute the process. The non-transitory computer-readable storage medium can include one or more solid-state storage devices (e.g., flash memory), disk or optical disc storage devices, or other non-volatile memory devices. The instructions can include source code, assembly language code, object code, or any other instruction format that can be interpreted by one or more processors. Some operations in the process can be combined, and the order of some operations can be changed.

[0100] After obtaining a medical image (e.g., whole slide image) (e.g., 402, Figures 4A - 4C ), the A.I. DICOM compliance engine 118 of the server system 102 removes patient-specific data from the image. The server system 102 uses the aggregated TPI and scores (706) to identify (704) the tissue type and separate the diagnostic and / or treatment-related TPIα layer (e.g., 404, Figures 4A - 4C ). The server system 102 creates (708) a metadata layer (e.g., 406, Figures 4A - 4C ) to notify subsequent super-resolution. The server system 102 creates (710) a pixel-shifted downsampled retrosource proxy layer (e.g., 408, Figures 4A - 4C ). The server system 102 tests (712) and pre-verifies the super-resolution reconstruction, thereby adding corrections (e.g., Figures 4A - 4C steps 432, 434 and path “A”). The server system 102 determines (714) the fidelity, isolates and retains the final key (e.g., steps 434 and 436, Figures 4A - 4C)。The server system 102 packs (716) the resulting smaller total file into a new wrapper (e.g., step 436, Figures 4A - 4C )。

[0101] In some embodiments, the input image 402 described above with reference to Figure 4A - 7 can be part of a z-stack (multiple images at each corresponding z-height of a sample). Typically, a sample is prepared and imaged by flattening the z-stack into a single layer. By flattening the z-stack, the user loses access to navigation within the z-field and any insights that can be observed from the ability to utilize such navigation. The following discussion describes embodiments for restoring and / or simulating a previously flattened z-stack, thereby providing a credible navigable z-field reconstruction.

[0102] In a scan where there is a complete z-stack (multiple images at each corresponding z-height), the image processing platform 110 can capture only the incremental pixels of a feature relative to the pixels of the same feature on the upper and lower layers. Such incremental pixels can include those pixels that are better focused, but the evaluation can also include diagnostic and therapeutic relevance. Thus, the incremental pixels can be pixels having a focus corresponding to a predetermined threshold of sharpness and / or pixels that are part of a feature corresponding to a predetermined threshold of diagnostic or therapeutic relevance.

[0103] Since the image mapping module 114 classifies the entire input image, the image processing platform 110 can determine and save the z-level corresponding to each feature (and each part of a feature) in the image. Thus, the image processing platform 110 can determine which features in the z-level are behind or on top of other features. For example, the image processing platform 110 can determine which blood cells are on top of other blood cells and then run one or more predictive GAN models to predict the pixel values of the occluded parts of the underlying blood cells. Thus, the image processing platform 110 can recover a navigable z-level from a planar image.

[0104] Specifically, the pixels occluded by overlapping cells or other features can be recovered using the techniques described herein, provided that those "underlying" pixels are different from the prediction model. Using this very low-cost few pixels reserved as an alpha layer, the image processing platform 110 can use controls such as a focus knob (e.g., the controls on the peripheral device 803 described below) or use a biomarker navigation feature (e.g., as described below with reference to Figures 14 - 16 ) to approximate the true navigation experience within the z-axis. For example, once a zoom gesture reaches a predefined or dynamically triggered maximum threshold, the zoom gesture can give way to z-navigation.

[0105] In some embodiments, using a very small number of pixels reserved as the alpha layer with very low cost, the image processing platform 110 can create virtual slices at a certain angle, or even create various non-planar virtual surfaces from deep within the z-field. This feature can be used for 3D imaging, such as lattice light sheet, or for simulating or overlaying slice images on 3D radiographic images. This use case can include not only features in the slice image that far exceed the resolution of the radiographic image, but also transferring stains from the slice image to adjacent radiographic pixels or voxels. Based on just a few consecutive portions of the slice image, such methods can insert or infer voxels through context overlay based on radiographic voxels and slice image pixels to virtually generate a 3D "slice image". Such 3D voxels (motion resolution elements) can be used as the main diagnostic workflow display, or can be used as the basis for rendering several "virtual slices" that have most or even all of the details and usefulness (relative to the radiographic image) that a stained tissue slice can otherwise provide.

[0106] In some embodiments, using the very small number of pixels reserved as the alpha layer as described above, the image processing platform 110 can not only determine the depth of multiple features in the z-field, but also determine their positions relative to the focus. Based on this, the image processing platform 110 can computationally correct other optical aberrations (such as spherical aberration), can eliminate spectral differences associated with different distances from the focus and different feature geometries, or leave the spectral differences in place to provide the user with a true navigation of the z-field. Further, the image processing platform 110 can place "underlying" features further behind a given feature by defocusing the "underlying" features more, in order to provide a more realistic z-navigation effect.

[0107] The following section discusses the economy of reserved pixels with the aid of a prediction model (e.g., GAN mapping). In other words, the image processing platform can save only the pixels (in the image data provided to the packer 436) that are different from the various predictions described above with reference to the GAN model in processes 400a, 400b, and 400c ( Figures 4A - 4C ).

[0108] In some embodiments, the image processing platform retains only the pixels that are different from the results predicted by the GAN. In some embodiments, the image processing platform can selectively replace some pixels in a way that enhances usability. The prediction efficiency can be informed by an increasing library of models and parameters to characterize and thus realistically reconstruct each pixel / feature.

[0109] Specifically, after predictive GAN-based reconstruction (“restoration”) of the image for fidelity checking (e.g., as discussed above with reference to modules 432, 434, and 114), and in some cases, fine-tuning to maximize fidelity until it reaches a specified fidelity threshold, the remainder (e.g., the final correction layer 412, also referred to as residual coding or residue coding) can be saved as a different file or as a layer in a tiled image file (e.g., packaged at step 436), or it can be included in any number of meta-layers contained within a file wrapper.

[0110] As GAN models continue to improve in accurately predicting what the super-resolution representation of any given cell or cell feature or biomarker will look like, the aforementioned remainder will become smaller and smaller. Thus, in addition to the pixels that are different from the prediction, the image processing platform may not need to save any pixels. This is particularly valuable in reducing the file size of volumetric images (also referred to as “z-stacks”) or the “voxels” of 3D radiological images. Thus, the image processing platform only needs to save those pixels that are different from the prediction of the generative model... which will be increasingly accurate predictions.

[0111] The aforementioned “remainder” efficiency is an important part of the image processing techniques described herein because the largest images at the highest magnification will conversely have the largest ratio of pixels (or voxels) per cell. And since the prediction model works best in predictively reconstructing “standard / normal / healthy” cells, this results in a very high compression ratio.

[0112] For example, a typical blood smear (primarily) includes healthy red blood cells and suspect white blood cells. The diagnostic relevance of healthy red blood cells is small, but their ratio to white blood cells can exceed 600 to 1. A GAN model as described herein can very accurately predict more than 4,500 pixels for each red blood cell based on only 36 concise parameters including approximately 72 bytes of data. This would constitute a compression ratio of approximately 99.9% for the red blood cell region of the image. And since the number of red blood cells is 600 times that of white blood cells, this would result in a rough potential compression of 99.9x(600 / 601). Even for tissue models, the compression ratio can be higher than 70%.

[0113] The aforementioned error remainder efficiency applies to the volumetric images discussed above, including z-stack slice images and the resulting volumetric images of light sheet microscopy and / or radiology.

[0114] In other words, any pixel that can be accurately predicted by the various prediction models described herein (models working alone or in any combination) can be downsampled and precisely reconstructed thereafter (e.g., as discussed above with reference to modules 116 and 432). This can include the edges of entire red blood cells, cells, or cell nuclei, or organelles within the cell or chromatids within the cell and their corresponding granularity, Auer rods, mitotic chromosomes, etc. This can also include having an alpha layer (image region) only for cells below individual layers within a z-stack (stereoscopic image). For all the foregoing examples, the image processing platform only needs to save those pixels that deviate from the prediction model.

[0115] Generally, the above error residue efficiency applies to anything that can be imaged, including readings of NGS flow cells, karyotypes of chromosomes (which sometimes overlap each other), stereoscopic layers within lattice light sheet images, etc.

[0116] Figure 8A FIG. 7 is a diagram of a system 800 for displaying and interacting with medical images according to some embodiments. The server system 102 transmits an image file (e.g., 414, Figures 4A - 4C ) via the network 130 to the client device 150 / 160. The client device 150 / 160 includes, for example, a smartphone 801 and is optionally communicatively coupled to a peripheral device 803 and a display device 802 for interacting with and viewing the restored image (e.g., 402, Figure 5 ). In some embodiments, for example, in Figure 8B , the peripheral device is unnecessary, and the client device can be only the smartphone 801, only the display device 802, or the smartphone 801 coupled to the display device 802.

[0117] Figures 9A - 9D FIGS. 10A - 10B are diagrams of a peripheral device (e.g., 803, FIG. 8) configured to interact with medical images according to some embodiments. The peripheral devices depicted in these figures can be used to advance through the image region as part of a diagnostic workflow specified in a metadata layer associated with the image (e.g., 406, Figures 4A - 4C ). Additional details regarding the peripheral device are disclosed below.

[0118] Figure 11 and 12A FIGS. 12A - 12C are diagrams depicting multiple usage modes of a peripheral device (e.g., 803, FIG. 8) configured to interact with medical images according to some embodiments. Additional details regarding these usage modes are disclosed below.

[0119] Figure 13FIG. for a system 1300 for collaborative interaction with medical images according to some embodiments. In some embodiments, movement of a peripheral device at a first client device 150 is transmitted to one or more second client devices 160, thereby causing a peripheral device associated with the one or more second client devices 160 to perform the same movement as the peripheral device at the first client device 150. In such embodiments, a lead pathologist (150) can train others (160) to perform a diagnostic workflow in a manner that allows the others to have the same viewing and haptic experience as the lead pathologist as the lead pathologist navigates through image regions as part of a diagnostic workflow. Additional details regarding such collaborative interactions are disclosed below.

[0120] Figures 14 - 16 FIG. for a schematic system for interacting with medical images using facial expressions according to some embodiments. In some embodiments, facial expressions can enable a user of a client device to control viewing and navigation of an image region of a medical image on a display. Additional details regarding such systems are disclosed below.

[0121] Some embodiments of the present disclosure improve compression by using A.I. to create downsampled and pixel-shifted pseudo-source or "inverse source proxy layer" images for reconstructing and upsampling or super-resolving a proper accurate replica of an original high-resolution source image or a portion of the source, such as a tile or a region within a tile or a group of tiles or regions. Such embodiments can also retain a reference portion of the original image for a machine learning discrimination circuit during a high-resolution process. Such embodiments can also use such reference portions of the original image for a machine learning discrimination circuit in a priori checksums and / or validations during an upsampling process.

[0122] Some embodiments of the present disclosure improve compression by using an A.I. system to analyze regions and features of high-resolution pathology slide images and / or biopsy samples to compare them to a continuously updated "known tissue library" that contains raster and / or vector and / or wavelet data instances of various types of cells, cytoplasm, organelles, angiogenesis, tumors, cysts, lumens, glands, crypts, lamina, and other diagnostically relevant features, in various states of being, such as metastasis, mitosis, meiosis, carcinogenesis, apoptosis, etc. Such libraries can contain discrete, specific, or general or random parameter data for such instances, such as morphological type, area, width, diameter, contrast, rotational state, distortion, aspect ratio, presence of proteins, etc. For each instance or parameter of the instance, such libraries can contain statistical data such as median, mean, standard deviation, etc. For one or more of the instances, such libraries can contain correlations between the instances with each other or between the instances and various factors found in patient data and / or metadata. In such embodiments, the A.I. system records the resulting attributes and parameters associated with the cells or regions as one or more metadata layers, maps the metadata to the Cartesian coordinates of the regions of the sample and / or maps the metadata to the index positions of the structures detected in the image or sample, and / or maps the metadata to the pixel positions within the image or portion of the image. Such metadata is then used by the aforementioned A.I. system or by another A.I. system or subsystem to support subsequent decompression and / or reconstruction and / or upscaling or super-resolution of a suitably accurate replica of the original image. Such aforementioned metadata can be retained and stored and / or transmitted and / or mined as an image layer, or can be stored steganographically within the pixels of the image layer, or can be retained and stored simultaneously as a data file or array, such as XML or HTML, or as an ASCII text file or DB2 or DBF or CSV or JSON or MDB or other format or data modality that can be indexed and searched and / or mined in other ways.

[0123] Some embodiments of the present disclosure improve compression by using an A.I. system that uses the analyzed metadata layer of previous embodiments to simultaneously reconstruct an upscaled or super-resolved copy of the original image to guide the upscaling or super-resolution process. Such actions are used to confirm that such copies have sufficient fidelity and consistency upon subsequent re-execution and are consistent with the original high-resolution image. Such embodiments may employ such super-resolution for a library of different and specialized generative adversarial networks (GANs), each GAN dedicated to a cell type, tissue type, state of the tissue or cell, or other useful, different, and diagnostically relevant aspects of the image and / or sample or a portion thereof. In such embodiments, the system determines which dedicated GAN most accurately approximates the original image or region of the image and associates it or "maps" it to a validated region or sample location or feature. Such "GAN mapping" is then retained as metadata that is indexed as a dataset or retained as an image layer. If retained as an image layer, such metadata may utilize compression, such as run-length encoding (RLE), as such attributes may tend to apply to many contiguous or adjacent pixels or regions or sample features.

[0124] Some embodiments of the present disclosure use an A.I. software and / or hardware system to aggregate groups of adjacent pixels from an original or synthetic image into lower-resolution pseudo-pixels for a "downscaled" pseudo-source image (or "inverse-source proxy"), and then repeat the aggregation, thereby shifting the next pseudo-source image by a fraction of the aggregated pseudo-pixel size in an approximation of the traditional "pixel shift" process. When recombining and upscaling using one or more available algorithms, GANs, or other types of neural networks, the system validates that the "inverse-source" image precisely reconstructs the original image. In such validation, the machine learning system uses portions of the original source image as a reference in a discriminator circuit or comparison loop to reconstruct a suitably accurate copy of the original high-resolution source or a portion of the source, such as a tile or a region within a tile or a group of tiles or regions.

[0125] Some embodiments of the present disclosure use an AI software system to computationally or algorithmically combine multiple optically coincident exposures to remove sensor noise, thereby creating a "denoised" source image. The denoised source is then down-resolved by combining adjacent pixels in four, nine, or sixteen square clusters. This process is repeated to generate a series of down-resolved images, each of which is shifted by a small fraction of clustered pseudo pixels, typically by a shift distance corresponding to one of the original native source pixels. The AI ​​system runs the reorganization and up-resolution process to verify that the product of the newly created pixel-shifted "reverse source proxy" image accurately recreates or sufficiently approximates the original native resolution and / or denoised multiple exposure synthesis that was the source of the de-resolution and pixel-shifting process. Such embodiments provide file size advantages: the total size of the reverse source proxy layer image is proportional to the number of such images (N), but the up-resolution result will have a file size proportional to the square of the number (N). Up-resolution results are generated as needed, and the smaller reverse source proxy layer images (individually and in aggregate) are versions of the subject content that are stored and transmitted. And because the AI ​​pre-verifies fidelity, such embodiments minimize the number of reverse-source proxy images (N) required for sufficient fidelity.

[0126] Some embodiments of the present disclosure improve compression by using AI to select pixel regions within tiles of multi-focal plane source images (also referred to as a "z-stack" image set) that represent preferred image quality and / or diagnostic relevance and / or suitability for a reduced and optimized palette for any given Cartesian coordinate location of the imaged specimen, thereby aggregating such selected pixels or pixel regions into a pseudo source image of specimen features and / or portions that may not be imaged by the sensor due to their non-existence in coincident planes or in coincident lines of a line scan sensor.

[0127] Some embodiments of the present disclosure improve compression by using AI to select pixel regions within a source image, or within such aforementioned selectively aggregated pseudo-source image, that indicate preferred suitability of a tissue-specific and / or pathology-specific graphical token palette.

[0128] Some embodiments of the present disclosure improve compression by using AI to select pixel regions within a source image, or within such aforesaid selectively aggregated pseudo-source image, that indicate preferred suitability for compression by means of run length encoding.

[0129] Some embodiments of the present disclosure improve compression by using AI to select pixel regions within a source image, or within such aforementioned selectively aggregated pseudo-source images, that indicate preferred suitability for wavelet compression.

[0130] Some embodiments of the present disclosure improve compression by using A.I. to select pixel regions within a source image or within a pseudo-source image of such a foregoing selective aggregation, which, if computationally and / or extracted from said source or pseudo-source, would leave a remainder having a preferred suitability for one or more types of compression.

[0131] Some embodiments of the present disclosure improve compression by using A.I. to compare such a foregoing extracted layer with an original source image or a partially extracted pseudo-source image or an aggregated pseudo-source image and generate a correction factor that, when applied to the extracted layer and / or the remaining layer, improves the fidelity of the reconstructed and / or upscaled image.

[0132] Some embodiments of the present disclosure improve the transmission and cloud-hosted viewing of stored slice images by selectively caching more diagnostically relevant image portions or reference tiles in such locations or such infrastructure to provide superior speed or lower latency to a user pathologist during their diagnostic workflow or during a collaborative consultation.

[0133] Some embodiments of the present disclosure improve the transmission and cloud-hosted viewing of stored slice images by using an A.I. system or subsystem to predictively preload images or image portions in such locations or by using such infrastructure that can directly facilitate one or more suitable collaborative resources.

[0134] Some embodiments of the present disclosure improve the transmission and cloud-hosted viewing of stored slice images by selectively preloading whole downscaled slice images or portions of said images in such locations or such infrastructure to provide superior speed or lower latency to a user pathologist during their diagnostic workflow or during a collaborative consultation.

[0135] Some embodiments of the present disclosure use an A.I. system or subsystem to improve the diagnostic workflow to a priori select and / or isolate and / or extract and / or retain diagnostically relevant reference portions of an original source image or a pseudo-source image. Such portions and sample features contained therein are pre-indexed relative to the reconstructed image, and those indexed regions are mapped to detent features of a rotary roller (e.g., 803) for fast and precise navigation of a large number of such features and locations.

[0136] Some embodiments of the present disclosure use AI systems or subsystems to improve diagnostic workflows to provide hands-free navigation, area selection, and annotations with the aid of voice commands, voice-to-text annotation capabilities, and through eye and face tracking, particularly through measurement and precise tracking of vestibular-ocular reflexes. Complete navigation and annotation actions can be shared simultaneously with collaborative colleagues or virtual colleagues over a network and across long distances, thereby enabling real-time consultation service exchange, where diagnostic services can be aggregated and delivered to those people and regions where such resources are in short supply. Such real-time collaborative diagnosis is different from second opinion networks, such as Soenksen (US11,211,170), because it provides increased skill growth and credibility for less experienced and / or non-Western personnel. Such guidance is essential to improving the actual and perceived quality of care within emerging countries and / or economically disadvantaged communities.

[0137] Some embodiments of the present disclosure use an AI system or subsystem to improve the visual quality and reconstruction fidelity of the resulting image, the system or subsystem is preferably adapted and / or dedicated to a specific type or combination of types of tissue, morphology and / or pathology. One embodiment of such a dedicated system may include a generative adversarial network (GAN) for up-resolution of cells of a given tissue type, which has been determined to exemplify polymorphism. Another embodiment of such a dedicated system may include a generative adversarial network (GAN) for up-resolution of healthy and regular cells of a given tissue type. Another embodiment of such a dedicated system may include a generative adversarial network (GAN) for up-resolution of cells of a given tissue type, which has been determined to exemplify cancer metastasis.

[0138] Some embodiments of the present disclosure use an AI system or subsystem to map pixels or pixel groups or image portions or pseudo images or cells or cell groups or Cartesian coordinates or defined regions of coordinates of an imaged sample into a library and / or palette of such aforementioned dedicated tissue-specific GANs and / or morphology-specific GANs or other dedicated GANs to improve the visual quality and reconstruction fidelity of the resulting image.

[0139] Some embodiments of the present disclosure use an A.I. system or subsystem to map pixels or groups of pixels or portions of an image or pseudo-image or cells or groups of cells or Cartesian coordinates or defined regions of coordinates within an imaging sample to a library and / or palette of tissue-specific and / or morphology-specific or other specialized graphical tokens to improve the visual quality and reconstruction fidelity of the resulting image. In one such embodiment, the A.I. system or subsystem can be used to dynamically update the specialized token library, the A.I. system or subsystem detecting the repetition and / or widespread occurrence of potential new graphical tokens and using a GAN to create new such tokens and thereby append them to the existing token library and / or token palette. Such a system can then use a discriminator network and a reference portion of the image to retroactively apply the improved and / or expanded token library to a previously processed image or portion of the image to ensure excellent and / or satisfactory resulting reconstructed image quality.

[0140] Some embodiments of the present disclosure improve compression by using A.I. to select pixel regions within a source image or a selectively aggregated pseudo-source image that, if computationally extracted from the source or pseudo-source, would leave a remainder with preferred applicability for reducing color depth, including but not limited to optimal palletized colors for hematoxylin and eosin stain (H&E) or other stains.

[0141] Some embodiments of the present disclosure improve compression by using A.I. to select pixel regions within a source image or a selectively aggregated pseudo-source image that, if computationally extracted from the source or pseudo-source, would have preferred applicability for reducing color depth, including but not limited to optimal palletized colors for tissue treated with hematoxylin and eosin stain (H&E) or other stains or color profiles for algorithmic compression.

[0142] Some embodiments of the present disclosure use a camera and an A.I. system or subsystem to improve the diagnostic workflow to provide hands-free navigation, region selection, and annotation by means of the vestibulo-ocular reflex, where the user's line of sight is intentionally fixed on a selector element, such as a crosshair or selection box or color-highlighted region, line, circle, point, or polygon, or a dimmed or flickering or flashing highlighted region, line, circle, point, or polygon, and then the user's head and / or face is intentionally moved to indicate the intention to shift an image or a portion of the image into and / or under the aforementioned selector element. The aforementioned camera and A.I. system detect such VOR activity and shift the displayed image accordingly. In some implementations, in response to receiving a user input to temporarily or permanently disable the head and / or face navigation feature, tracking of the user's head and / or face can be lifted or disabled. During this time, the user can recenter his or her face (re-establish a new origin) and re-enable head and / or face tracking via a second user input that indicates the system to resume head and / or face tracking.

[0143] Some embodiments of the present disclosure use a camera and an A.I. system or subsystem to improve the diagnostic workflow to provide hands-free highlighting and annotation by means of the vestibulo-ocular reflex, where the user's line of sight is intentionally fixed on a portion of the displayed image, and then the user's head and / or face is intentionally moved to indicate the intention to highlight or select that portion of the image. The aforementioned camera and A.I. system detect such VOR activity and accordingly select or track or select that portion of the displayed image. Subsequent speech-to-text capture annotates the actively selected region or sample features when the pathologist verbally indicates.

[0144] Some embodiments of the present disclosure use a camera, a microphone, and an A.I. system or subsystem to improve the diagnostic workflow to provide hands-free navigation, highlighting, and annotation by means of the vestibulo-ocular reflex (VOR) in combination with voice commands such as "highlight", "select", "deselect", "annotate", "circle", "box", "navigate", "draw polygon", "draw spline", "touch draw", "new layer", "mark", "fix here", "pause", "save point", "compare", "split view", or other such typical commands for graphic editing and / or text editing.

[0145] Some embodiments of the present disclosure use a system or subsystem consisting of a camera, a microphone, and an A.I. software system to improve the diagnostic workflow, providing hands-free navigation by means of head and / or facial movements in combination with voice commands such as "makeup mirror". In such a mode, the user indicates the intention to increase the image magnification by tilting towards the display. The system detects and tracks this movement in real time, thereby adjusting the displayed image accordingly. Similarly, the user indicates the intention to pan left by turning their head to the left, or the intention to pan up by tilting their head upwards, and the intention to pan down by tilting their head downwards. The system detects and tracks this movement in real time, thereby adjusting the displayed image accordingly. In such a "makeup mirror" mode, the user can verbally instruct the system to apply or increase or decrease a "zoom factor" such that a slight movement can cause a large shift in the displayed image and vice versa. Similarly, in the above mode, the user can verbally instruct to "reverse" the relationship between their head movement and the resulting shift of the displayed image. In such a "makeup mirror" mode, the user can verbally instruct the system to apply or increase or decrease a "stabilization factor" such that the displayed image is shifted in a smooth and jitter-free manner regardless of the more subtle and / or seemingly less intentional movements of their face or head. This stabilization would be an important feature for users with degenerative neuromuscular conditions. In some embodiments, in response to receiving a user input that temporarily or permanently disables the head and / or facial navigation feature, tracking of the user's head and / or face is disengaged. During this period, the user can recenter his or her face (re-establish a new origin) and re-enable head and / or facial tracking via a second user input that indicates to the system to resume head and / or facial tracking.

[0146] In some embodiments, the line of sight of the user's eyes is fixed on a fixed element being displayed, and concurrent movements of the face and head are used to indicate the movement of a moving element being displayed and / or the selection of a selected element, which may also be associated with an actuation command.

[0147] In some embodiments, the indicated movement is the movement of an image relative to a fixed cursor, selection box, drawing tool, mask indicator, magnification selection area, or region-specifying graphical display element.

[0148] In some embodiments, the indicated movement is the movement of a cursor, selection box, drawing tool, mask indicator, magnification selection area, or region-specifying graphical display element relative to a fixed image or a portion of an image or sample area.

[0149] In some embodiments, the indicated movement is the movement of a file name, a folder name, or an icon or thumbnail representing a file or folder or multiple files or folders relative to a fixed cursor, selection box, magnified selection area, file designation, or folder designation graphical display element.

[0150] In some embodiments, the indicated movement is the movement of a cursor, selection box, magnified selection area, file designation, or folder designation graphical display element relative to a fixed file name, folder name, or an icon or thumbnail representing a file or folder or multiple files or folders.

[0151] In some embodiments, the indicated movement is the movement of a command, command list, hierarchical command category, or an icon or thumbnail or preview representing a command or command category or multiple commands or command categories relative to a fixed cursor, selection box, magnified selection area, file designation, or folder designation graphical display element.

[0152] In some embodiments, the indicated movement is the movement of a cursor, selection box, magnified selection area, file designation, or folder designation graphical display element relative to a fixed command, command list, hierarchical command category, or an icon or thumbnail or preview representing a command or command category or multiple commands or command categories.

[0153] In some embodiments, the indicated movement is the movement of a setting, setting list, hierarchical setting category, or an icon or thumbnail or preview representing a setting or setting category or multiple settings or setting categories relative to a fixed cursor, selection box, magnified selection area, setting value designation, or setting selection graphical display element.

[0154] In some embodiments, the indicated movement is the movement of a cursor, selection box, magnified selection area, selection designation, or setting designation graphical display element relative to a fixed setting, setting list, hierarchical setting category, or an icon or thumbnail or preview representing a setting value or setting category or multiple setting values or setting categories.

[0155] A version or (each / all of the foregoing technical solutions), wherein the actuation command is voice command activation, button, roller, keystroke, touchpad, touch screen, deliberate blink, foot switch, ball, non-verbal sound activation.

[0156] Some embodiments of the present disclosure use an A.I. system or subsystem to improve the diagnostic workflow. The A.I. system or subsystem consists of a smartphone and a two-part phone holder, where the lower base remains stationary and the movable upper holder secures the phone horizontally with the display facing up. When the user manipulates the phone with their fingertips, the system displays magnified slide images on the phone's display as if the phone were an extreme magnifier sliding around on the actual sample or, conversely, as if the slide were sliding around under an optical microscope. The rear camera of the smartphone senses the movement of the lower base passing beneath it, which can be illuminated by the rear LED of the smartphone as needed. Commutation can be additionally sensed using the phone's on-board sensors or via a Bluetooth paired peripheral device having similar functions, features, and construction to a wireless optical scroll mouse. The upper holder facilitates smooth and precise movement by means of low-friction pads and / or rollers between the upper holder and the lower base.

[0157] Some embodiments of the present disclosure can implement the aforementioned "desktop" mode, where the lower base is the surface of a table or desk, and the aforementioned scroll mouse feature is integrated with the upper holder.

[0158] Some embodiments of the present disclosure can implement the aforementioned "desktop" mode, where the upper holder is a typical smartphone case.

[0159] Some embodiments of the present disclosure can implement the aforementioned "desktop" mode, where the smartphone is configured to be in an inclined state or an adjustable inclined state, i.e., not parallel to the underlying surface. Such an inclination can be oriented to facilitate more effective face tracking by the front camera of the phone.

[0160] Some embodiments of the present disclosure can implement A.I. diagnostic recommendations as draft annotations that the user can confirm, modify, or reject themselves. Such draft pre-annotations can be presented in a synonymous manner to the annotations of collaborating colleagues. Such annotations can be presented without distinction between the recommendations of one or more human colleagues or the recommendations of an A.I. "virtual pathologist" or the anonymized previous annotations of the same user / pathologist. The system can resubmit previously evaluated slides to the user in order to truly measure its own consistency. Such a consistency test can be secretly conducted by the system for a certain part of the workflow process.

[0161] Some embodiments of the present disclosure can implement the aforementioned "desktop" mode, where the stepwise selection, viewing, and annotation of pre-identified diagnostic relevant sample features are controlled by face movement and / or VOR and / or the touch screen of the phone. In such a modality, the user can imaginarily complete the entire diagnostic workflow without holding the phone in their hand as an agent for traditional slide manipulation.

[0162] Some embodiments of the present disclosure can implement the above-mentioned facial tracking navigation in a manner that can be smoothly carried out through various pre-identified diagnostic-related features, units, positions, or annotations.

[0163] Some embodiments of the present disclosure can implement the above-mentioned facial tracking navigation in a non-linear manner through various pre-identified diagnostic-related features, units, positions, or annotations, and "capture" or "pop up" each indexed feature or position or annotation in a manner similar to the navigation behavior associated with the above-mentioned roller embodiments. Such "capture" or "pop up" navigation will be used to accelerate the user's review of samples and images. In such navigation, auditory and visual cues will indicate the step-by-step selection of the corresponding position or feature, and such cues are determined according to the context by the system and / or user-configurable settings. In such embodiments, the system can temporarily or continuously change the sensitivity and / or scale of facial movement tracking to contribute to a more stable or smooth review experience for the user. In such embodiments, the navigation can be between features and / or positions while remaining at or near a single magnification, or alternatively, the magnification can be reduced before advancing to the next position, or alternatively, the navigation can continue to visually approximate apparent flight or boundary arcs in the z-axis. In such embodiments, the system can reduce or attenuate or completely ignore the overshoot aspect of facial tracking in an instantaneous or temporary or persistent or modal manner.

[0164] Some embodiments of the present disclosure can implement the above-mentioned "desktop" mode while mirroring the display output (Miracasting) to a TV.

[0165] Some embodiments of the present disclosure use a system or subsystem composed of a camera, a microphone, and an A.I. software system to improve the diagnostic workflow to provide manual tracking or manual tracking control for navigation, highlighting, and annotation.

[0166] Some embodiments of the present disclosure use a system or subsystem composed of a touch-sensitive sensor, a camera, a microphone, and an A.I. software system to improve the diagnostic workflow to provide the above control modalities in any combination by combining a touch screen, and the sensors are, for example, placed on a table or desktop or a smartphone held in a vertical or semi-vertical bracket.

[0167] Some embodiments of the present disclosure use a system composed of an A.I. software system or subsystem and a 5G smartphone to improve the diagnostic workflow. The 5G smartphone wirelessly interfaces with a nearby large-screen TV in a display mode called "screen mirroring". In such embodiments, the slice images are cloud-hosted and streamed via a 5G mobile network. The voice commands and the speech-to-text transcription of the annotations are completed by the functions of the smartphone. The VOR and facial tracking navigation and / or selection are also implemented by means of the smartphone, one or more cameras, and / or an infrared tracking point graphics projector, and are correspondingly displayed on the TV.

[0168] Some embodiments of the present disclosure use a system consisting of an A.I. software system or subsystem, a 5G smartphone wirelessly interfaced with a nearby large-screen TV, and one or more Bluetooth or WiFi peripheral devices paired with the smartphone, such as a roller, a joystick, a foot switch or variable foot pedal, a trackball, a simple selector button, a mouse, a keyboard, a capacitive proximity sensor, an infrared or ultrasonic motion detector or proximity sensor, one or more discrete or integrated motion-sensing MEMs, accelerometers or strain gauges, a stylus, a mouse, a haptic VR glove, a wand, a laser pointer, VR / AR display goggles, one or more speakers, one or more LED or LCD displays, a head-mounted microphone and / or a head-mounted headset, a remote control handset, a reflective or fluorescent ball or strip or other sensing or motion capture control or feedback devices commercially available for mobile or desktop computing.

[0169] Some embodiments of the present disclosure use a system consisting of an A.I. software system or subsystem, a 5G smartphone wirelessly interfaced with a nearby large-screen TV, and a detent roller to improve the diagnostic workflow. In such embodiments, each detent is indexed to a specific feature or region of diagnostic relevance within the imaging sample, such as tissue features, and / or Cartesian coordinates within the imaging sample, and / or highlighted or annotated portions of the image, and / or an external document or portion of a document or media file or active chat session or collaborative resource with an annotation or external message or hyperlink. Such roller modalities provide unique precise but fast and efficient navigation of a large number of discrete regions or features or image portions.

[0170] Some embodiments of the present disclosure use dynamically detented rollers to improve the diagnostic workflow, the devices changing their audiovisual and tactile behavior as each indexed feature or region is reviewed and annotated, in such a way as to indicate progress and provide a review and / or revision of the progress. Such dynamically detented rollers, consisting of a brushless DC motor integrated with one or more electronic circuit boards and knobs or wheels, may also include LEDs, buttons, membrane buttons, touch sensors, OLED or LCD displays, palm rests, speakers or sound transducers, Hall effect sensors or strain gauges, microphones or piezoelectric transducers or sensors, optical or magnetic commutation sensors or other elements of peripheral devices commercially available for mobile or desktop computing.

[0171] Some embodiments of the present disclosure use a dynamic detent roller to improve the diagnostic workflow, the dynamic detent roller using a brushless DC motor for simulated and dynamically configurable kinetic behavior, such as momentum, resistive inertia, and soft damping. Such embodiments may incorporate strain gauges or other sensors in the base of the roller to detect the manual force axially applied to the top center of the knob for the purpose of XY navigation. Such sensors may also detect a tap action on the top of the roller for select and deselect functions.

[0172] Some embodiments of the present disclosure use a dynamic kinetics roller to improve the diagnostic workflow, the device dynamically changing the apparent inertia and / or soft damping of the wheel or knob by means of a brushless or brushed DC motor or stepper motor or actuated mechanical features (such as friction elements) in combination with a relay, solenoid, or electromagnet or electronically actuated ferrofluid. In such embodiments, the roller may be moved under the control of a remotely collaborating colleague or to indicate their progress for collaborative diagnostic and / or training purposes. Such aforementioned simulated inertia can provide customization of the feel of the device to better suit various users or reduce hand and wrist fatigue. For reasons of similar fatigue and / or user preference, the embodiments may also provide the strength of the detent via user settings.

[0173] Some embodiments of the present disclosure use an A.I. system or subsystem to improve the collaborative diagnostic workflow to select and interconnect one or more collaborative and available resources from a real-time registry of current active pathologists and / or virtual pathologists within a cloud-hosted network. In such embodiments, the aforementioned dynamic roller may be used to obtain synchronous feedback of opinions or ratings from multiple pathologists and / or virtual pathologists, the synchronous feedback being tabulated and / or aggregated by the system according to one or more consistency algorithms or other preferred criteria and practices, either computationally or via a neural network. In such a mode, the roller may serve as a remote haptic and tactile handshake between and among collaborative participants. Such collaborative sessions may be recorded for subsequent review.

[0174] Some embodiments of the present disclosure use an A.I. system or subsystem to improve its own consistency, to measure and evaluate the differences between the diagnostic preferences of a pathologist and the diagnostic tendencies of similar or identical sample images. In the context of training consultations, the system can induce such a re-review of the previously diagnosed slide images. The cases of sample similarity and consistency can be based on the inter-sample similarity in the system's own classification of its diagnostic-related attributes, or according to the standards and practices of medical boards or other regulatory authorities. The system can facilitate the improvement of its own consistency or the continuous due diligence of the pathologist's review by means of a subtle, suggestive flicker of nearby image portions that are diagnostically relevant. The system can continuously improve its own diagnostic relevance criteria to more closely approximate the judgment and workflow patterns of individual pathologists until almost every slide review is an affirmation of the recommended annotations and diagnostic conclusions. This value proposition increases productivity rather than "replacing" the pathologist.

[0175] Some embodiments of the present disclosure use an A.I. system to improve its own consistency, which compares the diagnostic-related features of each slide in a diagnosis with previously similar features, images, and cases diagnosed by pathologists and / or other recognized pathologists and / or committee standards and practices. Such embodiments can pre-fill the annotations with recommended text, which the user can freely confirm, modify, or reject. By continuously monitoring the actual diagnostic workflow and machine learning, the acuity of such systems in diagnostic sensitivity and specificity may ultimately reach an indistinguishable level comparable to that of humans.

[0176] The following description includes multiple embodiments of systems and methods for implementing the concepts described above with reference to Figures 1 - 16 Those described. These embodiments are provided as non-limiting examples.

[0177] In some embodiments, a method includes, at a server system: obtaining an image of a sample (e.g., including a synthetic image derived from a stereo z-stack, composed of pixels, regions, or features selected for diagnostic or therapeutic relevance) (e.g., slicing is not necessarily required; a tissue tape can be scanned directly without mounting onto a slide) (e.g., including specific circumstances associated with the protocol of one or more drug trials and suitable participating candidates matching therewith); identifying one or more cell morphologies of the sample; mapping multiple regions of the image corresponding to the one or more cell morphologies; assigning a diagnostic or therapeutic relevance level to each of the multiple regions; for each region, compressing the multiple regions using a compression level negatively correlated with the assigned diagnostic or therapeutic relevance level of the region (e.g., or a compression type / method negatively correlated with fidelity); receiving a request to view the image from a first client device; and in response to receiving the request to view the image from the first client device, transmitting (i) the compressed multiple regions and (ii) metadata to the first client device, the metadata including an index of the assigned diagnostic or therapeutic relevance levels of the multiple regions.

[0178] In some embodiments, assigning a diagnostic or therapeutic relevance level includes: submitting the image to one or more diagnostic machine vision systems (or human pathologist review); in response to submitting the image, receiving diagnostic or therapeutic relevance data associated with the multiple regions from the one or more diagnostic machine vision systems; and aggregating the diagnostic or therapeutic relevance data received from the one or more diagnostic machine vision systems; wherein the assignment of the diagnostic or therapeutic relevance level is based on the aggregated diagnostic or therapeutic relevance data received from the one or more diagnostic machine vision systems (there can be various methods for combining several such inputs to achieve optimal sensitivity, specificity, and consistency).

[0179] In some embodiments, the method further includes: extracting the multiple regions into multiple discrete alpha layers or images, wherein compressing the multiple regions includes compressing the multiple discrete alpha layers or images; associating a portion of the metadata with each of the multiple discrete alpha layers or images; and encoding or encrypting the portion of the metadata into the multiple discrete alpha layers or images, respectively.

[0180] In some embodiments, identifying one or more cell morphologies of the sample includes compiling a cell index of features of the image using a predefined library of tissue-specific or pathology-specific neural networks.

[0181] In some embodiments, assigning a diagnostic or therapeutic relevance level to each region includes assigning multiple diagnostic or therapeutic relevance grades; and compressing the multiple regions includes using compression levels corresponding respectively to each of the multiple diagnostic or therapeutic relevance grades.

[0182] In some embodiments, the method further comprises: prioritizing the plurality of regions into an ordered sequence of distinct image regions or sample features based on the diagnostic or therapeutic relevance of each of the plurality of regions; and wherein the metadata includes instructions for displaying the plurality of regions in a sequence-based order.

[0183] In some embodiments, the ordered sequence of distinct image regions is optimized based on one or more of: review efficiency; review thoroughness; directionality from one side of the image to the other; linear review of cell morphology; and categorical review of cell morphology.

[0184] In some embodiments, the method further comprises: rendering the ordered sequence of distinct image regions on a display as a three-dimensional fly-through rendering of the image; wherein a first horizontal axis and a second horizontal axis of the three-dimensional fly-through rendering correspond to spatial components of the image, and a vertical axis of the three-dimensional fly-through rendering corresponds to the assigned level of diagnostic or therapeutic relevance of each region of the image.

[0185] In some embodiments, the metadata includes a parameter-based characterization of cells, cell organelles, cell groups or regions, cell states, or tissue morphology of the sample.

[0186] In some embodiments, for each region, the metadata includes a designation of a dedicated generative adversarial network (GAN) model for subsequent reconstruction of the region.

[0187] In some embodiments, for each region, the metadata includes one or more instances from a library of dedicated GAN models for subsequent reconstruction of the region (such a GAN library may consist of hierarchical classes and various levels of categories and specializations).

[0188] In some embodiments, compressing the plurality of regions comprises: downsampling regions of the plurality of regions having a diagnostic or therapeutic relevance below a threshold; and maintaining the original resolution of regions of the plurality of regions having a diagnostic or therapeutic relevance at or above the threshold.

[0189] In some embodiments, downsampling regions of the plurality of regions having a diagnostic or therapeutic relevance below a threshold comprises: downsampling them to an inverse source image layer of hierarchical pixel shifting for subsequent recombined pixel shifting super-resolution at a first client device.

[0190] In some embodiments, the method further comprises, prior to receiving a request to view the image from a first client device: decompressing the plurality of regions into a plurality of reconstructed regions using one or more dedicated GANs; comparing the plurality of reconstructed regions with a pre-compressed version of the plurality of regions; and based on the comparison, determining a difference between the reconstructed regions and the pre-compressed version of the plurality of regions.

[0191] In some embodiments, the method further includes, before receiving a request to view an image from a first client device: determining that a difference between a reconstructed region and a pre-compressed version of a plurality of regions reaches a threshold; updating one or more dedicated GANs based on the determination that the difference between the reconstructed region and the pre-compressed version of the plurality of regions reaches the threshold; and for each region, re-compressing the plurality of regions using the dedicated GAN in the updated one or more dedicated GANs; wherein transmitting the compressed plurality of regions includes transmitting the re-compressed plurality of regions.

[0192] In some embodiments, the method further includes, before receiving a request to view an image from a first client device: determining that a difference between a reconstructed region and a pre-compressed version of a plurality of regions does not reach a threshold; wherein transmitting the compressed plurality of regions is based on the determination that the difference between the reconstructed region and the pre-compressed version of the plurality of regions does not reach the threshold.

[0193] In some embodiments, the method further includes at a server system: storing the compressed plurality of regions and metadata; and deleting the image before receiving a request to view an image from a first client device.

[0194] In some embodiments, the method further includes at a server system: packing the compressed plurality of regions and metadata into a file wrapper; wherein transmitting the compressed plurality of regions and metadata to the first client device includes transmitting the file wrapper to the first client device.

[0195] In some embodiments, the method further includes at a first client device: receiving the compressed plurality of regions and metadata from the server system; decompressing the compressed plurality of regions and metadata; combining the decompressed regions into a reconstructed version of the image or a requested portion thereof; attaching characteristic data corresponding to features of a sample included in the metadata to corresponding regions of the reconstructed version of the image; and displaying, on a display integrated in or communicatively coupled to the first client device, portions of the reconstructed version of the image in an order based on an assigned diagnostic or treatment relevance level indicated by the metadata.

[0196] In some embodiments, the plurality of regions includes a first region having a first-level diagnostic or treatment relevance and a second region having a second-level diagnostic or treatment relevance lower than the first-level diagnostic or treatment relevance; and compressing the plurality of regions includes compressing the first region using a first compression ratio M:1 and compressing the second region using a second compression ratio N:1, where N > M ≥ 1.

[0197] In some embodiments, a plurality of regions includes a first region having a first level of diagnostic or therapeutic relevance and a second region having a second level of diagnostic or therapeutic relevance that is lower than the first level; and compressing the plurality of regions includes compressing the first region using a lossless compression algorithm and compressing the second region using a lossy compression algorithm.

[0198] In some embodiments, a plurality of regions includes a first region having a first level of diagnostic or therapeutic relevance and a second region having a second level of diagnostic or therapeutic relevance that is lower than the first level; and compressing the plurality of regions includes reducing the resolution of the first region to level M and reducing the resolution of the second region to level N, where N > M ≥ 0.

[0199] On the other hand, a method of compressing and transmitting, reconstructing, and presenting an image for diagnostic annotation includes, at a server system including one or more processors: obtaining an image of a sample (e.g., including a synthetic image derived from a stereo z-stack, composed of pixels, regions, or features selected for diagnostic or therapeutic relevance) (e.g., sectioning is not necessarily required; tissue strips can be scanned directly without embedding in sections) (e.g., including specific cases associated with the protocol of one or more drug trials and suitable participating candidates matched thereto); identifying one or more cell morphologies of the sample; mapping a plurality of regions of the image corresponding to the one or more cell morphologies; assigning corresponding diagnostic or therapeutic relevance levels to the plurality of regions; reducing or maintaining the corresponding resolutions of the plurality of regions based on the assigned diagnostic or therapeutic relevance levels, thereby generating a plurality of processed regions; receiving a request to view the image from a first client device; and in response to receiving the request to view the image from the first client device, transmitting (i) the plurality of processed regions and (ii) metadata to the first client device, the metadata including an index of the assigned diagnostic or therapeutic relevance levels of the plurality of processed regions.

[0200] In some embodiments, reducing or maintaining the corresponding resolutions of the plurality of regions based on the assigned diagnostic or therapeutic relevance levels includes reducing the resolution of at least one region of the plurality of regions, including performing inverse pixel shifting on the at least one region.

[0201] In some embodiments, performing reverse pixel shifting on at least one region includes: dividing adjacent pixels of an image into a plurality of pixel groups; combining adjacent pixels of each pixel group among the plurality of pixel groups into a pixel group value (e.g., the combination includes taking an average or other mathematical function or algorithm, including a neural network to predict and mitigate de-bayering artifacts or sensor noise); dividing adjacent pixels of the image into a plurality of shifted pixel groups; averaging adjacent pixels of each shifted pixel group among the plurality of shifted pixel groups into a shifted pixel group value; and replacing adjacent pixels of the image with a plurality of layers, the plurality of layers including (i) a first layer including the pixel group value of each pixel group and (ii) a second layer including the shifted pixel group value of each shifted pixel group.

[0202] In some embodiments, assigning respective diagnostic or treatment relevance levels to a plurality of regions includes assigning a first-level diagnostic or treatment relevance to a first region among the plurality of regions and assigning a second-level diagnostic or treatment relevance lower than the first level to a second region among the plurality of regions; and reducing or maintaining the respective resolutions of the plurality of regions based on the assigned diagnostic or treatment relevance levels includes: reducing the resolution of the first region to level M; and reducing the resolution of the second region to level N, where N > M ≥ 0.

[0203] In some embodiments, assigning respective diagnostic or treatment relevance levels to a plurality of regions includes assigning a first-level diagnostic or treatment relevance to a first region among the plurality of regions and assigning a second-level diagnostic or treatment relevance lower than the first level to a second region among the plurality of regions; and reducing or maintaining the respective resolutions of the plurality of regions based on the assigned diagnostic or treatment relevance levels includes: maintaining the original resolution of the first region based on determining that the diagnostic or treatment relevance level of the first region reaches a threshold; and reducing the resolution of the second region based on determining that the diagnostic or treatment relevance level of the second region does not reach the threshold.

[0204] In some embodiments, reducing or maintaining the respective resolutions of a plurality of regions includes: de-resolving regions among the plurality of regions having a diagnostic or treatment relevance lower than a threshold; and maintaining the original resolution of regions among the plurality of regions having a diagnostic or treatment relevance reaching the threshold.

[0205] In some embodiments, de-resolving a region having a diagnostic or treatment relevance lower than a threshold includes: de-resolving it into an inverse source image layer of hierarchical pixel shifting for subsequent recombined super-resolution at a first client device.

[0206] In some embodiments, assigning a diagnostic or treatment relevance level includes: submitting an image to one or more diagnostic machine vision systems (or for human pathologist review); in response to submitting the image, receiving diagnostic or treatment relevance data associated with multiple regions from one or more diagnostic machine vision systems; and aggregating the diagnostic or treatment relevance data received from one or more diagnostic machine vision systems; wherein the assignment of the diagnostic or treatment relevance level is based on the aggregated diagnostic or treatment relevance data received from one or more diagnostic machine vision systems.

[0207] In some embodiments, the method further includes: extracting the multiple regions into multiple discrete alpha layers or images, wherein reducing or maintaining the respective resolution of the multiple regions includes reducing or maintaining the respective resolution of the multiple discrete alpha layers or images; associating a portion of metadata with each of the multiple discrete alpha layers or images; and separately encoding or encrypting the portion of metadata into the multiple discrete alpha layers or images.

[0208] In some embodiments, identifying one or more cell morphologies of a sample includes compiling a cell index of features of an image using a predefined library of tissue-specific or pathology-specific neural networks.

[0209] In some embodiments, assigning a diagnostic or treatment relevance level to each region includes assigning multiple diagnostic or treatment relevance grades; and reducing or maintaining the respective resolution of the multiple regions includes reducing or maintaining the respective resolution using a degree of resolution reduction corresponding to each of the multiple diagnostic or treatment relevance grades.

[0210] In some embodiments, the method further includes: prioritizing the multiple regions into an ordered sequence of distinct image regions or sample features based on the diagnostic or treatment relevance of each region in the multiple regions; and wherein the metadata includes instructions for displaying the multiple regions in a sequence-based order.

[0211] In some embodiments, the method further includes: rendering the ordered distinct image regions on a display as a three-dimensional fly-through rendering of the image; wherein a first horizontal axis and a second horizontal axis of the three-dimensional fly-through rendering correspond to spatial components of the image, and a vertical axis of the three-dimensional fly-through rendering corresponds to the assigned diagnostic or treatment relevance level of each region of the image.

[0212] In some embodiments, the metadata includes a parameter-based characterization of cells, cell organelles, cell groups or cell regions, cell states, or tissue morphologies of the sample.

[0213] In some embodiments, for each region, the metadata includes a designation of a dedicated generative adversarial network (GAN) model for subsequent reconstruction of the region.

[0214] In some embodiments, for each region, the metadata includes one or more instances from a dedicated GAN model library for subsequent reconstruction of the region (such GAN libraries can consist of hierarchical classes and various categories and levels of specialization).

[0215] In some embodiments, the method further includes, before receiving a request to view an image from a first client device: using one or more dedicated GANs to up-resolve a plurality of regions into a plurality of reconstructed regions; comparing the plurality of reconstructed regions with an original version of the plurality of regions; and based on the comparison, determining a difference between the reconstructed regions and the original version of the plurality of regions.

[0216] In some embodiments, the method further includes, before receiving a request to view an image from a first client device: determining that the difference between the reconstructed regions and the original version of the plurality of regions reaches a threshold; based on the determination that the difference between the reconstructed regions and the original version of the plurality of regions reaches the threshold, updating one or more dedicated GANs; and for each region, using the dedicated GAN in the updated one or more dedicated GANs to re-down-resolve or maintain the corresponding resolution of the plurality of regions; wherein transmitting the plurality of processed regions includes transmitting the plurality of regions at the re-down-resolved or maintained corresponding resolution.

[0217] In some embodiments, the method further includes, before receiving a request to view an image from a first client device: determining that the difference between the reconstructed regions and the original version of the plurality of regions does not reach a threshold; wherein transmitting the plurality of processed regions is based on the determination that the difference between the reconstructed regions and the original version of the plurality of regions does not reach the threshold.

[0218] In some embodiments, the method further includes at the server system: storing the plurality of processed regions and the metadata; and deleting the image before receiving a request to view an image from a first client device.

[0219] In some embodiments, the method further includes at the server system: packing the plurality of processed regions and the metadata into a file wrapper; wherein transmitting the plurality of processed regions and the metadata to the first client device includes transmitting the file wrapper to the first client device.

[0220] In some embodiments, the method further includes at the first client device: receiving the plurality of processed regions and the metadata from the server system; up-resolving at least a subset of the plurality of processed regions and the metadata; combining the up-resolved regions into a reconstructed version of the image; attaching characteristic data corresponding to features of the sample included in the metadata to the corresponding regions of the reconstructed version of the image; and on a display integrated in or communicatively coupled to the first client device, displaying portions of the reconstructed version of the image in an order based on the assigned diagnostic or treatment relevance level indicated by the metadata.

[0221] In some embodiments, the method further comprises, for each region, compressing the plurality of regions using a compression level that is negatively correlated with the assigned diagnostic or therapeutic relevance level of the region.

[0222] In some embodiments, the plurality of regions includes a first region having a first level of diagnostic or therapeutic relevance and a second region having a second level of diagnostic or therapeutic relevance that is lower than the first level; and compressing the plurality of regions includes compressing the first region using a first compression ratio M:1 and compressing the second region using a second compression ratio N:1, where N > M ≥ 1.

[0223] In some embodiments, the plurality of regions includes a first region having a first level of diagnostic or therapeutic relevance and a second region having a second level of diagnostic or therapeutic relevance that is lower than the first level; and compressing the plurality of regions includes compressing the first region using a lossless compression algorithm and compressing the second region using a lossy compression algorithm.

[0224] On the other hand, a method of compressing and transmitting, reconstructing, and presenting an image for diagnostic annotation includes, at a server system including one or more processors: obtaining an image of a sample (e.g., including a synthetic image derived from a stereo z-stack, composed of pixels, regions, or features selected for diagnostic or therapeutic relevance) (e.g., slicing is not necessarily required; a tissue strip can be directly scanned without embedding in a slice) (e.g., including a specific case associated with a protocol of one or more drug trials and a suitable participating candidate matching thereto); identifying one or more cell morphologies of the sample; mapping a plurality of regions of the image corresponding to the one or more cell morphologies; compressing or downsampling at least a subset of the plurality of regions into a plurality of compressed or downsampled image segments; determining a corresponding generative adversarial network (GAN) model corresponding to the respective cell morphologies associated with the respective compressed or downsampled image segments among the plurality of compressed or downsampled image segments; and assigning the corresponding GAN model to the respective compressed or downsampled image segments; receiving a request to view the image from a first client device; and in response to receiving the request to view the image from the first client device, transmitting (i) the plurality of compressed or downsampled image segments and (ii) the corresponding GAN models assigned to the plurality of compressed or downsampled image segments to the first client device.

[0225] In some embodiments, the method further comprises, at the server system: constructing a mapping of the corresponding GAN models assigned to the plurality of compressed or downsampled image segments, wherein segments of the mapping of the corresponding GAN models are linked to the corresponding image segments among the plurality of compressed or downsampled image segments; wherein transmitting the corresponding GAN models includes transmitting the mapping of the corresponding GAN models.

[0226] In some embodiments, the method further comprises, at the server system: compressing the original resolution of at least one of the plurality of regions using a lossless compression algorithm or maintaining the original resolution; ceasing to determine and allocate a corresponding GAN model for the at least one of the plurality of regions; and in response to receiving a request to view an image from a first client device, transmitting to the first client device (iii) at least one region compressed using the lossless compression algorithm or maintaining the original resolution. Algorithm.

[0227] In some embodiments, the method further comprises, at the server system: allocating corresponding diagnostic or treatment relevance levels to the plurality of regions; determining that at least one of the plurality of regions reaches a diagnostic or treatment relevance threshold; determining that a subset of the plurality of regions does not reach a diagnostic or treatment relevance threshold; wherein compressing the original resolution of at least one of the plurality of regions using a lossless compression algorithm or maintaining the original resolution is based on the determination that at least one of the plurality of regions reaches a diagnostic or treatment relevance threshold; and wherein compressing or downsampling the subset of the plurality of regions and allocating a corresponding GAN model to the corresponding compressed or downsampled image segments is based on the determination that the subset of the plurality of regions does not reach a diagnostic or treatment relevance threshold.

[0228] In some embodiments, identifying one or more cell morphologies of a sample comprises compiling a cell index of features of an image using a predefined library of tissue-specific or pathology-specific neural networks.

[0229] In some embodiments, compressing or downsampling comprises downsampling a subset of the plurality of regions to an inverse source image layer of hierarchical pixel shifting for subsequent recombinant pixel shifting super-resolution at a first client device.

[0230] In some embodiments, the method further comprises, before receiving a request to view an image from a first client device: decompressing or super-resolving a subset of regions to a plurality of reconstructed regions using a corresponding GAN model; comparing the plurality of reconstructed regions with a pre-compressed or pre-downsampled version of the subset of regions; and based on the comparison, determining a difference between the reconstructed regions and the pre-compressed or pre-downsampled version of the subset of regions.

[0231] In some embodiments, the method further comprises, before receiving a request to view an image from a first client device: determining that a difference between the reconstructed regions and the pre-compressed or pre-downsampled version of the subset of regions reaches a threshold; based on the determination that the difference between the reconstructed regions and the pre-compressed or pre-downsampled version of the subset of regions reaches a threshold, updating the corresponding GAN model; and re-compressing or re-downsampling a subset of the plurality of regions using the updated corresponding GAN model; wherein transmitting the plurality of compressed or downsampled image segments comprises transmitting the re-compressed or re-downsampled subset of the plurality of regions.

[0232] In some embodiments, the method further includes, before receiving a request to view an image from a first client device: determining that a difference between a reconstructed region and a pre-compressed or pre-downsampled version of a subset of regions does not reach a threshold; wherein transmitting the plurality of compressed or downsampled image segments is based on the determination that the difference between the reconstructed region and the pre-compressed or pre-downsampled version of the subset of regions does not reach the threshold.

[0233] In some embodiments, the method further includes, at the server system: storing the plurality of compressed or downsampled image segments and corresponding GAN models assigned to the plurality of compressed or downsampled image segments; and deleting the image before receiving a request to view an image from a first client device.

[0234] In some embodiments, the method further includes, at the server system: packing the plurality of compressed or downsampled image segments and corresponding GAN models assigned to the plurality of compressed or downsampled image segments into a file wrapper; wherein transmitting the plurality of compressed or downsampled image segments and corresponding GAN models assigned to the plurality of compressed or downsampled image segments to the first client device includes transmitting the file wrapper to the first client device.

[0235] In some embodiments, the method further includes, at the first client device: receiving the plurality of compressed or downsampled image segments and corresponding GAN models assigned to the plurality of compressed or downsampled image segments from the server system; decompressing or super-resolving the compressed or downsampled image segments using the corresponding GAN models assigned to the plurality of compressed or downsampled image segments; combining the decompressed or super-resolved image segments into a reconstructed version of the image or a requested portion thereof; and displaying a portion of the reconstructed version of the image on a display integrated in or communicatively coupled to the first client device.

[0236] In another aspect, a method of processing and transmitting an image for diagnostic analysis includes, at a server system including one or more processors: obtaining an input image of a sample; globally downsampling the input image to a downsampled image; after globally downsampling the input image to a downsampled image, simultaneously: globally upsampling the downsampled image to an upsampled image using a generative adversarial network (GAN) model configured to reconstruct an image including features corresponding to the sample; and classifying a plurality of regions of the downsampled image based on cell morphology and / or diagnostic relevance; and transmitting the upsampled image to a communication network for delivery to a client device.

[0237] In some embodiments, the method further includes dividing the input image into a plurality of tiles, wherein: globally downsampling the input image includes downsampling each of the plurality of tiles; and globally upsampling the downsampled image includes upsampling each of the plurality of tiles.

[0238] In some embodiments, globally upsampling the downsampled image includes predictively enhancing the clarity of the downsampled image using a GAN model. In some embodiments, globally upsampling the downsampled image includes restoring deleted pixels by using a GAN model to predict pixel values corresponding to the deleted pixels. In some embodiments, globally upsampling the downsampled image includes overwriting the downsampled pixel values with the pixel values predicted by the GAN model.

[0239] In some embodiments, the method further includes compressing the upsampled image using a run-length encoding scheme before transmitting the upsampled image to the communication network.

[0240] In some embodiments, the method further includes manipulating a portion of the input image based on the classification of the plurality of regions for subsequent processing. In some embodiments, the subsequent processing includes re-globally downsampling the input image having the manipulated portion, and simultaneously globally upsampling and classifying the plurality of regions of the re-globally downsampled image.

[0241] In some embodiments, a system includes: one or more processors of a server or a client device, and a memory storing instructions that, when executed by the one or more processors, cause the server or the client device to perform any of the above methods.

[0242] In some embodiments, a non-transitory computer-readable storage medium stores instructions that, when executed by a server or a client device, cause the server or the client device to perform any of the above methods.

[0243] In another aspect, a method of processing and transmitting images for diagnostic analysis includes, at a server system including one or more processors: obtaining an input image of a sample, where the input image includes image data representing a flattened z-stack; classifying spectral differences of multiple features of the input image; assigning z-levels of the z-stack to each of the multiple features based on the classification, including assigning one or more first z-levels (blood cells in lower z-levels) to a first subset of the multiple features and one or more second z-levels (blood cells in higher z-levels) to a second subset of the multiple features, where the one or more first z-levels are below the one or more second z-levels, thereby obscuring portions of the first subset of the multiple features; (e.g., at least a portion of the lower blood cells is obscured by at least a portion of the higher blood cells); predicting pixel values associated with the obscured portions of the first subset of the multiple features using a generative adversarial network (GAN) model configured to reconstruct image features; generating three-dimensional (3D) image data that includes the predicted pixel values and includes image data from the one or more first z-levels and the one or more second z-levels, thereby representing a virtual reconstructed 3D z-stack; and providing the generated 3D image data for display on a client device.

[0244] In some embodiments, generating the 3D image data includes: selecting multiple pixel values that span multiple z-levels and include at least a portion of the predicted pixel values that reach a predetermined sharpness threshold; and replacing pixel values corresponding to the obscured pixels with the selected pixel values.

[0245] In some embodiments, generating the 3D image data includes: selecting multiple pixel values that span multiple z-levels and include at least a portion of the predicted pixel values that reach a predetermined diagnostic or therapeutic relevance threshold; and replacing pixel values corresponding to the obscured pixels with the selected pixel values.

[0246] In some embodiments, classifying the spectral differences includes classifying the boundaries of the features based on which spectral portions are dominant.

[0247] In some embodiments, providing the generated 3D image data for display includes implementing approximate navigation in a z-field of the z-stack by mapping the multiple z-levels of the z-stack to corresponding control levels associated with control user input elements at the client device. In some embodiments, the control user input elements are sliders, knobs, zoom controls, or z-field navigation controls. In some embodiments, the approximate navigation in the z-stack is triggered after a zoom threshold is met. In some embodiments, generating the 3D image data includes generating virtual slices or non-planar virtual surfaces at an angle that bisects the multiple z-levels.

[0248] In some embodiments, a system includes: one or more processors of a server or a client device, and a memory storing instructions that, when executed by the one or more processors, cause the server or the client device to perform any of the above methods.

[0249] In some embodiments, a non-transitory computer-readable storage medium stores instructions that, when executed by a server or a client device, cause the server or the client device to perform any of the above methods.

[0250] Those skilled in the art will appreciate that the exemplary embodiments can be modified without departing from the broad inventive concept shown and described above. Accordingly, it is to be understood that the invention is not limited to the exemplary embodiments shown and described, but is intended to cover modifications within the spirit and scope of the invention as defined by the claims.

[0251] For example, the specific features of the exemplary embodiments may or may not be part of the claimed invention, different components contrary to those specifically mentioned may perform at least some of the features described herein, and the features of the disclosed embodiments may be combined.

[0252] As used herein, the terms "about" and "approximately" may refer to + / - 10% of the referenced value. For example, "about 9" should be understood to cover 8.2 and 9.9.

[0253] It should be understood that at least some of the figures and descriptions of the present invention have been simplified to focus on elements relevant to a clear understanding of the present invention, while eliminating other elements that those of ordinary skill in the art will understand may also be part of the present invention for the sake of clarity. However, since these elements are well known in the art and since they do not necessarily contribute to a better understanding of the present invention, descriptions of these elements are not provided herein.

[0254] It should be understood that although the terms "first", "second", etc. are sometimes used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another.

[0255] For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without changing the meaning of the description, as long as all occurrences of the "first element" are consistently renamed and all occurrences of the second element are consistently renamed. The first element and the second element are both elements, but they are not the same element.

[0256] As used herein, depending on the context, the term "if" may optionally be construed to mean "when" or "in response to determining" or "in response to detecting" or "in accordance with a determination". Similarly, depending on the context, the phrase "if determined" or "if [stated condition or event] is detected" is optionally construed to mean "after determining" or "in response to determining" or "after detecting [stated condition or event]" or "in response to detecting [stated condition or event]" or "in accordance with a determination detecting [stated condition or event]".

[0257] The terms used herein are for the following purposes: to describe particular embodiments only and are not intended to limit the claims. For example, in addition to or as an alternative to the above medical imaging examples, the above image processing concepts can be used for non-medical images. Any image data, regardless of its content (medical or non-medical), can be processed by the image processing platform described herein using the same functions and modules.

[0258] As used in the description of the embodiments and the appended claims, the singular forms "a / an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0259] It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0260] It will be further understood that the term "comprises and / or comprising", when used in this specification, specifies the presence of the stated features, integers, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, operations, elements, components, and / or groups thereof.

[0261] As used herein, depending on the context, the term "if" can be construed to mean "when" or "after" or "in response to determining" or "in accordance with a determination" or "in response to detecting".

[0262] Similarly, the phrase "if determined (the stated prerequisite is true)" or "if (the stated prerequisite is true)" or "when (the stated prerequisite is true)" can be construed to mean "after determining" or "in response to determining" or "in accordance with a determination" or "after detecting..." or "in response to detecting", depending on the context, the stated prerequisite is true.

[0263] In addition, insofar as the method is not dependent on a particular order of the steps set forth herein, the particular order of the steps should not be construed as limiting the claims. The claims for the method of the present invention should not be limited to the steps being performed in the order written, and those skilled in the art can readily understand that the steps can be changed and still remain within the spirit and scope of the present invention.

Claims

1. A method for compressing, transmitting, reconstructing, and presenting an image for diagnostic annotation, the method comprising: at a server system including one or more processors: obtaining an image of a sample; identifying one or more cell morphologies of the sample; mapping a plurality of regions of the image corresponding to the one or more cell morphologies; assigning a level of diagnostic or therapeutic relevance to each of the plurality of regions; for each region, compressing the plurality of regions using a compression level negatively correlated with the assigned level of diagnostic or therapeutic relevance of the region; receiving a request to view the image from a first client device; and in response to receiving the request to view the image from the first client device, transmitting (i) the compressed plurality of regions and (ii) metadata to the first client device, the metadata including an index of the assigned levels of diagnostic or therapeutic relevance of the plurality of regions.

2. The method according to claim 1, wherein assigning the level of diagnostic or therapeutic relevance comprises: submitting the image to one or more diagnostic machine vision systems; in response to submitting the image, receiving diagnostic or therapeutic relevance data associated with the plurality of regions from the one or more diagnostic machine vision systems; and aggregating the diagnostic or therapeutic relevance data received from the one or more diagnostic machine vision systems; wherein the assignment of the level of diagnostic or therapeutic relevance is based on the aggregated diagnostic or therapeutic relevance data received from the one or more diagnostic machine vision systems.

3. The method according to any one of the preceding claims, further comprising: extracting the plurality of regions into a plurality of discrete alpha layers or images, wherein compressing the plurality of regions comprises compressing the plurality of discrete alpha layers or images; associating a portion of the metadata with each of the plurality of discrete alpha layers or images; and encoding or encrypting the portion of the metadata into the plurality of discrete alpha layers or images, respectively.

4. The method according to any one of the preceding claims, wherein identifying the one or more cell morphologies of the sample comprises compiling a cell index of features of the image using a predefined library of tissue-specific or pathology-specific neural networks.

5. The method according to any one of the preceding claims, wherein: assigning the level of diagnostic or therapeutic relevance to each region comprises assigning a plurality of diagnostic or therapeutic relevance grades; and compressing the plurality of regions comprises using compression levels respectively corresponding to each of the plurality of diagnostic or therapeutic relevance grades.

6. The method according to any one of the preceding claims, further comprising: sorting the plurality of regions into an ordered sequence of distinct image regions or sample features based on the diagnostic or therapeutic relevance of each region of the plurality of regions; and wherein the metadata includes instructions for displaying the plurality of regions in an order based on the sequence.

7. The method according to claim 6, wherein the ordered sequence of distinct image regions is optimized based on one or more of the following: review efficiency; review thoroughness; Directionality from one side to the other side of the image; Linear review of cell morphology; and Categorical review of cell morphology.

8. The method according to claim 6, further comprising: Rendering the ordered different image regions on a display as a three-dimensional fly-through rendering of the image; Wherein a first horizontal axis and a second horizontal axis of the three-dimensional fly-through rendering correspond to spatial components of the image, and a vertical axis of the three-dimensional fly-through rendering corresponds to the assigned diagnostic or therapeutic relevance level of each region of the image.

9. The method according to any one of the preceding claims, wherein the metadata contains a parameter-based characterization of cells, cell organelles, cell groups or cell regions, cell states or tissue morphology of the sample.

10. The method according to any one of the preceding claims, wherein for each region, the metadata contains a designation of a dedicated generative adversarial network (GAN) model for subsequent reconstruction of the region.

11. The method according to claim 10, wherein for each region, the metadata contains one or more instances from a library of dedicated GAN models for subsequent reconstruction of the region.

12. The method according to any one of the preceding claims, wherein compressing the plurality of regions comprises: Downsampling regions in the plurality of regions with a diagnostic or therapeutic relevance below a threshold; and Maintaining the original resolution of regions in the plurality of regions with a diagnostic or therapeutic relevance reaching the threshold.

13. The method according to claim 12, wherein downsampling the regions with a diagnostic or therapeutic relevance below the threshold comprises: downsampling them to an inverse source image layer of hierarchical pixel shift for subsequent recombined pixel shift super-resolution at the first client device.

14. The method according to any one of the preceding claims, further comprising before receiving the request to view the image from the first client device: Using one or more dedicated GANs to decompress the plurality of regions into a plurality of reconstructed regions; Comparing the plurality of reconstructed regions with a pre-compressed version of the plurality of regions; And Based on the comparison, determining a difference between the reconstructed regions and the pre-compressed version of the plurality of regions.

15. The method according to claim 14, further comprising before receiving the request to view the image from the first client device: Determining that the difference between the reconstructed regions and the pre-compressed version of the plurality of regions reaches a threshold; Update the one or more dedicated GANs based on the determination that the difference between the reconstructed region and the pre-compressed version of the plurality of regions reaches the threshold; And For each region, re-compressing the plurality of regions using a dedicated GAN in an updated one or more dedicated GANs; Wherein transmitting the compressed plurality of regions comprises transmitting the re-compressed plurality of regions.

16. The method according to claim 14, further comprising before receiving the request to view the image from the first client device: Determining that the difference between the reconstructed regions and the pre-compressed version of the plurality of regions does not reach the threshold; wherein transmitting the compressed plurality of regions is based on the determination that the difference between the reconstructed region and the pre-compressed version of the plurality of regions does not reach the threshold.

17. The method according to any one of the preceding claims, further comprising: at the server system: storing the compressed plurality of regions and the metadata; and deleting the image before receiving the request to view the image from the first client device.

18. The method according to any one of the preceding claims, further comprising: at the server system: packing the compressed plurality of regions and the metadata into a file wrapper; wherein transmitting the compressed plurality of regions and the metadata to the first client device comprises transmitting the file wrapper to the first client device.

19. The method according to any one of the preceding claims, further comprising: at the first client device: receiving the compressed plurality of regions and the metadata from the server system; decompressing the compressed plurality of regions and the metadata; combining the decompressed regions into a reconstructed version of the image or a requested portion thereof; appending the characteristic data corresponding to the characteristics of the sample included in the metadata to the corresponding regions of the reconstructed version of the image; and displaying, on a display integrated in or communicatively coupled to the first client device, portions of the reconstructed version of the image in an order based on the assigned diagnostic or treatment relevance level indicated by the metadata.

20. The method according to any one of the preceding claims, wherein: the plurality of regions includes a first region having a first level of diagnostic or treatment relevance and a second region having a second level of diagnostic or treatment relevance lower than the first level; and compressing the plurality of regions includes compressing the first region using a first compression ratio M:1 and compressing the second region using a second compression ratio N:1, where N > M ≥ 1.

21. The method according to any one of the preceding claims, wherein: the plurality of regions includes a first region having a first level of diagnostic or treatment relevance and a second region having a second level of diagnostic or treatment relevance lower than the first level; and compressing the plurality of regions includes compressing the first region using a lossless compression algorithm and compressing the second region using a lossy compression algorithm.

22. The method according to any one of the preceding claims, wherein: the plurality of regions includes a first region having a first level of diagnostic or treatment relevance and a second region having a second level of diagnostic or treatment relevance lower than the first level; and compressing the plurality of regions includes reducing the resolution of the first region to level M and reducing the resolution of the second region to level N, where N > M ≥ 0.

23. A system, comprising: One or more processors of a server or a client device, and a memory storing instructions that, when executed by the one or more processors, cause the server or the client device to perform any of the methods according to claims 1 to 22.

24. A non-transitory computer-readable storage medium storing instructions that, when executed by a server or a client device, cause the server or the client device to perform any of the methods according to claims 1 to 22.

25. A method for compressing and transmitting, reconstructing and presenting an image for diagnostic annotation, the method comprising: At a server system including one or more processors: Obtaining an image of a sample; Identifying one or more cell morphologies of the sample; Mapping a plurality of regions of the image corresponding to the one or more cell morphologies; Assigning a corresponding diagnostic or treatment relevance level to the plurality of regions; Reducing or maintaining the corresponding resolution of the plurality of regions based on the assigned diagnostic or treatment relevance level, thereby generating a plurality of processed regions; Receiving a request to view the image from a first client device; And In response to receiving the request to view the image from the first client device, transmitting (i) the plurality of processed regions and (ii) metadata to the first client device, the metadata including an index of the assigned diagnostic or treatment relevance levels of the plurality of processed regions.

26. The method according to claim 25, wherein: Reducing or maintaining the corresponding resolution of the plurality of regions based on the assigned diagnostic or treatment relevance level includes reducing the resolution of at least one of the plurality of regions, including performing reverse pixel shifting on the at least one region.

27. The method according to claim 26, wherein performing reverse pixel shifting on the at least one region includes: Dividing adjacent pixels of the image into a plurality of pixel groups; Combining adjacent pixels of each pixel group of the plurality of pixel groups into a pixel group value; Dividing adjacent pixels of the image into a plurality of shifted pixel groups; Averaging adjacent pixels of each shifted pixel group of the plurality of shifted pixel groups into a shifted pixel group value; And Replacing the adjacent pixels of the image with a plurality of layers, the plurality of layers including (i) a first layer including the pixel group values of each pixel group and (ii) a second layer including the shifted pixel group values of each shifted pixel group.

28. The method according to any of the preceding claims, wherein: Assigning a corresponding diagnostic or treatment relevance level to the plurality of regions includes assigning a first-level diagnostic or treatment relevance to a first region of the plurality of regions, and assigning a second-level diagnostic or treatment relevance lower than the first level to a second region of the plurality of regions; and Reducing or maintaining the corresponding resolution of the plurality of regions based on the assigned diagnostic or treatment relevance level includes: Reducing the resolution of the first region to level M; and Reducing the resolution of the second region to level N, where N > M ≥ 0.

29. The method according to any one of the preceding claims, wherein: Assigning a respective diagnostic or therapeutic relevance level to the plurality of regions includes assigning a first-level diagnostic or therapeutic relevance to a first region among the plurality of regions, and assigning a second-level diagnostic or therapeutic relevance lower than the first level to a second region among the plurality of regions; and Reducing or maintaining the respective resolution of the plurality of regions based on the assigned diagnostic or therapeutic relevance level includes: Maintaining the original resolution of the first region based on determining that the diagnostic or therapeutic relevance level of the first region reaches a threshold; and Reducing the resolution of the second region based on determining that the diagnostic or therapeutic relevance level of the second region does not reach the threshold.

30. The method according to any one of the preceding claims, wherein reducing or maintaining the respective resolution of the plurality of regions includes: Downsampling regions among the plurality of regions having a diagnostic or therapeutic relevance lower than a threshold; and Maintaining the original resolution of regions among the plurality of regions having a diagnostic or therapeutic relevance reaching the threshold.

31. The method according to claim 30, wherein downsampling the regions having a diagnostic or therapeutic relevance lower than the threshold includes: downsampling them to an inverse source image layer of hierarchical pixel shifting for subsequent reconstruction super-resolution at the first client device.

32. The method according to any one of the preceding claims, wherein assigning the diagnostic or therapeutic relevance level includes: Submitting the image to one or more diagnostic machine vision systems; Receiving, in response to submitting the image, diagnostic or therapeutic relevance data associated with the plurality of regions from the one or more diagnostic machine vision systems; and Aggregating the diagnostic or therapeutic relevance data received from the one or more diagnostic machine vision systems; wherein the assignment of the diagnostic or therapeutic relevance level is based on the aggregated diagnostic or therapeutic relevance data received from the one or more diagnostic machine vision systems.

33. The method according to any one of the preceding claims, further comprising: Extracting the plurality of regions into a plurality of discrete alpha layers or images, wherein reducing or maintaining the respective resolution of the plurality of regions includes reducing or maintaining the respective resolution of the plurality of discrete alpha layers or images; Associating a part of the metadata with each of the plurality of discrete alpha layers or images; And Encoding or encrypting the part of the metadata into the plurality of discrete alpha layers or images respectively.

34. The method according to any one of the preceding claims, wherein identifying the one or more cell morphologies of the sample includes compiling a cell index of features of the image using a predefined tissue-specific or pathology-specific neural network library.

35. The method according to any one of the preceding claims, wherein: Assigning the diagnostic or therapeutic relevance level to each region includes assigning a plurality of diagnostic or therapeutic relevance grades; and Reducing or maintaining the respective resolution of the plurality of regions includes reducing or maintaining the respective resolution using a degree of downsampling corresponding respectively to each of the plurality of diagnostic or therapeutic relevance grades.

36. The method according to any one of the preceding claims, further comprising: Based on the diagnostic or therapeutic relevance of each of the plurality of regions, prioritizing the plurality of regions into an ordered sequence of distinct image regions or sample features; And Wherein the metadata includes instructions for displaying the plurality of regions in the order based on the sequence.

37. The method according to claim 36, further comprising: Rendering the ordered distinct image regions on a display as a three-dimensional fly-through rendering of the image; Wherein a first horizontal axis and a second horizontal axis of the three-dimensional fly-through rendering correspond to spatial components of the image, and a vertical axis of the three-dimensional fly-through rendering corresponds to the assigned diagnostic or therapeutic relevance level of each region of the image.

38. The method according to any one of the preceding claims, wherein the metadata includes a parameter-based characterization of cells, cell organelles, cell groups or cell regions, cell states or tissue morphology of the sample.

39. The method according to any one of the preceding claims, wherein for each region, the metadata includes a specification of a dedicated generative adversarial network (GAN) model for subsequent reconstruction of the region.

40. The method according to claim 39, wherein for each region, the metadata includes one or more instances from a library of dedicated GAN models for subsequent reconstruction of the region.

41. The method according to any one of the preceding claims, further comprising before receiving the request to view the image from the first client device: Upscaling the plurality of regions to a plurality of reconstructed regions using one or more dedicated GANs; Comparing the plurality of reconstructed regions with an original version of the plurality of regions; And Based on the comparison, determining a difference between the reconstructed regions and the original version of the plurality of regions.

42. The method according to claim 41, further comprising before receiving the request to view the image from the first client device: Determining that the difference between the reconstructed regions and the original version of the plurality of regions reaches a threshold; Based on the determination that the difference between the reconstructed region and the original version of the plurality of regions reaches the threshold, update the one or more dedicated GANs; And For each region, using a dedicated GAN in an updated one or more dedicated GANs to re-downscale or maintain the corresponding resolution of the plurality of regions; Wherein transmitting the plurality of processed regions includes transmitting the plurality of regions at the corresponding re-downscaled or maintained resolution.

43. The method according to claim 41, further comprising before receiving the request to view the image from the first client device: Determining that the difference between the reconstructed regions and the original version of the plurality of regions does not reach the threshold; Wherein transmitting the plurality of processed regions is based on the determination that the difference between the reconstructed regions and the original version of the plurality of regions does not reach the threshold.

44. The method according to any one of the preceding claims, further comprising: At the server system: Storing the plurality of processed regions and the metadata; And Delete the image before receiving the request to view the image from the first client device.

45. The method according to any one of the preceding claims, further comprising: At the server system: Pack the plurality of processed regions and the metadata into a file wrapper; Wherein transmitting the plurality of processed regions and the metadata to the first client device comprises transmitting the file wrapper to the first client device.

46. The method according to any one of the preceding claims, further comprising: At the first client device: Receive the plurality of processed regions and the metadata from the server system; Upscale at least a subset of the plurality of processed regions and the metadata; Combine the upscaled regions into a reconstructed version of the image; Append the characteristic data corresponding to the features of the sample included in the metadata to the corresponding regions of the reconstructed version of the image; And On a display integrated in or communicatively coupled to the first client device, display portions of the reconstructed version of the image in an order based on the assigned diagnostic or treatment relevance levels indicated by the metadata.

47. The method according to any one of the preceding claims, further comprising: For each region, compress the plurality of regions using a compression level negatively correlated with the assigned diagnostic or treatment relevance level of the region.

48. The method according to claim 47, wherein: The plurality of regions includes a first region having a first level of diagnostic or treatment relevance and a second region having a second level of diagnostic or treatment relevance lower than the first level of diagnostic or treatment relevance; and Compressing the plurality of regions includes compressing the first region using a first compression ratio M:1 and compressing the second region using a second compression ratio N:1, where N > M ≥ 1.

49. The method according to claim 47, wherein: The plurality of regions includes a first region having a first level of diagnostic or treatment relevance and a second region having a second level of diagnostic or treatment relevance lower than the first level of diagnostic or treatment relevance; and Compressing the plurality of regions includes compressing the first region using a lossless compression algorithm and compressing the second region using a lossy compression algorithm.

50. A system, comprising: One or more processors of a server or a client device, and a memory storing instructions that, when executed by the one or more processors, cause the server or the client device to perform any one of the methods according to claims 25 to 49.

51. A non-transitory computer-readable storage medium storing instructions that, when executed by a server or a client device, cause the server or the client device to perform any one of the methods according to claims 25 to 49.

52. A method for compressing and transmitting, reconstructing and presenting an image for diagnostic annotation, the method comprising: At a server system comprising one or more processors: Obtain an image of a sample; Identify one or more cell morphologies of the sample; Map multiple regions of the image corresponding to the one or more cell morphologies; Compress or downsample at least a subset of the multiple regions into multiple compressed or downsampled image patches; Determine a corresponding generative adversarial network (GAN) model corresponding to the respective cell morphology associated with the respective compressed or downsampled image patch among the multiple compressed or downsampled image patches; and Assign the respective GAN model to the respective compressed or downsampled image patch; Receive a request to view the image from a first client device; And In response to receiving the request to view the image from the first client device, transmit (i) the multiple compressed or downsampled image patches and (ii) the respective GAN models assigned to the multiple compressed or downsampled image patches to the first client device.

53. The method according to any one of the preceding claims, further comprising: At the server system: Construct a mapping of the respective GAN models assigned to the multiple compressed or downsampled image patches, wherein a segment of the mapping of the respective GAN model is linked to the corresponding image patch among the multiple compressed or downsampled image patches; Wherein transmitting the respective GAN model includes transmitting the mapping of the respective GAN model.

54. The method according to any one of the preceding claims, further comprising: At the server system: Compress the original resolution of at least one of the multiple regions using a lossless compression algorithm or maintain the original resolution; Stop determining and assigning respective GAN models for the at least one of the multiple regions; And In response to receiving the request to view the image from the first client device, transmit (iii) the at least one region compressed with the lossless compression algorithm or maintained at the original resolution to the first client device algorithm.

55. The method according to claim 54, further comprising: At the server system: Assign respective diagnostic or treatment relevance levels to the multiple regions; Determine that the at least one of the multiple regions reaches a diagnostic or treatment relevance threshold; Determine that the subset of the multiple regions does not reach the diagnostic or treatment relevance threshold; Wherein compressing the original resolution of the at least one of the multiple regions using the lossless compression algorithm or maintaining the original resolution is based on the determination that the at least one of the multiple regions reaches the diagnostic or treatment relevance threshold; And Wherein compressing or downsampling the subset of the multiple regions and assigning the respective GAN models to the respective compressed or downsampled image patches is based on the determination that the subset of the multiple regions does not reach the diagnostic or treatment relevance threshold.

56. The method according to any one of the preceding claims, wherein identifying the one or more cell morphologies of the sample includes compiling a cell index of features of the image using a predefined tissue-specific or pathology-specific neural network library.

57. The method according to any one of the preceding claims, wherein said compression or said downsampling comprises downsampling said subset of said plurality of regions to an inverse source image layer of hierarchical pixel shifting for subsequent reconstruction of pixel shifting super-resolution at said first client device.

58. The method according to any one of the preceding claims, further comprising, before receiving said request to view said image from said first client device: using said respective GAN model to decompress or super-resolve said subset of regions into a plurality of reconstructed regions; comparing said plurality of reconstructed regions with a pre-compressed or pre-downsampled version of said subset of regions; and based on said comparison, determining a difference between said reconstructed regions and said pre-compressed or pre-downsampled version of said subset of regions.

59. The method according to claim 58, further comprising, before receiving said request to view said image from said first client device: determining that said difference between said reconstructed regions and said pre-compressed or pre-downsampled version of said subset of regions reaches a threshold; Updating the corresponding GAN model based on the determination that the difference between the reconstructed region and the pre-compressed or pre-downsampled version of the region subset reaches the threshold; and using an updated respective GAN model to re-compress or re-downsample said subset of said plurality of regions; wherein transmitting said plurality of compressed or downsampled image segments comprises transmitting said re-compressed or re-downsampled subset of said plurality of regions.

60. The method according to claim 58, further comprising, before receiving said request to view said image from said first client device: determining that said difference between said reconstructed regions and said pre-compressed or pre-downsampled version of said subset of regions does not reach said threshold; wherein transmitting said plurality of compressed or downsampled image segments is based on said determination that said difference between said reconstructed regions and said pre-compressed or pre-downsampled version of said subset of regions does not reach said threshold.

61. The method according to any one of the preceding claims, further comprising: at said server system: storing said plurality of compressed or downsampled image segments and said respective GAN model assigned to said plurality of compressed or downsampled image segments; and deleting said image before receiving said request to view said image from said first client device.

62. The method according to any one of the preceding claims, further comprising: at said server system: packing said plurality of compressed or downsampled image segments and said respective GAN model assigned to said plurality of compressed or downsampled image segments into a file wrapper; wherein transmitting said plurality of compressed or downsampled image segments and said respective GAN model assigned to said plurality of compressed or downsampled image segments to said first client device comprises transmitting said file wrapper to said first client device.

63. The method according to any one of the preceding claims, further comprising: at said first client device: receiving said plurality of compressed or downsampled image segments and said respective GAN model assigned to said plurality of compressed or downsampled image segments from said server system; Decompress or super-resolve the compressed or downsampled image segments using the respective GAN models assigned to the multiple compressed or downsampled image segments; Combine the decompressed or super-resolved image segments into a reconstructed version of the image or the requested portion thereof; and Display a portion of the reconstructed version of the image on a display integrated in or communicatively coupled to the first client device.

64. A system, comprising: One or more processors of a server or client device, and a memory storing instructions that, when executed by the one or more processors, cause the server or the client device to perform any of the methods according to claims 50 to 63.

65. A non-transitory computer-readable storage medium storing instructions that, when executed by a server or client device, cause the server or the client device to perform any of the methods according to claims 52 to 63.

66. A method of processing and transmitting images for diagnostic analysis, the method comprising: At a server system comprising one or more processors: Obtain an input image of a sample; Globally downsample the input image to a downsampled image; After globally downsampling the input image to the downsampled image, simultaneously: Use a generative adversarial network (GAN) model to globally upsample the downsampled image to an upsampled image, the GAN model being configured to reconstruct an image containing features corresponding to the sample; And Classify multiple regions of the downsampled image based on cell morphology and / or diagnostic relevance; and Transmit the upsampled image to a communication network for delivery to a client device.

67. The method according to any of the preceding claims, further comprising dividing the input image into a plurality of tiles, wherein: Globally downsampling the input image comprises downsampling each of the plurality of tiles; And Globally upsampling the downsampled image comprises upsampling each of the plurality of tiles.

68. The method according to any of the preceding claims, wherein globally upsampling the downsampled image comprises predictively enhancing the clarity of the downsampled image using the GAN model.

69. The method according to any of the preceding claims, wherein globally upsampling the downsampled image comprises restoring deleted pixels by using the GAN model to predict pixel values corresponding to the deleted pixels.

70. The method according to any of the preceding claims, wherein globally upsampling the downsampled image comprises overwriting the downsampled pixel values with pixel values predicted by the GAN model.

71. The method according to any of the preceding claims, further comprising compressing the upsampled image using a run-length encoding scheme before transmitting the upsampled image to the communication network.

72. The method according to any one of the preceding claims, further comprising manipulating a portion of the input image based on the classification of the plurality of regions for subsequent processing.

73. The method according to claim 72, wherein the subsequent processing includes re-performing global downsampling on the input image having the manipulated portion, and simultaneously performing global upsampling and classification on a plurality of regions of the re-globally downsampled image.

74. A system, comprising: One or more processors of a server or a client device, and a memory storing instructions which, when executed by the one or more processors, cause the server or the client device to perform any one of the methods according to claims 66 to 73.

75. A non-transitory computer-readable storage medium storing instructions which, when executed by a server or a client device, cause the server or the client device to perform any one of the methods according to claims 66 to 73.

76. A method of processing and transmitting an image for diagnostic analysis, the method comprising: at a server system including one or more processors: obtaining an input image of a sample, wherein the input image includes image data representing a flattened z-stack; classifying spectral differences of a plurality of features of the input image; assigning a z-level of the z-stack to each of the plurality of features based on the classification, including assigning one or more first z-levels to a first subset of the plurality of features and assigning one or more second z-levels to a second subset of the plurality of features, wherein the one or more first z-levels are below the one or more second z-levels, thereby obscuring a portion of the first subset of the plurality of features; predicting pixel values associated with the occluded portion of the first subset of the plurality of features using a generative adversarial network (GAN) model configured to reconstruct image features; generating three-dimensional (3D) image data, the 3D image data including the predicted pixel values and including image data from the one or more first z-levels and the one or more second z-levels, thereby representing a virtual reconstructed 3D z-stack; and providing the generated 3D image data for display on a client device.

77. The method according to any one of the preceding claims, wherein generating the 3D image data includes: selecting a plurality of pixel values spanning a plurality of the z-levels and including at least a portion of the predicted pixel values that reach a predetermined sharpness threshold; and replacing pixel values corresponding to occluded pixels with the selected pixel values.

78. The method according to any one of the preceding claims, wherein generating the 3D image data includes: selecting a plurality of pixel values spanning a plurality of the z-levels and including at least a portion of the predicted pixel values that reach a predetermined diagnostic or treatment relevance threshold; and replacing pixel values corresponding to occluded pixels with the selected pixel values.

79. The method according to any one of the preceding claims, wherein classifying the spectral differences includes classifying boundaries of the features based on which spectral portions are dominant.

80. The method according to any one of the preceding claims, wherein providing the generated 3D image data for display includes implementing approximate navigation in a z-field including the z-stack by mapping a plurality of z-levels of the z-stack to respective control levels associated with a control user input element at the client device.

81. The method according to claim 80, wherein the control user input element is a slider, a knob, a zoom control, or a z-field navigation control.

82. The method according to claim 80, wherein approximate navigation in the z-stack is triggered after a zoom threshold is met.

83. The method according to any one of the preceding claims, wherein generating the 3D image data includes generating virtual slices or non-planar virtual surfaces at an angle bisecting a plurality of the z-levels.

84. A system, comprising: One or more processors of a server or a client device, and a memory storing instructions that, when executed by the one or more processors, cause the server or the client device to perform any of the methods according to claims 76 to 83.

85. A non-transitory computer-readable storage medium storing instructions that, when executed by a server or a client device, cause the server or the client device to perform any of the methods according to claims 76 to 83.

Images