Systems and methods for efficient latent representations

Abstract

The present disclosure provides a method for processing images, comprising: encoding a medical image using an encoder to generate a latent representation; processing the latent representation using trainable projection layers to generate a projected latent representation; computing feature embeddings for the medical image and the projected latent representation; calculating a loss between the feature embeddings; and optimizing parameters of the projection layers to minimize the loss, wherein the encoder is trained using perceptual loss, patch-based adversarial objectives, and domain-specific embedding consistency loss.

Description

Systems and Methods for Efficient Latent RepresentationsCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The current application claims the benefit of and priority under 35 U.S.C. § 119(e) to U.S. Provisional Patent Application No. 63 / 676,850 entitled “Medical Autoencoders and Image Compression” filed July 29, 2024. The disclosure of U.S. Provisional Patent Application No. 63 / 676,850 is hereby incorporated by reference in its entirety for all purposes.FIELD OF THE INVENTION

[0002] The present disclosure relates to image processing systems, and more particularly to autoencoders for generating compressed latent representations of medical images while preserving clinically relevant features.BACKGROUND

[0003] Medical images are essential diagnostic tools in clinical practice. These images are typically acquired at high resolutions with large fields of view to capture finegrained features necessary for accurate diagnoses. However, the resulting high- resolution medical images can lead to substantial data storage requirements and increased computational complexity for downstream analysis and processing tasks. As medical imaging volumes continue to grow rapidly, there is increasing interest in developing efficient methods to represent and process these large datasets while preserving diagnostically relevant information.

[0004] Autoencoder neural networks have shown promise for learning compact representations of complex data. When applied to medical images, autoencoders can potentially generate downsized latent representations that capture salient features while reducing storage and computational demands. However, preserving clinically important details across diverse imaging modalities and anatomical regions remains challenging. Improved techniques are desired for compressing medical images in ways that maintain high fidelity to the original data and enable efficient downstream processing, without compromising diagnostic quality or clinical utility.SUMMARY OF THE INVENTION

[0005] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0006] Systems and methods for image processing in accordance with embodiments of the invention are illustrated. One embodiment includes a method for processing images, including encoding an image using an encoder to generate a latent representation, processing the latent representation using trainable projection layers to generate a projected latent representation, computing feature embeddings for the image and the projected latent representation, calculating a loss between the feature embeddings, and optimizing parameters of the projection layers to minimize the loss.

[0007] In a further embodiment, the image is a two-dimensional (2D) image selected from the group consisting of X-rays, computed tomography (CT) scans, and magnetic resonance imaging (MRI) scans.

[0008] In still another embodiment, the image is a three-dimensional (3D) volume selected from the group consisting of CT scans and MRI scans.

[0009] In a still further embodiment, the encoder is a convolutional neural network.

[0010] In yet another embodiment, the convolutional neural network includes residual blocks.

[0011] In a yet further embodiment, the trainable projection layers are implemented using Low-Rank Adaptation (LoRA) techniques.

[0012] In another additional embodiment, computing feature embeddings includes using a pre-trained medical vision-language model.

[0013] In a further additional embodiment, the pre-trained medical vision-language model is BiomedCLIP.

[0014] In another embodiment again, the method further includes steps for decoding the latent representation using a decoder to generate a reconstructed image.

[0015] In a further embodiment again, the decoder includes transposed convolutions.

[0016] In still yet another embodiment, the method further includes steps for evaluating the quality of the reconstructed image using perceptual quality metrics.

[0017] In a still yet further embodiment, the perceptual quality metrics include peak signal-to-noise ratio (PSNR) and multi-scale structural similarity index measure (MS- SSIM).

[0018] In still another additional embodiment, the method further includes steps for evaluating the quality of the latent representation using computer-aided diagnosis (CAD) tasks.

[0019] In a still further additional embodiment, the CAD tasks include at least one of malignancy detection on full-field digital mammograms (FFDMs), calcification detection on FFDMs, Breast Imaging-Reporting and Data System (BI-RADS) prediction on FFDMs, bone age prediction on X-rays, fracture detection on wrist X-rays, fracture detection on spine CT scans, fracture classification on head CT scans, and anterior cruciate ligament (ACL) and meniscal tear detection on knee MRIs.

[0020] In still another embodiment again, the encoder and a decoder are trained using a two-stage process comprising a base autoencoder training stage, and a feature preservation stage.

[0021] In a still further embodiment again, the base autoencoder training stage utilizes at least one selected from the group consisting of a perceptual loss, patch-based adversarial objectives, and a domain-specific embedding consistency loss.

[0022] In yet another additional embodiment, the feature preservation stage for 2D imaging modalities comprises freezing parameters of the encoder and the decoder, and fine-tuning the trainable projection layers.

[0023] In a yet further additional embodiment, the feature preservation stage for 3D imaging modalities includes lifting a 2D architecture to 3D using kernel centering inflation.

[0024] In yet another embodiment again, the method further includes steps for storing the latent representation, wherein the latent representation requires less storage space than the image.

[0025] In a yet further embodiment again, the method further includes steps for using the latent representation in downstream computer-aided diagnosis (CAD) tasks,wherein using the latent representation improves computational efficiency compared to using the image.

[0026] In another additional embodiment again, encoding the image includes downsizing the image by a factor selected from the group consisting of 16, 64, and 512.

[0027] In a further additional embodiment again, the method further includes steps for selecting an autoencoder from a plurality of autoencoders based on characteristics of the image, wherein the plurality of autoencoders comprises at least two 2D autoencoders with different downsizing factors and latent channel configurations, and at least two 3D autoencoders with different downsizing factors.

[0028] In still yet another additional embodiment, the plurality of autoencoders comprises four 2D autoencoders with configurations selected from the group consisting of downsizing factor of 16 with 1 latent channel, downsizing factor of 16 with 3 latent channels, downsizing factor of 64 with 1 latent channel, and downsizing factor of 64 with 4 latent channels, and two 3D autoencoders with configurations selected from the group consisting of downsizing factor of 64 with 1 latent channel, and downsizing factor of 512 with 1 latent channel.

[0029] In a further embodiment, encoding the image comprises determining whether the image is a 2D image or a 3D volume, selecting a 2D autoencoder when the image is a 2D image, and selecting a 3D autoencoder when the image is a 3D volume.

[0030] In still another embodiment, the 3D autoencoder is derived from a 2D autoencoder architecture using kernel centering inflation.

[0031] In a still further embodiment, the encoder is trained using at least one selected from the group consisting of perceptual loss, patch-based adversarial objectives, and domain-specific embedding consistency loss.

[0032] One embodiment includes a non-transitory machine readable medium for processing images, including encoding an image using an encoder to generate a latent representation, processing the latent representation using trainable projection layers to generate a projected latent representation, computing feature embeddings for the image and the projected latent representation, calculating a loss between the feature embeddings, and optimizing parameters of the projection layers to minimize the loss. One embodiment includes an image processing system comprising a set of one or moreprocessors, and a set of one or more non-transitory computer-readable media. The set of non-transitory computer-readable media includes program instructions that are executable by the one or more processors such that the apparatus is configured to encode an image using an encoder to generate a latent representation, process the latent representation using trainable projection layers to generate a projected latent representation, compute feature embeddings for the image and the projected latent representation, calculate a loss between the feature embeddings, and optimize parameters of the projection layers to minimize the loss.

[0033] Additional embodiments and features are set forth in part in the description that follows, and in part will become apparent to those skilled in the art upon examination of the specification or may be learned by the practice of the invention. A further understanding of the nature and advantages of the present invention may be realized by reference to the remaining portions of the specification and the drawings, which forms a part of this disclosure.BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The description and claims will be more fully understood with reference to the following figures and data graphs, which are presented as exemplary embodiments of the invention and should not be construed as a complete recitation of the scope of the invention.

[0035] FIG. 1 illustrates a flowchart for a method of processing medical images in accordance with an embodiment of the invention.

[0036] FIG. 2 illustrates an example of optimizing an encoder and a decoder in accordance with an embodiment of the invention.

[0037] FIG. 3 illustrates a flowchart for a process of training an autoencoder in accordance with an embodiment of the invention.

[0038] FIG. 4 illustrates an example of refining latent representations in accordance with an embodiment of the invention.

[0039] FIG. 5 illustrates a flowchart of a method for training projection layers in accordance with an embodiment of the invention.

[0040] FIG. 6 illustrates an example of a system for processing three-dimensional (3D) images in accordance with an embodiment of the invention.

[0041] FIG. 7 illustrates a flowchart for a method of processing 3D images in accordance with an embodiment of the invention.

[0042] FIG. 8 illustrates an example of evaluation tasks for medical image autoencoders in accordance with an embodiment of the invention.

[0043] FIG. 9 illustrates an example of CAD model efficiency in accordance with an embodiment of the invention.

[0044] FIG. 10 illustrates a graph showing manual perceptual quality evaluations with expert readers in accordance with an embodiment of the invention.

[0045] FIG. 11 illustrates qualitative examples of reconstructed medical images in accordance with an embodiment of the invention.

[0046] FIG. 12 illustrates an example of an image processing system that processes images in accordance with an embodiment of the invention.

[0047] FIG. 13 illustrates an example of an image processing element that processes images in accordance with an embodiment of the inventionDETAILED DESCRIPTION

[0048] The following description sets forth exemplary aspects of the present disclosure. It should be recognized, however, that such description is not intended as a limitation on the scope of the present disclosure. Rather, the description also encompasses combinations and modifications to those exemplary aspects described herein.

[0049] In many embodiments, image processing systems utilize autoencoders for processing images. Autoencoders in accordance with several embodiments of the invention include an encoder and a decoder. In a number of embodiments, the encoder and decoder are implemented as variational autoencoders (VAEs). VAEs may offer certain advantages in generating latent representations of medical images.

[0050] In a number of embodiments, encoders generate latent representations of input medical images. Latent representations may be a compressed or lower-dimensional representations of an original image. In several embodiments, latent representations undergo further processing using trainable projection layers to generate a projected latent representation from the initial latent representation. Projected latent representations in accordance with a number of embodiments of the invention can improve the preservationof clinically relevant features while maintaining the computational efficiency benefits of the compressed format.

[0051] Image processing systems in accordance with some embodiments of the invention employ a two-stage training approach for handling both two-dimensional (2D) and three-dimensional (3D) images. In a number of embodiments, the training approach includes a first stage of training a set of one or more autoencoders and a second stage of refining the trained autoencoders. Refining in accordance with numerous embodiments of the invention can include training trainable projection layers on projected latent representations, lifting 2D autoencoder architectures to 3D, and / or training a 3D autoencoder on 3D volumes. In several embodiments, encoders perform an initial training using one or more objectives. Objectives in accordance with some embodiments of the invention may include (but are not limited to) perceptual loss, patch-based adversarial objectives, and / or domain-specific embedding consistency loss. Such training approaches may enhance the encoder's ability to capture relevant image features.

[0052] Refinement for image processing systems in accordance with a number of embodiments of the invention can perform optimizations of trainable projection layers based on projected latent representations. In certain embodiments, refinements can be based on a loss between feature embeddings of a source image and its projected latent representation. Feature embeddings may capture relevant characteristics or patterns within the images. Loss calculations in accordance with a variety of embodiments of the invention can assess the similarity (or difference) between an image and its compressed representation. Image processing systems in accordance with a variety of embodiments optimize parameters of the projection layers to minimize the calculated loss. In many embodiments, optimization helps preserve important features from an original image in its latent representation.

[0053] Image processing systems in accordance with many embodiments are capable of handling various types of medical images. Medical images in accordance with several embodiments of the invention include 2D images (e.g., X-rays, computed tomography (CT) scans, and / or magnetic resonance imaging (MRI) scans) and / or 3D volumes (e.g., CT scans and / or MRI scans). Although many of the examples described herein describe medical applications, one skilled in the art will recognize that similarsystems and methods can be used in a variety of imaging applications, without departing from this invention.

[0054] An example of a process for processing images in accordance with an embodiment of the invention is illustrated in FIG. 1. In many embodiments, image processing systems employ a two-stage training process for handling both two- dimensional (2D) and three-dimensional (3D) images. Process 100 optimizes (105) an encoder and a decoder. In a variety of embodiments, optimizing an encoder and / or a decoder may be performed using one or more objectives. Objectives in accordance with certain embodiments of the invention include (but are not limited to) perceptual loss, patch-based adversarial objectives, and / or domain-specific embedding consistency loss. In several embodiments, encoders are initialized with weights from a previously- developed natural image autoencoder, such as (but not limited to) KL-VAE.

[0055] Process 100 refines (110) a latent space to preserve features of an input image in a latent representation. In a number of embodiments, refining latent spaces to preserve clinically-relevant features can be performed across multiple imaging modalities (e.g., across 2D and 3D images). For 2D imaging modalities, refining latent spaces in accordance with several embodiments of the invention is based on trained autoencoders (e.g., trained in the first stage) and freezing their parameters to train projection layers. This may allow for the preservation of learned features while adapting to specific characteristics of medical images.

[0056] In various embodiments, refining latent spaces for 3D imaging modalities involves lifting a 2D architecture to 3D. This lifting process may be accomplished using techniques such as (but not limited to) kernel centering inflation. By adapting 2D architectures to 3D, image processing systems in accordance with numerous embodiments of the invention may efficiently handle volumetric data while leveraging knowledge gained from 2D image processing. Refining latent spaces for 3D imaging modalities in accordance with numerous embodiments of the invention includes training 3D encoders on domain-specific 3D images. For example, 3D medical images may include (but are not limited to) CT scans and MRI scans.

[0057] Training processes in accordance with many embodiments of the invention enable image processing systems to effectively handle diverse imaging modalities whilepreserving clinically-relevant features. Such approaches may contribute to the development of robust and versatile autoencoder models for medical image analysis.

[0058] An example of optimizing an encoder and a decoder in accordance with an embodiment of the invention is illustrated in FIG. 2. The first step 201 illustrates an encoder 205 and a decoder 210. In many embodiments, encoders include (but are not limited to) convolutional neural networks (CNNs). In this example, encoder 205 receives an input image and generates a latent representation. Latent representations in accordance with a variety of embodiments of the invention are compressed or lowerdimensional representations of an image. Decoder 210 receives the latent representation and reconstructs an output image from the latent representation. Decoders in accordance with a number of embodiments of the invention are trained to reconstruct an image based on latent representations generated by an encoder.

[0059] The second step 202 illustrates that the input image and the generated reconstruction are fed to optimizer 220. Optimizer 220 optimizes encoder 205 and decoder 210 based on a computed loss between the input image and the generated reconstruction. Losses in accordance with a number of embodiments of the invention are computed based on a set of one or more objectives. Objectives in accordance with certain embodiments of the invention include (but are not limited to) perceptual loss, patch-based adversarial objectives, and / or domain-specific embedding consistency loss.

[0060] An example of a process for training an autoencoder in accordance with an embodiment of the invention is illustrated in FIG. 3. Process 300 generates (305) a latent representation from an input image using an encoder. Latent representations in accordance with several embodiments of the invention are compressed, lowerdimensional representation of the input image that captures essential features while reducing data size. In many embodiments, encoders transform high-dimensional input images into compact latent representations through a series of convolutional operations that progressively extract and condense relevant image features. In a variety of embodiments, generating latent representations can include a downsizing process that reduces the dimensionality of the input image by factors (e.g., 16, 64, 512, etc.) significantly decreasing storage requirements while maintaining clinically relevant information.

[0061] Process 300 feeds (310) the latent representations from the encoder to the decoder to generate a reconstruction. In many embodiments, decoders may employ a series of transposed convolutional operations to progressively upsample the latent representation and reconstruct an image with dimensions matching the original input. Reconstruction in accordance with many embodiments of the invention may involve multiple stages of feature expansion and refinement to enhance the quality of the reconstructed image.

[0062] Process 300 calculates (315) a loss between the input image and the reconstruction. Loss calculation in accordance with many embodiments of the invention quantifies the difference between the original input image and its reconstructed version. In several embodiments, loss functions may include pixel-wise metrics such as (but not limited to) mean squared error (MSE) or mean absolute error (MAE), as well as perceptual metrics that assess structural and contextual similarities. Calculated losses can serve as a quantitative measure of reconstruction quality and guides the optimization process by providing a gradient for parameter updates.

[0063] Process 300 optimizes (320) the encoder and the decoder to minimize the calculated loss. Optimization in accordance with some embodiments of the invention iteratively improves the quality of both latent representations and reconstructed images. In various embodiments, optimization may involve adjusting the weights and biases of the neural network layers within the encoder and decoder to improve their ability to generate accurate latent representations and high-quality reconstructions.

[0064] Process 300 determines (325) whether to continue. If the process determines (325) to continue, process 300 returns to step 305 to continue processing the next input image. Otherwise, process 300 ends.

[0065] In some embodiments, image processing systems may utilize a set of autoencoders with different configurations to handle various types of medical images. For example, the system may include four 2D autoencoders and two 3D autoencoders. The 2D autoencoders may be configured with different downsizing factors and latent channel combinations, such as a downsizing factor of 16 with 1 latent channel, a downsizing factor of 16 with 3 latent channels, a downsizing factor of 64 with 1 latent channel, and a downsizing factor of 64 with 4 latent channels. This diverse set of autoencoders may allowimage processing systems to adapt to different image types and resolutions, potentially improving the overall performance and flexibility of the image processing pipeline. In several embodiments, image processing systems may select the most appropriate autoencoder based on the characteristics of the input medical image, such as dimensionality, resolution, and / or modality.

[0066] In various embodiments, the process of training autoencoders may be performed for multiple autoencoders simultaneously or sequentially, each with different configurations. Configurations in accordance with a variety of embodiments of the invention may include varying parameters such as network architecture, layer sizes, activation functions, and / or learning rates. In a number of embodiments, autoencoders may be initialized with different random weights or pre-trained weights from various sources. In numerous embodiments, training objectives for each autoencoder may differ, potentially combining different loss functions or emphasizing certain aspects of image reconstruction or feature preservation. In certain embodiments, the number of training iterations may vary between autoencoders, allowing for exploration of different convergence patterns. In some cases, the training process for a single model may evolve over time. For instance, after a certain number of iterations, the learning rate may be adjusted, additional loss terms may be introduced, and / or the focus of training may shift from general reconstruction to preserving specific medical features. Such an adaptive approach may allow the model to first learn broad image characteristics before refining its ability to capture clinically relevant details.

[0067] While specific processes for optimizing an encoder and a decoder are described above, any of a variety of processes can be utilized to optimize an encoder and a decoder as appropriate to the requirements of specific applications. In certain embodiments, steps may be executed or performed in any order or sequence not limited to the order and sequence shown and described. In a number of embodiments, some of the above steps may be executed or performed substantially simultaneously where appropriate or in parallel to reduce latency and processing times. In some embodiments, one or more of the above steps may be omitted.

[0068] An example of refining latent representations in accordance with an embodiment of the invention is illustrated in FIG. 4. Step 401 illustrates an encoder 405and projection layers 415. In this example, an input image is passed through encoder 405 to generate a latent representation. Latent representations in accordance with several embodiments of the invention are compact, lower-dimensional representations of input images that capture essential features while reducing data size. Encoders and / or decoders in accordance with various embodiments of the invention may be fine-tuned using Low-Rank Adaptation (LoRA) techniques. The latent representation is then passed through projection layers 415 to generate a projected representation. In certain embodiments, parameters (or weights) of an encoder may be frozen when optimizing projection layers.

[0069] Step 402 illustrates an optimization based on the generated projected representation. In this example, both the input image and the projected representation are passed through feature embeddings engine 425 to compute feature embeddings for each. In various embodiments, feature embeddings engines utilize pre-trained medical vision-language models such as (but not limited to) BiomedCLIP. BiomedCLIP is a foundation model capable of extracting relevant features from medical images.

[0070] This step further shows that feature embeddings for the input image and for the projected representation are passed to optimizer 420. Optimizers in accordance with numerous embodiments of the invention compute a loss based on a difference between the computed feature embeddings for the input image and the projected representation. In many embodiments, optimization aims to adjust the projection layers (e.g., weights) to minimize the difference between feature embeddings of the input image and feature embeddings of the projected latent representation generated by the projection layers, thereby preserving clinically-relevant features in the latent space.

[0071] An example of a process for refining latent representations in accordance with an embodiment of the invention is illustrated in FIG. 5. Process 500 generates (510) a latent representation from an input image using an encoder. In several embodiments, encoders may include convolutional neural networks trained to extract relevant features from medical images.

[0072] Process 500 feeds (515) the latent representation from the encoder to trainable projection layers to generate a projected latent representation. Projection layersin accordance with many embodiments of the invention are designed to further refine the latent representation while preserving clinically-relevant features.

[0073] Process 500 computes (520) feature embeddings for the original input image and the projected latent representation. In a number of embodiments, computing feature embeddings may involve using a pre-trained medical vision-language model (e.g., BiomedCLIP). Pre-trained medical vision-language models in accordance with various embodiments of the invention may be capable of extracting meaningful features from diverse medical imaging modalities.

[0074] Process 500 calculates (525) a loss between the feature embeddings of the input image and the projected latent representation. Loss calculations in accordance with several embodiments of the invention quantify the difference in feature representations between the original image and its projected latent form. In some embodiments, losses may be calculated using various methods such as (but not limited to) cosine similarity, Euclidean distance, or cross-entropy between the feature embeddings.

[0075] Process 500 optimizes (530) the projection layers to minimize the embedding consistency loss. Embedding consistency loss may be defined as a measure of the discrepancy between the feature embeddings of the original input image and those of its projected latent representation, quantifying how well the latent space preserves the essential characteristics of the input data. Optimization of the projection layers in accordance with numerous embodiments of the invention aims to ensure that the projected latent representation retains clinically-relevant features present in the original input image.

[0076] Process 500 determines (535) whether to continue refining the latent representation. If the process determines to continue, process 500 returns to step 510 to process the next input image. Otherwise, process 500 ends.

[0077] In various embodiments, refinement processes may be performed iteratively over a large dataset of medical images to improve the generalization capabilities of the projection layers. In some cases, the refinement process may be tailored to specific medical imaging modalities or clinical tasks, allowing for specialized latent representations that capture domain-specific features.

[0078] While specific processes for refining latent representations are described above, any of a variety of processes can be utilized to refine a latent representation as appropriate to the requirements of specific applications. In certain embodiments, steps may be executed or performed in any order or sequence not limited to the order and sequence shown and described. In a number of embodiments, some of the above steps may be executed or performed substantially simultaneously where appropriate or in parallel to reduce latency and processing times. In some embodiments, one or more of the above steps may be omitted.

[0079] Once latent representations have been refined in accordance with some embodiments of the invention, projected latent representations can be generated and stored in a more compact form. This process may allow for the preservation of clinically relevant features while maintaining the computational efficiency benefits of the compressed latent representation format. In a number of embodiments, projected latent representations can be used as inputs to other models for various applications, such as (but not limited to) computer-aided diagnosis (CAD) tasks.

[0080] An example of a system for processing three-dimensional (3D) images in accordance with an embodiment of the invention is illustrated in FIG. 6. Step 601 illustrates 3D encoder 605 and 3D decoder 610. In this example, 3D encoder 605 receives 3D input volumes and generates a volumetric latent representation. 3D decoder 610 receives the latent representation and reconstructs a 3D reconstruction of the input. In several embodiments, 3D encoders and / or 3D decoders are derived from 2D autoencoder architectures using various techniques, such as (but not limited to) kernel centering inflation techniques. This approach allows image processing systems to leverage knowledge gained from 2D image processing while efficiently handling volumetric data. In a number of embodiments, kernel centering inflation involves expanding 2D convolutional kernels into 3D by replicating weights along the depth dimension and normalizing to maintain the overall magnitude of activations.

[0081] In step 602, optimizer 620 receives an input image and a reconstruction output, and provides optimization signals to both 3D encoder 605 and 3D decoder 610. Optimizers in accordance with a number of embodiments of the invention optimize 3D encoders and / or 3D decoders based on comparisons between the input volumes andreconstructions. Optimization in accordance with various embodiments of the invention may involve calculating losses and adjusting model parameters to improve the quality of latent representations and reconstructions.

[0082] In many embodiments, optimizations (e.g., perceptual loss, patch-based adversarial objectives, etc.) are calculated per-slice for 3D volumes. This approach can allow image processing systems to capture both global volumetric structure and finegrained details within each slice. Calculating losses per-slice in accordance with numerous embodiments of the invention may involve iterating through the depth dimension of the 3D volume and computing 2D loss metrics for each slice, then aggregating these losses to guide the optimization of the 3D autoencoder.

[0083] An example of a process for processing 3D images in accordance with an embodiment of the invention is illustrated in FIG. 7. Process 700 generates (705) a latent representation from an input image using an encoder. In various embodiments, generating latent representations for 3D volumes may involve a series of 3D convolutional operations that progressively reduce spatial dimensions while increasing the number of feature channels. Processes in accordance with a variety of embodiments of the invention the encoder is a 3D encoder generated based on 2D encoders trained in the first stage.

[0084] Process 700 feeds (710) the latent representations from the encoder to a decoder to generate a reconstruction. Reconstructions of 3D volumes in accordance with several embodiments of the invention may involve a series of 3D transposed convolutions that progressively increase spatial dimensions while decreasing the number of feature channels.

[0085] Process 700 calculates (715) a loss between the input image and the reconstruction. In many embodiments, loss calculation for 3D volumes may include volumetric versions of common loss functions such as (but not limited to) mean squared error, as well as perceptual losses.

[0086] Process 700 optimizes (720) parameters of the encoder and decoder to minimize the calculated loss. Optimization in accordance with numerous embodiments of the invention may involve updating the weights of 3D convolutional kernels to improve the quality of both latent representations and reconstructed volumes.

[0087] Process 700 determines (725) whether to continue processing. If the process determines to continue, process 700 returns to step 705 to process the next input image. Otherwise, process 700 ends.

[0088] In a number of embodiments, image processing systems may train on random cubic patches of size 64x64x64 for 3D autoencoders. Training on cubic patches in accordance with various embodiments of the invention may allow the system to efficiently process large 3D volumes while capturing local spatial relationships. In several embodiments, during training, random cubic patches may be extracted from full 3D volumes, and the autoencoder may be optimized to reconstruct these patches. This approach may help the model learn to preserve important local features and structures within the 3D space.

[0089] Processes in accordance with many embodiments of the invention may train multiple 3D autoencoders with different characteristics. 3D autoencoders in accordance with a variety of embodiments of the invention may be configured with different downsizing factors, such as a downsizing factor of 64 with 1 latent channel and a downsizing factor of 512 with 1 latent channel.

[0090] Image processing systems in accordance with many embodiments of the invention may employ various techniques to handle the increased computational demands of 3D data. In some cases, systems may utilize distributed computing or GPU acceleration to parallelize the processing of 3D volumes. In certain embodiments, systems may implement memory-efficient techniques such as (but not limited to) gradient checkpointing or mixed-precision training to manage the larger memory footprint of 3D data.

[0091] While specific processes for processing 3D medical images are described above, any of a variety of processes can be utilized to process 3D medical images as appropriate to the requirements of specific applications. In certain embodiments, steps may be executed or performed in any order or sequence not limited to the order and sequence shown and described. In a number of embodiments, some of the above steps may be executed or performed substantially simultaneously where appropriate or in parallel to reduce latency and processing times. In some embodiments, one or more of the above steps may be omitted.

[0092] Image processing systems in accordance with many embodiments of the invention have applications in various healthcare scenarios. In several embodiments, image processing systems may be utilized for computer-aided diagnosis (CAD) tasks. CAD tasks in accordance with numerous embodiments of the invention include (but are not limited to) malignancy detection on full-field digital mammograms (FFDMs), calcification detection on FFDMs, Breast Imaging-Reporting and Data System (BI-RADS) prediction on FFDMs, bone age prediction on X-rays, fracture detection on wrist X-rays, fracture detection on spine CT scans, fracture classification on head CT scans, and / or anterior cruciate ligament (ACL) and meniscal tear detection on knee MRIs.

[0093] An example of evaluation tasks for medical image autoencoders in accordance with an embodiment of the invention is illustrated in FIG. 8. FIG. 8 includes an optimization step 805, a refinement step 810, and examples 815 of some evaluations of results from trained autoencoders. Examples of optimization and refinement are described in greater detail throughout this description.

[0094] In many embodiments, image processing systems evaluate the quality of latent representations using latent quality evaluation. Latent quality evaluation in accordance with several embodiments of the invention assesses how well latent representations preserve clinically-relevant features from input images.

[0095] Image processing systems in accordance with numerous embodiments of the invention perform storage efficiency evaluation. Storage efficiency evaluation in accordance with a variety of embodiments of the invention may assess the reduction in storage requirements achieved by using latent representations instead of high-resolution images. In a number of embodiments, storage efficiency evaluation quantifies the decrease in data size when storing latent representations compared to original images.

[0096] Medical image processing systems in accordance with various embodiments of the invention evaluate reconstruction quality of reconstructions generated from latent representations. Reconstruction quality evaluation may assess how well decoded images from latent representations match original input images. In several embodiments, reconstruction quality evaluation utilizes perceptual quality metrics such as (but not limited to) peak signal-to-noise ratio (PSNR) and / or multi-scale structural similarity index measure (MS-SSIM). Evaluations of processes in accordance with someembodiments of the invention have shown significant improvements and quality in latent quality, storage and efficiency, and reconstruction quality.

[0097] An example of CAD model efficiency in accordance with an embodiment of the invention is illustrated in FIG. 9. This figure illustrates how using latent representations in downstream CAD tasks may improve computational efficiency compared to using high- resolution images. In many embodiments, image processing systems encode images by downsizing them by factors such as (but not limited to) 16, 64, and / or 512. Downsizing in accordance with numerous embodiments of the invention reduces storage requirements while preserving clinically-relevant features.

[0098] Medical image processing systems in accordance with several embodiments of the invention select an autoencoder from a plurality of autoencoders based on characteristics of the input image. In various embodiments, the plurality of autoencoders may include four 2D autoencoders and two 3D autoencoders. The 2D autoencoders in accordance with a number of embodiments of the invention may have configurations such as (but not limited to) a downsizing factor of 16 with 1 latent channel, a downsizing factor of 16 with 3 latent channels, a downsizing factor of 64 with 1 latent channel, and / or a downsizing factor of 64 with 4 latent channels. The 3D autoencoders in accordance with various embodiments of the invention may have configurations such as (but not limited to) a downsizing factor of 64 with 1 latent channel and / or a downsizing factor of 512 with 1 latent channel.

[0099] An example of manual perceptual quality evaluations with expert readers in accordance with an embodiment of the invention is illustrated in FIG. 10. In many embodiments, image processing systems conduct manual perceptual quality evaluations with expert radiologists to assess the quality of reconstructed images. Manual evaluations in accordance with several embodiments of the invention may provide insights into how well reconstructed images preserve clinically-relevant features from a radiologist's perspective.

[0100] Qualitative examples of reconstructed medical images in accordance with an embodiment of the invention are illustrated in FIG. 11 . The top section provides qualitative examples of a reconstructed chest X-ray. The bottom section provides qualitative examples of a reconstructed brain MRI slice. Residual figures show pixel-leveldifferences between reconstructed images and original, high-resolution images; brighter colors represent larger differences.

[0101] A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. Accordingly, other implementations are within the scope of the following claims.A. Systems for Image Processing1. Image Processing System

[0102] An example of an image processing system that processes images in accordance with an embodiment of the invention is illustrated in FIG. 12. Network 1200 includes a communications network 1260. The communications network 1260 is a network such as the Internet that allows devices connected to the network 1260 to communicate with other connected devices. Server systems 1210, 1240, and 1270 are connected to the network 1260. Each of the server systems 1210, 1240, and 1270 is a group of one or more servers communicatively connected to one another via internal networks that execute processes that provide cloud services to users over the network 1260. One skilled in the art will recognize that an image processing system may exclude certain components and / or include other components that are omitted for brevity without departing from this invention.

[0103] For purposes of this discussion, cloud services are one or more applications that are executed by one or more server systems to provide data and / or executable applications to devices over a network. The server systems 1210, 1240, and 1270 are shown each having three servers in the internal network. However, the server systems 1210, 1240 and 1270 may include any number of servers and any additional number of server systems may be connected to the network 1260 to provide cloud services. In accordance with various embodiments of this invention, an image processing system that uses systems and methods that process images in accordance with an embodiment of the invention may be provided by a process being executed on a single server system and / or a group of server systems communicating over network 1260.

[0104] Users may use personal devices 1280 and 1220 that connect to the network 1260 to perform processes that process images in accordance with various embodiments of the invention. In the shown embodiment, the personal devices 1280 are shown as desktop computers that are connected via a conventional “wired” connection to the network 1260. However, the personal device 1280 may be a desktop computer, a laptop computer, a smart television, an entertainment gaming console, or any other device that connects to the network 1260 via a “wired” connection. The mobile device 1220 connects to network 1260 using a wireless connection. A wireless connection is a connection that uses Radio Frequency (RF) signals, Infrared signals, or any other form of wireless signaling to connect to the network 1260. In the example of this figure, the mobile device 1220 is a mobile telephone. However, mobile device 1220 may be a mobile phone, Personal Digital Assistant (PDA), a tablet, a smartphone, or any other type of device that connects to network 1260 via wireless connection without departing from this invention.

[0105] As can readily be appreciated the specific computing system used to process images is largely dependent upon the requirements of a given application and should not be considered as limited to any specific computing system(s) implementation.2. Image Processing Element

[0106] An example of an image processing element that executes instructions to perform processes that process images in accordance with an embodiment of the invention is illustrated in FIG. 13. Image processing elements in accordance with many embodiments of the invention can include (but are not limited to) one or more of mobile devices, cameras, and / or computers. Image processing element 1300 includes processor 1305, peripherals 1310, network interface 1315, and memory 1320. One skilled in the art will recognize that an image processing element may exclude certain components and / or include other components that are omitted for brevity without departing from this invention.

[0107] The processor 1305 can include (but is not limited to) a processor, microprocessor, controller, or a combination of processors, microprocessor, and / or controllers that performs instructions stored in the memory 1320 to manipulate data stored in the memory. Processor instructions can configure the processor 1305 to perform processes in accordance with certain embodiments of the invention. In variousembodiments, processor instructions can be stored on a non-transitory machine readable medium.

[0108] Peripherals 1310 can include any of a variety of components for capturing data, such as (but not limited to) cameras, displays, and / or sensors. In a variety of embodiments, peripherals can be used to gather inputs and / or provide outputs. Image processing element 1300 can utilize network interface 1315 to transmit and receive data over a network based upon the instructions performed by processor 1305. Peripherals and / or network interfaces in accordance with many embodiments of the invention can be used to gather inputs that can be used to process images.

[0109] Memory 1320 includes an image processing application 1325, image data 1330, and model data 1335. Image processing applications in accordance with several embodiments of the invention can be used to process images.

[0110] In many embodiments, image data includes images of various modalities (e.g., 2D and 3D images). Image data in accordance with numerous embodiments of the invention includes medical images. In many embodiments, medical images can include (but are not limited to) X-rays, computed tomography (CT) scans, and / or magnetic resonance imaging (MRI) scans.

[0111] In several embodiments, model data can store various parameters and / or weights for various models (e.g., encoders, decoders, projection layers, etc.) that can be used for various processes as described in this specification. Model data in accordance with many embodiments of the invention can be updated through training on multimedia data captured on an image processing element or can be trained remotely and updated at an image processing element.

[0112] Although a specific example of an image processing element 1300 is illustrated in this figure, any of a variety of image processing elements can be utilized to perform processes for image processing similar to those described herein as appropriate to the requirements of specific applications in accordance with embodiments of the invention.

[0113] Although specific methods of image processing are discussed above, many different methods of image processing can be implemented in accordance with many different embodiments of the invention. It is therefore to be understood that the present invention may be practiced in ways other than specifically described, without departingfrom the scope and spirit of the present invention. Thus, embodiments of the present invention should be considered in all respects as illustrative and not restrictive. Accordingly, the scope of the invention should be determined not by the embodiments illustrated, but by the appended claims and their equivalents.

Claims

WHAT IS CLAIMED IS:1 . A method for processing images, comprising: encoding an image using an encoder to generate a latent representation; processing the latent representation using trainable projection layers to generate a projected latent representation; computing feature embeddings for the image and the projected latent representation; calculating a loss between the feature embeddings; and optimizing parameters of the projection layers to minimize the loss.

2. The method of claim 1 , wherein the image is a two-dimensional (2D) image selected from the group consisting of X-rays, computed tomography (CT) scans, and magnetic resonance imaging (MRI) scans.

3. The method of claim 1 , wherein the image is a three-dimensional (3D) volume selected from the group consisting of CT scans and MRI scans.

4. The method of claim 1 , wherein the encoder is a convolutional neural network.

5. The method of claim 4, wherein the convolutional neural network comprises residual blocks.

6. The method of claim 1 , wherein the encoder is fine-tuned using Low-Rank Adaptation (LoRA) techniques.

7. The method of claim 1 , wherein computing feature embeddings comprises using a pre-trained medical vision-language model.

8. The method of claim 7, wherein the pre-trained medical vision-language model isBiomedCLIP.

9. The method of claim 1 , further comprising: decoding the latent representation using a decoder to generate a reconstructed image.

10. The method of claim 9, wherein the decoder comprises transposed convolutions.11 . The method of claim 9, further comprising: evaluating the quality of the reconstructed image using perceptual quality metrics.

12. The method of claim 11 , wherein the perceptual quality metrics include peak signal-to-noise ratio (PSNR) and multi-scale structural similarity index measure (MS- SSIM).

13. The method of claim 1 , further comprising: evaluating the quality of the latent representation using computer-aided diagnosis (CAD) tasks.

14. The method of claim 13, wherein the CAD tasks include at least one of: malignancy detection on full-field digital mammograms (FFDMs), calcification detection on FFDMs,Breast Imaging-Reporting and Data System (BI-RADS) prediction on FFDMs, bone age prediction on X-rays, fracture detection on wrist X-rays, fracture detection on spine CT scans, fracture classification on head CT scans, and anterior cruciate ligament (ACL) and meniscal tear detection on knee MRIs.

15. The method of claim 1 , wherein the encoder and a decoder are trained using a two-stage process comprising: a base autoencoder training stage, and a feature preservation stage.

16. The method of claim 15, wherein the base autoencoder training stage utilizes at least one selected from the group consisting of a perceptual loss, patch-based adversarial objectives, and a domain-specific embedding consistency loss.

17. The method of claim 15, wherein the feature preservation stage for 2D imaging modalities comprises: freezing parameters of the encoder and the decoder, and fine-tuning the trainable projection layers.

18. The method of claim 15, wherein the feature preservation stage for 3D imaging modalities comprises lifting a 2D architecture to 3D using kernel centering inflation.

19. The method of claim 1 , further comprising storing the latent representation, wherein the latent representation requires less storage space than the image.

20. The method of claim 19, further comprising using the latent representation in downstream computer-aided diagnosis (CAD) tasks, wherein using the latent representation improves computational efficiency compared to using the image.

21. The method of claim 1 , wherein encoding the image comprises downsizing the image by a factor selected from the group consisting of 16, 64, and 512.

22. The method of claim 1 , further comprising: selecting an autoencoder from a plurality of autoencoders based on characteristics of the image, wherein the plurality of autoencoders comprises: at least two 2D autoencoders with different downsizing factors and latent channel configurations; and at least two 3D autoencoders with different downsizing factors.

23. The method of claim 22, wherein the plurality of autoencoders comprises: four 2D autoencoders with configurations selected from the group consisting of: downsizing factor of 16 with 1 latent channel, downsizing factor of 16 with 3 latent channels, downsizing factor of 64 with 1 latent channel, and downsizing factor of 64 with 4 latent channels; and two 3D autoencoders with configurations selected from the group consisting of: downsizing factor of 64 with 1 latent channel, and downsizing factor of 512 with 1 latent channel.

24. The method of claim 1 , wherein encoding the image comprises: determining whether the image is a 2D image or a 3D volume; selecting a 2D autoencoder when the image is a 2D image; and selecting a 3D autoencoder when the image is a 3D volume.

25. The method of claim 24, wherein the 3D autoencoder is derived from a 2D autoencoder architecture using kernel centering inflation.

26. The method of claim 1 , wherein the encoder is trained using at least one selected from the group consisting of perceptual loss, patch-based adversarial objectives, and domain-specific embedding consistency loss.

27. A non-transitory machine readable medium for processing images, comprising: encoding an image using an encoder to generate a latent representation; processing the latent representation using trainable projection layers to generate a projected latent representation; computing feature embeddings for the image and the projected latent representation; calculating a loss between the feature embeddings; and optimizing parameters of the projection layers to minimize the loss.

28. An image processing system comprising: a set of one or more processors; a set of one or more non-transitory computer-readable media comprising program instructions that are executable by the one or more processors such that the apparatus is configured to: encode an image using an encoder to generate a latent representation; process the latent representation using trainable projection layers to generate a projected latent representation; compute feature embeddings for the image and the projected latent representation; calculate a loss between the feature embeddings; and optimize parameters of the projection layers to minimize the loss.

Images