Image compression using variational auto-encoder

Abstract

The disclosed approach performs image compression using a variational auto-encoder that is capable of greater compression than conventional approaches while maintaining high fidelity of the decompressed image. Examples persist a bottleneck layer output of a variational auto-encoder into a compressed image in the form of potential tensors. The potential tensor is decompressed into a restored image in the pixel space by a variational self-decoder. In some examples, different encoder / decoder pairs are trained on a particular image type based on feature attributes. For example, maps have lines that are narrower compared to their length (e.g., with high aspect ratios), which differ from features within photographs of persons and scenes. Some examples utilize contrast language image pre-training (CLIP) and / or bootstrap language image pre-training (BLIP) models to store inserts, each of which is associated with a compressed image to enable a natural language search of a set of compressed images without requiring decompression.

Description

Background Technology

[0001] Traditional image compression solutions employ one or more compression algorithms, such as Fourier Transform (FT), Principal Component Analysis (PCA) dimensionality reduction, fractal compression based on chaos theory, and others. However, these solutions are significantly lossy (e.g., blurred images and loss of information) when achieving high compression ratios.

[0002] FT-based solutions compress images by discarding high-frequency information, which may be considered less important but blurs the edges between pixels of different colors. PCA-based solutions compress images by reducing feature dimensionality, thus discarding image features. Fractal compression compresses images by determining a fractal function that simulates the image data during reconstruction, resulting in a difference between the actual content in the image and the simulated reconstruction. Summary of the Invention

[0003] The disclosed examples are described in detail below with reference to the accompanying drawings. The following summary of the invention is provided to illustrate some of the examples disclosed herein.

[0004] The example solution uses a variational autoencoder to perform image compression. The example receives a first image of a first image type for compression, the first image being in pixel space; provides the first image to a first variational autoencoder having a bottleneck layer; persists the output of the bottleneck layer of the first variational autoencoder as a first compressed image, the first compressed image including a latent tensor; and, based on at least receiving a request to decompress the first compressed image, a first variational autodecoder decompresses the first compressed image into a first restored image, the first restored image being in pixel space. Attached Figure Description

[0005] The disclosed example is described in detail below with reference to the accompanying drawings:

[0006] Figure 1 An example architecture that advantageously uses a variational autoencoder to perform image compression is illustrated;

[0007] Figure 2 The illustration shows that it can be used in, for example Figure 1 Further details on the example architecture used for compression with a variational autoencoder and decompression with a variational autodecoder;

[0008] Figure 3 The illustration shows, for example, in Figure 1 The example architecture demonstrates how compression and decompression can be customized for different image types;

[0009] Figure 4 The diagram illustrates the training methods used in, for example, Figure 1Further details on the variational autoencoder and variational autodecoder used in the example architecture;

[0010] Figure 5 The illustration shows that it can be used in, for example Figure 1 The example architecture performs a search for compressed images without requiring decompression.

[0011] Figure 6 The diagram illustrates that it can be used in, for example... Figure 1 A flowchart of an exemplary operation performed in the example architecture;

[0012] Figure 7A , 7B 7C and 7D are shown Figure 6 The flowchart provides further details on the various operations within the flowchart;

[0013] Figure 8 The diagram illustrates that it can be used in, for example... Figure 1 Another flowchart illustrating the exemplary operations performed in the example architecture; and

[0014] Figure 9 A block diagram of an example computing device suitable for implementing some of the various examples disclosed herein is shown.

[0015] The corresponding reference numerals indicate the corresponding parts throughout the figures. Any figures can be combined to form a single example or embodiment. Detailed Implementation

[0016] The disclosed solution uses a variational autoencoder to perform image compression, which achieves greater compression than traditional methods while maintaining high fidelity in the decompressed image. Examples persist the bottleneck layer output of the variational autoencoder as a compressed image in the form of a latent tensor. The latent tensor is then decompressed into a reconstructed image in pixel space using a variational autodecoder. In some examples, different encoder / decoder pairs are trained on specific image types based on feature attributes. For example, maps have lines that are narrow compared to their length (e.g., have a high aspect ratio), which differs from the features within photographs of people and scenes. Some examples leverage contrastive language image pre-training (CLIP) and / or bootstrap language image pre-training (BLIP) models to store embeddings, each associated with a compressed image, to enable natural language search of a set of compressed images without requiring decompression.

[0017] Various aspects of this disclosure provide novel techniques for performing image compression, including specific processes for training encoders / decoders to handle different image types that achieve performance superior to conventional image compression methods, and for creating and using searchable datasets to quickly locate details of compressed images that satisfy natural language searches. Examples include persisting the output of the bottleneck layer of a variational autoencoder as a compressed image; training a variational autoencoder / decoder to compress / decompress images by image type (e.g., by distinguishing them by identifiable features within each image type, such as aspect ratio); and associating CLIP and / or BLIP embeddings with compressed images. These aspects of the disclosure address several problems inherently rooted in computer technology, such as high compression of digital images (saving significant storage space) and technically efficient searching of highly compressed sets of digital images (e.g., in terms of processing and memory usage), without requiring computationally expensive decompression.

[0018] Various examples are described in detail with reference to the accompanying drawings. Wherever preferred, the same reference numerals will be used throughout the drawings to refer to the same or similar parts. References throughout this disclosure relating to specific examples and implementations are provided for illustrative purposes only and are not intended to limit all examples unless indicated otherwise.

[0019] Figure 1 An example architecture 100 is illustrated, which advantageously uses variational autoencoder 201 to perform image compression to compress image 301 into compressed image 311. In some examples, when image 302 is an image of a different type than image 301, different variational autoencoders within multiple variational autoencoders 102 are used to compress different types of images, such as using variational autoencoder 202 to compress image 302 into compressed image 312. The following is about... Figure 3 This describes what defines the images as different types in order to select which variational autoencoder among the multiple variational autoencoders 102 is used for compression. Selector 130 selects from available variational autoencoders (e.g., variational autoencoders 201 and 202) for compression based on the image type. However, some examples may use only a single variational autoencoder for all image types.

[0020] Compressed images 311 and 312 are persistently stored (e.g., stored) within a plurality of compressed images 104 in storage device 106. Upon receiving a decompression request, such as request 111 for decompressing compressed image 311 or request 112 for decompressing compressed image 312, the identified compressed image is decompressed using a plurality of variational self-decoders within variational self-decoders 108 corresponding to the variational autoencoders used to compress the compressed image. For example, variational self-decoder 211 corresponding to variational autoencoder 201 is used to decompress compressed image 311 into restored image 321, and variational self-decoder 212 corresponding to variational autoencoder 202 is used to decompress compressed image 312 into restored image 322. Selector 130 also selects from available variational self-decoders (e.g., variational self-decoders 211 and 212) for decompression (e.g., restoration), based on which the variational autoencoder is used for compression. However, some examples may use only a single variational self-decoder for all image types.

[0021] Considering the fidelity maintained in the decompressed image, architecture 100 provides a relatively high compression ratio. For example, a 512x512 color image in pixel space (e.g., the domain of images 301 and 302 and the recovered images 321 and 322) is a 3x512x512 tensor—a 3-layer 512x512 pixel matrix, with one layer for each color (red, green, blue) in RGB space or another color space (e.g., YCbCr). The corresponding compressed image using architecture 100 is a 4x64x64 tensor in latent space (e.g., the domain of compressed images 311 and 312).

[0022] Using the example of Architecture 100, a 401 kilobyte (KB) image is compressed to 33 KB, a compression ratio of 12:1, and occupies only 8.2% of the storage space of the uncompressed image. This high compression ratio is achieved using a mean squared error (MSE) of only 11.5. The MSE is calculated using the following formula: Equation (1) in W It's the width. H It's about height. X These are the pixel values ​​in the original image (e.g., image 301), and Y It is the corresponding pixel value in the restored original image (e.g., restored image 321). A lower MSE indicates a higher fidelity of the restored image.

[0023] In some examples, the compression ratio varies with image size; for instance, a 548KB image is compressed to 60KB, which is a compression ratio close to 10:1 with an MSE of only 8.2, and a 6,671KB image is compressed to 939KB, which is a compression ratio of 7:1 with an MSE of only 2.8. These MSE values ​​use pixel values ​​from [0 to 255], although pixel value conversion may be required in some examples. In some examples, each pixel value in the [0 to 255] pixel value range in the RGB color space is converted to the [-1 to +1] value range to be input to the variational autoencoder, and the output from the variational autodecoder is converted back from the [-1 to +1] value range to the common [0 to 255] pixel value range before being used for display or other output.

[0024] Additional preprocessing may be needed to convert the tensor of the original image into a NumPy array, which is a value grid of the same type, all indexed by tuples of non-negative integers. The dimension is the rank of the array, and the shape of the array is a set of integer tuples, given the size of the array along each dimension. Post-processing may then be required to convert it back to a common image format.

[0025] In some scenarios, it may be necessary to search for images that satisfy a certain search criterion among multiple compressed images 104, such as by using natural language (NL) text descriptions. An example search query might be "images of a cat chasing a bird". In examples where there are a large number of compressed images within multiple compressed images 104, decompressing each compressed image for the search is computationally expensive. However, in architecture 100, the compressed images reside in a latent space that is indistinguishable from noise in human and many automated search processes.

[0026] To enable fast searching, multiple embeddings 510 are stored in storage device 106, which contains image embeddings and / or text embeddings indexed to compressed images 311 and 312 in multiple compressed images 104. Storage device 106 typically represents storage capacity and may include distributed and / or virtual storage devices. Search interface 500 receives search criteria and searches for the best match (or multiple matches) with the search criteria among the multiple embeddings 510. As illustrated, search interface 500 has located the best match as compressed image 311, and therefore generates a request 111 for decompressing compressed image 311. This will be discussed below regarding... Figure 5 Further details are shown and described.

[0027] To generate entries corresponding to the compressed image 311 (or image 301) in multiple embeddings 510, image 301 is provided to CLIP model 120a. CLIP model 120a generates image embedding 511, which is added to the multiple embeddings 510 and persisted in storage device 106. In some examples, image 301 is also provided to BLIP model 122, which generates image caption 531 based on a prompt such as “describe what the image contains”. CLIP model 120b generates text embedding 521 from image caption 531, and text embedding 521 is also added to the multiple embeddings 510 and persisted in storage device 106. In some examples, a single CLIP model is used as CLIP models 120a and 120b, while in some examples, different CLIP models are used.

[0028] CLIP models 120a and 120b, and BLIP model 122, are transformer-based image-to-text models. BLIP model 122 generates captions of 4 to 8 words describing an image. CLIP model 120a (and CLIP model 120b, if different models) produce shorter text descriptions. For example, for an image of a cat chasing a bird, BLIP model 122 might output "The cat is walking on the grass while the bird is flying in the air," while CLIP model 120a might output "The cat is chasing the bird." In some examples, BLIP model 122 includes bootstrapping language image pre-training for a unified visual language understanding and generation model. Some examples also persist image captions 531 across multiple captions 530, and the search interface 500 also uses multiple captions 530 to locate matches of search criteria. Embeddings and captions for compressed image 312 (or image 302) and other compressed images within multiple compressed images 104 can be generated similarly.

[0029] Figure 2 Further details of compression using variational autoencoder 201 and decompression using variational autodecoder 211 are illustrated. Compression and decompression using variational autoencoder 202 and variational autodecoder 212 are similar. Using a 3xWxH tensor and RGB, YCbCr color space, image 301 and restored image 321 are located in pixel space 221. Variational autoencoder 201 has a bottleneck layer 220 from which the compressed image 311 is extracted as a latent vector in latent space 222.

[0030] A variational autoencoder (VAE) is an artificial neural network architecture. Bottleneck layers (e.g., bottleneck layer 220 of the variational autodecoder 211) provide a statistical way to describe samples of the dataset in the latent space 222 and output a probability distribution instead of a single output value as in a traditional autoencoder. In some examples, the variational autodecoder 211 imposes constraints on the output probability distribution, forcing it to be normally distributed to ensure that the latent space 222 is regularized. The variational autodecoder 211 maps from the latent space 222 back to the pixel space 221. In some examples, the variational autodecoder 211 has nearly complementary layers corresponding to the layers of the variational autoencoder 201, but in reverse order.

[0031] Figure 3 The diagram illustrates how compression and decompression can be customized for different image types. Variational autoencoders perform nonlinear dimensionality reduction or manifold learning, projecting high-dimensional data into a low-dimensional latent space. Variational autoencoders 201 and 202 are trained to approximate the high-dimensional data of images 301 and 302 with low-dimensional data. This is how high compression ratios are achieved.

[0032] Therefore, different image types, defined according to the most common features within the image (as opposed to file formats), may produce different levels of MSE performance for a single variational autoencoder and variational autodecoder pair. Thus, some examples of architecture 100 will train different pairs of variational autoencoders and variational autodecoders on different image types. Then, when an incoming image for compression is received, selector 130 selects from the available variational autoencoders (e.g., variational autoencoders 201 and 202, and other variational autoencoders within the plurality of variational autoencoders 102).

[0033] Image 301 is an image of image type 331, such as a photograph of an outdoor scene or a person. Image 301 has a portion 361 with an identifiable feature 341, which is a face (or a stone carving of a face). Feature annotation 363 summarizes the identifiable feature 341 as a bounding polygon or bounding ellipse, as a subset of bounding boxes that can be generated using common object detection processes. Feature annotation 363 has an aspect ratio (e.g., a conceptual representation of a mathematical quantity) represented by two orthogonal measurement indicators as aspect ratio 351. The aspect ratio 351 is not significantly different from the value 1, making the height and width of the identifiable feature 341 within orders of magnitude of each other.

[0034] In contrast, image 302 is an image of image type 332, such as a map with text and lines that are thin relative to their length. Image 302 has a portion 362 with identifiable features 342, which are part of the mapped railway line. Feature annotation 364 summarizes the identifiable features 342 as bounding boxes, which can be generated using common object detection processes. Boundary polygons can be used for curves. Feature annotation 364 has an aspect ratio 352 that is significantly different from the value 1. The height and width of the identifiable features 342 are not within orders of magnitude of each other, making the aspect ratio 352 exceed the value 10 (e.g., when normalized to a value of 1 or greater).

[0035] In some examples, selector 130 may use object detection and determine the aspect ratio of larger, more common features in order to classify the image into image type 331 or image type 332. In some examples, different categories and counts of image types and different classification schemes may be used.

[0036] Figure 4 Further details for training variational autoencoders 201 and 202, and variational autodecoders 211 and 212, are illustrated. In some examples, the variational autoencoders and variational autodecoders are trained in pairs, such that variational autoencoder 201 and variational autodecoder 211 are trained as one pair, and variational autoencoder 202 and variational autodecoder 212 are trained as a different pair, as illustrated. The training of each pair (variational autoencoder 201 and variational autodecoder 211 as one pair, and variational autoencoder 202 and variational autodecoder 212 as another pair) may be similar.

[0037] The trainer 400 has training data 401 of image type 331 and training data 402 of image type 332. Each training data in training data 401 and training data 402 may include thousands or millions of images. Some examples may not require labeling of the training data images because the loss function 410 can be calculated directly from the images in training recovery result 403 (for training data 401) or training recovery result 404 (for training data 402).

[0038] The loss function 410 used for training is the example MSE calculated in equation (1), where X Each pixel comes from an image within training data 401 (or training data 402), and Y Each pixel is taken from the image within the training recovery result 403 (or training recovery result 404). This encourages the reconstruction of the input at the output with minimal error. Backpropagation 411 is applied to the variational autoencoder 201 and the variational autodecoder 211 to generate complementary layers (e.g., in reverse order).

[0039] Figure 5 The illustration shows a search for a compressed image without the need for decompression to test against search criteria. The search material is pre-calculated and persisted in multiple embeddings 510 and / or multiple captions 530. The multiple embeddings 510 include image embedding 511, image embedding 512, text embedding 521, and text embedding 522. The multiple captions 530 include image captions 531 and image captions 532. Image embedding 512, text embedding 522, and image caption 532 are generated for image 302 similarly to those described above for image embedding 511, text embedding 521, and image caption 531.

[0040] The association between the embedded image 515 and the compressed image 311 enables the compressed image 311 to be identified using image embedding 511 or text embedding 521, and the compressed image 312 to be identified using image embedding 512 or text embedding 522. The association between the caption and the compressed image 535 enables the compressed image 311 to be identified using image caption 531, and the compressed image 312 to be identified using image caption 532.

[0041] Users can type an image search query 502 as an NL description in the search interface 500 to search for a desired image. The search interface 500 uses CLIP model 120c to convert the image search query 502 into a query embedding 504. CLIP model 120c can be the same model as CLIP models 120a and 120b or a different model.

[0042] The matching function 506 compares the query embedding 504 with each of the multiple embeddings 510 for similarity. Some examples use vector dot products, as shown in equations (2) and / or (3): Similarity = query_embedding·image_embedding Equation (2) Similarity = query_embedding·text_embedding Equation (3) in query_embedding It's a query embedding 504. image_embedding It is either image embedding 511 or image embedding 512, and query_embedding It is either text embedding 521 or text embedding 522.

[0043] In some examples, the matching function 506 also directly compares the image search query 502 with image captions 531 and 532 within multiple captions 530. When the best match is found from any of these comparisons or a weighted combination of these comparisons, the matching function 506 alerts the search interface 500, which identifies the compressed image 311 using the association between the embedded image and the compressed image 515 and / or the association between the caption and the compressed image 535. The search interface 500 generates a request 111 to decompress the compressed image 311. Alternatively, in some examples, the user can manually or using some other procedure type request 112 to decompress the compressed image 312.

[0044] Figure 6 Flowchart 600 is shown, illustrating exemplary operations that can be performed by architecture 100, and Figures 7A to 7D Flowcharts 700a to 700d illustrate further details of flowchart 600. In some examples, the operations described for flowcharts 600 and 700a to 700d are... Figure 9 The computational device 900 performs the operation. Flowchart 600 begins with pairwise training of the variational autoencoder and variational autodecoder (e.g., ...) in operation 602. Figure 4 (As shown).

[0045] That is, operation 602 trains variational autoencoder 201 to compress images of image type 331, trains variational autodecoder 211 to decompress compressed images of image type 331, trains variational autoencoder 202 to compress images of image type 332, and trains variational autodecoder 212 to decompress compressed images of image type 332. In some examples, the difference between image type 331 and image type 332 lies in the median aspect ratio of identifiable features within each image type (e.g., image type 301 is a photograph of people or scenes, and image type 302 is a map with a set of lines having a median aspect ratio greater than 10).

[0046] Operation 604 Figure 7A Flowchart 700a compresses images 301 and 302. Operation 606 uses... Figure 7B Flowchart 700b constructs a search dataset comprising multiple embeddings 510 and / or multiple captions 530. In operation 610, operation 608 uses... Figure 7C The search process in flowchart 700c selects the image to be decompressed, or a decompression request is generated by another process (e.g., request 112 for decompressing compressed image 312). In operation 612, using... Figure 7D Flowchart 700d decompresses the requested (multiple) images, and operation 614 returns the recovered images, such as recovered image 321 and / or recovered image 321.

[0047] Figure 7A Flowchart 700a is shown as a further detail of operation 604 of flowchart 600. Flowchart 700a is executed for both images 301 and 302. In operation 702, image 301 is received for compression, and operation 704 selects a variational autoencoder 201 from a plurality of variational autoencoders 102 based on image type 331 of image 301. Operation 706 compresses image 301 into a compressed image 311 using the variational autoencoder 201, and can be performed using operations 708 and 710. Operation 708 provides image 301 to the variational autoencoder 201, and operation 710 persists the output of the bottleneck layer 220 of the variational autoencoder 201 as the compressed image 311 among the plurality of compressed images 104.

[0048] In operation 702, image 302 is received for compression, and in operation 704, a variational autoencoder 202 is selected from a plurality of variational autoencoders 102 based on image type 332 of image 302. Operation 706 uses the variational autoencoder 202 to compress image 302 into compressed image 312, and can be performed using operations 708 and 710. Operation 708 provides image 302 to the variational autoencoder 202, and operation 710 persists the output of the bottleneck layer 220 of the variational autoencoder 202 as compressed image 312 among the plurality of compressed images 104.

[0049] Figure 7B Flowchart 700b is shown as further detail of operation 606 of flowchart 600. Flowchart 700b is performed on both images 301 and 302. Operation 720 generates an image embedding 511 from image 301 using CLIP model 120a, and operation 722 persists the image embedding 511 in multiple embeddings 510. Operation 724 associates the image embedding 511 with the compressed image 311.

[0050] Operation 726 generates an image caption 531 from image 301 using BLIP model 122 and image caption 531. Operation 728 persists the image caption 531 in multiple captions 530 and associates the image caption 531 with the compressed image 311. Operation 732 generates a text embedding 521 from image caption 531 using CLIP model 120b. Operation 734 persists the text embedding 521 in multiple embeddings 510 and associates the text embedding 521 with the compressed image 311.

[0051] Operation 720 generates an image embedding 512 from image 302 using CLIP model 120a, and operation 722 persists the image embedding 512 in multiple embeddings 510. Operation 724 associates the image embedding 512 with the compressed image 312. Operation 726 generates an image caption 532 from image 302 using CLIP model 122, operation 728 persists the image caption 532 in multiple captions 530, and operation 730 associates the image caption 532 with the compressed image 312. Operation 732 generates a text embedding 522 from image caption 532 using CLIP model 120b, operation 734 persists the text embedding 522 in multiple embeddings 510, and operation 736 associates the text embedding 522 with the compressed image 312.

[0052] Figure 7C Flowchart 700c is shown as further details of operation 608 of flowchart 600. In operation 740, an image search query 502 is received, and in operation 742, a query embedding 504 is generated from the image search query 502 using CLIP model 120c. Operation 744 uses one or more of operations 746, 748, and 750 to select the best similarity. Operation 746 selects a text embedding 521 from multiple embeddings 510 by determining the similarity between the image search query 502 and the text embedding 521. Operation 748 selects an image embedding 511 from multiple embeddings 510 by determining the similarity between the image search query 502 and the image embedding 511. Operation 750 selects an image caption 531 by determining the similarity between the image search query 502 and the image caption 531.

[0053] Operation 752 selects a compressed image 311 from a plurality of compressed images 104 based on at least selecting an image embedding 511, a text embedding 521, or an image caption 531 as a match for the image search query 502. Operation 754 generates a request 111 for decompressing the compressed image 311 based on at least the compressed image 311 selected in operation 752.

[0054] Figure 7D Flowchart 700d is shown as further details of operation 612 of flowchart 600. Flowchart 700d is executed for both compressed image 311 and compressed image 312. In operation 760, a request 111 for decompressing compressed image 311 is received, and operation 762 uses operations 764 and 766 to decompress compressed image 311. Operation 764 selects variational self-decoder 211 from a plurality of variational self-decoders 108 based on the fact that compressed image 311 is a compressed version of an image of image type 331. Operation 766 uses variational self-decoder 211 to decompress compressed image 311 into restored image 321.

[0055] In operation 760, a request 112 for decompressing compressed image 312 is received, and in operation 762, operations 764 and 766 are used to decompress compressed image 312. Operation 764 selects variational self-decoder 212 from a plurality of variational self-decoders 108 based on the fact that compressed image 312 is a compressed version of an image of image type 332. Operation 766 uses variational self-decoder 212 to decompress compressed image 312 into restored image 322.

[0056] Figure 8 Flowchart 800 is shown, illustrating exemplary operations that can be performed by architecture 100. In some examples, the operations described for flowchart 800 are performed by... Figure 9 The computing device 900 performs the operation. Flowchart 800 begins with operation 802, which includes receiving a first image of a first image type for compression, the first image being in pixel space.

[0057] Operation 804 includes providing a first image to a first variational autoencoder having a bottleneck layer. Operation 806 includes persisting the output of the bottleneck layer of the first variational autoencoder as a first compressed image, the first compressed image including a latent tensor. Operation 808 includes decompressing the first compressed image into a first restored image in pixel space by a first variational autodecoder based on at least receiving a request to decompress the first compressed image. Additional examples

[0058] An example system includes: a processor; and a computer-readable medium storing instructions operable, when executed by the processor, to: receive a first image of a first image type for compression, the first image being in pixel space; provide the first image to a first variational autoencoder having a bottleneck layer; persist the output of the bottleneck layer of the first variational autoencoder as a first compressed image, the first compressed image including a latent tensor; and, based on at least receiving a request to decompress the first compressed image, decompress the first compressed image by a first variational autodecoder to a first restored image, the first restored image being in pixel space.

[0059] An example computer-implemented method includes: receiving a first image of a first image type for compression, the first image being in pixel space; providing the first image to a first variational autoencoder having a bottleneck layer; persisting the output of the bottleneck layer of the first variational autoencoder as a first compressed image, the first compressed image including a latent tensor; and decompressing the first compressed image into a first restored image, the first restored image being in pixel space, by a first variational autodecoder based on at least receiving a request to decompress the first compressed image.

[0060] One or more example computer storage devices store computer-executable instructions thereon, which are computer-implementable to cause a computer to perform operations when executed by a computer, including: receiving a first image of a first image type for compression, the first image being in pixel space; providing the first image to a first variational autoencoder having a bottleneck layer; persisting the output of the bottleneck layer of the first variational autoencoder as a first compressed image, the first compressed image including a latent tensor; and decompressing the first compressed image into a first restored image in pixel space by a first variational autodecoder based on at least receiving a request to decompress the first compressed image.

[0061] Alternatives, or examples other than those described herein, include any combination of the following: - Select the first variational autoencoder from multiple variational autoencoders, each of which is trained to compress images of different image types; - The first variational autoencoder is trained to compress images of the first image type; - Select the first variational autodecoder from multiple variational autodecoders, each of which is trained to decompress compressed images of different image types; - The first variational autodecoder is trained to decompress compressed images of the first image type; - Receive a second image of a second image type for compression, the second image being in pixel space; - The second image type is different from the first image type; - Select the second variational autoencoder from a plurality of variational autoencoders; - The second variational autoencoder is trained to compress images of the second image type; - Provide the second image to the second variational autoencoder; - The second variational autoencoder has a bottleneck layer; - Persist the output of the bottleneck layer of the second variational autoencoder as a second compressed image; - The second compressed image has a potential tensor; - Based on receiving at least a request to decompress the second compressed image, select a second variational autodecoder from a plurality of variational autodecoders; - The second variational autodecoder is trained to decompress compressed images of the second image type; - The second compressed image is decompressed into a second restored image by the second variational autodecoder; - The second restored image is in pixel space; - Train the first variational autoencoder to compress images of the first image type; - Train the first variational autodecoder to decompress compressed images of the first image type; - Train a second variational autoencoder to compress images of a second image type; - Train a second variational autodecoder to decompress compressed images of a second image type; - Generate a first image embedding from the first image using the CLIP model; - Persist the first image embedding; - Embed the first image and associate it with the first compressed image; - Receive image search queries; - Select a first compressed image from multiple compressed images based on selecting a first image embedding or a first text embedding from at least multiple embeddings as a match for an image search query; - Based on selecting at least a first compressed image, generate a request to decompress the first compressed image; - Generate a caption for the first image from the first image using the BLIP model; - Use the CLIP model to generate the first text embedding from the first image caption; - Persistent first text embedding; - Associate the first text embedding with the first compressed image; - Selecting a first text embedding from multiple embeddings includes determining the similarity between the image search query and the first text embedding; - Selecting a first image embedding from multiple embeddings includes determining the similarity between the image search query and the first image embedding; - Select a first compressed image from multiple compressed images based on selecting a first text embedding as a match for an image search query from at least multiple embeddings; - Use the CLIP model to generate query embeddings from image search queries; - Determine the similarity between the query embedding and each embedding within multiple embeddings; - Selecting a first image embedding from multiple embeddings includes selecting the first image embedding based at least on the similarity between the query embedding and the first image embedding; - The first and second image types differ in the median aspect ratio of identifiable features within each image type; - The first image type is a photograph of people or scenes; - The second image type is a map with a set of lines having a median aspect ratio greater than 10; - The first image is compressed into a first compressed image by a first variational autoencoder; - The second image is compressed into a second compressed image by a second variational autoencoder; - The same CLIP model generates the first image embedding, the first text embedding, and the query embedding; - The CLIP model that generates the first image embedding is not the CLIP model that generates the first text embedding and / or is not the CLIP model that generates the query embedding; - The BLIP model includes bootstrapping language image pre-training for a unified visual language understanding and generation model; and - Multiple embeddings include a first image embedding, a first text embedding, a second image embedding for a second image, and a second text embedding for a second image.

[0062] Although various aspects of this disclosure have been described with reference to various examples and their associated operations, those skilled in the art will understand that combinations of operations from any number of different examples are also within the scope of various aspects of this disclosure. Example operating environment

[0063] Figure 9 This is a block diagram of an example computing device 900 (e.g., a computer storage device) used to implement the various aspects disclosed herein, and is generally designated as computing device 900. In some examples, one or more computing devices 900 are provided for a local computing solution. In some examples, one or more computing devices 900 are provided for a cloud computing solution. In some examples, a combination of local and cloud computing solutions is used. Computing device 900 is merely one example of a suitable computing environment and is not intended to impose any limitation on the scope or functionality of the examples disclosed herein, whether used alone or as part of a larger set.

[0064] The computing device 900 should not be construed as having any dependencies or requirements relating to any or all of the components / modules illustrated. The examples disclosed herein can be described in the general context of computer code or machine-usable instructions, including computer-executable instructions, such as program components executed by a computer or other machine (such as a personal data assistant or other handheld device). Generally, program components, including routines, programs, objects, components, data structures, etc., refer to code that performs a specific task or implements a specific abstract data type. The disclosed examples can be practiced in a variety of system configurations, including personal computers, laptops, smartphones, mobile tablets, handheld devices, consumer electronics, professional computing devices, etc. The disclosed examples can also be practiced in distributed computing environments when the task is performed by a remote processing device linked via a communication network.

[0065] Computing device 900 includes a bus 910 that directly or indirectly couples to the following devices: computer storage memory 912, one or more processors 914, one or more presentation components 916, input / output (I / O) ports 918, I / O components 920, power supply 922, and network components 924. Although computing device 900 is depicted as a single device, multiple computing devices 900 can work together and share the depicted device resources. For example, memory 912 can be distributed across multiple devices, and processors 914 can be housed with different devices.

[0066] Bus 910 indicates that it may be one or more buses (such as an address bus, a data bus, or a combination thereof). Although for clarity... Figure 9 The various blocks are shown with lines, but the various components can be depicted using alternative representations. For example, in some examples, presentation components such as display devices are I / O components, and some examples of processors have their own memory. There is no distinction between categories such as "workstation," "server," "laptop," and "handheld device," because all of these are envisioned as... Figure 9 Within the scope of this document and the reference to "computing device". Memory 912 may take the form of a computer storage medium referenced below and is operable to provide storage for computer-readable instructions, data structures, program modules, and other data for computing device 900. In some examples, memory 912 stores one or more of an operating system, a general-purpose application platform, or other program modules and program data. Thus, memory 912 is capable of storing and accessing data 912a and instructions 912b, which are executable by processor 914 and configured to perform the various operations disclosed herein. Therefore, computing device 900 includes a computer storage device on which computer-executable instructions 912b are stored.

[0067] In some examples, memory 912 includes computer storage media. Memory 912 may include any number of memories associated with or accessible by computing device 900. Memory 912 may be internal to computing device 900 (e.g., Figure 9 As shown, the memory 912 may be located outside the computing device 900 (not shown), or both (not shown). Alternatively or additionally, the memory 912 may be distributed across multiple computing devices 900, for example in a virtualization environment where instruction processing is performed on multiple computing devices 900. For the purposes of this disclosure, "computer storage medium," "computer storage memory," "memory," and "memory device" are synonyms for memory 912, and none of these terms include a carrier wave or propagating signal.

[0068] The processors 914 may include any number of processing units that read data from various entities such as memory 912 or I / O components 920. Specifically, the processors 914 are programmed to execute computer-executable instructions to implement various aspects of this disclosure. These instructions may be executed by a processor, multiple processors within computing device 900, or a processor external to client computing device 900. In some examples, the processors 914 are programmed to execute instructions such as those discussed below and illustrated in the flowcharts depicted in the accompanying drawings. Furthermore, in some examples, the processors 914 represent an implementation of analog technology to perform the operations described herein. For example, the operations may be performed by analog client computing device 900 and / or digital client computing device 900. The presentation components 916 present data indications to a user or other device. Exemplary presentation components include display devices, speakers, printing components, vibration components, etc. Those skilled in the art will understand and appreciate that computer data can be presented in a variety of ways, such as visually in a graphical user interface (GUI), audibly through speakers, wirelessly between computing devices 900, over a wired connection, or otherwise. I / O port 918 allows computing device 900 to be logically coupled to other devices including I / O components 920, some of which may be built-in. Example I / O components 920 include, for example, but not limited to, microphones, joysticks, gamepads, satellite dishes, scanners, printers, wireless devices, etc.

[0069] Computing device 900 can operate in a networked environment via network component 924 using a logical connection to one or more remote computers. In some examples, network component 924 includes a network interface card and / or computer-executable instructions (e.g., a driver) for operating the network interface card. Communication between computing device 900 and other devices can occur over any wired or wireless connection using any protocol or mechanism. In some examples, network component 924 is operable to use short-range communication technologies (e.g., Near Field Communication (NFC), Bluetooth) wirelessly using transmission protocols. TM Data is transmitted between devices (such as brand communication devices) or combinations thereof via public, private, or hybrid (public and private) means. Network component 924 communicates with remote resource 928 (e.g., cloud resource) on network 930 via wireless communication link 926 and / or wired communication link 926a. Various examples of communication links 926 and 926a include wireless connections, wired connections, and / or dedicated links, and in some examples, at least a portion is routed over the Internet.

[0070] Although described in conjunction with example computing device 900, the examples of this disclosure can be implemented with many other general-purpose or special-purpose computing system environments, configurations, or devices. Examples of well-known computing systems, environments, and / or configurations that may be applicable to various aspects of this disclosure include, but are not limited to, smartphones, mobile tablet computers, mobile computing devices, personal computers, server computers, handheld or laptop devices, multiprocessor systems, game consoles, microprocessor-based systems, set-top boxes, programmable consumer electronics, mobile phones, wearable or accessory form factor mobile computing and / or communication devices (e.g., watches, glasses, headsets, or earphones), network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, virtual reality (VR) devices, augmented reality (AR) devices, mixed reality devices, holographic devices, etc. Such systems or devices can accept input from users in any manner, including from input bars such as keyboards or pointing devices, via gesture input, proximity input (such as by hover), and / or via voice input.

[0071] Examples of this disclosure can be described in the general context of computer-executable instructions (such as program modules) that are executed by one or more computers or other devices as software, firmware, hardware, or a combination thereof. Computer-executable instructions can be organized into one or more computer-executable components or modules. Typically, program modules include, but are not limited to, routines, programs, objects, components, and data structures that perform a particular task or implement a particular abstract data type. Various aspects of this disclosure can be implemented with any number and organization of such components or modules. For example, various aspects of this disclosure are not limited to specific computer-executable instructions or specific components or modules illustrated in the accompanying drawings and described herein. Other examples of this disclosure may include different computer-executable instructions or components that have more or less functionality than those illustrated and described herein. In examples involving general-purpose computers, various aspects of this disclosure transform a general-purpose computer into a special-purpose computing device when configured to execute the instructions described herein.

[0072] By way of example and not limitation, computer-readable media include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, etc. Computer storage media are tangible and mutually exclusive with communication media. Computer storage media are implemented in hardware and do not include carrier waves and propagating signals. Computer storage media used for the purposes of this disclosure do not include signals. Exemplary computer storage media include hard disks, flash drives, solid-state storage, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital universal disc (DVD) or other optical storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other non-transfer medium that can be used to store information for access by a computing device. In contrast, communication media typically implement computer-readable instructions, data structures, program modules, etc., in modulated data signals (such as carrier waves) or other transport mechanisms, and include any information delivery medium.

[0073] The order in which operations are performed or executed in the examples of this disclosure illustrated and described herein is not required and may be performed in different orders in various examples. For example, it is contemplated that a particular operation is performed or executed before, simultaneously with, or after another operation within the scope of various aspects of this disclosure. When elements of various aspects of this disclosure or examples thereof are introduced, the articles “a,” “an,” “the,” and “described” are intended to indicate the presence of one or more of these elements. The terms “comprising,” “including,” and “having” are intended to be inclusive and indicate that additional elements other than those listed may be present. The term “exemplary” is intended to mean “an example of…”. The phrase “one or more of the following: A, B, and C” means “at least one of A and / or at least one of B and / or at least one of C.”

[0074] Various aspects of this disclosure have been described in detail, and modifications and variations are obviously possible without departing from the scope of the aspects of this disclosure as defined in the appended claims. Since various changes can be made to the above-described constructions, products, and methods without departing from the scope of the aspects of this disclosure, everything contained in the above description and shown in the accompanying drawings is intended to be interpreted as illustrative rather than restrictive.

Claims

1. A system comprising: Processor (914); as well as Computer-readable medium (912) storing instructions (912b) that, when executed by the processor, are operable to: Receive (802) a first image (301) of a first image type (331) for compression, the first image being in pixel space (221); The first image is provided (804) to a first variational autoencoder (201), which has a bottleneck layer (220). The output of the bottleneck layer of the first variational autoencoder is persisted (806) as a first compressed image (311), the first compressed image including a latent tensor; as well as Based on receiving at least a request (111) to decompress the first compressed image, the first compressed image is decompressed (808) by the first variational autodecoder (211) into a first restored image (321), the first restored image being in pixel space.

2. The system according to claim 1, wherein the instructions are further operable to: The first variational autoencoder is selected from a plurality of variational autoencoders, each of which is trained to compress images of different image types, wherein the first variational autoencoder is trained to compress images of the first image type. The first variational autodecoder is selected from a plurality of variational autodecoders, each of which is trained to decompress compressed images of different image types, wherein the first variational autodecoder is trained to decompress compressed images of the first image type. A second image of a second image type is received for compression, the second image being in pixel space, wherein the second image type is different from the first image type; A second variational autoencoder is selected from the plurality of variational autoencoders, and the second variational autoencoder is trained to compress images of the second image type; The second image is provided to the second variational autoencoder, which has a bottleneck layer; The output of the bottleneck layer of the second variational autoencoder is persisted as a second compressed image, the second compressed image including a latent tensor; as well as Based on receiving at least a request to decompress the second compressed image: A second variational autodecoder is selected from the plurality of variational autodecoders, the second variational autodecoder being trained to decompress compressed images of the second image type; as well as The second compressed image is decompressed into a second restored image by the second variational autodecoder, and the second restored image is in pixel space.

3. The system according to claim 2, wherein the instructions are further operable to: Train the first variational autoencoder to compress images of the first image type; Train the first variational autodecoder to decompress a compressed image of the first image type; Train the second variational autoencoder to compress images of the second image type; as well as The second variational autodecoder is trained to decompress compressed images of the second image type.

4. The system according to claim 1, wherein the instructions are further operable to: A first image embedding is generated from the first image using a contrastive language image pre-training (CLIP) model; Persist the first image embedding; The first image is embedded and associated with the first compressed image; Receive image search queries; The first compressed image is selected from multiple compressed images based on selecting either the first image embedding or the first text embedding as a match for the image search query from at least multiple embeddings; as well as Based on selecting at least the first compressed image, a request is generated to decompress the first compressed image.

5. The system according to claim 4, wherein the instructions are further operable to: The first image caption is generated from the first image using a Bootstrap Language Image Pre-trained (BLIP) model; The CLIP model is used to generate a first text embedding from the first image caption; Persist the first text embedding; and The first text is embedded and associated with the first compressed image. Selecting the first text embedding from the plurality of embeddings includes determining the similarity between the image search query and the first text embedding.

6. The system of claim 4, wherein the instructions are further operable to: Generate query embeddings from the image search query using the CLIP model; and Determine the similarity between the query embedding and each of the plurality of embeddings. Selecting the first image embedding from the plurality of embeddings includes selecting the first image embedding based at least on the similarity between the query embedding and the first image embedding.

7. The system according to claim 1, wherein the instructions are further operable to: The first image caption is generated from the first image using a Bootstrap Language Image Pre-trained (BLIP) model; The first text embedding is generated from the first image caption using a contrastive language image pre-trained (CLIP) model; Persist the first text embedding; The first text is embedded and associated with the first compressed image; Receive image search queries; The first compressed image is selected from multiple compressed images based on selecting the first text embedding as a match for the image search query from at least multiple embeddings; as well as Based on selecting at least the first compressed image, a request is generated to decompress the first compressed image.

8. A computer-implemented method, comprising: Receive (802) a first image (301) of a first image type (331) for compression, the first image being in pixel space (221); The first image is provided (804) to a first variational autoencoder (201), which has a bottleneck layer (220). The output of the bottleneck layer of the first variational autoencoder is persisted (806) as a first compressed image (311), the first compressed image including a latent tensor; as well as Based on receiving at least a request (111) to decompress the first compressed image, the first compressed image is decompressed (808) by the first variational autodecoder (211) into a first restored image (321), the first restored image being in pixel space.

9. The computer-implemented method according to claim 8, further comprising: The first variational autoencoder is selected from a plurality of variational autoencoders, each of which is trained to compress images of different image types, wherein the first variational autoencoder is trained to compress images of the first image type. The first variational autodecoder is selected from a plurality of variational autodecoders, each of which is trained to decompress compressed images of different image types, wherein the first variational autodecoder is trained to decompress compressed images of the first image type. A second image of a second image type is received for compression, the second image being in pixel space, wherein the second image type is different from the first image type; A second variational autoencoder is selected from the plurality of variational autoencoders, wherein the second variational autoencoder is trained to compress images of the second image type; The second image is provided to the second variational autoencoder, which has a bottleneck layer; The output of the bottleneck layer of the second variational autoencoder is persisted as a second compressed image, the second compressed image including a latent tensor; as well as Based on receiving at least a request to decompress the second compressed image: A second variational autodecoder is selected from the plurality of variational autodecoders, wherein the second variational autodecoder is trained to decompress compressed images of the second image type; as well as The second compressed image is decompressed into a second restored image by the second variational autodecoder, and the second restored image is in pixel space.

10. The computer-implemented method according to claim 9, further comprising: Train the first variational autoencoder to compress images of the first image type; Train the first variational autodecoder to decompress a compressed image of the first image type; Train the second variational autoencoder to compress images of the second image type; as well as The second variational autodecoder is trained to decompress compressed images of the second image type.

11. The computer-implemented method according to claim 8, further comprising: A first image embedding is generated from the first image using a contrastive language image pre-training (CLIP) model; Persist the first image embedding; The first image is embedded and associated with the first compressed image; Receive image search queries; The first compressed image is selected from multiple compressed images based on selecting either the first image embedding or the first text embedding as a match for the image search query from at least multiple embeddings; as well as Based on selecting at least the first compressed image, a request is generated to decompress the first compressed image.

12. The computer-implemented method according to claim 11, further comprising: The first image caption is generated from the first image using a Bootstrap Language Image Pre-trained (BLIP) model; The CLIP model is used to generate a first text embedding from the first image caption; Persist the first text embedding; The first text is embedded and associated with the first compressed image; and Selecting the first text embedding from the plurality of embeddings includes determining the similarity between the image search query and the first text embedding.

13. The computer-implemented method according to claim 11, further comprising: The CLIP model is used to generate query embeddings from the image search query; Determine the similarity between the query embedding and each of the plurality of embeddings; and Selecting the first image embedding from the plurality of embeddings includes selecting the first image embedding based at least on the similarity between the query embedding and the first image embedding.

14. The computer-implemented method according to claim 8, further comprising: The first image caption is generated from the first image using a Bootstrap Language Image Pre-trained (BLIP) model; The first text embedding is generated from the first image caption using a contrastive language image pre-trained (CLIP) model; Persist the first text embedding; The first text is embedded and associated with the first compressed image; Receive image search queries; The first compressed image is selected from multiple compressed images based on selecting the first text embedding as a match for the image search query from at least multiple embeddings; as well as Based on selecting at least the first compressed image, a request is generated to decompress the first compressed image.

15. A computer storage device having computer-executable instructions stored thereon, the computer-executable instructions causing the computer to perform operations when executed by a computer, comprising: Receive (802) a first image (301) of a first image type (331) for compression, the first image being in pixel space (221); The first image is provided (804) to a first variational autoencoder (201), which has a bottleneck layer (220). The output of the bottleneck layer of the first variational autoencoder is persisted (806) as a first compressed image (311), the first compressed image including a latent tensor; as well as Based on receiving at least a request (111) to decompress the first compressed image, the first compressed image is decompressed (808) by the first variational autodecoder (211) into a first restored image (321), the first restored image being in pixel space.

16. The computer storage device of claim 15, wherein the operation further comprises: The first variational autoencoder is selected from a plurality of variational autoencoders, each of which is trained to compress images of different image types, wherein the first variational autoencoder is trained to compress images of the first image type. The first variational autodecoder is selected from a plurality of variational autodecoders, each of which is trained to decompress compressed images of different image types, wherein the first variational autodecoder is trained to decompress compressed images of the first image type. A second image of a second image type is received for compression, the second image being in pixel space, wherein the second image type is different from the first image type; A second variational autoencoder is selected from the plurality of variational autoencoders, wherein the second variational autoencoder is trained to compress images of the second image type; The second image is provided to the second variational autoencoder, which has a bottleneck layer; The output of the bottleneck layer of the second variational autoencoder is persisted as a second compressed image, the second compressed image including a latent tensor; as well as Based on receiving at least a request to decompress the second compressed image: A second variational autodecoder is selected from the plurality of variational autodecoders, the second variational autodecoder being trained to decompress compressed images of the second image type; as well as The second compressed image is decompressed into a second restored image by the second variational autodecoder, and the second restored image is in pixel space.

17. The computer storage device of claim 15, wherein the operation further comprises: A first image embedding is generated from the first image using a contrastive language image pre-training (CLIP) model; Persist the first image embedding; The first image is embedded and associated with the first compressed image; Receive image search queries; The first compressed image is selected from multiple compressed images based on selecting either the first image embedding or the first text embedding as a match for the image search query from at least multiple embeddings; as well as Based on selecting at least the first compressed image, a request is generated to decompress the first compressed image.

18. The computer storage device of claim 17, wherein the operation further comprises: The first image caption is generated from the first image using a Bootstrap Language Image Pre-trained (BLIP) model; The CLIP model is used to generate a first text embedding from the first image caption; Persist the first text embedding; The first text is embedded and associated with the first compressed image; and Selecting the first text embedding from the plurality of embeddings includes determining the similarity between the image search query and the first text embedding.

19. The computer storage device of claim 17, wherein the operation further comprises: The CLIP model is used to generate query embeddings from the image search query; Determine the similarity between the query embedding and each of the plurality of embeddings; and Selecting the first image embedding from the plurality of embeddings includes selecting the first image embedding based at least on the similarity between the query embedding and the first image embedding.

20. The computer storage device of claim 15, wherein the operation further comprises: The first image caption is generated from the first image using a Bootstrap Language Image Pre-trained (BLIP) model; The first text embedding is generated from the first image caption using a contrastive language image pre-trained (CLIP) model; Persist the first text embedding; The first text is embedded and associated with the first compressed image; Receive image search queries; The first compressed image is selected from multiple compressed images based on selecting the first text embedding as a match for the image search query from at least multiple embeddings; as well as Based on selecting at least the first compressed image, a request is generated to decompress the first compressed image.

Images