Extensible learning type image compression based on adaptive learning type sparse image representation

Abstract

The present invention relates to a method for performing an extensible learning type image compression (LIC) based on an adaptive learning type sparse image representation (LSIR), in particular to a method for performing an extensible learning type image compression (LIC) based on an adaptive learning type sparse image representation (LSIR). The method is implemented by a transmitting device, comprising: K input encoders of the transmitting device encoding an input image x to generate K image feature tensors, where K is an integer greater than 1; based on the K image feature tensors and the corresponding base codebooks, respectively calculating K sparse hidden features based on the codebooks; a weight map is calculated, the weight map provides information used for carrying out weighted combination on K decoded image embedding features, and the K decoded image embedding features are respectively based on the K codebook-based sparse hidden features and the corresponding base codebook; calculating a compact weight map based on the weight map and a target compression rate; and sending the compact weight map and the K sparse hidden features based on the codebook to a receiving device.

Description

Cross-reference to related applications

[0001] This application claims priority to U.S. Provisional Application No. 63 / 527,278, filed July 17, 2023. The entire disclosure of the above application is incorporated herein by reference. Technical Field

[0002] This disclosure relates to learned image compression (LIC), and more particularly to scalable LIC based on adaptive learned sparse image representation (LSLR). Background Technology

[0003] In recent years, neural network-based LIC (Sparse Image Coding) has been extensively studied, demonstrating superior performance compared to traditional coding methods such as Joint Photographic Experts Group (JPEG), Versatile Video Coding (VVC), and High Efficiency Video Coding (HEVC). Sparse representation learning has always been a hot research topic in general image modeling. Learned Sparse Image Representation (LSIR) learns a highly compressed codebook in the image domain through a vector quantization autoencoder trained based on adversarial and perceptual losses. Summary of the Invention

[0004] The first aspect relates to a method implemented by a transmitting device, the method comprising: the transmitting device K An input encoder for the input image x Encode to generate K Image feature tensor ,in, K It is an integer greater than 1; based on the above K Image feature tensor and the corresponding base codebook Calculate separately K A codebook-based sparse hidden feature ; Calculate the weighted graph W The weight graph W Provide for K Decoded image embedding features The information for weighted merging, the K Decoded image embedding features Based on the above K A codebook-based sparse hidden feature and the corresponding basecode Based on the weight graph W and target compression ratio R Calculate the compact weighted graph The compact weighted graph and stated K A codebook-based sparse hidden feature Send to the receiving device.

[0005] Optionally, in the first implementation according to the first aspect, based on the weight graph W and the target compression ratio R Calculate the compact weight graph Includes: based on the target compression ratio R Calculate the mask rate Based on the mask rate Generate a weight mask M Based on the weight graph W And the weight mask, calculate the compact weight graph. .

[0006] Optionally, according to the first aspect or any of its implementations, in the second implementation, the mask rate... τ It is the ratio of the masked weight to the total weight.

[0007] Optionally, according to the first aspect or any of its implementations, in a third implementation, the method further includes: based on the mask rate τ And a masking strategy, to generate the weight mask. M .

[0008] Optionally, according to the first aspect or any of its implementations, in a fourth implementation, the method further includes: based on the target compression ratio R Calculate the target number of bits Based on the target number of bits The masking strategy is determined.

[0009] Optionally, according to the first aspect or any of its implementations, in the fifth implementation, the method further includes: applying the weight mask... M It is sent to the receiving device.

[0010] Optionally, according to the first aspect or any implementation thereof, in the sixth implementation, the method further includes: calculating and measuring the input image. x With reconstructed output image Merging latent features between and / or targets Merging latent features Distortion loss between D Modify the compact weight graph The weights corresponding to the unmasked pixels are assigned to reduce the distortion loss. D .

[0011] Optionally, according to the first aspect or any of its implementations, in the seventh implementation, the method further includes: calculating and measuring the input image based on multiple candidate masking strategies. x With reconstructed output image Merging latent features between and / or targets Merging latent features Distortion loss between D Select the masking strategy with the minimum distortion loss from the multiple candidate masking strategies. D The masking strategy.

[0012] The second aspect relates to a method implemented by a receiving device. The method includes: receiving from a transmitting device... K A codebook-based sparse hidden feature and compact weighted graph ,in, K It is an integer greater than 1; based on the above K A codebook-based sparse hidden feature ,calculate K Decoded image embedding features Based on the compact weight graph and stated K Decoded image embedding features Calculate the fill weight map Based on the filled weight map and stated K Decoded image embedding features Calculate merged latent features Based on the aforementioned merged hidden features Calculate and reconstruct the output image .

[0013] Optionally, according to the second aspect, in the first implementation, based on the compact weight graph and stated K Decoded image embedding features Calculate the fill weight map Includes: for the compact weight graph The masked weights and the predicted weights.

[0014] Optionally, according to the second aspect or any of its implementations, in the second implementation, based on the compact weight graph... and stated K Decoded image embedding features Calculate the fill weight map Includes: in calculating the K Decoded image embedding features At that time, predict weights for unused basecodes.

[0015] Optionally, according to the second aspect or any of its implementations, in the third implementation, based on the compact weight graph... and stated K Decoded image embedding features Calculate the fill weight map Includes: calculating the latent features used to recover the target merged feature A new set of weights.

[0016] Optionally, according to the second aspect or any of its implementations, in a fourth implementation, the method further includes: using a neural network model based on the compact weight graph. and stated K Decoded image embedding features Calculate the fill weight map .

[0017] Optionally, according to the second aspect or any of its implementations, in the fifth implementation, the neural network model is designed for the compact weight graph. The corresponding masking strategy is optimized.

[0018] Optionally, according to the second aspect or any of its implementations, in the sixth implementation, based on the filling weight graph... and stated K Decoded image embedding features Calculate the merged hidden features Including: targeting Each pixel in the image is calculated as follows: ( ):

[0019] in, It is the filled weight map The k-th weight corresponding to the l-th pixel.

[0020] The third aspect relates to an apparatus. The apparatus includes: a memory for storing instructions; and one or more processors coupled to the memory and configured to execute the instructions, such that the apparatus performs the method according to the first aspect, the second aspect, or any implementation thereof.

[0021] The fourth aspect relates to a computer program product. The computer program product includes computer-executable instructions stored in a non-transitory computer-readable storage medium, wherein, when executed by one or more processors of the device, the computer-executable instructions cause the device to perform the method according to the first aspect, the second aspect, or any implementation thereof.

[0022] For clarity, any of the above embodiments may be combined with any one or more of the other embodiments described above to create new embodiments within the scope of this disclosure.

[0023] These and other features and their advantages will become clearer from the following detailed description in conjunction with the accompanying drawings and claims. Attached Figure Description

[0024] To gain a more complete understanding of this disclosure, the following brief description is provided in conjunction with the accompanying drawings and specific embodiments, wherein similar reference numerals in the drawings denote similar parts.

[0025] Figure 1 This is a schematic diagram of the general framework of LIC.

[0026] Figure 2 This is a schematic diagram of the general framework of LSIR.

[0027] Figure 3 This is a schematic diagram of a general framework for adaptive LSIR according to an embodiment of the present disclosure.

[0028] Figure 4 This is a schematic diagram of a general framework for a scalable LIC based on adaptive LSLR according to an embodiment of the present disclosure.

[0029] Figure 5 This is a schematic diagram of an adaptive weighted compression module according to an embodiment of the present disclosure.

[0030] Figure 6 This is a flowchart of the processing workflow of a transmitting device according to an embodiment of the present disclosure.

[0031] Figure 7 This is a flowchart of the processing workflow of a receiving device according to an embodiment of the present disclosure.

[0032] Figure 8 This is a schematic diagram of an apparatus according to an embodiment of the present disclosure. Detailed Implementation

[0033] First, it should be understood that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods can be implemented using any number of techniques, whether currently known or existing. This disclosure should not be limited in any way to the illustrative implementations, drawings, and techniques described below, including the exemplary designs and implementations illustrated and described herein, but can be modified within the full scope of the appended claims and their equivalents.

[0034] This paper discloses various systems and methods for encoding and decoding images. Specifically, this disclosure describes a general LIC framework implemented through adaptive LSLR. The disclosed framework provides scalability, enabling the utilization of increasingly larger and more complex LSIRs, thereby significantly improving the compression performance of LIC while achieving high compression ratios and high reconstruction quality. Specifically, the disclosed framework employs an adaptive weighted method with effective weighted compression, which can flexibly select the compression ratio based on the input image and compression requirements.

[0035] Figure 1 This is a schematic diagram of the general framework of LIC. LIC is a modern image compression method that utilizes deep learning techniques to learn efficient representations of images. Traditional image compression techniques rely on manually designed algorithms to convert image data into compressed formats. However, LIC aims to improve compression efficiency by automatically learning the most efficient compression strategies directly from the data through training neural networks. Figure 1 As shown, at the transmitting end (i.e., the end implemented by the transmitting device), the input image... x Image embedding features are generated by input encoder 102. This feature is the input image. x The numerical representation of the input image. For example, in one embodiment, the input encoder 102 is a neural network used to process the input image. x Convert the original pixel values ​​to the input image x Compressed and semantically meaningful numerical representations within a high-dimensional vector space. In some embodiments, image embedding features The data is further compressed into a data string through quantization and arithmetic encoding, which is convenient for efficient storage and transmission from the transmitter to the receiver.

[0036] At the receiving end (i.e., the end implemented by the receiving device), image embedding features The received data string is recovered (i.e., decoded) through arithmetic decoding and dequantization. The decoded image embedding features are then processed. The input is fed into decoder 104. Decoder 104 is a neural network that is used to embed features based on the decoded image. Reconstructed output image The goal of LIC is to minimize the reconstruction output image. With the original input x The recovery loss between them, and minimizing the amount of data used to represent image embedding features. Bits used for storage and transmission.

[0037] like Figure 1 The current LIC framework relies on learning a general, compact image representation, i.e., a latent space, where image embedding features... Capable of capturing input x Detailed information for reconstruction The LIC framework has several serious limitations. First, it struggles with learning general priors in the image domain. (i.e., learning to generate representations of input images) x Image embedding features When dealing with image features and attributes (i.e., prior image information), compression performance is inherently limited by model capacity (such as network structure and the number of parameters in the input encoder and decoder). Due to limited model capacity, training data, and computational resources during both training and testing phases, it is difficult to further improve compression performance beyond a good benchmark. Second, learning a LIC model aims to balance mutually exclusive objectives in rate-distortion (RD) loss, where reducing reconstruction distortion and lowering the bit rate are contradictory. Therefore, because it is difficult to balance different loss terms in end-to-end training, using LIC makes it difficult to simultaneously improve compression performance and perceptual quality.

[0038] Figure 2 This is a schematic diagram of the general framework of LSIR. In LSIR, the image-domain vector quantization autoencoder is trained based on adversarial and perceptual losses (e.g., using a Vector Quantized Generative Adversarial Network (VQGAN) method) to learn a highly compressed codebook 202. The learned codebook 202 includes the set of codewords used in the compression algorithm. The goal is to represent the image using a set of codewords from codebook 202, which is more efficient than directly encoding every vector or pixel group within the image. The learned codebook 202 is optimized end-to-end to balance codebook efficiency and reconstruction quality. Figure 2 As shown, the input image x Image embedding features are generated by input encoder 102. ,like Figure 1 As described above. For LSIR, at the transmitting end, image embedding features... The learned codebook 202 is mapped to a series of codeword indices. Codeword Index This is an integer, which can be efficiently stored or sent from the sender to the receiver. At the receiver, the same learned codebook 202 is used for embedding image features. Perform recovery / decoding (e.g., using the codebook 202 with the received codeword index). (Corresponding codewords). Then, decoder 104 is used to embed features based on the decoded image. Reconstructed output image .

[0039] In Figure 2 The LSIR and Figure 1 When compared with the LIC, LSIR performs worse in image compression. This is because, when used for compression, LSIR uses a compact codebook to handle complex general image priors. Modeling is performed to achieve the ambitious goal of high compression rates. As a result, the reconstructed images often lack rich and faithful detail.

[0040] Figure 3 This is a schematic diagram of a general framework for adaptive LSIR according to an embodiment of the present disclosure. The goal of adaptive LSIR is to improve the performance of general image restoration for arbitrary image content. In adaptive LSIR, such as Figure 2 The learned codebook 202 is split into K A basic codebook Each codebook is geared towards images of a specific classification category. For example, in some embodiments, these classification categories are semantic categories of the images, such as indoor objects, natural scenes, and portraits. Adaptive LSIR can also be applied to face images (i.e., only for face semantic categories), where the classification categories used to split the base codebook are the image distribution. Figure 3 As shown, at the sending end, the input image x pass K Input encoders (e.g., input encoder 1 302A... input encoders) K Encode 302K to generate separately K Image embedding features In some embodiments, input encoder 1 302A... input encoder K 302K are the same, therefore The same applies. In other embodiments, the input encoder 1 302A... input encoder K 302K is different, therefore They are different. Then, K Image embedding features Codeword generation module 1 306A... Codeword generation module K306K uses the corresponding basecode. Mapped to respectively K Codeword index of a sequence Furthermore, on the sending end, K Image embedding features The input is passed to the weight predictor module 310, which is used to calculate the weight graph. W Weighted graph W Used to guide how to embed features into the decoded image at the receiving end. Weighted merging is performed to calculate the final image embedding features. Codeword Index and weighted graph W From the sender to the receiver.

[0041] At the receiving end, feature retrieval module 1 308A... Feature retrieval module K 308K is used to employ the corresponding basecode. For the corresponding image embedding features Perform recovery / decoding to obtain the decoded image embedding features. Weighted graph W and decoded image embedding features The result is passed to the weighted merging module 312. The weighted merging module 312 is used for weighted graph-based merging. W Embedded features of the decoded image Perform weighted merging to generate merged latent features. Merging latent features The data is passed to decoder 314, which uses it to merge latent features. Reconstructed output image .

[0042] Adaptive LSIR has been applied to face-centric image compression. For example, due to the highly structured features of faces, high-quality (HQ) codebooks can be robustly learned for face reconstruction. However, this approach is difficult to extend to general images that include common image content. For instance, in some applications of adaptive LSIR, image reconstruction performance for general images can be improved by merging features from multiple base codebooks split according to semantic categories. That is, semantic classification-guided training is used to learn a semantic classification-guided codebook, thereby achieving better semantic classification-independent restoration performance. However, image restoration performance depends on the post-predicted adaptive weight map (e.g., ...). Figure 3 Weighted graph in W These weighted graphs are not conducive to compression. Therefore, dense weighted graphs are difficult to transmit in practice due to excessive transmission overhead.

[0043] To address one or more of the aforementioned technical problems, this disclosure describes a general framework for achieving efficient LIC using an adaptive LSIR. The disclosed framework employs an adaptive weighting method with weighted compression, providing scalability by leveraging increasingly larger and more complex adaptive LSIRs, which comprise basecoders arbitrarily split according to semantic category, image distribution, quality, etc. Therefore, the disclosed adaptive weighting mechanism can flexibly select the compression ratio based on the input image and compression requirements. Ultimately, the compression performance of LIC is significantly improved, achieving both high compression ratio and high reconstruction quality.

[0044] Figure 4 This is a schematic diagram of a general framework for a scalable LIC based on adaptive LSLR according to an embodiment of this disclosure. Figure 4 As shown, the disclosed scalable LIC based on the adaptive LSLR framework includes Figure 3 The general framework for adaptive LSIR (i.e. Figure 3 and Figure 4 Similar reference numerals in the figures indicate similar parts and perform similar functions. Figure 3 (Similar functions as described above). Figure 3 Compared to the adaptive LSIR framework in [the text], Figure 4 The scalable LIC based on the adaptive LSLR framework disclosed herein includes an adaptive weight compression module 402 at the transmitting end and a weight padding module 404 at the receiving end. As described herein, the adaptive weight compression module 402 can achieve an efficient LIC based on adaptive LSLR and also has the flexibility to adaptively select the compression rate according to the actual input image and compression requirements.

[0045] like Figure 3 The input image is described above. x pass K Input encoders (e.g., input encoder 1 302A... input encoders) K Encode 302K to generate separately K Image embedding features In one embodiment, the image x A general three-dimensional (3D) tensor representation is used. A 3D tensor is a mathematical object that extends the concept of a matrix to three dimensions. For example, in one embodiment, a general framework for scalable LIC based on adaptive LSLR is used for images. x For input, where, x It is shaped as The general 3D tensor, w , h , c This indicates the width, height, and number of channels of the image. For example, for a color image, For spectral images, For color depth (RGB-D) images, As mentioned above, the image x pass K Input encoders (e.g., input encoder 1 302A... input encoders) K Encode 302K to generate separately K Image embedding feature tensor In some embodiments, K One input encoder (i.e., input encoder 1 302A... input encoder) K The 302K input encoders can be different or identical. For example, in some embodiments, the same input encoder can be used to encode the input image. x It is embedded within a visual embedding space that models the visual representations of all natural images, including those with different content and domain distributions. In some embodiments, different input encoders can be used to process the input image. x Embedded into different visual embedding spaces, which are oriented towards each basecodec. The learned image represents a specific classification category. Furthermore, in some embodiments, K Some (but not all) of the input encoders can be the same to model a shared visual embedding space for several classification categories of an image.

[0046] In one embodiment, to Figure 4 The Middle k The first base codebook k Taking a single processing pipeline as an example, the input encoder K 302K will be entered x Encoding as shape Feature tensor , where width and height Depending on the input width and height, and the input encoder. K 302K network architecture, It refers to the number of feature channels. In some embodiments, for different k , width and height The design is the same, therefore They have the same width and height. Various neural networks can be used as input encoders. K 302K, for example, but not limited to, the Visual Transformer (ViT), which takes an input image xThe image is divided into patches and these patches are encoded into a sequence. In another embodiment, a Convolutional Neural Network (CNN) structure is used, where the entire image is encoded in parallel.

[0047] Code word generation module K 306K is used for feature tensor-based applications. and the corresponding base codebook Calculate sparse latent features based on codebook For example, in one embodiment, the basecode include Each codeword has... Each dimension. ( Each "pixel" in ) Corresponding codeword The codeword and its corresponding hidden feature Closest: , in, It is a distance metric, such as the L1 or L2 norm. According to the disclosed embodiments, the entire... Including with The index of each codeword corresponds to A number of integers can be sent to the decoder efficiently with very low bit overhead (i.e., using a small number of bits) in a lossless manner.

[0048] At the receiving end, the received Input to feature retrieval module K 308K, based on the same basecode as the sender. 304K obtains the shape as Decoded image embedding features .For example, ( Each "pixel" in the array is assigned an index of . The code words. For example Figure 4 As shown, for the first k Processing line (i.e.) k =1…… k = K Perform a similar process to generate the corresponding feature tensor. Sparse Hidden Features Based on Codebook and decoded image embedding features .

[0049] In one embodiment, based on feature tensors The weighted predictor module 310 is used to calculate the shape as Weighted graph W For example, a weight prediction method with transformer blocks can be used. In one embodiment, the weight graph... W Used to guide how to embed features into the decoded image at the receiving end. Perform weighted merging to calculate the target merged latent features. Target merging latent features Through dense weight graph W By utilizing the rich information from all base codes in the adaptive LSIR, the target (optimal) reconstructed output image is obtained. Specifically, in one embodiment, for Each "pixel" in ( The calculation is as follows:

[0050] in, It is a weighted graph W Middle and the first l The corresponding "pixel" k Each weight. According to the disclosed embodiments, due to the dense weight graph W The data size is too large to transmit, so an adaptive weighted compression module 402 is used, based on the weighted graph. W and target compression ratio R Computing a compact weight graph that is easy to transmit An embodiment of the adaptive weighted compression module 402 is as follows: Figure 5 As stated above.

[0051] At the receiving end, the weight filling module 404 is based on a compact weight graph. and decoded image embedding features Calculate the fill weight map Fill the weight map Used to populate compact weighted graph Missing weights in the map. There are many ways to implement the weight filling module 404. For example, in some embodiments, the weights of the masked “pixels” are predicted, and the weights of unused basecodes are predicted. In some embodiments, a neural network model (such as a transformer) is used to implement the weight filling module 404. In some embodiments, a model can be optimized separately for each masking strategy. In other embodiments, different masking strategies can share the same model. This disclosure does not limit the neural network architecture or its training method. In one embodiment, the weight filling module 404 calculates a new set of weights to fill the weight map. Capable of optimally recovering / computing target merging latent features In some embodiments, when filling the weight map With compact weighted graph If they are the same, the weight padding module 404 can be skipped.

[0052] like Figure 4 As shown, the weighted merging module 312 is used to use a filled weight map. Based on the embedded features of the decoded image Calculate merged latent features In one embodiment, features are merged with the target. The calculation is similar, in order to calculate the merged latent features ,for Each "pixel" in ( The calculation is as follows:

[0053] in, It is a weighted map. Middle and the first l The corresponding "pixel" k Each weight. Then, decoder 314 is based on merging latent features. Calculate and reconstruct the output image In some embodiments, the decoder 314 may employ a network structure with multiple CNN layers, similar to the decoding network of a Variable Autoencoder (VAE). In other embodiments, the decoder 314 may be a decoder diffusion model, such as an image generation model.

[0054] In some embodiments, Figure 4 The general framework for scalable LIC based on adaptive LSLR may also include a computational criterion module 406, which is used to measure the raw input. x With reconstructed output image Merging latent features between and / or targets Merging latent features Distortion between D .like Figure 5 Furthermore, in some embodiments, distortion D Or the loss can be minimized by the adaptive weight compression module 402 to achieve the target compression quality. It should be noted that, although Figure 4 The calculation criterion module 406 is shown to be located at the receiving end, but this disclosure does not limit the location of the calculation criterion module 406 or whether the calculation criterion module 406 is a standalone module or implemented as part of other modules (e.g., within the adaptive weight compression module 402).

[0055] Figure 5This is a schematic diagram of an adaptive weight compression module 500 according to an embodiment of the present disclosure. The adaptive weight compression module 500 is... Figure 4 An example of the adaptive weight compression module 402 in the example. The adaptive weight compression module 500 includes a mask rate selection module 502 and a mask module 504. Figure 5 As shown, the mask rate selection module 502 selects the mask rate at a given target compression rate. R The following is used to calculate the mask rate. (Also known as mask ratio). Mask rate Used to indicate the proportion or extent of a mask applied based on a target compression ratio R. Masking module 504 is used to apply a mask based on a given masking strategy. Generate weight mask M An example of a masking strategy could be balancing compression ratio and reconstruction performance. Weighted masking. M It can be applied to weighted graphs W To weight the graph W Masking the items in the graph (e.g., setting them to 0) reduces the weight of the graph. W The size. For example, such as Figure 5 As shown, the mask module 504 applies a weight mask. M Applied to weighted graphs W (Depend on Figure 4 The weight predictor module 310 calculates the weight map (using a masked version of the generated weight map as the compact weight map). For example, suppose the first k A basic codebook The index can be derived from Bit integer representation (e.g., if Including 256 or 1024 codewords, Or 10). If using the full weighted graph W To calculate target merging latent features The maximum number of bits that need to be transmitted is: , in, This represents a weighted graph. W The bits of each floating-point weight (e.g., for single-precision or double-precision floating-point), =16 or 32). Weight mask M Typically a binary mask, along with a weighted graph. W Having the same shape In one embodiment, the weights in W that are masked by mask M are set to zero. That is, the resulting compact weighted graph... It is a sparse weighted graph, which is larger than the original weighted graph. W Much lighter. Calculate the mask. MThere are many ways to do this. In one embodiment, for each "pixel"... ( The pixel is either completely masked, where the mask is... M Corresponding item in Set to zero; the pixel either uses J The codewords in the base codebook are calculated, where... For example, if a pixel is not masked, then The following can be calculated: , if If not masked, then Indicates used for The index of the base code, when hour, ;otherwise, .

[0056] If there are no fully masked "pixels", then the number of bits that need to be transmitted is: when hour,

[0057] when hour,

[0058] in, An index used to indicate the number of base codes that have been used; for example, if there are 8 or 16 base codes, then... Or 4.

[0059] When there are "pixels" that are completely masked, the corresponding entries will be removed from the number of bits to be transmitted, as shown in the following example: when hour,

[0060] when hour,

[0061] in, Or 1, indicating that the "pixel" is either completely masked or not masked.

[0062] As mentioned above, at a given target compression ratio R Next, the mask rate selection module 502 calculates the mask rate. In one embodiment, the mask rate It can be calculated as the ratio of the fully masked "pixels" to all pixels: /

[0063] For a given target compression ratioR It can calculate the corresponding target number of bits. .For example, Then, the corresponding masking strategy (i.e., the number of base codes per "pixel") can be determined. J (and the ratio of the "pixels" that are completely masked). For example, you can use a method that produces the closest compression ratio to the target. R The compression ratio masking strategy. Alternatively, a masking strategy that produces a compression ratio close to the target can be used. R A set of masking strategies for the compression ratio is determined as candidate strategies, and the optimal strategy that produces a balance between compression ratio and reconstruction performance can be used.

[0064] As described above, under a given masking strategy, the masking module 504 calculates the mask. M and compact weighted graph In one embodiment, for each "pixel" ( The base code with the maximum value corresponding to the weight of the "pixel" in the weight map W is selected for use. In some embodiments, uniform "pixel" sampling is performed according to the mask ratio. The pixels are masked. In other embodiments, a predictive neural network model can be used based on the mask ratio. and weighted graph W Predict the positions of the masked "pixels" and calculate the weights of the unmasked "pixels" to obtain a compact weight map. It can be restored better Figure 4 The target merging hidden features In some embodiments, the predictive neural network model can be trained by optimizing a separate model for each masking strategy, or by using the same model for all different masking strategies. This disclosure does not limit the neural network architecture or its training method.

[0065] In some embodiments, weight mask M and compact weighted graph Send directly to the receiving end. For example... Figure 4 At the receiving end, the weight filling module 404 is used to fill in the weights based on the compact weight graph. and decoded image embedding features Calculate the fill weight map The weighted merging module 312 then uses it to calculate the merged latent features. Then, decoder 314 is based on merging latent features. Calculate and reconstruct the output image .

[0066] In addition, such as Figure 4 and Figure 5As shown, in some embodiments, distortion loss D The original input can be calculated in the calculation criteria module 406. x With reconstructed output image Merging latent features between and / or targets Merging latent features Distortions, such as mean square error (MSE), can occur between components. In some embodiments, this loss can be minimized to achieve the target compression quality. There are several ways to minimize this loss. For example, in some embodiments, a mask is used... M Under the masking strategy, compact weight graph The weights corresponding to the unmasked "pixels" can be updated by backpropagating the gradient of the loss to these weights, while maintaining the masking policy (i.e., the masking method). M The number of codebook elements for each unmasked "pixel" remains unchanged. Alternatively, one approach is to use a mask that has already been calculated. M and compact weighted graph To calculate the loss of various candidate masking strategies without updating via backpropagation. And simply select the optimal strategy that minimizes the loss. Then, update the weight mask. M and the updated compact weighted graph The data is sent to the receiving end. This disclosure is not limited to calculating the loss or using the loss to update the mask in the calculation criterion module 406. M and compact weighted graph In this way.

[0067] Figure 6 This is a flowchart of a processing workflow 600 according to an embodiment of the present disclosure. Processing workflow 600 is an example of a method performed by a sender or sending device according to an embodiment of the present disclosure. In step 602, the sending device... K Input encoders (such as) Figure 4 Input encoder 1 302A... Input encoder K 302K) for the input image x Encode to generate K Image feature tensor ,in, K It is an integer greater than 1. In step 604, the transmitting device, through one or more processors of the transmitting device, based on... K Image feature tensor and the corresponding base codebook Calculate separately K A codebook-based sparse hidden feature (For example, by executing codeword generation module 1 306A... codeword generation module) K 306K related instructions and corresponding base codebook ,like Figure 4 Then, in step 606, the transmitting device calculates (e.g., by executing a process) using one or more processors of the transmitting device. Figure 4 (Instructions associated with the weight predictor module 310 in the weight graph) W Among them, the weighted graph W Provide for K Decoded image embedding features Information for weighted merging K Decoded image embedding features Based on respectively K A codebook-based sparse hidden feature and the corresponding base codebook In step 608, the transmitting device, through one or more processors of the transmitting device, based on the weight graph... W and target compression ratio R , calculation (e.g., by performing with Figure 4 (Instructions associated with the adaptive weight compression module 402 in the compact weight graph) In step 610, the transmitting device (e.g., via a transmitter or network communication device or module of the transmitting device) compacts the weighted graph. and K A codebook-based sparse hidden feature Send to the receiving device. Further details of the specific steps involved in processing workflow 600 have been described above in this disclosure.

[0068] Figure 7 This is a flowchart of a processing workflow 700 according to an embodiment of the present disclosure. Processing workflow 700 is an example of a method performed by a receiver or receiving device according to an embodiment of the present disclosure. In step 702, the receiving device (e.g., via a transceiver or network communication device or module of the receiving device) receives data from the sender or sending device. K A codebook-based sparse hidden feature and compact weighted graph ,in, K It is an integer greater than 1. As mentioned above, K A codebook-based sparse hidden feature It is based on the sender K Feature tensors and K A corresponding base codebook is generated. In step 704, the receiving device, through one or more processors of the receiving device, based on... KA codebook-based sparse hidden feature Use the corresponding K The base code was originally calculated K Decoded image embedding features (For example, by executing the feature retrieval module 1 308A... feature retrieval module) K 308K related instructions and corresponding base code ,like Figure 4 (as described above). In step 706, the receiving device, through one or more processors of the receiving device, based on a compact weight graph... and K Decoded image embedding features Calculate the fill weight map (For example, by executing with) Figure 4 (Instructions associated with the weight filling module 404 in the document). Then, in step 708, the receiving device, through one or more processors of the receiving device, fills the weight map based on the instructions. and K Decoded image embedding features Calculate merged latent features (For example, by executing with) Figure 4 (Instructions associated with the weighted merging module 312 in the document). In step 710, the receiving device, through one or more processors of the receiving device, performs a merging based on the implicit feature. Calculate and reconstruct the output image (For example, by executing with) Figure 4 (Instructions associated with decoder 314 in the document). Further details of the specific steps of the processing workflow 700 have been described above in this disclosure.

[0069] Figure 8 This is a schematic diagram of an apparatus 800 according to an embodiment of the present disclosure. The apparatus 800 can be used to implement embodiments of the present disclosure. For example, the apparatus 800 can be used to perform the functions of a transmitter or transmitting device and a receiver or receiving device according to any embodiment of the present disclosure. The apparatus 800 includes a receiving unit (RX) 820 or receiving component for receiving data through an input port 810. The apparatus 800 also includes a transmitting unit (TX) 840 or transmitting component for transmitting data through an output port 850. For example, at the transmitting end, the transmitting device can obtain the original image through the RX 820 or receiving component, and then compress the weighted image through the TX 840 or transmitting component. and K A codebook-based sparse hidden feature Send to, for example Figure 4 The aforementioned receiver. At the receiver, the decoder can receive the compact weighted map via an RX 820 or a receiving component. andK A codebook-based sparse hidden feature Then, via TX 840 or a sending component, the decoded image of the original image (such as the reconstructed output image) is sent. It is sent to a display device or other computing device.

[0070] The device 800 includes a memory 860 or data storage component for storing instructions and various types of data. The memory 860 can be any type or combination of storage components capable of storing data and / or instructions. For example, the memory 860 may include volatile and / or non-volatile memories such as read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM). The memory 860 may also include one or more disk drives, tape drives, and solid-state drives. In some embodiments, the memory 860 may be used as an overflow data storage device to store such a program when a program is selected for execution, and to store instructions and data read during program execution. In some embodiments, the memory 860 may be a memory integrated with the processor 830.

[0071] The apparatus 800 includes one or more processors 830 or other processing components (such as a central processing unit, CPU) to process instructions. The one or more processors 830 may be implemented as one or more CPU chips, cores (such as multi-core processors), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and digital signal processors (DSPs). The one or more processors 830 are communicatively coupled to ingress port 810, RX 820, TX 840, egress port 850, and memory 860 via a system bus. The one or more processors 830 may be used to execute instructions stored in memory 860. For example, in one embodiment, memory 860 stores an adaptive LSLR-based scalable LIC module 870. The adaptive LSLR-based scalable LIC module 870 includes data, executable instructions, and / or one or more sub-modules for implementing the disclosed embodiments. Therefore, the one or more processors 830 provide a component for performing any calculation, comparison, determination, initiation, configuration, or any other operation corresponding to the claims when appropriate instructions are executed by the processor 830. Therefore, including the scalable LIC module 870 based on adaptive LSLR substantially improves the functionality of the device 800.

[0072] While this disclosure provides several embodiments, it should be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of this disclosure. The examples in this disclosure should be considered illustrative rather than restrictive, and this disclosure is not limited to the details set forth herein. For example, various elements or components may be combined or integrated into other systems, or certain features may be omitted or not implemented.

[0073] Furthermore, without departing from the scope of this disclosure, the technologies, systems, subsystems, and methods described and illustrated as discrete or separate in the various embodiments may be combined or integrated with other systems, modules, technologies, or methods. Other items shown or described as coupled, directly coupled, or communicating with each other may be indirectly coupled or communicated electrically, mechanically, or otherwise through some interface, device, or intermediate component. Those skilled in the art can identify other examples of changes, substitutions, and modifications, and may make such changes, substitutions, and modifications without departing from the spirit and scope of this disclosure.

Claims

1. A method implemented by a transmitting device, characterized in that, The method includes: The transmitting device K An input encoder for the input image x Encode to generate K Image feature tensor ,in, K It is an integer greater than 1; Based on the above K Image feature tensor and the corresponding base codebook Calculate separately K A codebook-based sparse hidden feature ; Calculate the weighted graph W The weight graph W Provide for K Decoded image embedding features The information for weighted merging, the K Decoded image embedding features Based on the above K A codebook-based sparse hidden feature and the corresponding basecode ; Based on the weight graph W and target compression ratio R Calculate the compact weighted graph ; The compact weight map and stated K A codebook-based sparse hidden feature Send to the receiving device.

2. The method according to claim 1, characterized in that, Based on the weight graph W and the target compression ratio R Calculate the compact weight graph include: Based on the target compression ratio R Calculate the mask rate ; Based on the mask rate Generate a weight mask M ; Based on the weight graph W and the weight mask M Calculate the compact weight graph .

3. The method according to claim 2, characterized in that, The mask rate τ It is the ratio of the masked weight to the total weight.

4. The method according to claim 2 or 3, characterized in that, Also includes: Based on the mask rate And a masking strategy, to generate the weight mask. M .

5. The method according to any one of claims 2 to 4, characterized in that, Also includes: Based on the target compression ratio R Calculate the target number of bits ; Based on the target number of bits The masking strategy is determined.

6. The method according to any one of claims 2 to 5, characterized in that, Also includes: The weight mask M It is sent to the receiving device.

7. The method according to any one of claims 1 to 6, characterized in that, Also includes: Calculate and measure the input image x With reconstructed output image Merging latent features between and / or targets Merging latent features Distortion loss between D ; Modify the compact weight graph The weights corresponding to the unmasked pixels are assigned to reduce the distortion loss. D .

8. The method according to any one of claims 1 to 7, characterized in that, Also includes: Based on multiple candidate masking strategies, the measurement of the input image is calculated. x With reconstructed output image Merging latent features between and / or targets Merging latent features Distortion loss between D ; Select the masking strategy with the minimum distortion loss from the plurality of candidate masking strategies. D The masking strategy.

9. A method implemented by a receiving device, characterized in that, The method includes: Receive from transmitting device K A codebook-based sparse hidden feature and compact weighted graph ,in, K It is an integer greater than 1; Based on the above K A codebook-based sparse hidden feature ,calculate K Decoded image embedding features ; Based on the compact weight graph and stated K Decoded image embedding features Calculate the fill weight map ; Based on the filling weight map and stated K Decoded image embedding features Calculate merged latent features ; Based on the merging hidden features Calculate and reconstruct the output image .

10. The method according to claim 9, characterized in that, Based on the compact weight graph and stated K Decoded image embedding features Calculate the fill weight map Includes: for the compact weight graph The masked weights and the predicted weights.

11. The method according to claim 9 or 10, characterized in that, Based on the compact weight graph and stated K Decoded image embedding features Calculate the fill weight map Includes: in calculating the K Decoded image embedding features At that time, predict weights for unused basecodes.

12. The method according to claim 9 or 10, characterized in that, Based on the compact weight graph and stated K Decoded image embedding features Calculate the fill weight map Includes: calculating the latent features used to recover the target merged feature A new set of weights.

13. The method according to any one of claims 9 to 12, characterized in that, Also includes: Using a neural network model based on the compact weight graph and stated K Decoded image embedding features Calculate the fill weight map .

14. The method according to claim 13, characterized in that, The neural network model is designed for the compact weight graph. The corresponding masking strategy is optimized.

15. The method according to any one of claims 9 to 14, characterized in that, Based on the filling weight map and stated K Decoded image embedding features Calculate the merged hidden features Including: targeting Each pixel in the image is calculated as follows: ( ): in, It is the filled weight map Middle and the first l The corresponding pixel k Each weight.

16. An apparatus, characterized in that, include: A memory or storage component used to store instructions; One or more processors or processing components are coupled to the memory or storage component and are used to execute the instructions to cause the apparatus to perform the method according to any one of claims 1 to 8.

17. An apparatus, characterized in that, include: A memory or storage component used to store instructions; One or more processors or processing components are coupled to the memory or storage component and are used to execute the instructions to cause the apparatus to perform the method according to any one of claims 9 to 15.

18. A computer program product, characterized in that, Includes computer-executable instructions stored in a non-transitory computer-readable storage medium, wherein, when executed by one or more processors of the device, the computer-executable instructions cause the device to perform the method according to any one of claims 1 to 15.

Images