A multimedia data compression and decompression method based on semantic adaptive quantization
Patent Information
- Application Number
- CN202611050809.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-15
- Publication Date
- 2026-10-27
AI Technical Summary
[0008]针对现有技术中存在的语义描述性差、量化环节精度无法自适应分配以及重建质量堪忧的问题，本发明提出了一种基于语义自适应量化的多媒体数据压缩与解压缩方法，通过差异化矢量量化策略，在有限码率下实现高保真编码，并提高重构阶段的正确率，降低了模糊成块的概率
[0040]（1）本发明提出“语义可描述性”度量并用于指导压缩，有限码率能服务于对人类感知最重要的区域，在极低码率下优势明显；
Smart Images
Figure CN122994491A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of multimedia data compression technology, specifically involving encoding and decoding technologies for data such as images, videos, point clouds, and audio, and particularly relates to a multimedia data compression and decompression method based on semantic adaptive quantization. Background Technology
[0002] Traditional multimedia compression standards, such as JPEG for images and H.264 / HEVC for video, are primarily based on signal processing theory, utilizing predictive transforms (such as DCT and wavelet transform) and statistical coding to eliminate spatial and temporal redundancy in the data. However, these methods typically treat image or video frames as ordinary signal matrices, lacking an understanding of the semantic content of the data. They employ the same or similar compression strategies for all regions, resulting in severe quality loss in regions with important semantic information (such as faces and text) and complex background regions simultaneously at low bit rates (high compression ratios), failing to achieve optimal allocation of visual perception quality with limited bit resources.
[0003] In recent years, deep learning-based data compression methods have made significant progress. For example, compression models based on convolutional neural networks (CNNs) and variational autoencoders (VAEs) have achieved performance close to or surpassing that of some traditional encoders. Furthermore, vector quantization (VQ) techniques, due to their ability to effectively discretize feature spaces, are also widely used in these deep learning models. However, most existing deep compression methods still have limitations: they tend to focus on global rate-distortion optimization, failing to fully explore and utilize the inherent differences in compression requirements among different semantic regions within media data. For example, an object that can be clearly described in language (such as "a sitting cat") and a complex texture that is difficult to describe precisely in language (such as "sparkling water") are often encoded the same way in current methods.
[0004] More specifically, the existing technology has the following gaps or deficiencies:
[0005] (1) Lack of semantically guided adaptability: Existing compression technologies fail to dynamically adjust compression strategies based on the "semantic describability" of regions. For semantic regions that can be accurately summarized by text, their describability characteristics are not fully utilized, and they are represented with extremely low bitrates (such as generating text labels), thereby saving more bits to protect other visually detailed regions.
[0006] (2) Single quantization strategy: Most methods, whether traditional or deep learning-based, usually use fixed precision or a limited number of predefined quantization tables in the quantization process, which cannot achieve fine-grained, semantic content-based adaptive precision allocation.
[0007] (3) Poor balance between compression efficiency and reconstruction quality: Due to the above reasons, existing methods cannot achieve optimal semantic fidelity and visual quality at a given target bit rate. Either important semantic regions become blurred due to compression, or compression efficiency is sacrificed to ensure overall quality. Summary of the Invention
[0008] To address the problems of poor semantic descriptiveness, inability to adaptively allocate precision in the quantization stage, and poor reconstruction quality in existing technologies, this invention proposes a multimedia data compression and decompression method based on semantic adaptive quantization. By employing a differentiated vector quantization strategy, high-fidelity encoding is achieved at a limited bit rate, while improving the accuracy of the reconstruction stage and reducing the probability of blurred blocks.
[0009] A multimedia data compression and decompression method based on semantic adaptive quantization includes an encoding process at the sending end and a decoding process at the receiving end:
[0010] The encoding process includes the following steps:
[0011] S1: Obtain the source media data X to be compressed, perform semantic analysis on X, and divide it into N semantic regions. For each semantic region Calculate its semantic descriptibility metric. ;
[0012] The source media data can be images, video sequences, point clouds, or audio signals, etc.
[0013] The semantic analysis is implemented through a pre-trained semantic analysis network adapted to the modality of the source data. For images and video frames, visual semantic segmentation networks based on architectures such as U-Net can be used; for point clouds, 3D point cloud semantic segmentation networks based on architectures such as PointNet++ can be used; for audio, semantic analysis networks based on time-frequency graph convolution or audio event detection can be used. All networks include an encoder-decoder structure and use pre-trained models of the corresponding modality as the feature extraction backbone. The semantic analysis network outputs a semantic label for each pixel, and divides the image into continuous regions based on the labels. Each corresponds to a semantic object.
[0014] The semantic descriptibility metric is calculated in the following manner:
[0015] (1) For semantic regions The feature vector is obtained through its corresponding cross-modal encoder E. ;
[0016] (2) Calculation With each prototype concept vector in the text feature space Cosine similarity:
[0017]
[0018] The prototype concept vector is a predefined set of common semantic concept vectors extracted from descriptive text by a text encoder in the text feature space.
[0019] (3) Take the maximum value among all cosine similarities as the semantic descriptibility measure of the region:
[0020]
[0021] S2: Based on the measurement value With preset threshold The comparison categorizes the semantic regions: if Then determine Category I region This area can be efficiently described using text tokens; if Then determine Category II area This area is difficult to describe precisely using text tokens;
[0022] S3: For the first type of region Perform low-precision quantization encoding to generate its corresponding text token and first quantization data; for the second type of region Perform high-precision quantization encoding to generate second quantized data;
[0023] Both the low-precision quantization encoding and the high-precision quantization encoding adopt a codebook-based vector quantization method. The specific process is as follows: for the input feature vector F, find the codeword vector with the closest Euclidean distance in the codebook C. :
[0024]
[0025] The index j of the codeword is then output as the quantization result; where low-precision quantization encoding uses a smaller codebook, the codebook size is [size missing]. ,in High-precision quantization encoding uses a larger codebook, with a codebook size of [size missing]. ,in ,and .
[0026] S4: Multiplex the text token, the first quantization data, and the second quantization data to generate a compressed bitstream B;
[0027] The multiplexing method uses sequential storage: first, the number of regions N is stored, and then for each region, its type flag (1 bit), text token (variable length), or quantization index (fixed length) is stored. The bitstream structure can be further compressed using Huffman coding.
[0028] The decoding process includes the following steps:
[0029] S5: The receiving end receives the compressed bitstream B and reverse-parses it according to the multiplexing rules to separate the text token, the first quantized data and the second quantized data;
[0030] S6: For the first quantized data, combined with its corresponding text token, it is reconstructed by the first decoder to obtain the reconstructed data of the first type of region;
[0031] The process of data reconstruction by the first decoder includes:
[0032] (1) Codebook decoding: using a low-precision codebook Retrieve codeword based on index j , to obtain the quantized feature vector .
[0033] (2) Text feature fusion: Input the text token into the CLIP text encoder to obtain the text feature vector t.
[0034] (3) Generative Network: The data is concatenated with t and fed into a pre-trained generative adversarial network (GAN) to generate reconstructed region data.
[0035] S7: For the second quantized data, reconstruct it through the second decoder to obtain the reconstructed data of the second type of region;
[0036] The second decoder uses a high-precision codebook. Retrieve codeword based on index j , to obtain the quantized feature vector Then, the region data is reconstructed through an upsampling network (based on transposed convolution).
[0037] S8: Merge the reconstructed data of the first type of region with the reconstructed data of the second type of region to reconstruct the complete media data X'.
[0038] The fusion method is based on the inverse mapping of the original region division: for an image, the reconstructed region is pasted into the corresponding position and the boundary is smoothed using bilinear interpolation.
[0039] The advantages and beneficial effects of this invention are as follows:
[0040] (1) This invention proposes a "semantic describability" metric and uses it to guide compression. A limited bit rate can serve the region most important to human perception, and has obvious advantages at extremely low bit rates.
[0041] (2) This invention proposes a differentiated vector quantization strategy based on region category. It uses low bit rate encoding for the semantically clear first type of region and redistributes the large amount of saved bit rate space to the textured second type of region, enabling it to be encoded with high fidelity.
[0042] (3) This invention combines dual-path decoding and reconstruction of text tokens and quantized data. The text tokens provide powerful high-level semantic priors to guide the generative adversarial network (GAN) in generating structures and textures that conform to its category characteristics. Even with low quantization accuracy, semantically correct and visually natural content can be reconstructed, avoiding the blurring or block effects that may occur in such areas in traditional compression. Attached Figure Description
[0043] Figure 1 This is a diagram illustrating the overall architecture of a multimedia data compression and decompression method based on semantic adaptive quantization according to the present invention. Detailed Implementation
[0044] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.
[0045] This invention provides a multimedia data compression and decompression method based on semantic adaptive quantization. This method divides media data into different semantic regions through semantic analysis and adaptively selects the quantization precision based on the semantic describability of each region, thereby achieving efficient compression while ensuring reconstruction quality.
[0046] The method of this invention employs deep learning-based semantic segmentation and visual language modeling techniques. Semantic features are extracted through a pre-trained neural network, and a semantic descriptibility metric is calculated. Subsequently, regions are classified based on the metric, and vector quantization encoding with different precisions is applied. During decoding, the regions are reconstructed by combining text tokens and quantized data, ultimately fusing them to obtain complete data. This method is applicable to multimedia data such as images, videos, point clouds, and audio, and can reduce data volume in low-bandwidth environments.
[0047] For video, processing can be performed frame by frame or in spatiotemporal blocks; for point clouds, the 3D point set can be directly used as the processing unit; for audio, it can be first converted into a time-frequency representation such as a Mel spectrogram before region division.
[0048] Perform semantic analysis on X, dividing it into N semantic regions. Select the appropriate semantic analysis network based on the data modality:
[0049] Image / video frames: The U-Net architecture is adopted, the encoder uses ResNet-50, and the output is pixel-level semantic labels. Regions are formed through connected component analysis.
[0050] Point cloud: Using networks such as PointNet++, semantic labels (such as "table" and "chair") are assigned to each point, and spatial clustering is used to divide the set of adjacent points with the same label into three-dimensional semantic regions.
[0051] Audio: After converting the audio signal into a time-frequency graph, a semantic segmentation network similar to an image is used to identify and divide the time-frequency regions corresponding to different sound events (such as "speech", "music", "noise").
[0052] A multimedia data compression and decompression method based on semantic adaptive quantization, such as... Figure 1 As shown, the steps are as follows:
[0053] Step 1: Semantic Analysis and Region Partitioning
[0054] The source media data X to be compressed is obtained. In this embodiment, image data is used as an example (resolution is set to 512×512). First, semantic analysis is performed on image X, dividing it into N semantic regions. Semantic analysis is achieved through a pre-trained semantic analysis network adapted to the modality of the source data. For image data, the network employs a U-Net architecture, with the encoder using ResNet-50 as the backbone for feature extraction. The semantic analysis network outputs a semantic label for each pixel (e.g., "person," "sky," "road"), dividing the image into contiguous regions based on these labels. Each region... For each semantic object, region division is completed through connectivity component analysis.
[0055] Step 2: Calculate semantic descriptibility metrics and classify regions.
[0056] For each semantic region Calculate its semantic descriptibility metric. This value indicates whether a region can be effectively described using a text token. The calculation process is as follows:
[0057] (1) The region Cropped and resized to 224×224, the image is input into the pre-trained CLIP visual encoder E to obtain visual feature vectors. . It is a 512-dimensional vector.
[0058] (2) In the text feature space, a set of prototype concept vectors are predefined. These vectors correspond to common semantic concepts (such as "person", "animal", "building") and are extracted from descriptive text using the CLIP text encoder. In this embodiment, M=1000, covering ImageNet categories.
[0059] (3) Calculation With each Cosine similarity:
[0060]
[0061] in, This represents the L2 norm.
[0062] (4) Take the maximum value among all similarities as :
[0063] (5) Preset threshold Used for classifying regions. In this embodiment, Set to 0.7. If Then determine Category I region This indicates that the area can be efficiently described using a text token; if If so, it is classified as a second-class region. This indicates that the area is difficult to describe precisely using a text token.
[0064] Step 3: Adaptive Quantization Coding
[0065] For the first type of area Second Category Area Low-precision and high-precision quantization encoding are performed separately. Quantization encoding uses codebook-based vector quantization (VQ). First, feature vectors F (256 dimensions) for each region are extracted from the feature map of the semantic segmentation network.
[0066] (1) Low-precision quantization coding (used for) ):
[0067] Use a smaller codebook codebook size The codebook is obtained by clustering the training data using the k-means algorithm. The quantization process is as follows: for a feature vector F, in the codebook... Find the codeword vector with the closest Euclidean distance. : The codeword index j (occupying 7 bits) is output as the first quantization data. Simultaneously, a corresponding text token is generated, which is a region. Semantic labels (such as "person") are obtained by finding the text with the highest similarity in the prototype concept vector.
[0068] (2) High-precision quantization encoding (used for) ):
[0069] Use a larger codebook codebook size The quantification process is similar: Output codeword index j (occupying 12 bits) as the second quantization data.
[0070] Step 4: Data multiplexing to generate compressed bitstream
[0071] The text token (encoded in UTF-8), the first quantization data (low-precision quantization index), and the second quantization data (high-precision quantization index) are multiplexed to generate a compressed bitstream B. The multiplexing method uses sequential storage: first, the number of regions N is stored; then, for each region, its type flag (1 bit), text token (variable length), or quantization index (fixed length) is stored. The bitstream structure can be further compressed using Huffman coding.
[0072] Step 5: Parse the compressed bitstream
[0073] At the receiving end, the compressed bitstream B is received and parsed to separate the text token, the first quantized data, and the second quantized data. The parsing process is performed in reverse according to the multiplexing rules.
[0074] Step 6: Reconstruction of the first type of region
[0075] For the first quantized data, combined with the corresponding text token, it is reconstructed using the first decoder. The first decoder includes:
[0076] (1) Codebook decoding: using a low-precision codebook Retrieve codeword based on index j , to obtain the quantized feature vector .
[0077] (2) Text feature fusion: Input the text token into the CLIP text encoder to obtain the text feature vector t.
[0078] (3) Generative Network: The input is concatenated with t and fed into a pre-trained Generative Adversarial Network (GAN) to generate reconstructed region data. The GAN architecture is based on U-Net and uses L1 loss and adversarial loss during training.
[0079] Step 7: Reconstruction of the second type of region
[0080] The second quantized data is reconstructed using a second decoder. The second decoder uses a high-precision codebook. Retrieve codeword based on index j , to obtain the quantized feature vector Then, the region data is reconstructed through an upsampling network (based on transposed convolution). The upsampling network consists of 3 convolutional layers and 2 upsampling layers, outputting the region pixel values.
[0081] Step 8: Data Fusion
[0082] The reconstructed data from the first and second types of regions are fused to reconstruct the complete media data X'. The fusion method is based on the inverse mapping of the original region division: for the image, the reconstructed regions are pasted into the corresponding positions, and bilinear interpolation is used to smooth the boundaries.
[0083] Step 9: Compare the reconstructed data with the original data and calculate the loss value. Use artificial intelligence methods to reason and optimize the relevant parameters in the encoding and decoding processes to achieve high-fidelity encoding and accurate data reconstruction.
Claims
1. A multimedia data compression and decompression method based on semantic adaptive quantization, comprising an encoding process at the sending end and a decoding process at the receiving end, characterized in that, The implementation processes are as follows: The encoding process is as follows: S1: Obtain the source media data X to be compressed, perform semantic analysis on X, and divide it into N semantic regions. For each semantic region Calculate its semantic descriptibility metric. ; The semantic descriptibility metric is calculated in the following manner: (1) For semantic regions The feature vector is obtained through its corresponding cross-modal encoder E. ; (2) Calculation With each prototype concept vector in the text feature space Cosine similarity: (3) Take the maximum value among all cosine similarities as the semantic descriptibility measure of the region: S2: Based on the measurement value With preset threshold The comparison categorizes the semantic regions: like Then determine Category 1 area This area can be precisely described using text tokens; if Then determine Category II area This area is difficult to describe precisely using text tokens; S3: Quantize and encode the two types of regions separately to obtain the corresponding quantized data; For the first type of region Perform low-precision quantization encoding to generate its corresponding text token and first quantization data; for the second type of region Perform high-precision quantization encoding to generate second quantized data; S4: Multiplex the text token, the first quantization data, and the second quantization data to generate a compressed bitstream B; The decoding process is specifically as follows: S5: Receive the compressed bitstream B and reverse-parse it according to the multiplexing rules to separate the text token, the first quantized data and the second quantized data; S6: For the first quantized data, combined with its corresponding text token, it is reconstructed by the first decoder to obtain the reconstructed data of the first type of region; S7: For the second quantized data, reconstruct it through the second decoder to obtain the reconstructed data of the second type of region; S8: Merge the reconstructed data of the first type of region with the reconstructed data of the second type of region to reconstruct the complete media data X'.
2. The multimedia data compression and decompression method based on semantic adaptive quantization according to claim 1, characterized in that, The semantic analysis is achieved through a pre-trained semantic analysis network adapted to the modality of the source media data. Each of the networks contains an encoder-decoder structure and uses a pre-trained model of the corresponding modality as the feature extraction backbone. The semantic analysis network outputs a semantic label for each pixel, and divides the image into continuous regions based on these labels. Each corresponds to a semantic object.
3. The multimedia data compression and decompression method based on semantic adaptive quantization according to claim 2, characterized in that, The source media data includes images, video sequences, point clouds, or audio signal data. For images and video frames, a visual semantic analysis network based on the U-Net architecture is used; for point clouds, a three-dimensional point cloud semantic analysis network based on the PointNet++ architecture is used; and for audio, a semantic analysis network based on time-frequency graph convolution or audio event detection is used.
4. The multimedia data compression and decompression method based on semantic adaptive quantization according to claim 1, characterized in that, The prototype concept vector is a predefined set of common semantic concept vectors extracted from descriptive text by a text encoder in the text feature space.
5. The multimedia data compression and decompression method based on semantic adaptive quantization according to claim 1, characterized in that, Both the low-precision quantization encoding and the high-precision quantization encoding adopt a codebook-based vector quantization method. The specific process is as follows: for the input feature vector F, find the codeword vector in the codebook C that has the closest Euclidean distance to it. : The index j of the codeword is then output as the quantization result.
6. The multimedia data compression and decompression method based on semantic adaptive quantization according to claim 5, characterized in that, The codebook size used for low-precision quantization encoding is ,in The codebook size used for high-precision quantization encoding is [size missing]. ,in ,and .
7. The multimedia data compression and decompression method based on semantic adaptive quantization according to claim 1, characterized in that, The reuse method employs sequential storage: first, the number of regions N is stored; then, for each region, its type flag, text token, or quantization index is stored; and the bitstream structure is further compressed using Huffman coding.
8. The multimedia data compression and decompression method based on semantic adaptive quantization according to claim 1, characterized in that, The process of data reconstruction by the first decoder includes: (1) Codebook decoding: using a low-precision codebook Retrieve codeword based on index j , to obtain the quantized feature vector ; (2) Text feature fusion: Input the text token into the CLIP text encoder to obtain the text feature vector t; (3) Generative Network: The data is concatenated with t, and a pre-trained generative adversarial network is input to generate reconstructed region data.
9. A multimedia data compression and decompression method based on semantic adaptive quantization according to claim 1, characterized in that, The second decoder uses a high-precision codebook. Retrieve codeword based on index j , to obtain the quantized feature vector Then, the regional data is reconstructed through an upsampling network.