Calculation scalable image compression system based on Bayesian network structure learning
By constructing a computationally scalable image compression system based on Bayesian networks, the problem of high computational complexity in neural image compression technology is solved, achieving flexible allocation of computing resources and excellent compression performance on different devices.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2026-03-10
AI Technical Summary
Existing neural image compression (NIC) technologies are limited by high computational complexity, low computational efficiency, and lack of computational scalability, making them difficult to apply effectively in practical devices.
A computationally scalable image compression system based on Bayesian network structure learning is adopted. By constructing a heterogeneous bipartite Bayesian backbone network and a multipartite Bayesian context model, the computational complexity of the neural network is dynamically adjusted, thereby achieving flexible allocation of computational resources between the backbone neural network and the autoregressive model.
It achieves more precise allocation of computing resources on different devices, maintains better image compression performance, and saves bitrate and improves decoding speed under the same conditions compared with existing technologies.
Smart Images
Figure CN121639833A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application relates to the technical field of image compression, and particularly relates to a computing scalable image compression system based on Bayesian network structure learning. BACKGROUND
[0002] In recent years, with the development of neural network technology, the neural image compression (NIC) technology gradually starts to be superior to traditional image codecs in rate-distortion performance, however, the problem of computing complexity restricts its application in actual deployment, mainly including three problems of low computing efficiency of autoregressive model, high computing complexity of neural network and lack of computing scalability. SUMMARY
[0003] The application proposes a computing scalable image compression system based on Bayesian network structure learning to solve the limitation of the existing NIC framework in computing complexity, the computing complexity of the NIC framework is flexibly adjusted by learning the Bayesian network structure, so that the computing budget of different devices can be adapted, and the computing complexity of the backbone neural network and the autoregressive model in the NIC framework is respectively adjusted by decomposing the Bayesian network learning problem into two sub-problems of heterogeneous bipartite graph optimization and multi-partite graph optimization, so that the image compression performance can be ensured while the computing complexity is adjusted more finely.
[0004] The application is implemented by the following technical scheme:
[0005] The application relates to a computing scalable image compression system based on Bayesian network structure learning, which comprises an image compression model based on a heterogeneous bipartite graph Bayesian backbone network and a multi-partite graph Bayesian context model, wherein the heterogeneous bipartite graph Bayesian backbone network utilizes a variable-width network to dynamically adjust the channel width of the network, adjusts the conditional dependency relationship between the nodes of the Bayesian network, and controls the computing complexity of the backbone neural network; the multi-partite graph Bayesian context model adjusts the conditional dependency relationship inside the nodes of the Bayesian network, and controls the computing complexity of the context model through a dynamic mask convolution operation.
[0006] The image compression model comprises: a first-level encoding model, a second-level encoding model, a second-level decoding model, a context model based on dynamic mask convolution, and a first-level decoding model, all connected in sequence. The first-level encoding model, second-level encoding model, first-level encoding model, and second-level encoding model constitute a heterogeneous bipartite graph Bayesian backbone network for predicting conditional probabilities between Bayesian network nodes. Entropy encoding and decoding modules are provided between the context model and the first-level decoding model, and between the second-level encoding model and the second-level decoding model. Quantization modules are provided between the first-level encoding model and the context model, and between the second-level encoding model and the entropy encoding and decoding modules. During the encoding stage, the first-level encoding model determines the encoding based on the input image x and computational complexity control parameters. The image prior data y is calculated; y is then processed by the quantization module Q to obtain the quantized prior data. The two-level coding model controls parameters based on the input y and computational complexity. The image prior data z is calculated; z is then processed by the quantization module Q to obtain the quantized prior data. The second-level prior is used for entropy coding to obtain the second-level bitstream; the second-level decoding model is based on the input quantized super-prior data. and computational complexity control parameters The calculation is performed, and its output is compared with the quantified prior data. After merging, the input context model is combined with computational complexity control parameters. Calculate the first-level prior; quantify the super-prior data The first-level prior, combined with the output of the context model, is used for entropy coding to obtain the first-level bitstream. In the decoding stage, the second-level bitstream, combined with the second-level prior, is used for entropy decoding to obtain quantized super-prior data. The two-level decoding model is based on the input quantized prior data. and computational complexity control parameters The calculation is performed, and its output is compared with the quantized prior data to be decoded. After merging, the input context model is combined with computational complexity control parameters. The first-level prior is calculated and used for entropy decoding of the first-level bitstream to update the quantized prior data. This step, after several iterations, yields complete quantified prior data. The final decoding model is based on the input quantized prior data. and computational complexity control parameters The reconstructed image data x is calculated.
[0007] The image compression model described above uses a training set composed of images as samples and optimizes network parameters by calculating a loss function. Specifically, it calculates a weighted sum of bitrate loss, distortion loss, and computational complexity loss (FLOPs), and uses the Variational Inference of Multiple Relevance Result (VIMCO) function required for multi-part image training as the training loss function. Where: λ D ,λ C These are the weights for distortion loss and computational complexity loss, respectively, and the bitrate loss. Distortion loss Computational complexity loss VIMCO function required for multi-part graph training in: It is a heterogeneous bipartite graph Bayesian backbone network. For a multipart graph Bayesian context model; C() represents the computational complexity of the heterogeneous bipartite graph form or the multipartite graph Bayesian context model; M represents the number of samples in the Monte Carlo method. This represents the predicted probability of the i-th Monte Carlo sample for the context model input.
[0008] The heterogeneous bipartite graph Bayesian backbone network mentioned above refers to the backbone neural network that implements the corresponding image compression model. In Bayesian networks... arrive Taking the edge as an example, its corresponding heterogeneous bipartite graph parameters in: For Bayesian networks arrive Heterogeneous bipartite graph, represent arrive N-class distribution parameters of heterogeneous bipartite graph parameters The normalized sample representing the aforementioned distribution sampling, This represents the predicted probability of the neural network corresponding to the nth edge. Similarly, the remaining edges of the Bayesian network are also defined using corresponding parameters, for example... The heterogeneous bipartite graph from Bayesian network y to x also corresponds to an N-class distribution. During training, the parameters of this distribution can be optimized using the Gumbel-softmax reparameterization technique and mixed weights. Specifically, Gumbel-softmax reparameterization is a sampling algorithm that transforms discrete variables into differentiable continuous variables, which can be used to optimize discrete random variables. For other heterogeneous bipartite graphs corresponding to edges, including... and Similar methods can be used to define and optimize them.
[0009] The computational complexity of the heterogeneous bipartite graph form Bayesian network is... Where: C(p) n (y|z)) represents the computational complexity of the neural network corresponding to the nth edge in the heterogeneous bipartite graph from y to z.
[0010] The multi-graph Bayesian context model refers to the implementation of the context model in the corresponding image compression model. Its corresponding multi-graph parameters are defined as follows: Wherein: T c,h,w =Categorical(π) c,h,w ), p(G intra ) represents the distribution of multipart graph parameters, p(T) c,h,h ) represents the category distribution of the topological sort index of the current pixel position in the multipart graph.
[0011] The computational complexity of the multi-graph Bayesian context model refers to the fact that the computational complexity of the multi-graph Bayesian network implemented using dynamic mask convolution is a constant value, therefore no optimization is needed, and a constant of 0 can be used in the loss function.
[0012] The variable-width network described herein is implemented using, but is not limited to, the techniques described by Jiahui Yu et al. in their paper “Slimmable Neural Networks”.
[0013] The aforementioned context model employs an autoregressive model implemented using dynamic mask convolution. The convolution kernel mask can be dynamically adjusted at different positions based on the block division of each node to achieve a parallel autoregressive model, which facilitates adjusting the computational complexity of the autoregressive model based on the parameters of the multi-graph Bayesian network.
[0014] The dynamically masked convolution refers to covering part of the convolution kernel with a mask calculated based on the topological sorting index of nodes in a multi-graph network, used to implement an autoregressive model based on a multi-graph Bayesian network. The convolution kernel mask can be dynamically adjusted at different positions according to the topological sorting index of each node to achieve a parallel context model. For example, the convolution kernel of a 5x5 dynamically masked convolution is: Among them: W c,h,w The convolution kernel representing the current pixel position, T c,h,w This represents the topological sort index of the current pixel position in the multipart graph. Technical effect
[0015] This invention achieves scalable computation of the backbone neural network by constructing a Bayesian network based on heterogeneous bipartite graphs, and scalable computation of the autoregressive model by constructing a Bayesian network based on multipartite graphs. Compared with existing technologies, because each module of this invention can learn its own Bayesian network, more precise control of computational complexity can be achieved, thus enabling more accurate allocation of computational resources based on the computational budget of different devices. In addition, the autoregressive model learned by the multipartite graph Bayesian network performs better than manually designed autoregressive models, and can still maintain good compression performance under limited computational complexity. Attached Figure Description
[0016] Figure 1 This is a schematic diagram of the system of the present invention;
[0017] Figure 2 This is a schematic diagram of the Bayesian network used in the image compression system according to an embodiment of the present invention;
[0018] Figure 3 This is a flowchart of the online phase of an example.
[0019] Figure 4 This example compares the BD-Rate performance of different computationally scalable NIC frameworks under the MACs metric.
[0020] Figure 5 This example compares the BD-Rate performance of different computationally scalable NIC frameworks under the total encoding / decoding time metric.
[0021] Figure 6 This is a visual comparison of manually designed multipart graphs and multipart graph training in the example. Detailed Implementation
[0022] This embodiment relates to a construction process for a computationally scalable image compression system based on Bayesian network structure learning, specifically including:
[0023] Step 1: Based on the image compression framework, determine the node types in the Bayesian network and the corresponding inference and generation relationships of the Bayesian network nodes, and construct the Bayesian network structure, specifically including: based on Figure 2 The Bayesian network shown constructs an image compression model, where: Bayesian inference of the edge x to y corresponds to... Figure 1 In the first-level coding model, y corresponds to z Figure 1 Two-level coding model in; Bayesian edge generation arrive correspond Figure 1 The first-level decoding model in the middle, arrive correspond Figure 1 The two-level decoding model in Internal connection correspondence Figure 1 The context model in the model. Based on this, nodes corresponding to the heterogeneous bipartite graph are added, including... This is to facilitate control over the computational complexity of the backbone neural network and the nodes corresponding to the multipart graph parameters. This is to facilitate control over the computational complexity of the autoregressive model.
[0024] Step 2: Define and train the neural network implementation of the Bayesian network obtained in Step 1, specifically including:
[0025] 2.1) Constructing heterogeneous bipartite graph structures and backbone neural networks: For all heterogeneous bipartite graphs, N is 5, that is, 5 edges correspond to 5 different widths of variable-width networks; for the first-level encoding model, the second-level encoding model, the first-level decoding model, and the second-level decoding model, variable-width networks are used and the corresponding heterogeneous bipartite graph parameters are defined.
[0026] In this embodiment, the network structure and parameter settings of the variable-width network refer to the SlimCAE algorithm, that is, the channel width of all intermediate layers of the variable-width network is set to [48, 72, 96, 144, 192]. Except for the input channel of the first-level coding model and the output channel of the first-level decoding model, which have 3 channels, the input and output channels of the remaining layers are all 192. The SlimCAE uses, but is not limited to, the technology described by Feiyu Yang et al. in the paper "Slimmable Compressive Autoencoders for Practical Neural Image Compression".
[0027] 2.2) Constructing the multipart graph structure and context model: The structure of the context model is as follows: Figure 2 As shown in the dashed box, the layer includes one 5x5 dynamic mask convolution, three 1x1 dynamic mask convolutions, and one channel-dimensional parallel layer. The multi-part graph distribution uses three different block sizes [2, 4, 10], with each block size corresponding to a channel-dimensional grouping number of [2, 4, 12]. Three different distributions are trained to meet different computational complexity controls. Furthermore, in the VIMCO function required for multi-part graph training, the Monte Carlo sampling number M is set to 2.
[0028] 2.3) Using the training samples as input, run the Bayesian network constructed above, and according to the formula... Calculate the loss function: distortion loss weight λ D Choose one of the following values: [39.015, 75.8625, 145.2225, 281.775]; calculate the complexity loss weight λ. CWe set it to 0 because the magnitude of the computational complexity is much larger than the previous two terms, making it difficult to adjust. Therefore, we do not optimize it during training. After training, we use an exhaustive search to find the optimal model with the smallest average loss function under different computational complexity levels.
[0029] In this embodiment, the image compression system built on the open-source software PyTorch is trained on the ImageNet training set. The training process starts with a pre-trained model provided by the open-source software CompressAI and uses the Adam optimizer for 500,000 iterations with a learning rate of 0.0001. Note that, based on different distortion loss weights, a total of four models with different distortion loss weights are trained to adjust the image bitrate.
[0030] Step 3, as follows Figure 3 As shown, the online phase uses a trained Bayesian network structure for real-time image compression and tests the bitrate and reconstruction quality of the compressed image. Specifically, the trained image compression model is used to adjust the computational complexity of each module for different types of computing platforms (including mobile terminals, graphics processing units (GPUs), high-performance servers, etc.) to compress the input image to be compressed and output the bitstream, or to process the bitstream and output the reconstructed image.
[0031] This embodiment uses peak signal-to-noise ratio (PSNR) and bit rate (BPP) to evaluate compression performance and calculates the BD-rate relative to the benchmark model (Hyperprior). Computational complexity is evaluated using multiply-accumulate operations (MACs) or total encoding / decoding time (milliseconds).
[0032] The aforementioned benchmark model (Hyperprior) refers to an image compression framework consisting of a two-level autoencoder and an arithmetic encoder. It is the benchmark model for many NIC systems and was proposed by Johannes Balle et al. in the paper "Variational image compression with a scale hyperprior".
[0033] The aforementioned BD-rate refers to a scheme based on curve integrals to calculate the average bitrate saving relative to the baseline model under the same image quality, which employs, but is not limited to, Gisle. The technical implementation is described in the paper "Calculation of average PSNR differences between RD-curves".
[0034] Through specific practical experiments, the trained image compression model was tested on 24 images in the Kodak dataset for compression and decompression. The aforementioned metrics were calculated, and the test results were compared with similar scalable systems. The results are as follows: Figure 4 , Figure 5 As shown, compared to the SlimCAE framework, this invention can save 2%-10% of the bitrate under the same PSNR and MACs. Compared to the traditional WebP algorithm, although the encoding and decoding time is longer, the relative bitrate is reduced by more than 20%. Compared to the BPG algorithm, the bitrate is comparable but the encoding and decoding speed is faster. This demonstrates that the proposed scheme for adjusting the computational complexity of heterogeneous bipartite and multipartite Bayesian network parameters has the advantages of precise control and excellent compression performance.
[0035] In addition, this embodiment also compares different autoregressive model implementation schemes, including masked convolution, checkerboard, masked convolution, channel-wise masked convolution, spatial hierarchical masked convolution, and channel-wise checkerboard masked convolution. The bit rate and decoding rate corresponding to different autoregressive models are calculated for different systems using the same backbone neural network parameters (i.e., the same peak signal-to-noise ratio and computational complexity). The results are shown in Table 1.
[0036] Table 1 compares the bitrate and decoding speed performance of different autoregressive models in the embodiments under the framework of this invention.
[0037] like Figure 6 As shown, this is the result of training a multi-part image with 10 blocks, which is compared with the grouping result of channel-checkerboard mask convolution with the same 10 blocks. The gray value of the pixel in the image represents the topological sort index of the current pixel position, that is, pure black represents 0, pure white represents 9, and gray represents 1-8 according to gray level.
[0038] Compared to existing technologies, this invention achieves lower bitrates under various block count conditions while maintaining roughly the same speed. Visualization results show that the multi-graph block structure trained by this invention is more interleaved, and its lower bitrate demonstrates that the autoregressive model composed of interleaved multi-graph blocks is more advantageous for image compression. This proves that adjusting the computational complexity of the autoregressive model based on the multi-graph Bayesian network parameters offers superior compression performance.
[0039] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.
Claims
1. A system for computing scalable image compression based on Bayesian network structure learning, characterized in that, The image compression model comprises: a first-level encoding model, a second-level encoding model, a second-level decoding model, a multi-part graph Bayesian context model based on dynamic mask convolution, and a first-level decoding model connected in sequence, wherein the first-level encoding model, the second-level encoding model, the first-level encoding model, and the second-level encoding model constitute a heterogeneous bipartite graph Bayesian backbone network for predicting the conditional probability between Bayesian network nodes, and entropy encoding and decoding modules are arranged between the multi-part graph Bayesian context model and the first-level decoding model and between the second-level encoding model and the second-level decoding model, and a quantization module is arranged between the first-level encoding model and the context model and between the second-level encoding model and the entropy encoding and decoding module. The dynamic mask convolution refers to covering part of the convolution kernel with a mask calculated based on the multi-part graph node topology sorting index, which is used to implement an autoregressive model based on a multi-part graph Bayesian network; the convolution kernel mask is dynamically adjusted at different positions according to the topology sorting index of each node to implement a parallel context model; for example, the convolution kernel of a 5x5 dynamic mask convolution is: 2. The system for computing scalable image compression based on Bayesian network structure learning according to claim 1, characterized in that, In the encoding stage, the primary encoding model calculates the image prior data y according to the input image x and the calculation complexity control parameter The image prior data y is calculated, and the quantized prior data is obtained after the image prior data y passes through the quantization module Q The secondary encoding model calculates the image super-prior data z according to the input y and the calculation complexity control parameter The image super-prior data z is calculated, and the quantized super-prior data is obtained after the image super-prior data z passes through the quantization module Q The secondary prior data is combined for entropy encoding, and the secondary code stream is obtained. The secondary decoding model is based on the input quantized hyper-prior data and the calculation complexity control parameter is calculated, and the output thereof is combined with the quantized prior data to input into the context model, and the calculation complexity control parameter is combined to calculate the primary prior; and the quantized hyper-prior data is combined with the output of the context model to obtain the primary prior, and the primary prior is used for entropy coding to obtain a primary code stream; In the decoding stage, the secondary code stream is combined with the secondary prior for entropy decoding to obtain quantized hyper-prior data The secondary decoding model is calculated according to the input quantized hyper-prior data and the computational complexity control parameter The output is the quantized prior data to be decoded After merging, it is input into the context model combined with the computational complexity control parameter The primary prior is calculated for entropy decoding of the primary code stream to update the quantized prior data After several rounds of iteration, the complete quantized prior data is obtained The final decoding model is calculated according to the input quantized prior data and the computational complexity control parameter The reconstructed image data x is calculated.
3. The system for computing scalable image compression based on Bayesian network structure learning according to claim 1, characterized in that, The image compression model takes a training set composed of images as a sample, and optimizes network parameters by calculating a loss function, specifically: calculating the weighted sum of code rate loss, distortion loss, and computational complexity loss (FLOPs), and taking the VIMCO function required for multi-part image training as the loss function of the training wherein: λ S , λ C are the weights of the distortion loss and the computational complexity loss, respectively, the code rate loss the distortion loss the computational complexity loss VIMCO function required for multi-partite graph training wherein: is a heterogeneous bipartite Bayesian backbone network, is a multi-partite Bayesian context model; C() is the computational complexity of the heterogeneous bipartite Bayesian backbone network or the multi-partite Bayesian context model; M represents the number of Monte Carlo method sampling, represents the prediction probability of the i-th Monte Carlo sampling sample on the input of the context model.
4. The system for computing scalable image compression based on Bayesian network structure learning according to claim 1 or 3, characterized in that, The heterogeneous bipartite graph Bayesian backbone network refers to the backbone neural network that implements the corresponding image compression model, using Bayesian networks as the core. arrive Taking the edge as an example, its corresponding heterogeneous bipartite graph parameters in: For Bayesian networks arrive Heterogeneous bipartite graph, represent arrive N-class distribution parameters of heterogeneous bipartite graph parameters The normalized sample representing the aforementioned distribution sampling, This represents the predicted probability of the neural network corresponding to the nth edge; similarly, the remaining edges of the Bayesian network are also defined using corresponding parameters. This is a heterogeneous bipartite graph from y to x in a Bayesian network, which also corresponds to an N-class distribution.
5. The system for computing scalable image compression based on Bayesian network structure learning according to claim 1 or 3, characterized in that, The isomorphic bipartite Bayesian backbone network uses Gumbel-softmax reparameterization techniques and mixed weights to optimize the parameters of the distribution during training, specifically: Gumbel-softmax reparameterization is a sampling algorithm that converts discrete variables into continuous variables that are differentiable, which can be used to optimize discrete random variables; for other edge corresponding isomorphic bipartite graph, including and Similar methods are used to define and optimize.
6. The system for computing scalable image compression based on Bayesian network structure learning according to claim 1 or 3, characterized in that, The multi-part graph Bayesian context model adopts a dynamic mask convolution to realize an autoregressive model, a convolution kernel mask of which is dynamically adjusted according to the partition of each node at different positions, so as to realize a parallel autoregressive model, facilitate adjustment of the calculation complexity of the autoregressive model according to the multi-part graph Bayesian network parameters, and define corresponding multi-part parameters as Wherein, T c,h,w =Categorical(pi c,h,w ), p(G intra ) represents a distribution of the multi-part parameters, and p(T c,h,w ) represents a category distribution of the topological order index of the current pixel position in the multi-part graph.
7. The system for computing scalable image compression based on Bayesian network structure learning according to claim 6, characterized in that, where: W c,h,w represents the convolution kernel of the current pixel position, T c,h,w represents the topological ordering index of the current pixel position in the multi-part graph.