A light field image compression method based on multi-view representation and state space model

Abstract

The application discloses a light field image compression method based on a multi-view representation and a state space model, and belongs to the field of image processing and multimedia technologies. First, the application decomposes a light field image into four view branches based on the multi-dimensional characteristics of the light field image, independently extracts features of the view branches, and fully mines structural information of the light field image. Meanwhile, a high-dimensional light field image data is decomposed into multiple low-dimensional view data for processing in a view-by-view manner, so that the overall calculation cost of the model is reduced. In order to improve the expression ability and modeling ability of features of each view, a multi-scale Mamba module is designed. The module can extract global information of different branches with linear complexity, and obtain more compact feature representation. Finally, the method comprehensively considers the importance of different view features, establishes adaptive fusion multi-view features, and strengthens the expression ability of depth features to key information of the light field image. Finally, efficient and high-quality compression of the light field image is realized.

Description

Technical Field

[0001] This invention belongs to the field of image processing and multimedia technology, specifically relating to a light field image compression method based on multi-view representation and state space model. Background Technology

[0002] Light field images, as a novel image format capable of fully recording light radiation information, not only encompass the scene content of traditional two-dimensional images but also additionally retain the directional information of light rays. This makes light field images crucial for applications in 3D reconstruction, virtual reality, and view rendering. However, due to the extremely rich viewpoint information stored in light field images, their data volume increases significantly, posing substantial challenges to transmission and storage. Therefore, how to efficiently compress light field images has become a pressing technical problem to be solved.

[0003] Current general-purpose data, image, and video compression methods, such as ZIP, JPEG, JPEG 2000, and MPEG, have many limitations when processing light field images. These methods do not fully consider the complex structure and correlations of light field images across different views. Light field images not only contain a large amount of structural information in the spatial dimension, but also contain image information from multiple perspectives in the angular dimension, and further contain unique geometric and depth information in the epipolar plane direction. Traditional compression methods cannot fully exploit the inherent connections and data redundancy between different views, resulting in low data compression efficiency and difficulty in achieving high compression ratios while maintaining image quality.

[0004] In recent years, with the continuous development of deep learning technology, image compression methods based on deep models have gradually surpassed traditional methods in performance, especially deep compression networks based on super-prior structures, which have achieved superior compression performance compared to traditional methods and deep encoding / decoding architectures. However, when these methods are applied to light field image coding tasks, they typically only utilize convolutional operations to extract spatial and angular features within a limited disparity range, failing to consider the correlation between space and angle, resulting in unsatisfactory compression performance. Meanwhile, the Transformer deep model, due to its powerful long-range feature modeling capabilities, can effectively model the intrinsic correlation between spatial and angular features, thereby reducing the redundancy of feature representations. However, the high-dimensional data structure of light field images leads to a quadratic increase in computational complexity when applying the Transformer deep model, imposing very high computational resource requirements on its application.

[0005] Considering the issues mentioned above, existing light field image compression methods generally suffer from several major shortcomings: insufficient exploitation and utilization of inter-viewpoint correlations, high computational complexity, and difficulty in maintaining high-quality light field reconstruction results while pursuing high compression ratios. Therefore, current methods for light field image compression struggle to guarantee both performance and efficiency. Summary of the Invention

[0006] To effectively address the challenges in existing technologies, this invention proposes a light field image compression method based on multi-view representation and a state-space model. This method, tailored to the specific structural characteristics of light field images, aims to effectively mine and utilize the correlation between viewpoints while reducing computational complexity, thereby achieving a high compression ratio while maintaining high-quality reconstruction of the light field image.

[0007] First, based on the multidimensional characteristics of light field images, this invention decomposes them into four view branches and performs independent feature extraction on these branches to fully exploit the structural information of the light field image. Simultaneously, the view-splitting processing method decomposes high-dimensional light field image data into multiple low-dimensional view data for separate processing, reducing the overall computational overhead of the model. To improve the expressive and modeling capabilities of each view feature, this invention designs a multi-scale Mamba module. This module can extract global information from different branches with linear complexity, resulting in a more compact feature representation. Finally, the method of this invention integrates the importance of different view features, establishing an adaptive fusion of multi-view features to enhance the expressive power of depth features for key information in the light field image. Ultimately, this achieves efficient and high-quality compression of light field images.

[0008] To achieve the above objectives, the specific technical solution of the present invention is as follows:

[0009] A light field image compression method based on multi-view representation and state-space model is presented. The method focuses on inventing a feature extraction approach in image encoding / decoding networks and designs a multi-view representation and state-space model approach. This aims to effectively mine and utilize the correlation between viewpoints, achieving a high compression ratio with low computational overhead while ensuring data reconstruction quality. (See attached image.) Figure 1 The diagram shows the overall architecture of the light field image compression model proposed in this invention. The overall network architecture mainly includes: an image encoding / decoding network based on multi-view representation and state space model, a super-prior encoding / decoding network, a quantizer, a context network, an entropy parameter generation network, and an arithmetic encoder / decoder. The functions and collaborative mechanisms of each network part are as follows:

[0010] (1) Image encoding / decoding network based on multi-view representation and state space model: responsible for efficient feature extraction and reconstruction of light field images to obtain compact master latent variables. The multi-view representation state space model mainly includes a multi-view feature extraction module, a multi-scale Mamba module, and a multi-view feature fusion module. According to the structural characteristics of the light field image, the multi-view feature extraction module sets different convolution kernel sizes, strides, and dilation rates to form different feature extractors, which extract the spatial features, angular features, horizontal polar plane features, and vertical polar plane features of the original light field image respectively. Representing the light field image from multiple perspectives can capture the structural characteristics of the light field image more comprehensively. The multi-scale Mamba module adopts a multi-scale strategy to reduce feature redundancy and computational overhead in multiple scanning paths. This module introduces global information of each view feature with linear complexity, improving the model's ability to extract the data representation of master latent variables. The multi-view feature fusion module dynamically weights the extracted multi-view features based on the correlation and importance of features of different view branches, optimizes the interaction weights of multi-view features, and suppresses feature representation redundancy while retaining the key information of each view.

[0011] (2) Super-prior encoding / decoding network: This network is used to generate and extract prior information, which helps to improve compression efficiency and reconstruction quality. Super-prior information can be used as super-latent variables to learn the spatial dependencies of main latent variables, reduce feature redundancy, and improve the overall compression performance of the model.

[0012] (3) Quantizer: By rounding each element to the nearest integer, continuous value features are mapped to discrete integers, realizing parameterized vector quantization of features; during the model training stage, uniform noise is added to ensure the gradient differentiability of the quantization operation, thus ensuring end-to-end training of the model.

[0013] (4) Context network: By using mask convolution with stride of 1 and kernel of 5×5, the probability distribution parameters of the current location feature are predicted using the decoded or encoded neighborhood information, thereby improving the accuracy of probability estimation.

[0014] (5) Entropy parameter generation network: responsible for probabilistic modeling of the master latent variable output by the encoding network. This includes predicting the probability distribution parameters of each element in the master latent variable, such as mean and variance.

[0015] (6) Arithmetic encoder / decoder: The arithmetic encoder encodes characters based on their probabilities. The quantized master latent variable and super latent variable information are compressed into the bitstream by the arithmetic encoder and decompressed by the arithmetic decoder. The arithmetic decoder is the inverse operation of the arithmetic encoding process, and its core is to accurately recover the original data from the compressed bitstream.

[0016] Based on the above-described light field image compression model architecture, the method of this invention can achieve compression and decompression of light field images:

[0017] For light field image compression, multi-view representation and state-space model are used to extract latent representation features. A super-prior network is combined to estimate the global distribution, and a context network is used to model local correlations. Joint entropy coding is then used to generate an efficient binary bitstream, significantly improving compression efficiency and image quality while preserving the geometric and viewpoint information of the light field data for high-quality reconstruction. (See attached image) Figure 1 As shown in the solid line flowchart, firstly, the input is the original light field image. Image encoding networks with multi-view representation and state-space models are used to generate master latent variables. , The result obtained after quantization Advanced prior coding networks learn principal latent variables. Spatial distribution generates hyperhidden variables Quantized hyper-hidden variables The a priori bitstream is obtained through an arithmetic encoder. Secondly, The input is fed into the context network, where predictions are made sequentially using masked convolutions; simultaneously, the hyperlatent variables are quantized. The input is fed into the super-prior decoding network for prediction. The result is concatenated with the prediction result from the context network along the channel dimension and then fed into the entropy parameter generation network for estimation. mean of probability distribution and standard deviation Finally, the arithmetic encoder calculates the result based on its distribution parameters. ,right Encode to obtain image feature bitstream Image feature bitstream and super-prior stream Together, they constitute the final binary code stream.

[0018] For light field image decompression, the master latent variables are gradually recovered through an arithmetic decoder and a super-prior decoding network. By combining contextual network conditional probability modeling, decoding accuracy is improved. A multi-view visual state space image decoding network is used to reconstruct the light field image, accurately recovering its viewpoint correlation and geometric structure, achieving high-quality image restoration. (See attached image.) Figure 1 As shown by the dashed line flowchart, an arithmetic decoder is used to process the image feature bitstream. Decoding is performed, and an arithmetic decoder is used to process the a priori bitstream. Decode to obtain ,Will The parameters of the probability distribution of the first pixel are obtained by feeding them into the super-prior decoding network. and For the first decoded pixel, since there is no local context, a super-prior decoding network is used to decode it to obtain the main latent variables of the current pixel. Next, the loop will cycle to the next position. Decoding operation, at this position The data is fed into the prior decoding network for prediction, and the context network is based on the already decoded positions. The values ​​are predicted, and the results of the two predictions are concatenated along the channel dimension and then fed into the entropy parameter generation network to infer the current position. mean of the distribution and variance The arithmetic decoder is based on this. Decode and restore the current position's hidden variables Repeat this operation until the entire process is complete. Decoding complete. Finally, the decoded result... The light field image is reconstructed by an image decoding network based on multi-view representation and state-space model. .

[0019] The beneficial effects of this invention are:

[0020] In summary, this invention, based on the multidimensional characteristics of light field images, designs a multi-view representation and state space model in the image encoding / decoding network stage to obtain more compact latent variables. The multi-view feature extraction module deentangles the light field data into multiple low-dimensional subspaces, comprehensively acquiring the spatial, angular, and epiplanetary features of the light field image, and fully exploring the structural characteristics of different viewpoints. Secondly, the multi-scale Mamba module optimizes feature representation with linear complexity, reducing feature redundancy and obtaining global features. Furthermore, this invention proposes a multi-view feature fusion module that, using a dynamic weighting strategy, automatically adjusts the weights of each view feature in the fusion process according to the importance of different view features. Simultaneously, by combining a super-prior network with a context network for conditional probability modeling, the probability distribution of the latent variables is modeled more accurately, thereby improving encoding efficiency.

[0021] The method proposed in this invention realizes the compression and decompression of light field images, making full use of their spatial dimension, angular dimension and viewpoint correlation information, and completing the collaborative optimization of the encoding and decoding depth model of high compression ratio and high quality reconstructed light field image. This not only improves the overall rate-distortion performance of the model, but also effectively solves the core pain points of existing light field image compression technology. Attached Figure Description

[0022] Figure 1 Overall architecture diagram of light field image compression network based on multi-view representation and state-space model.

[0023] Figure 2 Multi-view representation and state-space model structure diagram

[0024] Figure 3 Multi-scale Mamba module structure diagram

[0025] Figure 4 Multi-view feature fusion module structure diagram Detailed Implementation

[0026] The following detailed description of embodiments of the present invention is provided in conjunction with the accompanying drawings:

[0027] A light field image compression method based on multi-view representation and state-space model is proposed. The detailed network structure diagram is attached. Figure 1 As shown. The overall process includes the compression encoding and decoding model architecture and training, compression encoding of light field images, and decoding and image reconstruction of the compressed bitstream of light field images.

[0028] 1. Compression Encoding / Decoding Model Architecture and Training

[0029] This deep model learns how to efficiently compress and reconstruct light field images through training. The model's construction and training process includes light field image encoding and decoding, as well as loss function calculation and network weight optimization based on the backpropagation algorithm.

[0030] Step 1: Obtain the original light field image to be compressed. Specifically, the light field image is represented as Where H and W represent the height and width resolutions of the two-dimensional imaging field of view, respectively, and U and V represent the light incident angle resolutions in two orthogonal dimensions during the imaging process. This invention uses macro-pixel representation of the light field image, that is, grouping pixels at the same two-dimensional spatial location together, represented as... The resulting light field image is represented as .

[0031] Step 2: Image the light field Image coding network based on multi-view representation and state-space model is fed into Get the main hidden variable Using a quantizer Will Quantified as Furthermore, during training, uniform noise is added to avoid the non-differentiability problem in the gradient descent algorithm iteration. Quantizer The implicit variable m, represented in floating-point format, is rounded to an integer representation for easier subsequent processing. Then, the arithmetic encoder calculates the value based on the implicit variable... The probability modeling results generate image feature bitstreams Similarly, during the decoding process, the image feature bitstream Recovered by the arithmetic decoder Then, through an image decoding network based on multi-view representation and state-space model The reconstructed light field image is obtained. .

[0032] Step 3: This invention employs a priori method to quantify the master latent variables. It performs probabilistic modeling and recovery. It utilizes a super-prior encoder-decoder network, a context network, and an entropy parameter generation network to... The model is based on a single Gaussian distribution, with the mean of which is denoted as . The standard deviation is denoted as . As prior information for light field image encoding and decoding, it serves as the input parameter for the operation of the arithmetic encoder and arithmetic decoder.

[0033] Super-prior coding network From the implicit variable of the master Extracting Hidden Variables , used to characterize Spatial dependencies, denoted as Then, use the quantizer Q to... Quantified as , The arithmetic encoder then encodes the a priori bitstream. . Image feature bitstream The final compressed bitstream is obtained by combining the data. .

[0034] In the process of super-prior decoding, quantizing the super-latent variables Through the a priori decoding network Calculate the initial prediction parameters Meanwhile, context network Using mask convolution operations, the quantized latent variables are processed sequentially. Perform a prediction and output the prediction result for the current position. Finally, the entropy parameter generation network... Combination and ,estimate probability distribution parameters The above operations can be formalized into the following formula:

[0035]

[0036] Step 4: Construct an end-to-end light field image compression model based on rate-distortion optimization objectives, using the model training loss function. for:

[0037]

[0038] In the formula, Different rate-distortion tradeoff parameters The values ​​correspond to different bit rates. It is a quantification of primary and implicit variables. The bit rate is defined as:

[0039]

[0040] In the formula Known Under the premise The conditional probability value, To obtain the expected value, Indicates that in the known Under the premise of, The average number of bits required for encoding, i.e., the number of symbols. Information entropy. It is a quantified hyper-hidden variable The bit rate is defined as:

[0041]

[0042] In the formula for The probability value, Indicates the quantization of hyperhidden variables The average number of bits required for encoding itself, i.e., the symbol Information entropy. This represents the distortion between the original image and the reconstructed image, calculated from the mean square error of the two images.

[0043] Given a dataset of light field images represented by macro pixels, based on the loss function The entire light field image encoding / decoding network is trained using the backpropagation algorithm. During training, a rate-distortion tradeoff parameter is used. Available The values ​​are selected from a range, and the AdamW optimizer is used for weight learning with a learning rate of 10. −4 Training was completed when the peak signal-to-noise ratio (PSNR) fluctuation of image reconstruction did not exceed 0.2 dB across three consecutive training epochs. This was based on the benchmark dataset EPFL in the field of light field imaging. At this time, an image reconstruction quality of 0.001 bpp (bits per pixel) and a peak signal-to-noise ratio (PSNR) of 30.5 dB can be obtained. It can achieve a compression bitrate of 0.0195 bpp and image reconstruction quality of PSNR=38.9dB.

[0044] The specific implementations of the image encoding / decoding network, hyperprior encoding / decoding network, context network, and entropy parameter generation network involved in the above process are as follows:

[0045] The image coding network based on multi-view representation and state-space model consists of a 4-level cascaded multi-view representation and state-space model, and a 5×5 convolution with a stride of 2. For light field images... After four sequential operations, we obtain the primary latent variable with an 8x downsampling factor. Each multi-view representation and state-space model consists of a multi-view feature extraction module, a multi-scale Mamba module, and a multi-view feature fusion module. Through the coordinated operation of these modules, the network efficiently integrates data from different perspectives to obtain a compact and robust light field image feature representation, i.e., the master latent variable. .

[0046] (1) Multi-view feature extraction module

[0047] For light field images with macropixel arrangement The multi-view feature extraction module extracts features from four views. Its structure includes a Spatial Feature Extractor (SFE), an Angle Feature Extractor (AFE), a Horizontal Equidistant Plane Feature Extractor (EFEH), and a Vertical Equidistant Plane Feature Extractor (EFEV), as shown in the attached figure. Figure 2 As shown.

[0048] Define parameters That is, d takes the maximum value of the light incident angle resolutions U and V. SFE uses dilated convolution with a kernel size of 3×3, a stride of 1, and an dilation rate of d to extract spatial features. AFE uses dilated convolution with a kernel size of 3×3, a stride of d, and an inflation rate of 1 to extract angular features. EFEH uses a convolution kernel size of 1×d. 2 Dilated convolution with a vertical stride of 1, a horizontal stride of d, and an inflation rate of 1 is used to extract features from the horizontal polar plane. EFEV uses a convolution kernel size of d. 2 A dilated convolution with a step size of ×1, a vertical stride of d, a horizontal stride of 1, and an inflation rate of 1 is used to extract vertical epipolar plane features. .

[0049] Spatial features Used to represent the spatial distribution and local details of an image, such as the shape and texture of objects, to preserve the spatial structure and visual quality of the image. Angular features It encompasses observational information of the same scene from different perspectives, characterizing changes in objects in different directions. By extracting angular information, light field images can better capture effects such as depth of field, parallax, and perspective, improving the understanding of 3D scenes. Horizontal epiplane features. It provides horizontal spatial and angular relationships, capturing the disparity values ​​of objects in light field images, and estimating disparity by analyzing the slope of different lighting patterns. Vertical epipolar plane features. This effectively captures the vertical feature variations and spatial structure of light field images, complementing the features of the horizontal epiplane. Extracting features from different views of the light field image reduces the computational complexity of high-dimensional data while forming a complete representation of the image structure.

[0050] (2) Multiscale Mamba module

[0051] As attached Figure 3 As shown, after the multi-view feature extraction module, each view feature is used as the input feature of the multi-scale Mamba module, denoted as... , . First, it undergoes a layer normalization operation. Then it splits into two computational paths. One path passes sequentially through the linear layer. A depthwise separable convolutional layer with a kernel size of 3×3 and a stride of 1. Activation function Multiscale selective scanning layer and layer normalization operations To obtain intermediate features :

[0052]

[0053] In another path, input features Intermediate features are obtained by processing only linear layers and SiLU activation. :

[0054]

[0055] Then, through element-wise multiplication right and The aggregated features are then processed through a linear layer to obtain the final output features. :

[0056]

[0057] Multiscale selective scanning layer This layer reduces feature redundancy caused by the multi-scan strategy used in traditional Mamba by introducing a multi-scale approach. The input feature of this layer is denoted as X. First, two parallel branches are used to compute a depthwise separable convolution with a kernel size of 3×3 and strides of 1 and 2, respectively, generating two scale features, denoted as X and X. and Next, and A total of four scan paths are expanded to 1D features, which are then processed by the SSM (Selective State Spaces) layer. The processing is represented as follows:

[0058]

[0059] In the formula, ( This indicates the operation of unfolding a 2D feature scan into a 1D sequence. To sort 1D sequences by The reverse process of unfolding is the operation of synthesizing 2D features. This indicates a horizontal raster scan from the top left to the bottom right. This indicates a vertical raster scan from the top left to the bottom right. This indicates a horizontal raster scan from the bottom right to the top left. This indicates a vertical raster scan from the bottom right to the top left. A general state-space model is used to model 1D sequence data. ( The 2D features are the result of processing four scan paths. and Maintain the original resolution of features and preserve high-resolution details. and By downsampling features at corresponding sizes, a wider range of feature information is captured. These features are then fused.

[0060]

[0061] In the formula, This is a 2x linear interpolation operation in both the horizontal and vertical directions. Multiscale selective scanning layer The output characteristics.

[0062] After optimization by the multi-scale Mamba module, the four view branch features each output their own enhanced feature representations, i.e., spatially enhanced features. Angle enhancement features Horizontal and vertical polar plane enhancement features and vertical polar plane enhancement features These features more comprehensively capture and integrate the spatial and angular information of the light field image, enhance the structural features under different views, make the feature expression more compact and global, and help with image encoding and reconstruction.

[0063] (3) Multi-view feature fusion module

[0064] As attached Figure 4 As shown, this module first enhances the features of four parallel views (i.e., after processing by the multi-scale Mamba module) , , , (Semblage at the channel dimension) This forms a feature representation matrix, which is then used to calculate the importance weight matrix of each element of the multi-view feature using a 1×1 convolution with a stride of 1 and a softmax function. This process is described as follows:

[0065]

[0066] Next, the obtained weight matrix Divide the channel into four equal parts to obtain the weight components corresponding to the four view features. Then, each component is multiplied element-wise with its corresponding view enhancement feature. The process involves adaptive feature weighting. Finally, the weighted features are summed element-wise to obtain the fused output features. :

[0067]

[0068] The multi-view feature fusion module uses an adaptive weighting mechanism to flexibly balance and fuse features from different view branches, ensuring effective information interaction between features and improving feature representation capabilities.

[0069] Image Decoding Network Based on Multi-View Representation and State Space Model It is the inverse structure of the encoding network, consisting of four cascaded 5×5 transposed convolutions with a stride of 2 and a multi-view representation with a state-space model. The multi-view representation and state-space model have the same structure as those in image coding networks. It is derived from the image feature bitstream. Quantized master implicit variable recovered by arithmetic decoder After passing through the above network structure four times consecutively, the reconstructed light field image is obtained. .

[0070] Super-prior coding network The following operations are included: main implicit variables After a 3×3 convolution with a stride of 1, the Leaky ReLU activation function is applied. Subsequently, two 3×3 convolutions with a stride of 2 are cascaded in the network; similarly, Leaky ReLU activation is applied after each convolution, ultimately generating the hyperlatent variables. . Modeling master-latent variables Spatial dependencies, capturing This reduces local spatial correlation, thereby further reducing feature redundancy.

[0071] Super-prior decoding network The operations include the following: given a quantized hyperhidden variable It can be derived from hyper-hidden variables. It comes from quantization, or from a priori bitstream. It was recovered using an arithmetic decoder. Through the a priori decoding network Perform predictions to obtain initial prediction parameters. . It contains two transposed convolutions with a stride of 2 and a kernel size of 3×3. Each transposed convolution is activated by the Leaky ReLU activation function, followed by a transposed convolution with a stride of 1 and a kernel size of 3×3, which is also activated by the Leaky ReLU activation function.

[0072] Contextual Networks A masked convolution operation with a stride of 1 and a kernel size of 5×5 is used to quantize the main latent variables in a serial manner. Perform a prediction and output the prediction result for the current position. During the model training phase, Image coding network Extracted implicit variables Quantification is achieved during the inference phase after model training. From the image feature stream by the arithmetic decoder The recovery process involves using already decoded pixels to predict the values ​​of undecoded pixels, thereby leveraging the correlation between adjacent pixels to improve compression efficiency. By incorporating contextual information into predictions, more accurate parameter estimation results can be obtained, reducing coding redundancy.

[0073] Entropy parameter generation network It contains three convolutional layers with a stride of 1 and a kernel size of 1×1. The first two convolutions are followed by a Leaky ReLU activation function to generate quantized latent variables. The probability distribution parameters (mean of a single Gaussian distribution) and standard deviation By modeling the probability of the master's latent variables, subsequent arithmetic coding can achieve more efficient bit rate allocation, thereby improving the overall compression performance.

[0074] 2. Compression coding of light field images

[0075] In the light field image compression stage, the trained model encodes the input image to generate a compressed bitstream that can be stored or transmitted. (Appendix) Figure 1 The solid line represents the compression phase process.

[0076] Step 1: Load attachment Figure 1 The network weight parameters corresponding to the depth obtained after training the network structure shown are used to perform a compression task.

[0077] Step 2: Obtain a light field image The input is fed into an image coding network based on multi-view representation and state-space model, and the master latent variables are transformed to obtain the master latent variables. During the compression stage, the quantizer uses rounding to quantize the variables, obtaining the quantized implicit variables. .

[0078] Step 3: The super-prior encoding network learns the principal latent variables. Spatial distribution generates hyperhidden variables The same rounding quantization method is used for quantization. The arithmetic encoder processes the quantized data... Encode to obtain the super-prior bitstream .

[0079] Step 4: Quantize the hyper-hidden variables The results are fed into the prior decoding network to obtain the initial prediction parameters. Meanwhile, the context network... Perform serial prediction and output the prediction result for the current position. . and These predictions are fed into an entropy parameter generation network, which combines them and models the results. The distribution parameters (mean of Gaussian distribution) and standard deviation Arithmetic encoder according to The probability distribution parameters are losslessly encoded to obtain the image feature bitstream. .

[0080] Step 5: Super-prior stream and image feature bitstream Together, they form the final binary bitstream file in the compression stage, completing image compression.

[0081] 3. Decoding and Image Reconstruction of Light Field Image Compressed Streams

[0082] In the light field image decompression stage, the bitstream obtained from the compression stage is used for decoding to reconstruct the original light field image. (Appendix) Figure 1 The section marked with a dashed line represents the decompression stage process.

[0083] Step 1: Load the same model as in the light field image compression stage and perform the decompression task. Ensure that the same network structure and parameters are used as in the compression stage to maintain compatibility between the encoding and decoding processes.

[0084] Step 2: Use an arithmetic decoder to process the a priori bitstream. Decode the variable to recover the quantized hyper-hidden variable. Decoding result Input the super-prior decoding network to obtain the initial prediction parameters. In decoding image feature streams At that time, the decoded portion is input into the context network; if it is the first decoded pixel, the initial input is 0, in order to obtain The entropy parameter generation network is based on... and Generate about The probability distribution parameters (mean of Gaussian distribution) and standard deviation The arithmetic decoder uses this probability distribution to analyze the image feature stream. Decode the image to obtain the decoded master-latent variables. Repeat the above process until the image feature stream is obtained. Decoding complete.

[0085] Step 3: Decode the quantized master implicit variables The image is input into an image decoding network based on multi-view representation and a state-space model to obtain a reconstructed light field image. The image decompression stage ends.

Claims

1. A light field image compression method based on multi-view representation and state-space model, characterized in that, The overall network architecture used in this method includes: an image encoder / decoder network based on multi-view representation and state-space model, a super-prior encoder / decoder network, a quantizer, a context network, an entropy parameter generation network, and an arithmetic encoder / decoder; the functions and collaborative mechanisms of each network component are as follows: (1) Image encoding / decoding network based on multi-view representation and state space model: responsible for feature extraction and reconstruction of light field image to obtain master latent variables; multi-view representation state space model includes multi-view feature extraction module, multi-scale Mamba module and multi-view feature fusion module; according to the structural characteristics of light field image, multi-view feature extraction module sets different convolution kernel size, stride and dilation rate to form different feature extractors, and extracts spatial features, angular features, horizontal polar plane features and vertical polar plane features of original light field image respectively; the light field image is represented from multiple perspectives to fully capture the structural characteristics of light field image; multi-scale Mamba module adopts multi-scale strategy, and this module introduces global information of each view feature with linear complexity; multi-view feature fusion module dynamically weights the extracted multi-view features based on the correlation and importance of features of different view branches; (2) Super-prior encoder / decoder network: This network is used to generate and extract prior information to help improve compression efficiency and reconstruction quality; (3) Quantizer: By rounding each element to the nearest integer, continuous value features are mapped to discrete integers, realizing parameterized vector quantization of features; during the model training stage, uniform noise is added to ensure the gradient differentiability of the quantization operation, thus ensuring end-to-end training of the model. (4) Context network: By using mask convolution with stride of 1 and kernel of 5×5, the probability distribution parameters of the current position feature are predicted using the decoded or encoded neighborhood information; (5) Entropy parameter generation network: responsible for probabilistic modeling of the master latent variable output by the encoding network; this includes predicting the probability distribution parameters of each element in the master latent variable, including the mean and variance; (6) Arithmetic encoder / decoder: The arithmetic encoder encodes the characters according to their probabilities. The quantized master latent variable and super latent variable information are compressed into the bitstream by the arithmetic encoder and decompressed by the arithmetic decoder.

2. The method according to claim 1, characterized in that: For light field image compression, multi-view representation and state-space model are used to extract latent representation features. A super-prior network is combined to estimate the global distribution, and a context network is used to model local correlations. Joint entropy coding is then used to generate an efficient binary bitstream, preserving the geometric and viewpoint information of the light field data. First, the input is the original light field image. Image encoding networks with multi-view representation and state-space models are used to generate master latent variables. , The result obtained after quantization ; Super-prior encoding networks learn principal latent variables Spatial distribution generates hyperhidden variables Quantized hyper-hidden variables The a priori bitstream is obtained through an arithmetic encoder. Secondly, The input is fed into the context network, where predictions are made sequentially using masked convolutions; simultaneously, the hyperlatent variables are quantized. The input is fed into the super-prior decoding network for prediction. The result is concatenated with the prediction result from the context network along the channel dimension and then fed into the entropy parameter generation network for estimation. mean of probability distribution and standard deviation Finally, the arithmetic encoder calculates the result based on its distribution parameters. ,right Encode to obtain image feature bitstream Image feature bitstream and super-prior stream Together they constitute the final binary code stream; For light field image decompression, the master latent variables are gradually recovered through an arithmetic decoder and a super-prior decoding network. Combined with context network conditional probability modeling; A multi-view visual state-space image decoding network is used to reconstruct the light field image, accurately recovering its viewpoint correlation and geometric structure, thus achieving image restoration; an arithmetic decoder is used to process the image feature stream. Decoding is performed, and an arithmetic decoder is used to process the a priori bitstream. Decode to obtain ,Will The parameters of the probability distribution of the first pixel are obtained by feeding them into the super-prior decoding network. and For the first decoded pixel, since there is no local context, a super-prior decoding network is used to decode it to obtain the main latent variables of the current pixel. ; Next, loop to the next position. Decoding operation, at this position The data is fed into the prior decoding network for prediction, and the context network is based on the already decoded positions. The values ​​are predicted, and the results of the two predictions are concatenated along the channel dimension and then fed into the entropy parameter generation network to infer the current position. mean of the distribution and variance The arithmetic decoder is based on this. Decode and restore the current position's hidden variables Repeat this operation until the entire Decoding complete; finally, the decoded result... The light field image is reconstructed by an image decoding network based on multi-view representation and state-space model. .

3. The method according to claim 1, characterized in that: Step 1: Obtain the original light field image to be compressed. Specifically, the light field image is represented as Where H and W are the height and width resolutions of the two-dimensional imaging field of view, respectively, and U and V are the incident angle resolutions of light rays in two orthogonal dimensions during the imaging process; macro-pixel representation of the light field image is used, that is, pixels at the same two-dimensional spatial location are grouped together and represented as The resulting light field image is represented as ; Step 2: Image the light field Image coding network based on multi-view representation and state-space model is fed into Get the main hidden variable Using a quantizer Will Quantified as Furthermore, uniform noise is added during training to avoid the non-differentiability problem in the gradient descent algorithm iteration; quantizer The implicit variable m, represented in floating-point format, is rounded to an integer representation for easier subsequent processing; then, the arithmetic encoder uses the implicit variable... The probability modeling results generate image feature bitstreams Similarly, during the decoding process, the image feature bitstream Recovered by the arithmetic decoder Then, through an image decoding network based on multi-view representation and state-space model The reconstructed light field image is obtained. ; Step 3: Use a priori method to quantify the main latent variables. It performs probabilistic modeling and recovery; it utilizes a super-prior encoding and decoding network, a context network, and an entropy parameter generation network to... The model is based on a single Gaussian distribution, with the mean of which is denoted as . The standard deviation is denoted as ; As prior information for light field image encoding and decoding, it serves as the input parameter for the operation of the arithmetic encoder and arithmetic decoder; Super-prior coding network From the implicit variable of the master Extracting Hidden Variables , used to characterize Spatial dependencies, denoted as Then, use the quantizer Q to... Quantified as , The arithmetic encoder then encodes the a priori bitstream. ; Image feature bitstream The final compressed bitstream is obtained by combining the data. ; In the process of super-prior decoding, quantizing the super-latent variables Through the a priori decoding network Calculate the initial prediction parameters Meanwhile, context network Using mask convolution operations, the quantized latent variables are processed sequentially. Perform a prediction and output the prediction result for the current position. Finally, the entropy parameter generation network Combination and ,estimate probability distribution parameters The above operations can be formalized into the following formula: Step 4: Construct an end-to-end light field image compression model based on rate-distortion optimization objectives, using the model training loss function. for: In the formula, Different rate-distortion tradeoff parameters The values ​​correspond to different bit rates; It is a quantification of primary and implicit variables. The bit rate is defined as: In the formula Known Under the premise The conditional probability value, To obtain the expected value, Indicates that in the known Under the premise of, The average number of bits required for encoding, i.e., the number of symbols. Information entropy; It is a quantified hyper-hidden variable The bit rate is defined as: In the formula for The probability value, Indicates the quantization of hyperhidden variables The average number of bits required for encoding itself, i.e., the symbol Information entropy; The distortion between the original image and the reconstructed image is represented by the mean square error of the two images; Given a dataset of light field images represented by macro pixels, based on the loss function The backpropagation algorithm is used to train the entire light field image encoding and decoding network; during training, the rate-distortion tradeoff parameter is used. Available The values ​​are selected from a range, and the AdamW optimizer is used for weight learning with a learning rate of 10. −4 Training is completed when the peak signal-to-noise ratio (PSNR) fluctuation of image reconstruction in three consecutive training rounds does not exceed 0.2 dB.

4. The method according to claim 3, characterized in that: The specific implementations of the image encoding / decoding network, super-prior encoding / decoding network, context network, and entropy parameter generation network based on multi-view representation and state space model are as follows: The image coding network based on multi-view representation and state-space model consists of a 4-level cascaded multi-view representation and state-space model and a 5×5 convolution with a stride of 2; for light field images After four sequential operations, we obtain the primary latent variable with an 8x downsampling factor. Each multi-view representation and state-space model consists of a multi-view feature extraction module, a multi-scale Mamba module, and a multi-view feature fusion module. Through the coordinated operation of these modules, the network can efficiently integrate data from different perspectives to obtain the master's latent variables. ; (1) Multi-view feature extraction module For light field images with macropixel arrangement The multi-view feature extraction module extracts features from four views. Its structure includes a spatial feature extractor (SFE), an angle feature extractor (AFE), a horizontal polar plane feature extractor (EFEH), and a vertical polar plane feature extractor (EFEV). Define parameters That is, d takes the maximum value of the light incident angle resolution U and V; SFE uses dilated convolution with a kernel size of 3×3, a stride of 1, and an dilation rate of d to extract spatial features. AFE uses dilated convolutions with a kernel size of 3×3, a stride of d, and an inflation rate of 1 to extract angular features. EFEH uses a convolution kernel size of 1×d. 2 Dilated convolution with a vertical stride of 1, a horizontal stride of d, and an inflation rate of 1 is used to extract features from the horizontal polar plane. ; EFEV uses a convolution kernel size of d 2 A dilated convolution with a step size of ×1, a vertical stride of d, a horizontal stride of 1, and an inflation rate of 1 is used to extract vertical epipolar plane features. ; Spatial features Used to represent the spatial distribution and local details of an image, including the shape and texture of objects, to preserve the spatial structure and visual quality of the image; angular features It encompasses observational information of the same scene from different perspectives, and can characterize changes in objects in different directions; horizontal polar plane features It provides horizontal spatial and angular relationships, can capture the disparity values ​​of objects in light field images, and performs disparity estimation by analyzing the slope of different lighting patterns; vertical epipolar plane features. This can effectively capture the characteristic changes and spatial structure of the light field image in the vertical direction; (2) Multiscale Mamba module After the multi-view feature extraction module, each view feature is used as the input feature of the multi-scale Mamba module, denoted as... , ; First, it undergoes a layer normalization operation. Then it splits into two computational paths; one path passes through the linear layer sequentially. A depthwise separable convolutional layer with a kernel size of 3×3 and a stride of 1. Activation function Multiscale selective scanning layer and layer normalization operations To obtain intermediate features : In another path, the input features Intermediate features are obtained by processing only linear layers and SiLU activation. : Then, through element-wise multiplication... right and The aggregated features are then processed through a linear layer to obtain the final output features. : Multiscale selective scanning layer This layer reduces feature redundancy caused by the multi-scan strategy used in traditional Mamba by introducing a multi-scale approach. The input feature of this layer is denoted as X. First, two parallel branches are used to compute a depthwise separable convolution with a kernel size of 3×3 and strides of 1 and 2, respectively, generating two scale features, denoted as X and X. and Next, and A total of four scan paths are expanded to 1D features, which are then processed by the SSM (Selective State Spaces) layer; the processing is represented as follows: In the formula, ( This indicates the operation of unfolding a 2D feature scan into a 1D sequence. To sort 1D sequences by The reverse process of unfolding is the operation of synthesizing 2D features. This indicates a horizontal raster scan from the top left to the bottom right. This indicates a vertical raster scan from the top left to the bottom right. This indicates a horizontal raster scan from the bottom right to the top left. This indicates a vertical raster scan from the bottom right to the top left. A general state-space model is used to model 1D sequence data; ( The 2D features are the result of processing four scan paths. and Maintain the original resolution of features and preserve high-resolution details. and Then, the features are downsampled according to their corresponding sizes, and these features are fused together: In the formula, This is a 2x linear interpolation operation in both the horizontal and vertical directions. Multiscale selective scanning layer The output characteristics; After optimization by the multi-scale Mamba module, the four view branch features each output their own enhanced feature representations, i.e., spatially enhanced features. Angle enhancement features Horizontal and vertical polar plane enhancement features and vertical polar plane enhancement features ;; (3) Multi-view feature fusion module This module first enhances the features of four parallel views (i.e., after processing by the multi-scale Mamba module) , , , (Semblage at the channel dimension) This forms a feature representation matrix, which is then used to calculate the importance weight matrix of each element of the multi-view feature using a 1×1 convolution with a stride of 1 and a softmax function. This process is described as follows: Next, the obtained weight matrix Divide the channel into four equal parts to obtain the weight components corresponding to the four view features. Then, each component is multiplied element-wise with its corresponding view enhancement feature. The process involves adaptive feature weighting; finally, the weighted features are summed element-wise to obtain the fused output features. : Image decoding network based on multi-view representation and state-space model It is the inverse structure of the coding network, consisting of four cascaded 5×5 transposed convolutions with a stride of 2 and a multi-view representation with a state-space model; the multi-view representation and state-space model have the same structure as those in image coding networks; it is composed of image feature bitstreams. Quantized master implicit variable recovered by arithmetic decoder After passing through the above network structure four times consecutively, the reconstructed light field image is obtained. ; Super-prior coding network The following operations are included: main implicit variables After a 3×3 convolution with a stride of 1, the Leaky ReLU activation function is applied. Subsequently, two 3×3 convolutions with a stride of 2 are cascaded in the network, and Leaky ReLU activation is also applied after each convolution, ultimately generating hyperlatent variables. ; Super-prior decoding network The operations include the following: given a quantized hyperhidden variable It can be derived from hyper-hidden variables. It comes from quantization, or from a priori bitstream. Recovered using an arithmetic decoder; Through the a priori decoding network Perform predictions to obtain initial prediction parameters. ; It contains two transposed convolutions with a stride of 2 and a kernel size of 3×3. Each transposed convolution is activated by the Leaky ReLU activation function, followed by a transposed convolution with a stride of 1 and a kernel size of 3×3, which is also activated by the Leaky ReLU activation function. Contextual Networks A masked convolution operation with a stride of 1 and a kernel size of 5×5 is used to quantize the main latent variables in a serial manner. Perform a prediction and output the prediction result for the current position. During the model training phase, Image coding network Extracted implicit variables Quantitative acquisition; in the inference phase after model training is deployed. From the image feature stream by the arithmetic decoder Recovered; Entropy parameter generation network It contains three convolutional layers with a stride of 1 and a kernel size of 1×1. The first two convolutions are followed by a Leaky ReLU activation function to generate quantized latent variables. The probability distribution parameters.

5. The method according to claim 3, characterized in that, The specific steps for light field image compression are as follows: Step 1: Load the corresponding depth network weight parameters obtained after training the network structure, and perform the compression task; Step 2: Obtain a light field image The input is fed into an image coding network based on multi-view representation and state-space model, and the master latent variables are transformed to obtain the master latent variables. During the compression stage, the quantizer uses rounding to quantize the variables, obtaining the quantized implicit variables. ; Step 3: The super-prior encoding network learns the principal latent variables. Spatial distribution generates hyperhidden variables The same rounding quantization method is used for quantization; the arithmetic encoder processes the quantized data... Encode to obtain the super-prior bitstream ; Step 4: Quantize the hyper-hidden variables The results are fed into the prior decoding network to obtain the initial prediction parameters. Meanwhile, the context network... Perform serial prediction and output the prediction result for the current position. ; and These predictions are fed into an entropy parameter generation network, which combines them and models the results. The distribution parameter is the mean of the Gaussian distribution. and standard deviation Arithmetic encoder according to The probability distribution parameters are losslessly encoded to obtain the image feature bitstream. ; Step 5: Super-prior stream and image feature bitstream Together, they form the final binary bitstream file in the compression stage, completing image compression.

6. The method according to claim 3, characterized in that, The decoding and image reconstruction of the compressed light field image stream are as follows: Step 1: Load the same model as in the light field image compression stage and perform the decompression task; ensure that the same network structure and parameters are used as in the compression stage to maintain compatibility between the encoding and decoding processes; Step 2: Use an arithmetic decoder to process the a priori bitstream. Decode the variable to recover the quantized hyper-hidden variable. Decoding result Input the super-prior decoding network to obtain the initial prediction parameters. In decoding image feature streams At that time, the decoded portion is input into the context network; if it is the first decoded pixel, the initial input is 0, in order to obtain The entropy parameter generation network is based on... and Generate about The probability distribution parameter is the mean of the Gaussian distribution. and standard deviation The arithmetic decoder uses this probability distribution to analyze the image feature bitstream. Decode the image to obtain the decoded latent variables; repeat the above process until the image feature bitstream is obtained. Decoding complete; Step 3: Decode the quantized master implicit variables The image is input into an image decoding network based on multi-view representation and state-space model to obtain a reconstructed light field image; the image decompression stage ends.

Images