Image coding and decoding method based on residual space Mama

Abstract

The invention discloses an image encoding and decoding method based on residual space Mama. Nonlinear transformation modules are inserted between adjacent down-sampling operations of an encoder of an encoding and decoding system and between down-sampling operations of a decoder of the encoding and decoding system; the nonlinear transformation module is a residual space Mama module or a mixed attention module, the residual space Mama module is formed by sequentially connecting a first LayerNorm layer, a structure sensing state space module, a second LayerNorm layer and a feedforward neural network, and jump residual connection is carried out at output; the hybrid attention module introduces a block-based channel attention mechanism module to the output of a feedforward neural network of the residual space Mamba module; according to the method, local and global spatial context information of the image can be effectively captured, and the calculation complexity is reduced to linear growth on the basis of ensuring adaptive weight adjustment; and in combination with a block-level channel attention mechanism, fine-grained modeling is performed on redundancy characteristics in the channel while global spatial information is captured, so that the compression capacity of a potential variable space is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and relates to an image compression coding technology, in particular to an image coding method based on residual space Mamba. BACKGROUND

[0002] With the continuous improvement of image resolution and color depth, the volume of image data is becoming increasingly large, and traditional image compression methods are difficult to meet the needs of modern efficient storage and transmission. In recent years, deep learning-based image compression methods (LIC) have emerged, which can achieve a better trade-off between compression rate and image quality through end-to-end optimization, and gradually surpass traditional compression standards such as JPEG and BPG, showing strong performance.

[0003] At present, LIC methods are mainly based on two types of backbone networks: one is a convolutional neural network (CNN), which is good at extracting local features and has a stable and efficient structure, but is limited by fixed convolution kernels, and its local receptive field is limited, making it difficult for the model to effectively capture long-distance context dependencies in images and lacking the ability to model global dependencies of images. In addition, its static weight mechanism also reduces the adaptability to different input contents. The second is a visual Transformer, which can capture global features through a self-attention mechanism and has global modeling capability and a dynamic weight updating mechanism, and has shown stronger performance in image compression tasks, but its computational complexity grows with the square of the input image size, resulting in a large inference overhead, which brings a large computational burden in high-resolution image processing and other practical applications, making it difficult to be applied to high-resolution image compression tasks.

[0004] To address the low computational efficiency of the Transformer, the recently proposed Mamba model introduces a state space model (SSM) to achieve linear complexity feature modeling with an attention-like mechanism, successfully demonstrating high performance and computational efficiency in the NLP field. Inspired by this, researchers have introduced Mamba structures into computer vision and built models such as Vim, VMamba, and LocalMamba, which have made positive progress in image recognition and segmentation tasks. However, the current research on Mamba in the field of image compression is still in its infancy, and its potential in compression tasks has not been systematically explored.

[0005] In addition, in images, there is usually a close correlation between channels, and traditional algorithms reduce redundant information by mapping pixels to the frequency domain; while modern end-to-end methods map image pixels to a high-dimensional latent space, where redundant information is not only present in the spatial domain but also widely distributed in the channel dimension. Previous methods have not adequately addressed these related information, and there is still residual redundant information. SUMMARY

[0006] In order to overcome the shortcomings of the prior art, the purpose of the present application is to provide a residual space Mamba-based image coding method, by introducing a designed residual space Mamba architecture, not only effectively capturing the local and global spatial context information of the image, but also reducing the computational complexity to linear growth on the basis of ensuring adaptive weight adjustment; and combining the mechanism of block-level channel attention, it can not only capture global spatial information, but also model the fine-grained characteristics of the channel, further improving the compression capability of the latent variable space.

[0007] In order to achieve the above purpose, the technical scheme adopted by the present application is:

[0008] A residual space Mamba-based image coding method, a nonlinear transformation module is inserted between adjacent down-sampling operations of an encoder of a coding system and between down-sampling operations of a decoder to perform nonlinear transformation operations;

[0009] The nonlinear transformation module is a residual space Mamba module, which is composed of a first LayerNorm layer, a structure-aware state space module, a second LayerNorm layer and a feedforward neural network connected in sequence, and performs a skip residual connection at the output; the flow of the nonlinear transformation operation is as follows:

[0010] The input feature X passes through the first LayerNorm layer to obtain the intermediate feature X N ; X N passes through the structure-aware state space module to obtain the intermediate feature X M ; X M passes through the second LayerNorm layer to obtain the feature X L ; X L then passes through the feedforward neural network and adds the input feature X to obtain the output feature Y;

[0011] For the encoder, the input feature X is the output of the previous down-sampling operation of the nonlinear transformation module, and the output feature Y is the input of the next down-sampling operation of the nonlinear transformation module;

[0012] For the decoder, the input feature X is the output of the previous up-sampling operation of the nonlinear transformation module, and the output feature Y is the input of the next up-sampling operation of the nonlinear transformation module.

[0013] In an embodiment of the present application, the nonlinear transformation module is a hybrid attention module, which introduces a block-based channel attention mechanism module at the output of the feedforward neural network of the residual space Mamba module; the flow of the nonlinear transformation operation is as follows:

[0014] The input feature X passes through a first LayerNorm layer to obtain an intermediate feature X N ; X N passes through a structure perception state space module to obtain an intermediate feature X M ; X M passes through a second LayerNorm layer to obtain a feature X L ; X L passes through a feedforward neural network to obtain a feature X K , X K is input into the block-based channel attention mechanism module and added to the input feature X to obtain an output feature Y.

[0015] In an embodiment of the present application, the encoder comprises a main path encoder and a hyper-prior path encoder, and the decoder comprises a main path decoder and a hyper-prior path decoder;

[0016] The main path encoder comprises four convolutional down-sampling blocks and three nonlinear transformation modules, and a multi-level representation extraction structure is alternately stacked;

[0017] The main path decoder comprises four convolutional up-sampling blocks and three nonlinear transformation modules, and a structure symmetric to the main path encoder is used to reconstruct image content;

[0018] The hyper-prior path encoder comprises two convolutional down-sampling blocks and one nonlinear transformation module, and a distribution prior is extracted from the latent variable of the main path encoder;

[0019] The hyper-prior path decoder comprises two convolutional up-sampling blocks and one nonlinear transformation module, and is used to restore auxiliary information for entropy modeling.

[0020] In an embodiment of the present application, in the main path encoder, the convolution kernel size of the first convolutional down-sampling block is 5 and the step size is 2; the convolution kernel size of the remaining convolutional down-sampling blocks is 3 and the step size is 2;

[0021] In the main path decoder, the convolution kernel size of the last convolutional up-sampling block is 5 and the step size is 2; the convolution kernel size of the remaining convolutional up-sampling blocks is 3 and the step size is 2;

[0022] In the hyper-prior path encoder, the convolution kernel size of the convolutional up-sampling block is 3 and the step size is 2;

[0023] In the hyper-prior path decoder, the convolution kernel size of the convolutional up-sampling block is 3 and the step size is 2;

[0024] In one embodiment of the present invention, the main path encoder has three nonlinear transformation modules stacked in sequence for 2, 2, and 9 layers; the super-prior path encoder has one nonlinear transformation module stacked in sequence for 4 layers; the main path decoder has three nonlinear transformation modules stacked in sequence for 2, 2, and 9 layers; and the super-prior path decoder has one nonlinear transformation module stacked in sequence for 4 layers.

[0025] In one embodiment of the present invention, in the main path encoder, the number of output channels of the four convolutional downsampling blocks is (C1, C2, C3, C4) = (128, 128, 128, 320), and in the hyperprior path encoder, the number of output channels of the two convolutional downsampling blocks is (C5, C6) = (128, 192), where C1, C2, and C3 are the channel dimensions in the main path, C4 is the channel dimension of the latent vector, C5 is the channel dimension of the hyperprior path, and C6 is the channel dimension of the hyperprior information; in contrast... It is stated that in the main path decoder, the number of output channels of the four convolutional upsampling blocks is (C'1,C'2,C'3,C'4) = (128,128,128,320), and in the super-prior path decoder, the number of output channels of the two convolutional upsampling blocks is (C5',C'6) = (128,192). Among them, C'1,C'2,C'3 are the channel dimensions in the main path, C'4 is the channel dimension of the latent vector, C'5 is the channel dimension of the super-prior path, and C'6 is the channel dimension of the super-prior information.

[0026] In one embodiment of the present invention, the encoding and decoding system further includes: a channel autoregressive entropy model, two sets of quantizers, and two pairs of arithmetic encoders and two pairs of arithmetic decoders;

[0027] The input image x first passes through the master path encoder to extract the latent variable y to be compressed. Then, the latent variable y is input into the super-prior path encoder to obtain the side information z of the latent variable. The side information z is discretized by a set of quantizers, and then passed through a pair of arithmetic encoders and arithmetic decoders to obtain the reconstructed side information. Reconstructed edge information The latent features F of the latent variable y are obtained through the super-prior path decoder. z The latent variable y is discretized by another set of quantizers, and then passed through another pair of arithmetic encoders and arithmetic decoders to obtain the discrete latent variable. The channel autoregressive entropy model for and F z Processing yields complete reconstructed features Will Input the main path decoder to obtain the reconstructed image.

[0028] In one embodiment of the present invention, the channel autoregressive entropy model for... and F z The processing is performed as follows:

[0029] First, the discrete latent variable is evenly divided into multiple slices along the channel dimension, and each slice is entropy encoded in order. For the slice being encoded the slice that has completed entropy encoding and the latent feature F z is input into the channel autoregressive entropy model to obtain the distribution parameter Φ of the current slice i and the prediction error r i and is arithmetically encoded to obtain the reconstructed slice

[0030] The encoding and decoding process of the slice is looped until all slices are reconstructed to obtain the complete reconstructed feature

[0031] In an embodiment of the present application, the number of slices of the channel autoregressive entropy model is set to S = 5.

[0032] Compared with the prior art, the present application has the following beneficial effects:

[0033] 1) Overcomes the defects of the prior art in insufficient modeling of global and local information.

[0034] Current image compression algorithms tend to focus on capturing global context information or extracting local structural features, and it is difficult to balance both. To this end, the present application proposes a spatial Mamba model combining global scanning and local state aggregation, which simultaneously extracts global and local context information in image compression, making up for the shortcomings of existing methods in information modeling.

[0035] 2) Overcomes the defect of the prior art that lacks a spatial Mamba structure suitable for image compression tasks.

[0036] There is no mature research on applying visual Mamba models to image compression. To fill this gap, the present application innovatively designs a residual spatial Mamba module to enhance the adaptability of Mamba in image compression, solving the problem of direct application of the original Mamba in compression tasks.

[0037] 3) Overcomes the defect of the prior art that does not fully exploit feature redundancy in the channel dimension.

[0038] Although the residual spatial Mamba effectively removes redundancy in the spatial dimension, it does not process the redundancy in the channels in the latent space. Therefore, the present application proposes a block-level channel attention mechanism to further compress useless information in the channel dimension, thereby improving the overall compression rate and image reconstruction quality.

[0039] 4) Overcome the defects of insufficient balance between model compression effect and complexity in the prior art.

[0040] While pursuing the improvement of compression performance, the complexity tends to rise significantly. To solve this problem, the present application introduces block-level channel attention, realizes performance improvement at a very small complexity cost, and effectively solves the contradiction between compression effect and calculation efficiency.

[0041] In summary, the present application introduces a residual space Mamba model and a hybrid attention module, solves the technical problems of insufficient integration of global and local information, insufficient channel redundancy compression, and lack of efficient backbone network structure in existing image compression algorithms, and thus realizes more efficient and higher quality image compression performance. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is the schematic diagram of the end-to-end image encoding framework structure of the present application.

[0043] Figure 2 is the schematic diagram of the residual space Mamba module structure of the present application.

[0044] Figure 3 is the schematic diagram of the hybrid attention module structure of the present application. DETAILED DESCRIPTION

[0045] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and examples.

[0046] The present application is an end-to-end image encoding and decoding method based on residual space Mamba, which proposes a corresponding improvement scheme for the defects existing in the existing end-to-end image compression method.

[0047] As shown in Figure 1 , the overall framework of the encoding and decoding system proposed by the present application includes a main path encoder g a , a main path decoder g s , a hyper-prior path encoder h a , a hyper-prior path decoder h s , a channel autoregressive entropy model, two sets of quantizers, and two pairs of arithmetic encoders and decoders. Among them, the main path encoder g a and the hyper-prior path encoder h a are the main parts of the encoder, and the main path decoder g s and the hyper-prior path decoder h s are the main parts of the decoder. The main path encoder g a and the main path decoder g s undertake the main information encoding and restoration tasks, and the hyper-prior path encoder h aand hyper-prior path decoder h s for modeling the latent variable distribution to optimize the precision of entropy estimation.

[0048] The working process of the system is described below in combination with the flow chart:

[0049] (1) The input image x is first subjected to the main path encoder g a to extract the to-be-compressed latent variable y, which is expressed by the formula:

[0050] y = g a (x; φ)

[0051] where φ is a learnable parameter.

[0052] (2) The to-be-compressed latent variable y is input into the hyper-prior path encoder h a to obtain the side information z of the latent variable, which is expressed by the formula:

[0053] z = h a (y; φ h )

[0054] where φ h is a learnable parameter.

[0055] (3) After the side information z is discretized by a set of quantizers, it is subjected to a set of arithmetic encoders and a set of arithmetic decoders to obtain the reconstructed side information which is expressed by the formula:

[0056]

[0057] (4) The reconstructed side information is subjected to the hyper-prior path decoder h s to obtain the latent feature F z of the to-be-compressed latent variable y, which is expressed by the formula:

[0058]

[0059] where θ h is a learnable parameter.

[0060] (5) After the to-be-compressed latent variable y is discretized by another set of quantizers, it is subjected to another set of arithmetic encoders and another set of arithmetic decoders to obtain the discrete latent variable which is expressed by the formula:

[0061]

[0062] (6) The channel autoregressive entropy model processes the discrete latent variable and the latent feature F z . First, the discrete latent variable Divide the channel into five slices evenly along its dimension. The five slices are entropy encoded sequentially. For the slice currently being encoded / decoded... Slices that have already undergone entropy encoding and decoding and latent features F z Input the channel autoregressive entropy model to obtain the distribution parameters Φ of the current slice. i and prediction error r i And perform arithmetic encoding and decoding to obtain the reconstructed slices. Repeat this process until all slices are reconstructed, resulting in complete reconstructed features. Formula expression:

[0063]

[0064] In the formula e i It is the slice network corresponding to the i-th slice;

[0065] (7) Finally, Input main path decoder g s The reconstructed image was obtained. Formula expression:

[0066]

[0067] In the formula θ h It is a learnable parameter θ.

[0068] The system as a whole consists of a main path and a priori path working together. The main path is used for feature compression and reconstruction of the image, while the priori path is used to model the distribution of latent variables of the main path, thereby achieving fine entropy estimation.

[0069] The main improvement of the image encoding and decoding method based on residual space Mamba in this invention is that a nonlinear transformation module is inserted between adjacent downsampling operations of the encoder and between downsampling operations of the decoder in the encoding and decoding system to perform nonlinear transformation operations.

[0070] For example, in the above encoding / decoding system framework, the main path encoder g a It contains four convolutional downsampling blocks and three nonlinear transform modules, which are alternately stacked to form a multi-level representation extraction structure. Main path decoder g s It contains four convolutional upsampling blocks and three nonlinear transform modules, and reconstructs image content using a structure symmetrical to the main path encoder. The hyper-prior path encoder h a It contains two convolutional downsampling blocks and a nonlinear transform module for extracting distributed prior and hyperprior path decoder h from the latent variables of the main path encoder. s It contains two convolutional upsampling blocks and a nonlinear transformation module to restore auxiliary information for entropy modeling.

[0071] The nonlinear transformation module is a key component in the structure of the present application, which is used to enhance the information interaction ability of spatial and channel dimensions. Specifically, the nonlinear transformation module of the present application can be in the following two structural forms:

[0072] 1. Residual spatial Mamba module (ReSMam): the core of this structure is a structure-aware state space model, which is increased with layer normalization for stable training before and after, connected with a feed-forward neural network to enhance the nonlinear ability of the model, and a skip connection is performed at the output.

[0073] 2. Hybrid attention module (MA): based on the residual spatial Mamba module, this structure introduces a block-based channel attention mechanism.

[0074] Both of these two modules are used to improve the representation ability and reconstruction quality in the image compression process. The hybrid attention module is an improved version of the residual spatial Mamba module, which shows better compression performance and visual effect in experiments.

[0075] The residual spatial Mamba module structure proposed in the present application is shown in Figure 2 , which is composed of a first LayerNorm layer (LN), a structure-aware state space module (SASSM), a second LayerNorm layer, and a feed-forward neural network (FFN) connected in sequence, and a skip connection is performed at the output. The structure-aware state space module is an existing work integrated as a tool module.

[0076] Workflow of the module: the input feature X passes through the first LayerNorm layer to obtain the intermediate feature X N ; X N passes through the structure-aware state space module to obtain the intermediate feature X M ; X M passes through the second LayerNorm layer to obtain the feature X L ; X L then passes through the feed-forward neural network and is added to the input feature X to obtain the output feature Y. The formula is expressed as:

[0077] X N =LN(X)

[0078] X M =SASSM(X N )

[0079] X L =LN(X M )

[0080] Y=FFN(X L )+X

[0081] Hybrid attention module:

[0082] The structure of the hybrid attention module proposed in the application is as shown in Figure 3 Based on ReSMam, a block-based channel attention mechanism module (BCA) is introduced. The entire module includes a first LayerNorm layer (LN), a structure-aware state space module (SASSM), a second LayerNorm layer and a feed-forward neural network (FFN) connected in turn, and a BCA block, and finally outputs a skip connection.

[0083] Workflow of the module: the input feature X is subjected to the first LayerNorm layer to obtain the intermediate feature X N ; X N is subjected to the structure-aware state space module to obtain the intermediate feature X M ; X M is subjected to the second LayerNorm layer to obtain the feature X L ; X L is subjected to the feed-forward neural network to obtain the feature X K ; finally, X K is input into the BCA module, and the result is added to the input feature X to obtain the output feature Y. The formula is expressed as:

[0084] X N = LN(X)

[0085] X M = SASSM(X N )

[0086] X L = LN(X M )

[0087] X K = FFN(X L )

[0088] Y = BCA(X K ) + X

[0089] Obviously, for the encoder, the input feature X is the output of the previous downsampling operation of the nonlinear transformation module, and the output feature Y is the input of the next downsampling operation of the nonlinear transformation module; for the decoder, the input feature X is the output of the previous upsampling operation of the nonlinear transformation module, and the output feature Y is the input of the next upsampling operation of the nonlinear transformation module.

[0090] Configuration of other modules:

[0091] Convolutional down-sampling blocks and convolutional up-sampling blocks: In the main path encoder, the first convolutional down-sampling block is a convolutional layer with a kernel size of 5 and a stride of 2, and the remaining convolutional down-sampling blocks are all convolutional layers with a kernel size of 3 and a stride of 2. In the main path decoder, the last convolutional up-sampling block is a deconvolutional layer with a kernel size of 5 and a stride of 2, and the remaining convolutional up-sampling blocks are all deconvolutional layers with a kernel size of 3 and a stride of 2.

[0092] Experimental results and evaluation:

[0093] The model training was performed on input images with a size of 256x256 pixels. The total number of iterations was set to 1000. The initial learning rate was set to 0.0001, which was verified through multiple experiments to ensure stable training while achieving faster convergence. During the training process, the Adam optimizer was used, which can adaptively adjust the learning rate to further optimize the training effect. In the training phase, the loss function was L = R + λ·D, where λ ∈ {0.00125, 0.0025, 0.0067, 0.013, 0.025, 0.05}.

[0094] For the residual space Mamba module and the mixed attention module proposed in the application, the number of layers is set to (2, 2, 9, 4). That is, in the main path encoder, three nonlinear transformation modules are stacked in sequence for 2, 2, and 9 layers, and in the hyper-prior path encoder, one nonlinear transformation module is stacked for 4 layers. For the channel number, it is set to (C1, C2, C3, C4, C5, C6) = (128, 128, 128, 320, 128, 192). That is, in the main path decoder, three nonlinear transformation modules are stacked in sequence for 2, 2, and 9 layers, and in the hyper-prior path decoder, one nonlinear transformation module is stacked for 4 layers, and the output channel numbers of the four convolutional downsampling blocks are (C1, C2, C3, C4) = (128, 128, 128, 320), and the output channel numbers of the two convolutional downsampling blocks in the hyper-prior path encoder are (C5, C6) = (128, 192). Among them, C1, C2, and C3 are the channel dimensions in the main path, C4 = 320 is the channel dimension of the latent vector, C5 is the channel dimension of the hyper-prior path, and C6 = 192 is the channel dimension of the hyper-prior information. The number of slices of the channel autoregressive entropy model is set to S = 5. Symmetrically, in the main path decoder, the output channel numbers of the four convolutional upsampling blocks are (C’1, C’2, C’3, C’4) = (128, 128, 128, 320), and in the hyper-prior path decoder, the output channel numbers of the two convolutional upsampling blocks are (C’5, C’6) = (128, 192), wherein C’1, C’2, and C’3 are the channel dimensions in the main path, C’4 is the channel dimension of the latent vector, C’5 is the channel dimension of the hyper-prior path, and C’6 is the channel dimension of the hyper-prior information.

[0095] The training data set contains 1663 diversified scene images constructed at different resolutions. In order to increase the diversity of the data and improve the generalization ability of the model, the training set data is subjected to horizontal and vertical flip data enhancement operations.

[0096] In the model performance evaluation section, Kodak data set and Tecnic data set are selected for testing. The evaluation index is BD-Rate.

[0097] As shown in Table 1, the BD-Rate of various methods is compared with BPG as the anchor point. The proposed ReSMam and MA methods are superior to the previous methods on the two data sets. ReSMam improves the modeling of global information and local information by improving the space mamba model, and the compression effect is better than the previous methods; and the MA method further introduces the channel attention mechanism, and the effect of eliminating redundancy is better.

[0098] Table 1 Comparison of BD-Rate of various methods with BPG as anchor point

[0099]

[0100] In summary, the present application realizes efficient capture and utilization of redundant information in image latent space from two aspects of structural design and attention mechanism, bringing significant performance improvement to the field of image compression. It can be summarized as:

[0101] 1. Image coding method based on residual space Mamba architecture (ReSMamLIC): In early image compression models constructed by convolutional neural networks (CNN), due to the inherent local receptive field limitation of CNN, it is difficult for the model to capture long-distance context information in the image, and the static weight also limits the adaptive adjustment ability of the model to the input content. To solve this problem, in recent years, the Transformer architecture has attracted attention due to its global modeling capability and dynamic weight mechanism, but its computational complexity grows with the input size as O(N2), which causes a considerable computational burden in practical applications. The ReSMam method proposed in the present application preserves the adaptive global modeling advantage of Transformer while effectively capturing the local and global spatial context information of the image by introducing the designed residual space Mamba architecture, and reducing the computational complexity to linear growth while ensuring adaptive weight adjustment. This design not only achieves a breakthrough in image reconstruction quality, but also significantly improves the compression efficiency, further verifying the effectiveness and efficiency of the Mamba visual variant in the image compression task.

[0102] 2. Image coding method based on hybrid attention mechanism (MALIC): In images, there is usually a close correlation between channels, and traditional algorithms reduce redundant information by mapping pixels to the frequency domain; while modern end-to-end methods map image pixels to a high-dimensional latent space, where redundant information is not only in the spatial domain, but also widely distributed in the channel dimension. To solve this problem, the present application introduces a block-level channel attention mechanism based on the ReSMamLIC model, and further integrates it with the spatial Mamba to form a hybrid attention mechanism. This mechanism can capture global spatial information while modeling the redundant characteristics within the channel in a fine-grained manner, thereby further improving the compression capability of the latent variable space. Through the information aggregation strategy of global scanning and local refinement, the model finds a balance between efficient compression and fine reconstruction.

[0103] The two methods of the present application realize better spatial-channel modeling capability and image reconstruction quality by introducing structure-aware state space modeling and channel attention mechanism, thereby improving the feature representation capability and compression performance of the image compression system.

[0104] Nevertheless, theoretically, to achieve similar image feature modeling and nonlinear compression representation functions, the residual spatial state modeling module (ReSMam) and the mixed attention module (MA) proposed by the present application can also be implemented in other neural network structures as functionally equivalent alternatives or variations according to requirements. Such alternative structures include but are not limited to the following categories:

[0105] 1. Simplified state space modeling module: A linear state space unit with a more simple structure (such as single-channel SSM, improved S4, etc.) can be used, or one-dimensional state modeling can be introduced only in the local path, so as to reduce the computational complexity while maintaining the modeling ability of long-distance dependencies.

[0106] 2. Gated neural module similar to Mamba principle: For example, gated convolution (Gated Convolution), gated activation unit (GLU), or a module that partially introduces a selective dynamic channel control mechanism. Such structures can achieve more stable training and deployment effects while maintaining nonlinear modeling capabilities.

[0107] In practical applications, the specific module replacement method can be flexibly selected according to system resource constraints (such as parameter size, inference speed), model training stability, or precision requirements of compression tasks. However, it should be noted that although there are multiple possible replacement paths, the ReSMam and MA modules proposed by the present application significantly improve the image representation capability and compression reconstruction quality while maintaining the compactness of the model, and thus belong to the preferred implementation scheme in the current task background.

Claims

1. An image encoding and decoding method based on residual space Mamba, characterized in that, A nonlinear transformation module is inserted between adjacent downsampling operations of the encoder and between downsampling operations of the decoder in the encoding and decoding system to perform nonlinear transformation operations; The nonlinear transformation module is a residual space Mamba module, which consists of a first LayerNorm layer, a structure-aware state space module, a second LayerNorm layer, and a feedforward neural network connected sequentially, with skip residual connections at the output; the flow of the nonlinear transformation operation is as follows: The input feature X passes through the first LayerNorm layer to obtain the intermediate feature X. N ;X N After passing through the structure-aware state space module, the intermediate feature X is obtained. M ;X M Feature X is obtained after the second LayerNorm layer. L ;X L The input feature Y is then passed through a feedforward neural network and added to the input feature X to obtain the output feature Y. For the encoder, the input feature X is the output of the previous sub-sampling operation of the nonlinear transformation module, and the output feature Y is the input of the next sub-sampling operation of the nonlinear transformation module. For the decoder, the input feature X is the output of the previous upsampling operation of the nonlinear transformation module, and the output feature Y is the input of the next upsampling operation of the nonlinear transformation module.

2. The image encoding and decoding method based on residual space Mamba according to claim 1, characterized in that, The nonlinear transformation module is a hybrid attention module, which introduces a block-based channel attention mechanism into the output of the feedforward neural network of the residual space Mamba module; the flow of the nonlinear transformation operation is as follows: The input feature X passes through the first LayerNorm layer to obtain the intermediate feature X. N ;X N After passing through the structure-aware state space module, the intermediate feature X is obtained. M ;X M Feature X is obtained after the second LayerNorm layer. L ;X L After passing through a feedforward neural network, feature X is obtained. K X K The block-based channel attention mechanism module is input and added to the input feature X to obtain the output feature Y.

3. The image encoding and decoding method based on residual space Mamba according to claim 1 or 2, characterized in that, The encoder includes a main path encoder and a super-prior path encoder, and the decoder includes a main path decoder and a super-prior path decoder. The main path encoder includes four convolutional downsampling blocks and three nonlinear transformation modules, which are alternately stacked to represent the extraction structure in multiple levels. The main path decoder includes four convolutional upsampling blocks and three nonlinear transformation modules, and reconstructs image content using a structure symmetrical to the main path encoder. The super-prior path encoder includes two convolutional downsampling blocks and a nonlinear transformation module to extract the distributed prior from the latent variables of the main path encoder. The super-prior path decoder includes two convolutional upsampling blocks and a nonlinear transformation module, used to restore auxiliary information for entropy modeling.

4. The image encoding and decoding method based on residual space Mamba according to claim 3, characterized in that, In the main path encoder, the kernel size of the first convolutional downsampling block is 5 and the stride is 2; the kernel size of the remaining convolutional downsampling blocks is 3 and the stride is 2. In the main path decoder, the kernel size of the last convolutional upsampling block is 5 and the stride is 2; the kernel size of the remaining convolutional upsampling blocks is 3 and the stride is 2. In the aforementioned hyperprior path encoder, the kernel size of the convolutional upsampling block is 3, and the stride is 2. In the aforementioned hyperprior path decoder, the kernel size of the convolutional upsampling block is 3, and the stride is 2.

5. The image encoding and decoding method based on residual space Mamba according to claim 3, characterized in that, In the main path encoder, the three nonlinear transformation modules are stacked in layers 2, 2, and 9 in sequence; in the super-prior path encoder, one nonlinear transformation module is stacked in layers 4; in the main path decoder, the three nonlinear transformation modules are stacked in layers 2, 2, and 9 in sequence; in the super-prior path decoder, one nonlinear transformation module is stacked in layers 4.

6. The image encoding and decoding method based on residual space Mamba according to claim 5, characterized in that, In the main path encoder, the number of output channels of the four convolutional downsampling blocks is (C1,C2,C3,C4)=(128,128,128,320), and in the super-prior path encoder, the number of output channels of the two convolutional downsampling blocks is (C5,C6)=(128,192), where C1,C2,C3 are the channel dimensions in the main path, C4 is the channel dimension of the latent vector, C5 is the channel dimension of the super-prior path, and C6 is the channel dimension of the super-prior information. In the main path decoder, the number of output channels of the four convolutional upsampling blocks is (C'1,C'2,C'3,C'4) = (128,128,128,320), and the number of output channels of the two convolutional upsampling blocks in the advanced prior path decoder is (C5',C'6) = (128,192). Here, C'1,C'2,C'3 are the channel dimensions in the main path, C'4 is the channel dimension of the latent vector, C'5 is the channel dimension of the advanced prior path, and C'6 is the channel dimension of the advanced prior information.

7. The image encoding and decoding method based on residual space Mamba according to claim 1 or 2, characterized in that, The encoding and decoding system also includes: a channel autoregressive entropy model, two sets of quantizers, two pairs of arithmetic encoders and two pairs of arithmetic decoders; The input image x first passes through the master path encoder to extract the latent variable y to be compressed. Then, the latent variable y is input into the super-prior path encoder to obtain the side information z of the latent variable. The side information z is discretized by a set of quantizers, and then passed through a pair of arithmetic encoders and arithmetic decoders to obtain the reconstructed side information. Reconstructed edge information The latent features F of the latent variable y are obtained through the super-prior path decoder. z The latent variable y is discretized by another set of quantizers, and then passed through another pair of arithmetic encoders and arithmetic decoders to obtain the discrete latent variable. The channel autoregressive entropy model for and F z Processing yields complete reconstructed features Will Input the main path decoder to obtain the reconstructed image.

8. The image encoding and decoding method based on residual space Mamba according to claim 7, characterized in that, The channel autoregressive entropy model for and F z The processing method is as follows: First, the discrete latent variables are... The channel is uniformly divided into multiple slices, and each slice is entropy encoded sequentially. For each slice currently being encoded / decoded... Slices that have already undergone entropy encoding and decoding and latent features F z Input the channel autoregressive entropy model to obtain the distribution parameters Φ of the current slice. i and prediction error r i And perform arithmetic encoding and decoding to obtain the reconstructed slices. The encoding and decoding process for each slice is repeated until all slices are reconstructed, resulting in a complete reconstructed feature.

9. The image encoding and decoding method based on residual space Mamba according to claim 7, characterized in that, The number of slices in the channel autoregressive entropy model is set to S = 5.

Images