Hybrid neural representation video compression method based on spatial-temporal characteristic modulation

Abstract

The invention discloses a hybrid neural representation video compression method based on spatial-temporal characteristic modulation, and belongs to the technical field of video compression. The method employs an encoder-decoder framework to map video content into compact neural network parameters. The core of the method is that a dual modulation mechanism of time sequence perception affine transformation (STAT) and wavelet space attention (WSA) is introduced into a decoder. The STAT module generates affine parameters through time sequence embedding, channel-level modulation is carried out on features, and accurate time sequence alignment is achieved; the WSA module decomposes the feature map through two-dimensional discrete wavelet transform, and generates a spatial attention map based on the high-frequency component to adaptively enhance spatial high-frequency details. The two modules work cooperatively, the problem of spectrum deviation in neural representation is effectively solved, and the time sequence coherence and the space detail quality of a reconstructed video are remarkably improved while the advantage of rapid decoding is kept.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of video compression, and particularly relates to a hybrid neural representation video compression method based on spatiotemporal feature modulation. BACKGROUND

[0002] With the rapid development of the digital era, video has become the core carrier of information, and has brought great pressure on network transmission and data storage. Efficient video compression technology is the key to relieving this pressure. At present, the video compression method based on implicit neural representation is widely concerned due to its lightweight and efficient characteristics. This method maps the video content to compact neural network parameters, directly reconstructs the frame through the decoding network by inputting the frame index, and realizes a highly simplified coding and decoding process and extremely fast decoding speed.

[0003] However, the existing neural representation video compression method has an inherent "spectral bias" problem, that is, the neural network tends to preferentially learn and fit low-frequency signals, resulting in a serious loss of high-frequency details such as complex textures and sharp edges in the reconstructed video, thereby causing visual blurring. The early NeRV method uses content-independent embedding representation, and the recovery ability of details and internal generalization are limited. Although the subsequent E-NeRV decomposes the representation into spatiotemporal context, it still fails to effectively solve the problem of high-frequency information loss.

[0004] In order to break through this limitation, hybrid neural representation methods such as HNeRV are proposed, which generate content-adaptive embeddings by introducing lightweight encoders, thereby enhancing the adaptive ability to motion patterns. However, such methods still have deficiencies in the full utilization of spatiotemporal redundancy: on the one hand, they lack an effective mechanism for fine alignment and modulation of features in the temporal dimension, making it difficult to ensure temporal stability in complex motion scenarios; on the other hand, the network structure fails to explicitly guide the model to focus on and recover spatial high-frequency information, and the detail reconstruction effect for fast motion scenes is still limited. Although some studies attempt to improve quality by introducing frequency domain constraints or analyzing inter-frame differences, they fail to actively and explicitly enhance high-frequency features in the decoding process.

[0005] Therefore, the existing technology fails to well solve the balance problem between "temporal coherence" and "spatial detail fidelity" in neural representation video compression, especially "high-frequency detail recovery". A new technical solution is urgently needed to cooperatively optimize spatiotemporal features in the decoding process, thereby significantly improving the visual quality of the reconstructed video while maintaining high compression efficiency and fast decoding advantages. SUMMARY

[0006] To solve the above technical problems, the application provides a hybrid neural representation video compression method based on spatio-temporal feature modulation, which enhances the modeling ability of video spatio-temporal correlation by constructing a hybrid neural representation video compression framework, and improves the video reconstruction quality by designing a double modulation mechanism to realize feature enhancement in the spatio-temporal dimension.

[0007] To achieve the above-mentioned purpose, the application adopts the following technical solutions:

[0008] A hybrid neural representation video compression method based on spatio-temporal feature modulation, comprising:

[0009] Step 1, input the original video sequence to be compressed;

[0010] Step 2, generate the corresponding time sequence embedding according to the index of each frame of video;

[0011] Step 3: use the encoder of the hybrid neural representation video compression network to extract and downsample the features of each frame of video, fuse the time sequence embedding and the extracted current frame features to generate content-related feature embedding;

[0012] Step 4: use the decoder of the hybrid neural representation video compression network to perform multi-stage upsampling on the feature embedding to reconstruct the video frame, wherein in the upsampling process, the time sequence modulation module is used to modulate the feature embedding, and the wavelet spatial attention module is used to enhance the spatial details of the feature embedding;

[0013] Step 5: calculate the loss between the reconstructed video frame and the corresponding original video frame, and optimize the parameters of the encoder and the decoder based on the loss;

[0014] Step 6: quantize and entropy encode the optimized decoder network parameters and the feature embedding of each frame generated by the encoder to generate the final compressed code stream and output.

[0015] Further, in step 3, the encoder comprises a plurality of ConvNeXt blocks and a time sequence perception modulation module connected in sequence, the input frame is first subjected to feature extraction and down-sampling by the ConvNeXt block, and then the time sequence perception modulation module performs time sequence perception feature enhancement and fusion according to the time sequence embedding generated by the current frame index, and finally outputs a compact feature embedding.

[0016] Further, in step 4, the decoder adopts a multi-stage upsampling architecture, uses SNeRV blocks for basic upsampling in the early stage, and accesses the time sequence perception modulation module to align the time sequence features, and introduces a plurality of spatio-temporal joint modulation modules in the later stage, which integrates the wavelet spatial attention module on the basis of inheriting the time sequence modulation capability of the time sequence perception modulation module.

[0017] Further, in step 4, the time-aware modulation module and the space-time joint modulation module both include a time-aware affine transformation module, and the specific steps of the time-aware affine transformation module for time modulation of the features include:

[0018] According to the input current frame time sequence embedding, a refined time sequence feature is generated through a first convolutional layer and an activation function;

[0019] Based on the refined time sequence feature, a channel-level scaling parameter and a bias parameter are generated through a second convolutional layer;

[0020] The scaling parameter and the bias parameter are used to perform affine transformation on the input feature map, realizing time alignment and feature enhancement.

[0021] Further, the refined time sequence feature is transmitted from the time-aware affine transformation module of the current layer to the time-aware affine transformation module in the subsequent upsampling stage of the decoder, forming a hierarchical time understanding and transmission path.

[0022] Further, in the space-time joint modulation module, the specific steps of the wavelet space attention module for spatial detail enhancement of the feature embedding include:

[0023] Performing two-dimensional discrete wavelet transform on the input feature map, decomposing to obtain a low-frequency subband and three high-frequency subbands in horizontal, vertical and diagonal directions;

[0024] The three high-frequency subbands are spliced and a spatial attention map is generated through upsampling and convolution operation;

[0025] The spatial attention map is multiplied by the original input feature map to adaptively enhance the spatial high-frequency details.

[0026] Further, the specific method for generating the spatial attention map is: after upsampling the spliced high-frequency components to the same spatial size as the input feature map, the spatial attention map is calculated through a convolutional layer and a Sigmoid function.

[0027] Further, in step 5, the loss function used for loss calculation is a multi-component loss function, which is composed of the weighted sum of the following components: L1 loss in the Fourier transform frequency domain, L1 loss in the pixel spatial domain, and the difference between 1 and the multi-scale structural similarity loss.

[0028] In a second aspect, the present application provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned hybrid neural representation video compression method based on space-time feature modulation.

[0029] In a third aspect, the present application provides a computer readable storage medium having stored thereon executable instructions that, when executed by a processor, enable the processor to implement the aforementioned hybrid neural representation video compression method based on spatio-temporal feature modulation.

[0030] The present application has the following beneficial effects:

[0031] The video reconstruction quality, especially the fidelity of high-frequency details, is effectively improved. By introducing the wavelet space attention (WSA) module, the present application can explicitly decompose and enhance the high-frequency components in the feature map, guiding the network to preferentially restore the texture, edge and other detail information, thereby effectively overcoming the spectral bias of the neural network and significantly reducing the blurring phenomenon of the reconstructed video.

[0032] The continuity and stability of the video time sequence are enhanced. The spatio-temporal attention transformation (STAT) module realizes accurate feature alignment between frames by fine modulation of the channels of the features, so that the reconstructed video has smoother transitions in dynamic scenes and reduces jitter and artifacts.

[0033] An optimal balance between compression efficiency and decoding speed is achieved. The present application, based on the natural advantages of fast decoding of implicit neural representation, uses the above-mentioned spatio-temporal modulation mechanism to greatly improve the reconstruction quality with relatively small parameter overhead. Combined with model quantization compression strategy, the rate-distortion performance is better than existing methods while maintaining high compression ratio and extremely fast decoding. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 is a flowchart of the hybrid neural representation video compression method based on spatio-temporal feature modulation of the present application;

[0035] Figure 2 is a schematic diagram of the hybrid neural representation video compression network structure of the present application;

[0036] Figure 3 is a schematic diagram of the spatio-temporal attention transformation module structure of the present application;

[0037] Figure 4 is a schematic diagram of the wavelet space attention module structure of the present application;

[0038] Figure 5 is a schematic diagram of the spatio-temporal attention modulation module and the spatio-temporal joint modulation module structure of the present application;

[0039] Figure 6 is a block structure diagram of the SNeRV in the prior art. DETAILED DESCRIPTION

[0040] The present application will be further described below in conjunction with the drawings and examples.

[0041] The present application represents video content as compact neural network parameters through an encoding-decoding framework, and particularly emphasizes the full exploitation of spatio-temporal redundancy in video to achieve high-quality reconstruction at high compression ratios. The flow of the entire technical solution is shown in Figure 1 The method comprises the following steps:

[0042] Step 1: input the original video sequence to be compressed;

[0043] Step 2: according to the index of each frame of video, convert the position encoding into a high-dimensional vector, and then generate the corresponding time sequence embedding through a multi-layer perception machine to provide basic temporal context information for the encoder and decoder;

[0044] Step 3: use the encoder of the hybrid neural representation video compression network to extract and downsample features of each frame of video, fuse the time sequence embedding and the extracted current frame features through a time sequence modulation module, and generate content-related feature embedding;

[0045] Step 4: use the decoder of the hybrid neural representation video compression network to perform multi-stage upsampling on the feature embedding to reconstruct the video frame, wherein during the upsampling process, feature alignment and high-frequency detail enhancement are realized through a time sequence perception modulation module and a space-time joint modulation module;

[0046] Step 5: calculate the loss between the reconstructed video frame and the corresponding original video frame, and optimize the parameters of the encoder and the decoder based on the loss;

[0047] Step 6: quantize and entropy encode the optimized decoder network parameters and the feature embedding of each frame generated by the encoder to generate the final compressed code stream and output.

[0048] As shown in Figure 2 The hybrid neural representation video compression network structure of the present application is shown, and the entire system is composed of an encoder and a decoder. The encoder is responsible for feature extraction of the input video frame and down-sampling into a compact feature embedding. The decoder outputs high-quality video frames through multi-stage upsampling and feature modulation.

[0049] The encoder comprises a plurality of ConvNeXt blocks and a time sequence perception modulation module. The input frame is first subjected to feature extraction and down-sampling through the ConvNeXt block, while the position encoding and the multi-layer perception machine map the input frame index to the time sequence embedding, and then the time sequence perception modulation (STAT-Res) module (structure as shown in Figure 5 ) will perform feature enhancement and fusion according to the time sequence embedding, and finally output a compact feature embedding. The decoder adopts a multi-stage upsampling architecture, and in the early stage uses the SNeRV block (structure as shown inFigure 6 The system performs channel number adjustment and basic upsampling, followed by integration with a STAT-Res block for further alignment of temporal features. In the later stages, a spatiotemporal joint modulation (STAT-W-Res) module is introduced. This module integrates a wavelet spatial attention (WSA) module on top of the STAT-Res block to collaboratively achieve precise temporal modulation and enhancement of spatial high-frequency details. Finally, after multiple upsampling and modulation stages, a high-quality reconstructed video frame is obtained through the output head.

[0050] Among them, such as Figure 3 As shown, the Time-Aware Affine Transform (STAT) module is an important component of the STAT-Res and STAT-W-Res blocks. The Time-Aware Affine Transform module first uses convolutional layers (Conv) and ReLU activation functions to embed the temporal sequence of the input. Refine the processing to generate more expressive and detailed temporal features. :

[0051] ,

[0052] Subsequently, Channel-level affine parameters are generated through another convolutional layer to modulate the input features:

[0053] ,

[0054] ,

[0055] This indicates the input characteristics of the current module. This is used to output features. This affine transformation operation allows the network to dynamically adjust the feature distribution based on the temporal position of the current frame, achieving precise temporal alignment. Simultaneously, it refines the temporal features. It is passed to the subsequent STAT layer, forming a hierarchical temporal understanding system.

[0056] like Figure 4 As shown, to further enhance the model's spatial detail recovery capability, a wavelet spatial attention module (WSA) is introduced into the STAT-W-Res block. First, a two-dimensional discrete wavelet transform (DWT) is used to extract the original features... Decomposed into four frequency sub-bands :

[0057] ,

[0058] in, Low-frequency components, , , These represent the high-frequency detail components in the horizontal, vertical, and diagonal directions, respectively. To enhance high-frequency information, these high-frequency components are extracted. Further processing will be carried out.

[0059] ,

[0060] in, This indicates a channel splicing operation. It is achieved through bilinear upsampling. After restoring the resolution of high-frequency features, convolutional layers and Function generates spatial attention graph :

[0061] ,

[0062] Finally, the spatial attention map Applied to original features :

[0063] ,

[0064] The WSA layer can adaptively enhance the feature response in high-frequency regions and suppress redundant activation in smooth regions, thereby improving the model's ability to recover spatial details.

[0065] In step 5, during the training process, a multi-component loss function is used to comprehensively optimize the quality of the reconstructed video in both the time and frequency domains.

[0066] ,

[0067] in, and This is a hyperparameter used to balance the weights of the various loss terms. For L1 loss, For multi-scale structural similarity loss, FFT stands for Fast Fourier Transform. For the input frame, This is the output frame.

[0068] In step 6, the model compression stage employs a consistency entropy minimization method. This method establishes entropy model consistency between the training and inference stages using a network-free Gaussian entropy model, minimizing bitrate overhead while ensuring reconstruction quality. After model compression, the decoder parameters corresponding to the video sequence and the feature embeddings for each frame are encoded into a bitstream for efficient storage and transmission.

[0069] As shown in Table 1, by comparing the rate-distortion performance with NeRV, HNeRV, and HNeRV-boost under different model scales, the compression method of this invention achieves the best reconstruction quality at three typical parameter scales (0.75M, 1.5M, and 3M), specifically with peak signal-to-noise ratios (PSNR) of 35.68dB, 39.11dB, and 41.15dB, respectively, comprehensively outperforming existing methods. Experimental results show that the spatiotemporal feature dual modulation mechanism proposed in this invention significantly improves the reconstruction details and temporal coherence of the video without significantly increasing model complexity, verifying its comprehensive advantages in compression efficiency and visual quality.

[0070] Table 1

[0071]

[0072] In a second aspect, the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned hybrid neural representation video compression method based on spatiotemporal feature modulation.

[0073] Thirdly, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enable the processor to implement the aforementioned hybrid neural representation video compression method based on spatiotemporal feature modulation.

[0074] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A video compression method based on spatiotemporal feature modulation using hybrid neural representation, characterized in that, include: Step 1: Input the original video sequence to be compressed; Step 2: Based on the index of each video frame, generate the corresponding temporal embedding through positional encoding and multilayer perceptron; Step 3: The encoder of the hybrid neural representation video compression network extracts and downsamples features for each frame of video. The temporal embedding is fused with the extracted current frame features by the temporal-aware modulation module to generate content-related feature embeddings. Step 4: The decoder of the hybrid neural representation video compression network performs multi-stage upsampling on the feature embedding to reconstruct the video frame, wherein feature alignment and high-frequency detail enhancement are achieved through the temporal-aware modulation module and the spatiotemporal joint modulation module during the upsampling process; Step 5: Calculate the loss between the reconstructed video frame and the corresponding original video frame, and optimize the parameters of the encoder and decoder based on the loss; Step 6: Quantize and entropy encode the optimized decoder network parameters and the feature embeddings of each frame generated by the encoder to generate the final compressed bitstream and output it.

2. The video compression method based on spatiotemporal feature modulation using hybrid neural representations according to claim 1, characterized in that, In step 3, the encoder includes multiple ConvNeXt blocks and a time-aware modulation module connected in sequence. The input frame first passes through the ConvNeXt blocks for feature extraction and downsampling. Then, the time-aware modulation module performs time-aware feature enhancement and fusion based on the time embedding generated by the current frame index, and finally outputs a compact feature embedding.

3. The video compression method based on spatiotemporal feature modulation using hybrid neural representations according to claim 1, characterized in that, In step 4, the decoder adopts a multi-stage upsampling architecture. In the early stage, the SNERV block is used for basic upsampling, and the timing-aware modulation module is connected to align the timing features. In the later stage, multiple spatiotemporal joint modulation modules are introduced. The spatiotemporal joint modulation module integrates a wavelet spatial attention module on the basis of inheriting the timing modulation capability of the timing-aware modulation module.

4. The video compression method based on spatiotemporal feature modulation using hybrid neural representation according to claim 1, characterized in that, In step 4, both the time-aware modulation module and the spatiotemporal joint modulation module include a time-aware affine transformation module. The specific steps for the time-aware affine transformation module to perform time-series modulation on the features include: Based on the temporal embedding of the current frame input, refined temporal features are generated through the first convolutional layer and activation function; Based on the refined temporal features, channel-level scaling and bias parameters are generated through the second convolutional layer; The scaling and bias parameters are used to perform an affine transformation on the input feature map to achieve temporal alignment and feature enhancement.

5. The video compression method based on spatiotemporal feature modulation using hybrid neural representation according to claim 4, characterized in that, The refined temporal features are passed from the temporal-aware affine transformation module of the current layer to the temporal-aware affine transformation module in the subsequent upsampling stage of the decoder, forming a hierarchical temporal understanding and transmission path.

6. The video compression method based on spatiotemporal feature modulation using hybrid neural representation according to claim 3, characterized in that, In the spatiotemporal joint modulation module, the specific steps of the wavelet spatial attention module to enhance the spatial details of the feature embedding include: A two-dimensional discrete wavelet transform is performed on the input feature map to decompose it into low-frequency sub-bands and high-frequency sub-bands in three directions: horizontal, vertical, and diagonal. The three high-frequency subbands are spliced ​​together, and a spatial attention map is generated through upsampling and convolution operations; The spatial attention map is multiplied with the original input feature map to adaptively enhance high-frequency spatial details.

7. The video compression method based on spatiotemporal feature modulation using hybrid neural representation according to claim 6, characterized in that, The specific method for generating the spatial attention map is as follows: after upsampling the spliced ​​high-frequency components to the same spatial size as the input feature map, it is calculated through a convolutional layer and a sigmoid function.

8. The video compression method based on spatiotemporal feature modulation using hybrid neural representation according to claim 1, characterized in that, In step 5, the loss function used to calculate the loss is a multi-component loss function, which consists of a weighted sum of the following components: the L1 loss of the original frame and the reconstructed frame in the Fourier transform frequency domain, the L1 loss in the pixel spatial domain, and the difference term of 1 minus the multi-scale structural similarity loss.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When one or more programs are executed by the one or more processors, the one or more processors implement the hybrid neural representation video compression method based on spatiotemporal feature modulation as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, It stores executable instructions that, when executed by a processor, enable the processor to implement the hybrid neural representation video compression method based on spatiotemporal feature modulation as described in any one of claims 1-8.

Images