Neural representation video coding method based on high-frequency feature enhancement

Abstract

The invention discloses a neural representation video coding method based on high-frequency feature enhancement, and belongs to the field of deep learning and video coding. The method comprises the following steps: preprocessing video frames, constructing a data set and inputting the data set into a neural network based on high-frequency feature enhancement; in the encoding process, video frames are sampled down through a content encoder and a wavelet high-frequency encoder, and embedding features are extracted. The wavelet high-frequency encoder separates high-frequency information by using a Haar wavelet decomposition block to generate independent high-frequency features. In a decoding stage, high-frequency information is fused through a high-frequency feature modulation layer, and a reconstructed video frame is generated by using harmonic block up-sampling, so that high-frequency details are enhanced. According to the method, end-to-end optimization is carried out on reconstruction loss of a space domain and a frequency domain, and after a preset round is met, trained network weights are saved and feature embedding is carried out. By introducing high-frequency information to guide up-sampling of the decoder, the quality and compression efficiency of the reconstructed video are remarkably improved.

Description

Technical Field

[0001] The present invention relates to a neural representation video encoding method based on high-frequency feature enhancement, and belongs to the technical field of deep learning and video encoding. Background Art

[0002] Video compression is a fundamental work that people have been studying for a long time. Nowadays, with the development of the Internet, video content has become the most important component of Internet traffic. In the past, video compression was usually achieved through traditional codecs, such as H.264 and H.265. Although these technologies are widely used because they achieve acceptable trade-offs, the manual algorithms in these encoding methods limit the encoding efficiency. In recent years, with the rapid development of deep learning, many methods have achieved advanced rate-distortion performance by replacing certain modules in traditional codecs with neural networks. However, these methods have low decoding efficiency due to the complex network structure and huge amount of calculation. In addition, these methods also have domain generalization problems and poor applicability, so they have not been widely used in practice. Therefore, it is necessary to actively explore new encoding methods.

[0003] Implicit neural representation is a novel approach that trains a neural network to learn a continuous function that maps input coordinates to corresponding values. Due to its efficiency and compactness, it has been used to represent and compress video content, achieving higher decoding speeds than previous learning-based methods. In this paradigm, instead of training a codec network to encode and decode video frames, a separate network is trained for each video, and after training, the video content is saved in the network weights. The encoding process of the video is the process of training the network to overfit, and the decoding process is the forward process of the neural network. Therefore, after training, the neural network compression technology can be used to compress the video to obtain the bitstream.

[0004] Dupont et al. proposed COIN (Dupont E, Goliński A, Alizadeh M, et al. Coin: Compression with implicit neural representations [J]. arXiv preprint arXiv: 2103.03123, 2021.), in which a simple MLP is used to map spatial coordinates to color information, and the network is compressed as the image bitstream using hyperparameter search and weight quantization techniques. Chen proposed NeRV (Chen H, He B, Wang H, et al. Nerv: Neural representations for videos [J]. Advances in Neural Information Processing Systems, 2021, 34: 21557-21568.), an image-level implicit neural representation-based video compression method, which greatly improves the quality of video reconstruction and the training speed of the network by using a convolutional neural network to learn the mapping function from frame coordinates to the entire frame image. NeRV successfully demonstrated that image-level implicit neural representation can achieve compression performance comparable to that of traditional codecs. Subsequently, HNeRv (Chen H, Gwilliam M, Lim SN, et al. Hnerv: A hybrid neural representation for videos[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition. 2023: 10270-10279.) proposed a hybrid neural coding scheme, which replaced the time coordinate t as the network input by encoding content-related tiny embeddings to provide visual priors, further improving the reconstruction quality and convergence speed of the model. DNeRv (Zhao Q, Asif MS, Ma Z. Dnerv: Modeling inherent dynamics viadifference neural representation for videos[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. 2023: 2031-2040.) proposed introducing differential flow to enhance the network's ability to fit dynamic scenes in response to the problem that HNeRv has poor ability to model complex motion scenes.

[0005] The spectral bias of neural networks will lead to insufficient reconstruction of high-frequency information. In addition, most previous methods use spatial domain loss, ignoring the importance of frequency in improving image quality. This results in overly smooth video frames reconstructed by existing solutions, lacking high-frequency details, and being unfriendly to human perception. Summary of the invention

[0006] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide a neural representation video encoding method based on high-frequency feature enhancement to improve the problem that the existing implicit neural representation video compression method has insufficient ability to reconstruct high-frequency information.

[0007] Preferably, the present invention provides a neural representation video coding method based on high-frequency feature enhancement, comprising:

[0008] S1, performing a preprocessing operation on the video to be encoded to obtain a preprocessed data set including video frames;

[0009] S2, constructing a neural representation network based on high-frequency feature enhancement, the neural representation network includes a content encoder, a wavelet high-frequency encoder and a frequency-aware decoder; wherein the frequency-aware decoder includes a harmonic block for feature size upsampling and a high-frequency feature modulation layer for feature modulation fusion;

[0010] S3, input the data set into the constructed neural representation network based on high-frequency feature enhancement for training, use the content encoder to downsample the video frame to obtain content feature embedding; use the wavelet frequency decomposition block in the wavelet high-frequency encoder to extract the high-frequency information of the video frame to obtain high-frequency feature embedding;

[0011] S4, embedding the content features and the high-frequency features into the frequency-aware decoder, using the high-frequency feature modulation layer in the frequency-aware decoder for feature fusion, and then upsampling the spatial size through the harmonic block in the frequency-aware decoder to obtain a reconstructed video frame;

[0012] S5, calculating the reconstruction loss between the reconstructed video frame and the original video frame, performing end-to-end optimization on the neural representation network through the Adan optimizer, and repeating the above steps S3-S5 until the preset training rounds are met;

[0013] S6, stop optimizing the neural representation network and save the weights and feature embeddings of the trained neural representation network;

[0014] S7, pruning, quantization, and entropy coding are performed on the neural representation network and then compressed into a video bit stream.

[0015] S8, during decoding, loads the saved weights of the neural representation network, and inputs the feature embedding into the decoder for forward calculation to obtain the reconstructed video frame.

[0016] Preferably, when encoding and decoding a video frame, high-frequency features of the current video frame are additionally extracted and utilized to enhance the encoding process.

[0017] Preferably, the content encoder comprises a first content encoding stage, a second content encoding stage, a third content encoding stage, a fourth content encoding stage and a fifth content encoding stage connected in sequence, wherein the first content encoding stage comprises a first strided convolutional layer, a first normalization layer, and a first ConvNeXt block connected in sequence;

[0018] The second content encoding stage includes the second strided convolutional layer and the second ConvNeXt block connected in sequence;

[0019] The third content encoding stage includes the third strided convolutional layer and the third ConvNeXt block connected in sequence;

[0020] The fourth content encoding stage includes the fourth strided convolutional layer and the fourth ConvNeXt block connected in sequence;

[0021] The fifth content encoding stage includes the fifth strided convolutional layer and the fifth ConvNeXt block connected sequentially;

[0022] Each ConvNeXt block consists of a depth-wise separable convolutional layer, a layer normalization layer, a first fully connected layer, a Gelu activation layer, and a second fully connected layer connected in sequence.

[0023] Preferably, the constructed neural representation network based on high-frequency feature enhancement includes a wavelet high-frequency encoder, a content encoder and a decoder;

[0024] The wavelet high frequency encoder includes a first high frequency encoding stage, a second high frequency encoding stage, a third high frequency encoding stage, a fourth high frequency encoding stage and a fifth high frequency encoding stage connected in sequence, wherein the first high frequency encoding stage includes a sixth strided convolution layer, a sixth normalization layer and a sixth ConvNeXt block;

[0025] The second high frequency coding layer includes a first Haar wavelet frequency decomposition block, a seventh stride convolution layer and a seventh ConvNeXt block, the input end of the first Haar wavelet frequency decomposition block is connected to the output end of the sixth ConvNeXt block, and the output end of the first Haar wavelet frequency decomposition block is connected to the seventh stride convolution layer;

[0026] The third high frequency coding layer includes a second Haar wavelet frequency decomposition block, an eighth stride convolution layer and an eighth ConvNeXt block, an input end of the second Haar wavelet frequency decomposition block is connected to an output end of the seventh ConvNeXt block, an output end of the second Haar wavelet frequency decomposition block is connected to the eighth stride convolution layer, and an output end of the first Haar wavelet frequency decomposition block is connected to an input end of the second Haar wavelet frequency decomposition block;

[0027] The fourth high frequency coding layer includes a third Haar wavelet frequency decomposition block, a ninth strided convolution layer and a ninth ConvNeXt block, the input end of the third Haar wavelet frequency decomposition block is connected to the output end of the eighth ConvNeXt block, the output end of the third Haar wavelet frequency decomposition block is connected to the ninth strided convolution layer, and the output end of the second Haar wavelet frequency decomposition block is connected to the input end of the third Haar wavelet frequency decomposition block;

[0028] The fifth high frequency coding layer includes a fourth Haar wavelet frequency decomposition block, a tenth strided convolution layer and a tenth ConvNeXt block. The input end of the fourth Haar wavelet frequency decomposition block is connected to the output end of the ninth ConvNeXt block. The output end of the fourth Haar wavelet frequency decomposition block is connected to the tenth strided convolution layer. The output end of the third Haar wavelet frequency decomposition block is connected to the input end of the fourth Haar wavelet frequency decomposition block.

[0029] Preferably, the decoder comprises a plurality of decoding stages, each decoding stage comprising a harmonic block and a high frequency characteristic modulation layer.

[0030] Prioritizing, the high-frequency feature modulation layer achieves deep fusion of high-frequency and content feature embedding through point-by-point convolutional layers, Gelu activation functions and feature modulation operations, thereby retaining the details and textures of video frames during video reconstruction; the harmonic block includes sequentially connected convolutional layers, pixel shuffling layers and adaptive harmonic activation functions, which are used to introduce periodic deviations while performing spatial upsampling.

[0031] Preferably, a frequency loss function is used to optimize high-frequency details in the video reconstruction process, including the following steps:

[0032] a) performing fast Fourier transform on the reconstructed video frame and the video frame in the data set respectively;

[0033] b) Calculate the difference in frequency domain between the reconstructed frame and the video frame in the dataset, and use a dynamic weighting strategy to weight the frequency difference to highlight the reconstruction quality of high-frequency details;

[0034] c) Optimizing the neural representation network by combining the frequency loss function with the spatial domain loss function to improve the overall detail performance of video reconstruction.

[0035] Preferably, the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods described in the first aspect when executing the program.

[0036] Preferably, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0037] The beneficial effects achieved by the present invention are:

[0038] 1. The present invention designs a hybrid neural representation network based on high-frequency enhancement. The method extracts and utilizes high-frequency information to enhance the synthesis of details in the decoding process through a neural representation network architecture including a high-frequency feature encoder, a content encoder, and a frequency-aware decoder. Specifically, the present invention proposes a wavelet high-frequency encoder module, which uses the Haar wavelet transform to separate the high-frequency information in the video frame, and generates a high-frequency feature embedding, which is provided to the decoder for use. In addition, the present invention designs a high-frequency feature modulation layer, which uses the extracted high-frequency embedding to modulate the content features to improve the network's ability to fit high-frequency details. At the same time, an adaptive harmonic activation function is used to optimize the decoder to enhance the model's ability to capture complex textures and periodic structures. Finally, based on the dynamic weighted frequency loss function, the loss of high-frequency information is further reduced through frequency domain supervision, thereby achieving high-quality compressed video restoration. After the training is completed, the high-frequency feature embedding, content feature embedding, and decoder are used as corresponding video representations. The present invention significantly improves the quality of the reconstructed video by encoding high-frequency information as prior information to guide the decoder upsampling process.

[0039] 2. The present invention makes improvements on the network structure. Based on the Haar wavelet transform, a wavelet high-frequency encoder is designed to extract high-frequency features for embedding; a high-frequency feature modulation layer is designed to realize feature fusion interaction; an adaptive harmonic activation function is used to replace the GELU activation function in the original upsampling block to improve the ability to capture high-frequency details; and the spatial domain and frequency domain losses are combined to supervise the complex high-frequency information of the network, thereby enhancing the network's ability to fit high-frequency details. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0041] Attached Figure 1 It is the coding flow chart of the present invention.

[0042] Attached Figure 2It is a structural diagram of the neural representation network for high-frequency feature enhancement in the present invention.

[0043] Attached Figure 3 It is a structural diagram of the content encoder in the present invention.

[0044] Attached Figure 4 It is a structural diagram of the wavelet high frequency encoder of the present invention.

[0045] Attached Figure 5 It is a structural diagram of the wavelet frequency decomposition block of the present invention.

[0046] Attached Figure 6 It is a structural diagram of the high-frequency characteristic modulation layer in the present invention.

[0047] Attached Figure 7 1 is a structural diagram of a harmonic block and a harmonic block without a pixel shuffling layer in the present invention.

[0048] Attached Figure 8 It is a schematic diagram of the visualization comparison result of the reconstructed frame of the present invention.

[0049] Attached Fig. 9 It is a rate-peak signal-to-noise ratio distortion curve diagram of the present invention.

[0050] Attached Fig.10 It is a rate-multi-scale structural similarity distortion curve diagram of the present invention. DETAILED DESCRIPTION

[0051] To facilitate the technical solution of the application, some concepts involved in the application are first explained below.

[0052] Embodiment 1

[0053] A neural representation video coding method based on high-frequency feature enhancement, comprising:

[0054] S1, performing a preprocessing operation on the video to be encoded to obtain a preprocessed data set including video frames;

[0055] S2, constructing a neural representation network based on high-frequency feature enhancement, the neural representation network includes a content encoder, a wavelet high-frequency encoder and a frequency-aware decoder; wherein the frequency-aware decoder includes a harmonic block for feature size upsampling and a high-frequency feature modulation layer for feature modulation fusion;

[0056] S3, input the data set into the constructed neural representation network based on high-frequency feature enhancement for training, use the content encoder to downsample the video frame to obtain content feature embedding; use the wavelet frequency decomposition block in the wavelet high-frequency encoder to extract the high-frequency information of the video frame to obtain high-frequency feature embedding;

[0057] S4, embedding the content features and the high-frequency features into the frequency-aware decoder, using the high-frequency feature modulation layer in the frequency-aware decoder for feature fusion, and then upsampling the spatial size through the harmonic block in the frequency-aware decoder to obtain a reconstructed video frame;

[0058] S5, calculating the reconstruction loss between the reconstructed video frame and the original video frame, performing end-to-end optimization on the neural representation network through the Adan optimizer, and repeating the above steps S3-S5 until the preset training rounds are met;

[0059] S6, stop optimizing the neural representation network and save the weights and feature embeddings of the trained neural representation network;

[0060] S7, pruning, quantization, and entropy coding are performed on the neural representation network and then compressed into a video bit stream.

[0061] S8, during decoding, loads the saved weights of the neural representation network, and inputs the feature embedding into the decoder for forward calculation to obtain the reconstructed video frame.

[0062] Preferably, when encoding and decoding a video frame, high-frequency features of the current video frame are additionally extracted and utilized to enhance the encoding process.

[0063] Preferably, the constructed neural representation network based on high-frequency feature enhancement includes a wavelet high-frequency encoder, a content encoder and a decoder;

[0064] The wavelet high frequency encoder includes a first high frequency encoding stage, a second high frequency encoding stage, a third high frequency encoding stage, a fourth high frequency encoding stage and a fifth high frequency encoding stage connected in sequence, wherein the first high frequency encoding stage includes a sixth strided convolution layer, a sixth normalization layer and a sixth ConvNeXt block;

[0065] The second high frequency coding layer includes a first Haar wavelet frequency decomposition block, a seventh stride convolution layer and a seventh ConvNeXt block, the input end of the first Haar wavelet frequency decomposition block is connected to the output end of the sixth ConvNeXt block, and the output end of the first Haar wavelet frequency decomposition block is connected to the seventh stride convolution layer;

[0066] The third high frequency coding layer includes a second Haar wavelet frequency decomposition block, an eighth stride convolution layer and an eighth ConvNeXt block, an input end of the second Haar wavelet frequency decomposition block is connected to an output end of the seventh ConvNeXt block, an output end of the second Haar wavelet frequency decomposition block is connected to the eighth stride convolution layer, and an output end of the first Haar wavelet frequency decomposition block is connected to an input end of the second Haar wavelet frequency decomposition block;

[0067] The fourth high frequency coding layer includes a third Haar wavelet frequency decomposition block, a ninth strided convolution layer and a ninth ConvNeXt block, the input end of the third Haar wavelet frequency decomposition block is connected to the output end of the eighth ConvNeXt block, the output end of the third Haar wavelet frequency decomposition block is connected to the ninth strided convolution layer, and the output end of the second Haar wavelet frequency decomposition block is connected to the input end of the third Haar wavelet frequency decomposition block;

[0068] The fifth high frequency coding layer includes a fourth Haar wavelet frequency decomposition block, a tenth strided convolution layer and a tenth ConvNeXt block. The input end of the fourth Haar wavelet frequency decomposition block is connected to the output end of the ninth ConvNeXt block. The output end of the fourth Haar wavelet frequency decomposition block is connected to the tenth strided convolution layer. The output end of the third Haar wavelet frequency decomposition block is connected to the input end of the fourth Haar wavelet frequency decomposition block.

[0069] Preferably, the content encoder comprises a first content encoding stage, a second content encoding stage, a third content encoding stage, a fourth content encoding stage and a fifth content encoding stage connected in sequence, wherein the first content encoding stage comprises a first strided convolutional layer, a first normalization layer, and a first ConvNeXt block connected in sequence;

[0070] The second content encoding stage includes the second strided convolutional layer and the second ConvNeXt block connected in sequence;

[0071] The third content encoding stage includes the third strided convolutional layer and the third ConvNeXt block connected in sequence;

[0072] The fourth content encoding stage includes the fourth strided convolutional layer and the fourth ConvNeXt block connected in sequence;

[0073] The fifth content encoding stage includes the fifth strided convolutional layer and the fifth ConvNeXt block connected sequentially;

[0074] Each ConvNeXt block consists of a depth-wise separable convolutional layer, a layer normalization layer, a first fully connected layer, a Gelu activation layer, and a second fully connected layer connected in sequence.

[0075] Preferably, the decoder comprises a plurality of decoding stages, each decoding stage comprising a harmonic block and a high frequency characteristic modulation layer.

[0076] Prioritizing, the high-frequency feature modulation layer achieves deep fusion of high-frequency and content feature embedding through point-by-point convolutional layers, Gelu activation functions and feature modulation operations, thereby retaining the details and textures of video frames during video reconstruction; the harmonic block includes sequentially connected convolutional layers, pixel shuffling layers and adaptive harmonic activation functions, which are used to introduce periodic deviations while performing spatial upsampling.

[0077] Preferably, a frequency loss function is used to optimize high-frequency details in the video reconstruction process, including the following steps:

[0078] a) performing fast Fourier transform on the reconstructed video frame and the video frame in the data set respectively;

[0079] b) Calculate the difference in frequency domain between the reconstructed frame and the video frame in the dataset, and use a dynamic weighting strategy to weight the frequency difference to highlight the reconstruction quality of high-frequency details;

[0080] c) Optimizing the neural representation network by combining the frequency loss function with the spatial domain loss function to improve the overall detail performance of video reconstruction.

[0081] The present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods described in the first aspect when executing the program.

[0082] The present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any one of the methods described in the first aspect are implemented.

[0083] Embodiment 2

[0084] See also Figure 1 Process, the present application discloses a neural representation video encoding method based on high-frequency feature enhancement, the method extracts high-frequency feature embedding while extracting the content feature embedding of the current input frame, and then integrates the extracted high-frequency features into the reconstruction process of the content features at the decoding end to perform high-frequency enhancement.

[0085] To this end, the main steps of this method are as follows:

[0086] Step 1: Perform preprocessing operations such as splitting the video to be encoded into frames and resizing to obtain a preprocessed data set including video frames. Specifically, for a 1920*1080 resolution video, we first use the ffmpeg tool to split it into images frame by frame, and then perform center cropping to 1920*960 to obtain images with an aspect ratio of 2:1 as the data set;

[0087] Step 2: Construct a neural representation network based on high-frequency feature enhancement. Figure 2 As shown, the neural representation network consists of three parts: wavelet high-frequency encoder, content encoder and frequency-aware decoder.

[0088] Structurally, the content encoder and wavelet high frequency encoder include five encoding stages. Each stage of the content encoder consists of sequentially connected ConvNeXt blocks and strided convolutional layers, such as Figure 3As shown, the ConvNeXt block includes a depth-separable convolutional layer, a layer normalization layer, a fully connected layer, a GELU activation layer, and a fully connected layer connected in sequence. The input image is downsampled in each stage in turn to obtain a content feature embedding of the required size. The downsampling multiple depends on the convolutional layer step size of each stage. In this embodiment, for a 1920*960 video dataset, the convolutional layer step size of the content encoder is set to [5,4,4,3,2], that is, the input size is reduced to 1 / 5, 1 / 4, 1 / 4, 1 / 3, and 1 / 2 of the original size, respectively, to generate a final content feature embedding of 4*2*16 size, where 4 and 2 are the length and width of the feature map, respectively, and 16 is the set number of feature embedding channels.

[0089] The high-frequency encoder consists of a sequentially connected ConvNeXt block, a strided convolution layer, and a wavelet frequency decomposition block. This module preserves the high-frequency details in the video through frequency decomposition and feature extraction. Figure 5 As shown, the wavelet frequency decomposition block is composed of a Haar wavelet transform and a point-by-point convolution layer. The Haar wavelet transform converts the feature map into multiple frequency components, namely LH (vertical subband), HL (horizontal subband), HH (diagonal subband) and LL (low-frequency subband), which represent the frequency features of the image in different directions. The point-by-point convolution layer is used to convert the frequency signal into the corresponding frequency feature. In this embodiment, for a 1920*960 video data set, the convolution layer step size of the high-frequency encoder is set to [3,2,2,2,2], that is, the input size is reduced to 1 / 3, 1 / 2, 1 / 2, 1 / 2, 1 / 2, respectively, and the Haar wavelet transform reduces the input size to 1 / 2 of the original size, generating a final high-frequency feature embedding of 20*10*2 size.

[0090] The decoder includes seven stages. The first decoding stage is composed of harmonic blocks without pixel shuffling layers to expand the number of embedded channels. This stage is used to control the decoder model, that is, the size of the video representation. The larger the number of hyperparameter channels is set, the larger the model size is, the better the reconstruction quality is, and the bit stream of the final model is also correspondingly larger. In this embodiment, the model size is set to 3M, where M is the storage unit, megabytes. In order to ensure uniform distribution of network parameters, the number of channels in subsequent stages is set to 0.8 times that of the previous stage. The second to sixth decoding stages are composed of harmonic blocks and high-frequency feature modulation layers for feature fusion and upsampling. Figure 7 As shown in FIG. 1 , the harmonic block is composed of a convolution layer, a pixel shuffling layer, and an adaptive harmonic activation layer. The convolution layer is used to learn and fit image features and increase the feature channel size. The pixel shuffling layer is used to amplify the spatial resolution according to the set upsampling factor. The adaptive harmonic activation layer is used to introduce nonlinearity and additional periodic bias. In this embodiment, due to the high frequency embedding and content embedding The initial sizes of are 20*10*2 and 4*2*16 respectively, so the pixel shuffle layer upsampling factors in the second to sixth harmonic blocks are set to [1,4,4,3,2] and [5,4,4,3,2] respectively to ensure that the features of each stage keep the same size. The convolution kernel size of the harmonic block convolution layer is set to 1, 3, 3, 3, 5, 5 to ensure that there are sufficient parameters to achieve image reconstruction in the later stage. The function of the adaptive harmonic activation layer is as follows: , and is a learnable parameter. Figure 6 As shown in the figure, the high-frequency feature modulation layer consists of a point-by-point convolution layer and a Gelu activation layer. The purpose is to learn a set of affine parameters γ, β through high-frequency features to perform affine transformation on the content features to obtain enhanced features. The output layer of the last stage consists of a convolution layer with a convolution kernel size of 3 with a Than activation function to achieve the conversion from the feature domain to the pixel domain.

[0091] Step 3: Input the data set obtained in step 1 into the neural representation network constructed in step 2 for the forward process. The video frame is first downsampled by the encoder to obtain feature embedding, and then upsampled by the decoder to obtain the reconstructed video frame. Specifically, the content encoder of the network and high frequency encoder From the current frame Extract the corresponding content feature embedding and high frequency feature embedding ,

[0092] ,

[0093] ,

[0094] Specifically, for each stage of the content encoder, the input features are downsampled by strided convolution layers in turn to reduce the spatial dimension and extract local detail information of the image. The input features are then processed by the ConvNeXt block to obtain the output content feature embedding of this stage. For each stage of the high-frequency encoder, the input is processed by the ConvNeXt block and the strided convolution layer, and then enters the wavelet frequency decomposition block for high-frequency and low-frequency decomposition. Figure 5As shown in the figure, the wavelet frequency decomposition block uses Haar wavelet transform to convert the feature map into multiple frequency components, namely LH (vertical subband), HL (horizontal subband), HH (diagonal subband) and LL (low-frequency subband). The high-frequency features are obtained by accumulating the high-frequency components LH+HL+HH and then performing point-by-point convolution operations. The low-frequency components LL are convolved point-by-point to obtain low-frequency features representing smooth overall structural information. The obtained high-frequency features are then concatenated with the output features of the previous stage and further processed through a new ConvNeXt block and a strided convolution layer to generate the high-frequency feature embedding of the current stage. At the same time, the low-frequency features are directly passed to the next stage through skip connections for decomposition and processing at a higher level.

[0095] The decoder takes content feature embedding and high-frequency feature embedding as input to reconstruct the video frame. Specifically, the content embedding and high frequency embedding After the first stage, the channel dimension is expanded to obtain the initial features and Then, in the second to fifth stages, spatial upsampling is performed through harmonic blocks to obtain content features. and high frequency features ,Then After the high-frequency feature modulation layer learns a set of modulation parameters γ and β The enhanced features are obtained by feature modulation , where γ is the multiplication factor and β is the additive bias.

[0096]

[0097] In the last output stage, the conversion from feature domain to pixel domain is realized to obtain the reconstructed frame :

[0098] ,

[0099] Step 4: Compare the reconstructed video frame obtained in step 3 with the video frame and calculate the reconstruction loss; specifically, the spatial domain loss is used and frequency domain loss The combination of supervises the network, the formula is as follows:

[0100] ,

[0101] x t is the original frame, is the reconstructed frame. The dynamic weight matrix introduced allows the network to put more weight on high-frequency features that are difficult to synthesize. and is the weight factor of each loss, which can be set by the user according to the needs. and Set them to 0.7 and 100 respectively.

[0102] Step 5: Using the Adan optimizer, the neural representation network is end-to-end optimized according to the reconstruction loss obtained in step 4. In this embodiment, the learning rate is set to 0.003, and the cosine annealing algorithm is used to adjust the learning rate, and 10% of the total training rounds are used for warm-up.

[0103] Step 6: Repeat steps 3 to 5 until the preset training rounds are met, and save the weights and feature embeddings of the trained neural representation network. Since this method is a training method based on network overfitting, the more training rounds, the better the reconstruction results. Considering the time-quality trade-off, the training rounds in this embodiment are set to 300, and the user can set the training rounds as needed. When the entire training process is completed, the content encoder and high-frequency encoder will be discarded, and the decoder and the content feature embedding and high-frequency feature embedding of all frames in the video will be used as compressed data to compress and obtain a bit stream.

[0104] Step 7: After pruning, quantization, and entropy coding operations on the neural representation network, the neural representation network is compressed into a video bitstream. Specifically, the network is first subjected to a global unstructured pruning operation. Here, the global pruning importance score proposed by Jaeho Lee et al. (Lee J, Park S, Mo S, et al. Layer-adaptive sparsity for the magnitude-basedpruning[J]. arXiv preprint arXiv:2010.07611, 2020.) is used to determine the pruned elements, that is, the layer-adaptive magnitude pruning score of each weight is first calculated, which approximates the distortion of the pruned model. Then global pruning is performed based on the score. Specifically,

[0105] ,

[0106] Here θ i is the initial weight, θ l is the pruned weight, that is, if the LAMP score of the weight is higher than the set threshold, the weight is retained; otherwise, it is set to zero. It is then fine-tuned to restore the network performance. The feature embedding and weights are then quantized to 8 bits to further reduce the size. Specifically, given a parameter tensor, the quantized value of the parameter μ is given by:

[0107] ,

[0108] where "round" rounds the value to the nearest integer, s is a scaling factor, and θ minand θ max are the minimum and maximum values ​​of the parameter tensor, respectively, and b represents the bit length used to quantize the model. The parameters can be dequantized using the following formula:

[0109] ,

[0110] Finally, lossless Huffman coding is used to generate a compact bit stream.

[0111] Step 8: When decoding, first decode the feature embedding and the weight of the neural representation network, build a decoder and load the saved weight of the neural representation network, then input the embedding into the decoder for the forward process to obtain the reconstructed video frame. Since the corresponding feature embedding is saved for each frame, any frame can be decoded separately, and finally all reconstructed image frames are arranged in order and converted into video by the ffmpeg tool.

[0112] This embodiment compares the neural representation video coding method based on high-frequency feature enhancement proposed in the present invention with the recent prior art. The data set used in the experiment is the UVG data set constructed by the Finnish academic video coding group Ultra Video Group. Seven videos with a resolution of 1080*1920 are selected, and the length is 600 or 300 frames. The present invention trains a model with 3M parameters for each video separately. Peak signal-to-noise ratio (PSNR) and multi-scale structural similarity (MS-SSIM) are used as indicators to measure the video reconstruction results, and bits per pixel (BPP) are used as indicators to measure video compression performance. The comparison results of the present invention with existing methods such as NeRV, E-NeRV, HNeRV, etc. on the UVG data set are shown in Table 1. The experimental results prove that the present invention improves the performance of the model and greatly improves the image reconstruction quality. We also trained and compared the low-resolution version (480*960) of the UVG data set. The results in Table 2 prove the superiority and robustness of the present invention.

[0113] Table 1

[0114]

[0115] Table 2

[0116] Figure 8 The visualization results in FIG. 1 can intuitively illustrate that the image reconstructed by the present invention has better visual quality, such as clearer details such as letters, windows, and buildings; the video compression comparison results of the present invention and the existing implicit neural representation video coding method and video coding standard are shown in FIG. Fig. 9 and Fig.10As shown in Table 3, the rate-distortion curve proves that the present invention is superior to the traditional HEVC method and other implicit neural representation-based video coding methods, where HEVC is implemented using the ffmpeg tool, the slow preset, and no B frame. To prove the effectiveness of the module design introduced in the scheme of the present invention, an ablation experiment is set in this embodiment, as shown in Table 3.

[0117] Table 3

[0118]

[0119] The present invention generates six variants, among which variant 1 is the test result of removing the high-frequency embedding branch; variant 2 is the test result of replacing the wavelet high-frequency encoder with the content encoder to verify the effectiveness of the proposed wavelet high-frequency encoder; variants 3 and 4 are the test results of replacing the activation function in the harmonic block with the commonly used Siren activation function and the commonly used activation GELU function to verify the effectiveness of the harmonic block proposed by the present invention; variants 5 and 6 are the test results of replacing the loss function adopted by the present invention with L1+SSIM and L2 to verify the effectiveness of the frequency domain and spatial domain combined loss adopted by the present invention. In summary, the present invention achieves better performance than other methods, especially in detail texture protection. Table 1 and Figure 7 Both the quantitative and qualitative results demonstrate the superiority of the scheme proposed in this invention. Figure 8 and Fig. 9 The rate-distortion curve shows that the compression performance of the present invention is better than the existing method and better than the traditional coding scheme HEVC.

[0120] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0121] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention invented herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not invented by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

[0122] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above are only specific implementation methods of the present application and are not used to limit the protection scope of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present application should be included in the protection scope of the present application.

Claims

1. A neural representation video coding method based on high-frequency feature enhancement, characterized in that: include: S1, performing a preprocessing operation on the video to be encoded to obtain a preprocessed data set including video frames; S2, constructing a neural representation network based on high-frequency feature enhancement, the neural representation network includes a content encoder, a wavelet high-frequency encoder and a frequency-aware decoder; wherein the frequency-aware decoder includes a harmonic block for feature size upsampling and a high-frequency feature modulation layer for feature modulation fusion; S3, input the data set into the constructed neural representation network based on high-frequency feature enhancement for training, use the content encoder to downsample the video frame to obtain content feature embedding; use the wavelet frequency decomposition block in the wavelet high-frequency encoder to extract the high-frequency information of the video frame to obtain high-frequency feature embedding; S4, embedding the content features and the high-frequency features into the frequency-aware decoder, using the high-frequency feature modulation layer in the frequency-aware decoder for feature fusion, and then upsampling the spatial size through the harmonic block in the frequency-aware decoder to obtain a reconstructed video frame; S5, calculating the reconstruction loss between the reconstructed video frame and the original video frame, performing end-to-end optimization on the neural representation network through the Adan optimizer, and repeating the above steps S3-S5 until the preset training rounds are met; S6, stop optimizing the neural representation network and save the weights and feature embeddings of the trained neural representation network; S7, pruning, quantizing, and entropy coding the neural representation network and then compressing it into a video bit stream; S8, during decoding, loads the saved weights of the neural representation network, and inputs the feature embedding into the decoder for forward calculation to obtain the reconstructed video frame.

2. The neural representation video coding method based on high-frequency feature enhancement according to claim 1, characterized in that: When encoding and decoding video frames, high-frequency features of the current video frame are additionally extracted and utilized to enhance the encoding process.

3. The neural representation video coding method based on high-frequency feature enhancement according to claim 1, characterized in that: The content encoder includes a first content encoding stage, a second content encoding stage, a third content encoding stage, a fourth content encoding stage, and a fifth content encoding stage connected in sequence, wherein the first content encoding stage includes a first strided convolutional layer, a first normalization layer, and a first ConvNeXt block connected in sequence; The second content encoding stage includes the second strided convolutional layer and the second ConvNeXt block connected in sequence; The third content encoding stage includes the third strided convolutional layer and the third ConvNeXt block connected in sequence; The fourth content encoding stage includes the fourth strided convolutional layer and the fourth ConvNeXt block connected in sequence; The fifth content encoding stage includes the fifth strided convolutional layer and the fifth ConvNeXt block connected sequentially; Each ConvNeXt block consists of a depth-wise separable convolutional layer, a layer normalization layer, a first fully connected layer, a Gelu activation layer, and a second fully connected layer connected in sequence.

4. The neural representation video coding method based on high-frequency feature enhancement according to claim 1, characterized in that: The constructed neural representation network based on high-frequency feature enhancement includes wavelet high-frequency encoder, content encoder and decoder; The wavelet high frequency encoder includes a first high frequency encoding stage, a second high frequency encoding stage, a third high frequency encoding stage, a fourth high frequency encoding stage and a fifth high frequency encoding stage connected in sequence, wherein the first high frequency encoding stage includes a sixth strided convolution layer, a sixth normalization layer and a sixth ConvNeXt block; The second high frequency coding layer includes a first Haar wavelet frequency decomposition block, a seventh stride convolution layer and a seventh ConvNeXt block, the input end of the first Haar wavelet frequency decomposition block is connected to the output end of the sixth ConvNeXt block, and the output end of the first Haar wavelet frequency decomposition block is connected to the seventh stride convolution layer; The third high frequency coding layer includes a second Haar wavelet frequency decomposition block, an eighth stride convolution layer and an eighth ConvNeXt block, an input end of the second Haar wavelet frequency decomposition block is connected to an output end of the seventh ConvNeXt block, an output end of the second Haar wavelet frequency decomposition block is connected to the eighth stride convolution layer, and an output end of the first Haar wavelet frequency decomposition block is connected to an input end of the second Haar wavelet frequency decomposition block; The fourth high frequency coding layer includes a third Haar wavelet frequency decomposition block, a ninth strided convolution layer and a ninth ConvNeXt block, the input end of the third Haar wavelet frequency decomposition block is connected to the output end of the eighth ConvNeXt block, the output end of the third Haar wavelet frequency decomposition block is connected to the ninth strided convolution layer, and the output end of the second Haar wavelet frequency decomposition block is connected to the input end of the third Haar wavelet frequency decomposition block; The fifth high frequency coding layer includes a fourth Haar wavelet frequency decomposition block, a tenth strided convolution layer and a tenth ConvNeXt block. The input end of the fourth Haar wavelet frequency decomposition block is connected to the output end of the ninth ConvNeXt block. The output end of the fourth Haar wavelet frequency decomposition block is connected to the tenth strided convolution layer. The output end of the third Haar wavelet frequency decomposition block is connected to the input end of the fourth Haar wavelet frequency decomposition block.

5. The neural representation video coding method based on high-frequency feature enhancement according to claim 1, characterized in that: The decoder includes multiple decoding stages, each of which includes a harmonic block and a high frequency feature modulation layer.

6. The neural representation video coding method based on high-frequency feature enhancement according to claim 6 is characterized in that: The high-frequency feature modulation layer achieves deep fusion of high-frequency and content feature embedding through point-by-point convolutional layers, Gelu activation functions and feature modulation operations, thereby retaining the details and textures of the video frames during video reconstruction; the harmonic block includes sequentially connected convolutional layers, pixel shuffling layers and adaptive harmonic activation functions, which are used to introduce periodic deviations while performing spatial upsampling.

7. The neural representation video coding method based on high-frequency feature enhancement according to claim 1, characterized in that: A frequency loss function is used to optimize high-frequency details: a) Reconstruct and perform Fourier transform on the original video frame; b) Calculate frequency domain differences and dynamically weight them to highlight high-frequency details; c) Combine frequency domain and spatial domain losses to optimize the neural network and improve detail performance.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images