A neural representation video coding method based on high-frequency feature enhancement

Abstract

The application discloses a neural representation video coding method based on high-frequency feature enhancement, and belongs to the fields of deep learning and video coding. The method pre-processes video frames, constructs a data set and inputs a neural network based on high-frequency feature enhancement. In the coding process, the video frames are down-sampled through a content encoder and a wavelet high-frequency encoder to extract embedded features. The wavelet high-frequency encoder separates high-frequency information by using a Haar wavelet decomposition block to generate independent high-frequency features. In the decoding stage, the high-frequency information is fused through a high-frequency feature modulation layer, and a harmonic block is used for up-sampling to generate reconstructed video frames and enhance high-frequency details. The method is optimized end to end in the spatial and frequency domain reconstruction loss, and the trained network weight and feature embedding are saved after a preset round. The application significantly improves the quality and compression efficiency of the reconstructed video by introducing high-frequency information to guide the up-sampling of the decoder.

Description

TECHNICAL FIELD

[0001] The present application relates to a neural representation video coding method based on high-frequency feature enhancement, belonging to the technical field of deep learning and video coding. BACKGROUND

[0002] Video compression is a basic work that people have been studying for a long time. Nowadays, with the development of the Internet, video content has become the most important part of Internet traffic. In the past, video compression was usually achieved through traditional codecs, such as H.264, H.265. Although these technologies are widely used because of the acceptable trade-off, the hand-crafted algorithms in these coding methods limit the coding efficiency. In recent years, with the rapid development of deep learning, many methods use neural networks to replace some modules in traditional codecs to achieve more advanced rate-distortion performance. However, these methods have low decoding efficiency due to the complex network structure and huge amount of calculation, in addition, these methods also have the problem of domain generalization, and the applicability is poor, so they have not been widely used in practice, so it is necessary to actively explore new coding methods.

[0003] Implicit neural representation is a novel method that trains a neural network to learn a continuous function to realize the mapping of input coordinates to corresponding values. Due to its efficiency and compactness, it has been used to represent and compress video content, and has achieved higher decoding speed compared with previous learning-based methods. Under this paradigm, instead of training a codec network to realize the coding of video frames, a separate network is trained for each video, and after the training is completed, the video content is saved to the network weights. The coding process of the video is the process of training the network to overfit, and the decoding process is the forward process of the neural network, so when the training is completed, the neural network compression technology can be used for compression to obtain the bit stream of the video.

[0004] Dupont et al. proposed COIN (Dupont E, Goliński A, Alizadeh M, et al. Coin: Compression with implicit neural representations [J]. arXiv preprint arXiv:2103.03123, 2021.) which uses a simple MLP to map spatial coordinates to color information and uses hyperparameter search and weight quantization techniques to compress the network as the code stream of the image. Chen proposed a video compression method based on image-level implicit neural representation NeRV (Chen H, He B, Wang H, et al. Nerv: Neural representations for videos [J]. Advances in Neural Information Processing Systems, 2021, 34: 21557-21568.) which greatly improves the reconstruction quality of the video and the training speed of the network by using a convolutional neural network to learn the mapping function from frame coordinates to the whole frame image. NeRV successfully proves that image-level implicit neural representation can achieve comparable compression performance with traditional codecs. Subsequently, HNeRv (Chen H, Gwilliam M, Lim S N, et al. Hnerv: A hybrid neural representation for videos [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023: 10270-10279.) proposes a hybrid neural coding scheme, which replaces the time coordinate t as the network input by encoding the content-related micro-embedding to provide visual priors, further improving the reconstruction quality and convergence speed of the model. DNeRv (Zhao Q, Asif M S, Ma Z. Dnerv: Modeling inherent dynamics via difference neural representation for videos [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023: 2031-2040.) addresses the problem of poor modeling of complex motion scenes in HNeRv by introducing a difference flow enhancement network to fit the ability of dynamic scenes.

[0005] Due to the spectral bias property of the neural network, the reconstruction ability of the network for high frequency information is insufficient, and in addition, the past method mostly uses the spatial domain loss, ignoring the importance of frequency to improve the image quality. This leads to the existing scheme to be over-smoothed and lack of high frequency details, which is not friendly to human perception. SUMMARY

[0006] The technical problem to be solved by the present application is to overcome the defects of the prior art and provide a neural representation video coding method based on high frequency feature enhancement to improve the problem of insufficient reconstruction of high frequency information in the existing implicit neural representation video compression method.

[0007] Preferably, the present application provides a neural representation video coding method based on high frequency feature enhancement, comprising:

[0008] S1, pre-processing the video to be encoded to obtain a pre-processed data set including video frames;

[0009] S2, constructing a neural representation network based on high frequency feature enhancement, the neural representation network comprising a content encoder, a wavelet high frequency encoder and a frequency-aware decoder; wherein the frequency-aware decoder comprises a harmonic block for feature size upsampling and a high frequency feature modulation layer for feature modulation fusion;

[0010] S3, inputting the data set into the constructed neural representation network based on high frequency feature enhancement for training, using the content encoder to downsample the video frames to obtain content feature embedding; using the wavelet frequency decomposition block in the wavelet high frequency encoder to extract the high frequency information of the video frames to obtain high frequency feature embedding;

[0011] S4, inputting the content feature embedding and the high frequency feature embedding into the frequency-aware decoder, using the high frequency feature modulation layer in the frequency-aware decoder for feature fusion, and then performing spatial size upsampling through the harmonic block in the frequency-aware decoder to obtain the reconstructed video frames;

[0012] S5, calculating the reconstruction loss between the reconstructed video frames and the original video frames, optimizing the neural representation network end-to-end through an Adan optimizer, and repeating the above steps S3-S5 until the preset training round is met;

[0013] S6, stopping optimizing the neural representation network, and saving the weights and feature embedding of the trained neural representation network;

[0014] S7, pruning, quantizing and entropy encoding the neural representation network to compress it into a video bit stream.

[0015] S8, in decoding, load the saved weight of the neural representation network, input the feature embedding into the decoder for forward calculation to obtain the reconstructed video frame.

[0016] Preferably, the high-frequency features of the current video frame are additionally extracted and utilized to enhance the encoding process when encoding and decoding the video frame.

[0017] Preferably, the content encoder comprises a first content encoding stage, a second content encoding stage, a third content encoding stage, a fourth content encoding stage and a fifth content encoding stage connected in sequence, wherein the first content encoding stage comprises a first stride convolution layer, a first layer normalization layer and a first ConvNeXt block connected in sequence;

[0018] The second content encoding stage comprises a second stride convolution layer and a second ConvNeXt block connected in sequence;

[0019] The third content encoding stage comprises a third stride convolution layer and a third ConvNeXt block connected in sequence;

[0020] The fourth content encoding stage comprises a fourth stride convolution layer and a fourth ConvNeXt block connected in sequence;

[0021] The fifth content encoding stage comprises a fifth stride convolution layer and a fifth ConvNeXt block connected in sequence;

[0022] Each ConvNeXt block comprises a depth separable convolution layer, a layer normalization layer, a first full connection layer, a Gelu activation layer and a second full connection layer connected in sequence.

[0023] Preferably, the neural representation network based on high-frequency feature enhancement comprises a wavelet high-frequency encoder, a content encoder and a decoder;

[0024] The wavelet high-frequency encoder comprises a first high-frequency encoding stage, a second high-frequency encoding stage, a third high-frequency encoding stage, a fourth high-frequency encoding stage and a fifth high-frequency encoding stage connected in sequence, wherein the first high-frequency encoding stage comprises a sixth stride convolution layer, a sixth layer normalization layer and a sixth ConvNeXt block;

[0025] The second high-frequency encoding layer comprises a first Haar wavelet frequency decomposition block, a seventh stride convolution layer and a seventh ConvNeXt block, the input end of the first Haar wavelet frequency decomposition block is connected to the output end of the sixth ConvNeXt block, and the output end of the first Haar wavelet frequency decomposition block is connected to the seventh stride convolution layer.

[0026] The third high-frequency coding layer comprises a second Haar wavelet frequency decomposition block, an eighth stride convolution layer and an eighth ConvNeXt block, the input end of the second Haar wavelet frequency decomposition block is connected to the output end of the seventh ConvNeXt block, the output end of the second Haar wavelet frequency decomposition block is connected to the eighth stride convolution layer, and the output end of the first Haar wavelet frequency decomposition block is connected to the input end of the second Haar wavelet frequency decomposition block.

[0027] The fourth high-frequency coding layer comprises a third Haar wavelet frequency decomposition block, a ninth stride convolution layer and a ninth ConvNeXt block, the input end of the third Haar wavelet frequency decomposition block is connected to the output end of the eighth ConvNeXt block, the output end of the third Haar wavelet frequency decomposition block is connected to the ninth stride convolution layer, and the output end of the second Haar wavelet frequency decomposition block is connected to the input end of the third Haar wavelet frequency decomposition block.

[0028] The fifth high-frequency coding layer comprises a fourth Haar wavelet frequency decomposition block, a tenth stride convolution layer and a tenth ConvNeXt block, the input end of the fourth Haar wavelet frequency decomposition block is connected to the output end of the ninth ConvNeXt block, the output end of the fourth Haar wavelet frequency decomposition block is connected to the tenth stride convolution layer, and the output end of the third Haar wavelet frequency decomposition block is connected to the input end of the fourth Haar wavelet frequency decomposition block.

[0029] Preferably, the decoder comprises a plurality of decoding stages, each decoding stage comprising a harmonic block and a high-frequency feature modulation layer.

[0030] Preferably, the high-frequency feature modulation layer realizes the deep fusion of high frequencies and content feature embedding through a point-by-point convolution layer, a Gelu activation function and a feature modulation operation, so as to retain the details and textures of the video frames in the process of reconstructing the video; the harmonic block comprises a convolution layer, a pixel shuffle layer and an adaptive harmonic activation function connected in sequence, for introducing periodic bias while performing spatial upsampling.

[0031] Preferably, a frequency loss function is used to optimize the high-frequency details in the process of reconstructing the video, comprising the following steps:

[0032] a) performing fast Fourier transform on the reconstructed video frames and the video frames in the data set, respectively;

[0033] b) calculating the difference in the frequency domain between the reconstructed frames and the video frames in the data set, and using a dynamic weighting strategy to weight the frequency difference to highlight the reconstruction quality of the high-frequency details;

[0034] c) optimizing the neural representation network by combining the frequency loss function with the spatial domain loss function to improve the overall detail performance of video reconstruction.

[0035] Preferably, the present application provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the method of any of the first aspect when executing the program.

[0036] Preferably, the present application provides a computer readable storage medium having stored thereon a computer program, the computer program implementing the steps of the method of any of the first aspect when executed by a processor.

[0037] The present application achieves the following beneficial effects:

[0038] 1、The present application designs a hybrid neural representation network based on high-frequency enhancement, which extracts and utilizes high-frequency information to enhance the detail synthesis in the decoding process through a neural representation network architecture containing a high-frequency feature encoder, a content encoder and a frequency-aware decoder. Specifically, the present application proposes a wavelet high-frequency encoder module, which separates the high-frequency information in the video frame by using Haar wavelet transform and generates high-frequency feature embedding for the decoder. In addition, the present application designs a high-frequency feature modulation layer to modulate the content features using the extracted high-frequency embedding to improve the network's fitting ability for high-frequency details. At the same time, an adaptive harmonic activation function is used to optimize the decoder to enhance the model's ability to capture complex textures and periodic structures. Finally, based on a dynamic weighted frequency loss function, the loss of high-frequency information is further reduced through frequency domain supervision, thereby realizing high-quality compressed video restoration. After training, the high-frequency feature embedding, the content feature embedding and the decoder serve as the corresponding video representation, and the present application guides the upsampling process of the decoder by encoding the high-frequency information as prior information, significantly improving the quality of the reconstructed video.

[0039] 2、The present application improves the network structure, designs a wavelet high-frequency encoder based on Haar wavelet transform to extract high-frequency feature embedding, designs a high-frequency feature modulation layer to realize the fusion and interaction of features, uses an adaptive harmonic activation function to replace the original GELU activation function in the upsampling block to improve the ability to capture high-frequency details, and combines spatial domain and frequency domain loss to supervise the network's complex high-frequency information, thereby enhancing the network's ability to fit high-frequency details. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed in the embodiments. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0041] FIG. 1 is a flowchart of the encoding process of the present application. Figure 1

[0042] FIG. 2 is a flowchart of the decoding process of the present application. Figure 2 ​is a structural diagram of a high-frequency feature enhanced neural representation network in the application.

[0043] attached Figure 3 is a structural diagram of a content encoder in the application.

[0044] attached Figure 4 is a structural diagram of a wavelet high-frequency encoder in the application.

[0045] attached Figure 5 is a structural diagram of a wavelet frequency decomposition block in the application.

[0046] attached Figure 6 is a structural diagram of a high-frequency feature modulation layer in the application.

[0047] attached Figure 7 is a structural diagram of a harmonic block and a harmonic block without pixel shuffle layer in the application.

[0048] attached Figure 8 is a schematic diagram of a reconstructed frame visualization comparison result in the application.

[0049] attached Figure 9 is a rate-peak signal-to-noise ratio distortion curve diagram of the application.

[0050] attached Figure 10 is a rate-multiscale structural similarity distortion curve diagram of the application. DETAILED DESCRIPTION

[0051] In order to facilitate the technical solutions of the application, the following first explains some concepts related to the application.

[0052] Embodiment one

[0053] A high-frequency feature enhanced neural representation video coding method, comprising:

[0054] S1, pre-processing the video to be coded to obtain a pre-processed data set including video frames;

[0055] S2, constructing a high-frequency feature enhanced neural representation network, the neural representation network including a content encoder, a wavelet high-frequency encoder and a frequency-aware decoder; wherein the frequency-aware decoder includes a harmonic block for feature size upsampling and a high-frequency feature modulation layer for feature modulation fusion;

[0056] S3, inputting the data set into the constructed high-frequency feature enhanced neural representation network for training, using the content encoder to downsample the video frames to obtain content feature embedding; using the wavelet frequency decomposition block in the wavelet high-frequency encoder to extract high-frequency information of the video frames to obtain high-frequency feature embedding;

[0057] S4, input the content feature embedding and the high frequency feature embedding into a frequency-aware decoder, perform feature fusion by using a high frequency feature modulation layer in the frequency-aware decoder, then perform spatial size upsampling by a harmonic block in the frequency-aware decoder to obtain a reconstructed video frame;

[0058] S5, calculate a reconstruction loss between the reconstructed video frame and the original video frame, optimize the neural representation network by an Adan optimizer in an end-to-end manner, and repeat the steps S3-S5 until a preset training round is met;

[0059] S6, stop optimizing the neural representation network, and save the weight and the feature embedding of the trained neural representation network;

[0060] S7, perform pruning, quantization and entropy encoding operations on the neural representation network, and compress the neural representation network into a video bitstream.

[0061] S8, during decoding, load the saved weight of the neural representation network, input the feature embedding into the decoder for forward calculation to obtain a reconstructed video frame.

[0062] Preferably, during encoding and decoding of the video frame, the high frequency feature of the current video frame is additionally extracted and utilized to enhance the encoding process.

[0063] Preferably, the constructed neural representation network based on high frequency feature enhancement comprises a wavelet high frequency encoder, a content encoder and a decoder.

[0064] The wavelet high frequency encoder comprises a first high frequency encoding stage, a second high frequency encoding stage, a third high frequency encoding stage, a fourth high frequency encoding stage and a fifth high frequency encoding stage connected in sequence, wherein the first high frequency encoding stage comprises a sixth stride convolution layer, a sixth normalization layer and a sixth ConvNeXt block.

[0065] The second high frequency encoding layer comprises a first Haar wavelet frequency decomposition block, a seventh stride convolution layer and a seventh ConvNeXt block, the input end of the first Haar wavelet frequency decomposition block is connected to the output end of the sixth ConvNeXt block, and the output end of the first Haar wavelet frequency decomposition block is connected to the seventh stride convolution layer.

[0066] The third high frequency encoding layer comprises a second Haar wavelet frequency decomposition block, an eighth stride convolution layer and an eighth ConvNeXt block, the input end of the second Haar wavelet frequency decomposition block is connected to the output end of the seventh ConvNeXt block, the output end of the second Haar wavelet frequency decomposition block is connected to the eighth stride convolution layer, and the output end of the first Haar wavelet frequency decomposition block is connected to the input end of the second Haar wavelet frequency decomposition block.

[0067] The fourth high-frequency coding layer comprises a third Haar wavelet frequency decomposition block, a ninth stride convolution layer and a ninth ConvNeXt block, the input end of the third Haar wavelet frequency decomposition block is connected to the output end of the eighth ConvNeXt block, the output end of the third Haar wavelet frequency decomposition block is connected to the ninth stride convolution layer, and the output end of the second Haar wavelet frequency decomposition block is connected to the input end of the third Haar wavelet frequency decomposition block.

[0068] The fifth high-frequency coding layer comprises a fourth Haar wavelet frequency decomposition block, a tenth stride convolution layer and a tenth ConvNeXt block, the input end of the fourth Haar wavelet frequency decomposition block is connected to the output end of the ninth ConvNeXt block, the output end of the fourth Haar wavelet frequency decomposition block is connected to the tenth stride convolution layer, and the output end of the third Haar wavelet frequency decomposition block is connected to the input end of the fourth Haar wavelet frequency decomposition block.

[0069] Preferably, the content encoder comprises a first content encoding stage, a second content encoding stage, a third content encoding stage, a fourth content encoding stage and a fifth content encoding stage connected in sequence, wherein the first content encoding stage comprises a first stride convolution layer, a first layer normalization layer and a first ConvNeXt block connected in sequence;

[0070] The second content encoding stage comprises a second stride convolution layer and a second ConvNeXt block connected in sequence;

[0071] The third content encoding stage comprises a third stride convolution layer and a third ConvNeXt block connected in sequence;

[0072] The fourth content encoding stage comprises a fourth stride convolution layer and a fourth ConvNeXt block connected in sequence;

[0073] The fifth content encoding stage comprises a fifth stride convolution layer and a fifth ConvNeXt block connected in sequence;

[0074] Each ConvNeXt block comprises a depth separable convolution layer, a layer normalization layer, a first full connection layer, a Gelu activation layer and a second full connection layer connected in sequence.

[0075] Preferably, the decoder comprises a plurality of decoding stages, each decoding stage comprising a harmonic block and a high-frequency feature modulation layer.

[0076] Preferably, the high-frequency feature modulation layer realizes deep fusion of high-frequency and content feature embedding through a pointwise convolution layer, a Gelu activation function and a feature modulation operation, so as to preserve the details and textures of the video frames in the process of reconstructing the video; the harmonic block comprises a convolution layer, a pixel shuffle layer and an adaptive harmonic activation function connected in sequence, for introducing periodic bias while performing spatial upsampling.

[0077] Preferably, a frequency loss function is employed to optimize the high frequency details in the process of video reconstruction, including the following steps:

[0078] a) Fast Fourier transform is performed on the reconstructed video frames and the video frames in the dataset, respectively;

[0079] b) The difference in the frequency domain between the reconstructed frames and the video frames in the dataset is calculated, and a dynamic weighting strategy is used to weight the frequency difference to highlight the reconstruction quality of high frequency details;

[0080] c) The neural representation network is optimized by combining the frequency loss function with the spatial domain loss function to improve the overall detail performance of video reconstruction.

[0081] The present application provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the method of any one of the first aspect when executing the program.

[0082] The present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the method of any one of the first aspect.

[0083] Embodiment two

[0084] Referring to Figure 1 the flow, the present application discloses a neural representation video coding method based on high frequency feature enhancement, which extracts high frequency feature embedding while extracting current input frame content feature embedding, and then fuses the extracted high frequency feature into the reconstruction process of content feature at the decoding end for high frequency enhancement.

[0085] To this end, the main steps of the method are as follows:

[0086] Step 1: Preprocessing operations such as frame splitting and size adjustment are performed on the video to be coded to obtain a preprocessed dataset including video frames. Specifically, for a 1920*1080 resolution video, we first use the ffmpeg tool to split it into images frame by frame, and then center crop each image to 1920*960 to obtain images with an aspect ratio of 2:1 as the dataset;

[0087] Step 2: Construct a neural representation network based on high frequency feature enhancement. As shown in Figure 2 , the neural representation network includes a wavelet high frequency encoder, a content encoder and a frequency-aware decoder.

[0088] In terms of structure, the content encoder and the wavelet high frequency encoder include five encoding stages, and each stage of the content encoder includes a ConvNeXt block and a stride convolution layer connected in sequence, as shown in Figure 3As shown, the ConvNeXt block includes a depth separable convolution layer, a layer normalization layer, a full connection layer, a GELU activation layer, and a full connection layer connected in sequence. The input image is sequentially subjected to each stage to obtain a content feature embedding of a required size. The down-sampling multiple depends on the convolution layer step length of each stage. In this embodiment, for a video data set of 1920*960, the convolution layer step length of the content encoder is set to [5, 4, 4, 3, 2], that is, the size of the input is reduced to 1 / 5, 1 / 4, 1 / 4, 1 / 3, and 1 / 2, respectively, to generate a final content feature embedding of 4*2*16 size, wherein 4 and 2 are the length and width of the feature map, respectively, and 16 is the set number of feature embedding channels.

[0089] The high-frequency encoder is composed of a ConvNeXt block, a stride convolution layer, and a wavelet frequency decomposition block connected in sequence. The module retains the high-frequency details in the video through frequency decomposition and feature extraction. Figure 5 As shown, the wavelet frequency decomposition block is composed of a Haar wavelet transform and a point-by-point convolution layer. The Haar wavelet transform converts the feature map into multiple frequency components, namely LH (vertical subband), HL (horizontal subband), HH (diagonal subband), and LL (low-frequency subband), which represent the frequency characteristics of the image in different directions, respectively. The point-by-point convolution layer is used to convert the frequency signal into the corresponding frequency feature. In this embodiment, for a video data set of 1920*960, the convolution layer step length of the high-frequency encoder is set to [3, 2, 2, 2, 2], that is, the size of the input is reduced to 1 / 3, 1 / 2, 1 / 2, 1 / 2, and 1 / 2, respectively, and the Haar wavelet transform reduces the size of the input to 1 / 2, to generate a final high-frequency feature embedding of 20*10*2 size.

[0090] The decoder includes seven stages. The first decoding stage is composed of a harmonic block without a pixel shuffle layer to expand the number of embedded channels. This stage is used to control the size of the decoder model, that is, the video representation. The larger the hyperparameter channel number is set, the larger the model size is, the better the reconstruction quality is, and the larger the final model bit stream is. In this embodiment, the model size is set to 3M, where M is a storage measurement unit, megabyte. In order to ensure the uniform distribution of network parameters, the channel number of the subsequent stage is set to 0.8 times that of the previous stage. The second to sixth decoding stages are composed of a harmonic block and a high-frequency feature modulation layer to perform feature fusion and up-sampling. Figure 7 As shown, the harmonic block is composed of a convolution layer, a pixel shuffle layer, and an adaptive harmonic activation layer. The convolution layer is used to learn and fit image features and increase the feature channel size. The pixel shuffle layer is used to enlarge the spatial resolution according to the set up-sampling factor. The adaptive harmonic activation layer is used to introduce nonlinearity while additionally introducing a periodic bias. In this embodiment, since the high-frequency embedding and the content embedding The initial sizes of the pixel shuffle layers in the second to sixth harmonic blocks are set to [1, 4, 4, 3, 2] and [5, 4, 4, 3, 2] respectively to ensure that the features remain the same size at each stage. The convolution kernel size of the harmonic block convolution layer is set to 1, 3, 3, 3, 5, 5 to ensure that there are sufficient parameters to implement image reconstruction at the later stages. The function of the adaptive harmonic activation layer is as follows: , and are learnable parameters. As shown in Figure 6 , the high-frequency feature modulation layer is composed of a point-by-point convolution layer and a Gelu activation layer. It aims to learn a set of affine parameters γ, β through high-frequency features to perform affine transformation on the content features to obtain enhanced features. The last stage output layer is composed of a convolution layer with a convolution kernel size of 3 with a Than activation function to realize the conversion of the feature domain to the pixel domain.

[0091] Step 3: Input the dataset obtained in step 1 into the neural representation network constructed in step 2 for a forward process. The video frame is first down-sampled by the encoder to obtain the feature embedding, and then up-sampled by the decoder to obtain the reconstructed video frame. Specifically, the content encoder and the high-frequency encoder extract the corresponding content feature embedding and high-frequency feature embedding from the current frame ,

[0092] ,

[0093] ,

[0094] Specifically, for each stage of the content encoder, the input features are sequentially down-sampled by the stride convolution layer to reduce the spatial dimension while extracting the local detail information of the image. Then the input features are processed by the ConvNeXt block to obtain the output content feature embedding of this stage. For each stage of the high-frequency encoder, after the input is processed by the ConvNeXt block and the stride convolution layer, it enters the wavelet frequency decomposition block for high-frequency and low-frequency decomposition. As shown in Figure 5As shown, the wavelet frequency decomposition block converts the feature map into multiple frequency components using the Haar wavelet transform, which are LH (vertical subband), HL (horizontal subband), HH (diagonal subband), and LL (low-frequency subband). The high-frequency components LH+HL+HH are accumulated and then subjected to a point-by-point convolution operation to obtain high-frequency features, and the low-frequency component LL is subjected to a point-by-point convolution to obtain low-frequency features representing smooth overall structural information. Then, the obtained high-frequency features are concatenated with the output features of the previous stage, and further processed through a new ConvNeXt block and a stride convolution layer to generate the high-frequency feature embedding of the current stage. At the same time, the low-frequency features are directly transmitted to the next stage through a skip connection for decomposition and processing at a higher level.

[0095] The decoder takes the content feature embedding and the high-frequency feature embedding as input to reconstruct the video frame. Specifically, the content embedding and the high-frequency embedding are expanded in the first stage to obtain the initial features and . Then, the content features and the high-frequency features are obtained by spatial upsampling through the harmonic block in the second to fifth stages, and then a set of modulation parameters γ and β are learned through the high-frequency feature modulation layer to modulate the features to obtain enhanced features , where γ is a multiplication factor and β is an addition bias.

[0096]

[0097] In the last output stage, the conversion from the feature domain to the pixel domain is realized to obtain the reconstructed frame :

[0098] ,

[0099] Step 4: Compare the reconstructed video frame obtained in step 3 with the video frame and calculate the reconstruction loss; specifically, use the combination of spatial domain loss and frequency domain loss to supervise the network, and the formula is as follows:

[0100] ,

[0101] x t is the original frame, is the reconstructed frame. is the introduced dynamic weight matrix, which allows the network to place more weight on high-frequency features that are difficult to synthesize. and are the weight factors of each loss, which are set by the user as needed, and in this embodiment and are set to 0.7 and 100, respectively.

[0102] Step 5: Based on the reconstruction loss obtained in step 4, the neural representation network is optimized end-to-end using an Adan optimizer. In this embodiment, the learning rate is set to 0.003, and the cosine annealing algorithm is used to adjust the learning rate, with 10% of the total training rounds used for preheating.

[0103] Step 6: Repeat steps 3 to 5 until the preset training rounds are met, save the trained neural representation network weights and feature embeddings. Since this method is based on network overfitting training, the more training rounds, the better the reconstruction result. Considering the time quality trade-off, the training rounds are set to 300 in this embodiment, and users can set the training rounds as needed. When the entire training process is complete, the content encoder and high-frequency encoder will be discarded, and the decoder and all frame content feature embeddings and high-frequency feature embeddings in the video will be compressed as compression data to obtain a bitstream.

[0104] Step 7: After pruning, quantizing, and entropy encoding the neural representation network, the neural representation network is compressed into a video bitstream. Specifically, first, a global unstructured pruning operation is performed on the network. Here, the global pruning importance score proposed by Jaeho Lee et al. (Lee J, Park S, Mo S, et al. Layer-adaptive sparsity for the magnitude-based pruning[J]. arXiv preprint arXiv:2010.07611, 2020.) is used to determine the pruning elements, i.e., first, the layer-adaptive magnitude pruning score of each weight is calculated, which approximates the distortion of the pruned model. Then, global pruning is performed according to the score. Specifically,

[0105] ,

[0106] Here, θ i is the initial weight, and θ l is the pruned weight, i.e., if the LAMP score of the weight is higher than the set threshold, the weight is retained; otherwise, it is set to zero. Subsequently, fine-tuning is performed to restore network performance. Then, the feature embedding and weight are quantized to 8 bits to further reduce the size. Specifically, given a parameter tensor, the quantized value of parameter μ is given by:

[0107] ,

[0108] where, where "round" is the value rounded to the nearest integer, s is a scaling factor, and θ minand theta max are the minimum and maximum values of the parameter tensor respectively b represents the bit length used to quantize the model. The parameters can be dequantized by the following formula:

[0109] ,

[0110] Finally, lossless Huffman coding is used to generate a compact bitstream.

[0111] Step 8: During decoding, first decode the feature embedding and the weight of the neural representation network, build the decoder and load the saved weight of the neural representation network, then input the embedding into the decoder for a forward process to obtain the reconstructed video frame. Since the corresponding feature embedding is saved for each frame, any frame can be decoded individually, and finally all the reconstructed image frames are arranged in order and converted into a video by the ffmpeg tool.

[0112] The embodiment compares the neural representation video coding method based on high-frequency feature enhancement proposed in the present application with the recent prior art. The data set used in the experiment is the UVG data set constructed by the Finnish Academic Video Coding Group Ultra Video Group, and 7 videos with a resolution of 1080*1920 and a length of 600 or 300 frames are selected. The present application trains a model with a parameter quantity of 3M for each video. Peak signal-to-noise ratio (PSNR) and multi-scale structural similarity (MS-SSIM) are used as indicators to measure the video reconstruction result, and bits per pixel (BPP) is used as an indicator to measure the video compression performance. The comparison results of the present application and the existing methods NeRV, E-NeRV, HNeRV and other existing methods on the UVG data set are shown in Table 1. The experimental results prove that the present application improves the performance of the model and greatly improves the image reconstruction quality. We also train and compare the low-resolution version (480*960) of the UVG data set, and the results in Table 2 prove the superiority and robustness of the present application.

[0113] Table 1

[0114]

[0115] Table 2

[0116] Figure 8 The visualization results in Table 2 can intuitively show that the reconstructed images of the present application have better visual quality, and the detail areas such as letters and window buildings are clearer; the comparison results of the present application and the existing implicit neural representation video coding method and video compression of the video coding standard are as follows Figure 9 and Figure 10As shown, the rate-distortion curve proves that the present application is superior to the conventional method of HEVC and other video coding methods based on implicit neural representation, wherein HEVC is implemented by using the ffmpeg tool, slow preset, and B frames are not used. To prove the effectiveness of the module design introduced in the present application scheme, the ablation experiment is set in this embodiment, as shown in Table 3.

[0117] Table 3

[0118]

[0119] The present application generates six variants, wherein variant 1 is the test result of removing the high-frequency embedded branch; variant 2 is the test result of replacing the wavelet high-frequency encoder with a content encoder to verify the effectiveness of the proposed wavelet high-frequency encoder; variants 3 and 4 are the test results of replacing the activation function in the harmonic block with the commonly used Siren activation function and the commonly used GELU activation function to verify the effectiveness of the proposed harmonic block; variants 5 and 6 are the test results of replacing the loss function used in the present application with L1+SSIM and L2 to verify the effectiveness of the combined loss in the frequency domain and the spatial domain. In summary, the present application achieves better performance compared to other methods, especially in the protection of details and textures. The quantitative and qualitative results of Table 1 and Figure 7 The rate-distortion curve of Table 1 and Figure 8 and Figure 9 The rate-distortion curve of Table 1 and

[0120] Each of the embodiments in the specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment mainly describes the differences from other embodiments.

[0121] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses or adaptations of the application including departures from the present disclosure that come within the scope of the application. The specification and examples are to be considered exemplary only, with the true scope and spirit of the application indicated by the following claims.

[0122] The above detailed description of the application has further described the purposes, technical solutions and beneficial effects of the application. It should be understood that the above is only a specific embodiment of the application, and is not used to limit the protection scope of the application. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the application shall be included in the protection scope of the application.

Claims

1. A neural representation video coding method based on high-frequency feature enhancement, characterized in that, The method comprises the following steps: S1, pre-processing the video to be encoded to obtain a pre-processed data set comprising video frames; S2, constructing a neural representation network based on high-frequency feature enhancement, the neural representation network comprising a content encoder, a wavelet high-frequency encoder and a frequency-aware decoder; wherein the frequency-aware decoder comprises a harmonic block for feature size upsampling and a high-frequency feature modulation layer for feature modulation fusion; S3, inputting the data set into the constructed neural representation network based on high-frequency feature enhancement for training, using the content encoder to downsample the video frames to obtain content feature embeddings; using the wavelet frequency decomposition block in the wavelet high-frequency encoder to extract high-frequency information of the video frames to obtain high-frequency feature embeddings; S4, inputting the content feature embeddings and the high-frequency feature embeddings into the frequency-aware decoder, using the high-frequency feature modulation layer in the frequency-aware decoder for feature fusion, and then using the harmonic block in the frequency-aware decoder for spatial size upsampling to obtain reconstructed video frames; S5, calculating the reconstruction loss between the reconstructed video frames and the original video frames, optimizing the neural representation network end-to-end through an Adan optimizer, and repeating the above steps S3-S5 until a preset training round is met; S6, stopping optimizing the neural representation network, and saving the weights and feature embeddings of the trained neural representation network; S7, pruning, quantizing and entropy encoding the neural representation network to compress it into a video bit stream; S8, when decoding, loading the saved weights of the neural representation network, inputting the feature embeddings into the decoder for forward calculation to obtain reconstructed video frames. 2.The neural representation video coding method based on high-frequency feature enhancement of claim 1, characterized in that, When encoding and decoding the video frames, the high-frequency features of the current video frames are additionally extracted and utilized to enhance the encoding process.

3. The neural representation video encoding method based on high-frequency feature enhancement according to claim 1, wherein The content encoder comprises a first content encoding stage, a second content encoding stage, a third content encoding stage, a fourth content encoding stage and a fifth content encoding stage connected in sequence, wherein the first content encoding stage comprises a first stride convolution layer, a first layer normalization layer and a first ConvNeXt block connected in sequence; The second content encoding stage comprises a second stride convolution layer and a second ConvNeXt block connected in sequence; The third content encoding stage comprises a third stride convolution layer and a third ConvNeXt block connected in sequence; The fourth content encoding stage comprises a fourth stride convolution layer and a fourth ConvNeXt block connected in sequence; The fifth content encoding stage comprises a fifth stride convolution layer and a fifth ConvNeXt block connected in sequence; Each ConvNeXt block comprises a depthwise separable convolution layer, a layer normalization layer, a first full connection layer, a GeLU activation layer and a second full connection layer connected in sequence.

4. The neural representation video coding method based on high-frequency feature enhancement according to claim 1, characterized in that, The constructed neural representation network based on high-frequency feature enhancement comprises a wavelet high-frequency encoder, a content encoder and a decoder; The wavelet high-frequency encoder includes a first high-frequency encoding stage, a second high-frequency encoding stage, a third high-frequency encoding stage, a fourth high-frequency encoding stage and a fifth high-frequency encoding stage connected in sequence, wherein the first high-frequency encoding stage includes a sixth stride convolution layer, a sixth layer normalization layer and a sixth ConvNeXt block; The second high-frequency encoding layer includes a first Haar wavelet frequency decomposition block, a seventh stride convolution layer and a seventh ConvNeXt block, the input end of the first Haar wavelet frequency decomposition block is connected to the output end of the sixth ConvNeXt block, and the output end of the first Haar wavelet frequency decomposition block is connected to the seventh stride convolution layer; The third high-frequency encoding layer includes a second Haar wavelet frequency decomposition block, an eighth stride convolution layer and an eighth ConvNeXt block, the input end of the second Haar wavelet frequency decomposition block is connected to the output end of the seventh ConvNeXt block, the output end of the second Haar wavelet frequency decomposition block is connected to the eighth stride convolution layer, and the output end of the first Haar wavelet frequency decomposition block is connected to the input end of the second Haar wavelet frequency decomposition block; The fourth high-frequency encoding layer includes a third Haar wavelet frequency decomposition block, a ninth stride convolution layer and a ninth ConvNeXt block, the input end of the third Haar wavelet frequency decomposition block is connected to the output end of the eighth ConvNeXt block, the output end of the third Haar wavelet frequency decomposition block is connected to the ninth stride convolution layer, and the output end of the second Haar wavelet frequency decomposition block is connected to the input end of the third Haar wavelet frequency decomposition block; The fifth high-frequency encoding layer includes a fourth Haar wavelet frequency decomposition block, a tenth stride convolution layer and a tenth ConvNeXt block, the input end of the fourth Haar wavelet frequency decomposition block is connected to the output end of the ninth ConvNeXt block, the output end of the fourth Haar wavelet frequency decomposition block is connected to the tenth stride convolution layer, and the output end of the third Haar wavelet frequency decomposition block is connected to the input end of the fourth Haar wavelet frequency decomposition block.

5. The neural representation video coding method based on high-frequency feature enhancement of claim 1, wherein the decoder includes a plurality of decoding stages, each of which includes a harmonic block and a high-frequency feature modulation layer. The high-frequency feature modulation layer realizes deep fusion of high-frequency and content feature embedding through a point-by-point convolution layer, a Gelu activation function and a feature modulation operation, so as to retain the details and textures of the video frames in the process of reconstructing the video; the harmonic block includes a convolution layer, a pixel shuffle layer and an adaptive harmonic activation function connected in sequence, and is used for introducing periodic bias while performing spatial upsampling.

6. The neural representation video coding method based on high-frequency feature enhancement according to claim 5, characterized in that, The frequency loss function is used to optimize the high-frequency details: a) Fourier transform is performed on the reconstructed and original video frames; b) the frequency domain difference is calculated and dynamically weighted to highlight the high-frequency details; c) the neural network is optimized in combination with the frequency domain and spatial domain loss to improve the detail performance.

7. The neural representation video coding method based on high-frequency feature enhancement according to claim 1, characterized in that, The processor implements the steps of the method of any one of claims 1 to 7 when executing the program.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The computer program implements the steps of the method of any one of claims 1 to 7 when executed by the processor.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, ​

Images