Implicit neural representation-based video coding method considering saliency

Abstract

The invention provides a saliency-considered video coding method based on implicit neural representation. The method comprises the following steps: preprocessing an original video; constructing a video implicit neural representation network based on the multi-scale feature grid, wherein the video implicit neural representation network comprises the multi-scale feature grid and a decoder; optimizing the model through a training strategy guided by significance; the multi-scale feature grid and the decoder are used as compressed data to be compressed, and a video code stream is obtained; and sending and decompressing the video code stream, generating feature embedding for the frame index of each frame through the feature grid, inputting the feature embedding into a decoder, outputting corresponding reconstructed images, and arranging the reconstructed images in sequence to obtain a decoded video. According to the method, the video is coded in the implicit neural network, the multi-scale feature grid and the decoder based on the lightweight convolutional neural network are provided, the objective quality of video reconstruction is remarkably improved, a training mode of significance preprocessing and significance guidance is introduced, and the visual quality of video reconstruction is comprehensively improved.

Description

Technical Field

[0001] The present invention relates to the fields of video coding and deep learning, and in particular to a video coding method based on implicit neural representation taking saliency into consideration. Background Art

[0002] With the rapid growth of video data, video storage and transmission face huge challenges. In order to reduce this burden, video compression technology has become an important research direction. Traditional video coding standards, such as H.264 and HEVC, rely on manually designed coding modules and achieve good compression effects. However, these methods have been unable to meet the increasing resolution and frame rate requirements when facing high-definition video, 4K and even higher resolution scenes. Therefore, new coding technologies have become a hot topic of research. In recent years, with the rapid development of deep learning, neural network-based video coding methods have achieved significant improvements in reconstruction quality. These methods usually use end-to-end network models to replace traditional hand-designed modules, providing a better balance between compression ratio and video quality. However, since these methods involve complex network architectures and calculation processes, the amount of calculation and decoding complexity increase, especially in scenarios with low latency or limited device computing power, it is difficult to achieve efficient decoding.

[0003] Implicit Neural Representations (INR), as a new type of signal representation, are gradually being applied to image and video compression tasks. Unlike traditional coding methods, video coding based on implicit neural representations trains a dedicated neural network to fit the video content instead of pixel-level coding. This method compresses the video into a weighted representation of the network by learning the mapping function from the video frame index to the image, achieving an efficient encoding process. The network weights are compressed as a bitstream. During decoding, the frame index is input into the network for forward propagation to reconstruct the video. Compared with end-to-end methods based on deep learning, this method has a lower number of parameters and a faster decoding speed. Some existing video coding methods based on implicit neural representations have achieved initial results. For example, Dupont et al. (Dupont E, Goliński A, Alizadeh M, et al. Coin: Compression with implicit neural representations [J]. arXiv preprint arXiv: 2103.03123, 2021.) proposed an image compression method based on implicit neural representation, which fits the mapping relationship between pixel position and RGB value by training a multi-layer perceptron (MLP), and quantizes the network weights and stores them as a bitstream. Rho et al. (Rho D, Cho J, Ko JH, et al. Neural residual flow fields for efficient video representations [C] / / Proceedings of the Asian Conference on Computer Vision. 2022: 3447-3463.) further proposed a pixel-level implicit neural representation video compression method, which trains the mapping relationship between pixel position and motion vector and RGB residual as an implicit function and encodes it by quantizing the network weights.Srinivasan et al. (Srinivasan P P, Deng B, Zhang X, et al. Nerv: Neural reflectance and visibility fields for relighting and view synthesis [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 7495-7504.) used convolutional neural network (CNN) as a video coding method based on implicit neural representation, and introduced the relationship between frame index and image mapping in video coding. Li et al. (Li Z, Wang M, Pi H, et al. E-nerv: Expedite neural video representation with disentangled spatial-temporal context [C] / / European Conference on Computer Vision. Cham: Springer Nature Switzerland. 2022: 267-284.) decomposed the implicit neural representation of the video into spatial and temporal contexts, improving the representation ability of the model. Chen et al. (Chen H, Gwilliam M, Lim SN, et al. Hnerv: A hybrid neural representation for videos [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023: 10270-10279.) proposed a hybrid implicit neural representation method, which further improved the reconstruction quality by introducing feature embedding related to the video content.

[0004] Although these methods have made some progress in improving video compression efficiency and reconstruction quality, most methods still face the problems of being too simple in structure, limited in representation ability, and insufficient in parameter efficiency, resulting in low reconstruction quality under limited parameters. The human eye exhibits different degrees of visual sensitivity in different areas. For example, moving objects in the foreground tend to attract more attention than background areas. Prioritizing the reconstruction quality of these areas can significantly improve the visual experience. However, existing video coding methods based on implicit neural representations use a unified weight allocation strategy for video reconstruction, ignoring this point. Summary of the invention

[0005] In order to solve the above problems, the present invention provides a video coding method based on implicit neural representation considering saliency.

[0006] The present invention designs a video implicit neural representation network based on a multi-scale feature grid and a saliency-guided training method to realize a video encoding method based on implicit neural representation that considers saliency. The video implicit neural representation network based on a multi-scale feature grid includes a multi-scale feature grid and a decoder based on lightweight convolution. The multi-scale feature grid utilizes the inter-frame correlation of the video, provides content-related rich feature embedding, and improves the representation ability of the network. The decoder part of the present invention is designed as a SNeRV block based on lightweight convolution, combined with heavy parameter technology to train a single structure deployment with a branch structure, improve parameter efficiency, and greatly improve reconstruction quality under the same parameter amount. The saliency-guided training method first pre-processes each frame of the video through saliency detection, outputs a saliency map, and then uses the saliency map to guide the training process of the network, preferentially fits the areas where the human eye is more sensitive, improves subjective visual quality, and achieves the effect of parameter allocation at the same time, and improves parameter efficiency.

[0007] The technical solution of the present invention is:

[0008] A video coding method based on implicit neural representation considering saliency comprises the following steps:

[0009] Step 1: Preprocess the original video to be encoded, including splitting the video into images frame by frame and constructing a data set;

[0010] Step 2: constructing a video implicit neural representation network model based on a multi-scale feature grid, wherein the video implicit neural representation network includes a multi-scale feature grid and a decoder;

[0011] Step 3: Input the training set images obtained in step 1 into the video implicit neural representation network constructed in step 2, and output the reconstructed image;

[0012] Step 4: Perform saliency prediction on each frame obtained in step 1 to obtain a saliency map; after masking the reconstructed image obtained in step 3, the input image and the saliency map, calculate the salient area reconstruction loss; compare the reconstructed image obtained in step 3 with the input image, and calculate the global reconstruction loss; weighted sum the global reconstruction loss and the salient area reconstruction loss to obtain the total loss;

[0013] Step 5: Based on the total loss obtained in step 4, use the Adam optimizer to perform saliency-guided training to optimize the video implicit neural representation network end-to-end;

[0014] Step 6: loop through steps 3 to 5, train for a specified number of rounds, save the trained video implicit neural representation network model, and compress the multi-scale feature grids and decoder network parameters in the trained video implicit neural representation network model as compressed data to obtain a video bitstream;

[0015] Step 7: Send and decompress the video bitstream to obtain a multi-scale feature grid and a decoder; input the frame index of each frame into the multi-scale feature grid, output the feature embedding of the frame, input the feature embedding into the decoder, output the corresponding reconstructed image, arrange the reconstructed images in sequence, and obtain the decoded video.

[0016] Furthermore, the step 2:

[0017] The video implicit neural representation network includes a multi-scale feature grid and a decoder.

[0018] The multi-scale feature grid acts as an embedding layer to positionally encode the input frame index t and output the feature embedding of the frame. The feature grid G ​​is a tensor of S×C×H×W, where S represents the number of frames, C represents the number of channels, H represents the height, and W represents the width. Input the frame index t, find two adjacent feature maps in the time dimension of the feature grid, and perform linear interpolation along the time dimension to obtain the feature embedding of the frame corresponding to index t in the feature grid G:

[0019]

[0020] Where φ(G,t) represents the embedding of frame index t in the feature grid G, t is the input index, s is the total number of frames in the feature grid, and T is the total number of frames in the video. is the normalized index of t in the feature grid G, m and n are the indices of two adjacent reference embeddings in the grid, using the base function and top function Calculate, G[x] is the xth feature embedding of G. We construct multiple feature grids with different spatiotemporal resolutions to encode the time index t. All the obtained feature embeddings are upsampled to match the spatial resolution of the largest embedding and concatenated in the channel dimension to form the final feature embedding E t :

[0021] E t =Concat(Upsample(φ(G1,t)……,φ(G K ,t)))

[0022] Where K is the total number of feature grids. Subsequently, a convolutional layer is used to project the embedding to the number of input channels of the decoder, outputting the initial feature X0:

[0023] X0=Conv(E t)

[0024] The decoder upsamples the initial feature X0 several times and finally outputs a reconstructed frame. The decoder includes N SNeRV blocks that gradually upsample and process the features and an output stage that maps the final features to the output reconstructed frame. SNeRV Block mainly includes two stages: upsampling stage and feature processing stage F n In the upsampling stage, we first use bilinear interpolation to upsample the output X of the previous block. n-1 Upsampling. Then, the position index of the block is encoded through a local feature grid, and then the number of channels is adjusted through a linear layer and added to the upsampled feature map. The feature processing stage includes a convnext module for adjusting the number of channels and two RepConv modules with residual connections. The RepConv module consists of a reparameterized depthwise separable convolution (repDWconv) and a feedforward network (FFN). RepDWconv applies reparameterization

[18] to deep convolution (DWConv). During training, RepDWconv adopts a multi-branch structure with multiple convolution paths, which helps better learning and optimization. During inference, these multiple paths are reparameterized into a single path DWConv, resulting in a simple and efficient architecture. This method can improve training results without introducing additional parameters. The feedforward network (FFN) mainly consists of two point-wise convolutions (PWConv) connected by residual connections. The processing process of SNeRV is shown in the formula:

[0025] X n =F n (Upsample(X n-1 )+localgrid(t))

[0026] The output of the last SNeRV block Xn is mapped to the final output frame through a convolutional layer and a sigmoid activation function

[0027]

[0028] Furthermore, the step 4:

[0029] The training based on saliency guidance includes two parts: saliency detection preprocessing and training based on saliency loss. In the saliency detection preprocessing stage, the original video frames are preprocessed using a pre-trained saliency detection model to generate a saliency map S for each frame, reflecting the visual importance of each region. According to the saliency of the pixel, it is binarized according to a certain threshold to generate a mask Mask. This step is performed only once before the start of training:

[0030] mask t =SaliencyDetection(y t )

[0031] Based on the training of saliency loss, the total loss function consists of two parts, global reconstruction loss and salient region reconstruction loss. globle It is used to guide network training to make the reconstructed frame image as close to the original image as possible. Specifically, it uses a combination of MSE loss and SSIM loss. The calculation formula is as follows:

[0032]

[0033] in To reconstruct the frame, y t The salient region reconstruction loss takes into account the different sensitivity of the human eye to different regions. The regions that are more sensitive to the human eye, that is, more salient, are given additional weights to guide the network to prioritize the reconstruction quality of more salient regions, so as to improve the overall visual quality. Specifically, the reconstructed frame and the original frame are multiplied point by point with the saliency mask, and then the MSE loss and SSIM loss are combined for the salient region. The calculation formula is as follows:

[0034]

[0035] Finally, the weighted sum of the global reconstruction loss and the salient area reconstruction loss is used to obtain the total loss function:

[0036] L=λ1*L globle +λ2*L saliency

[0037] By adopting the above scheme, the beneficial effects of the present invention are:

[0038] 1. The present invention proposes a video coding method based on implicit neural representation that takes saliency into account, designs a model structure with higher parameter efficiency to improve objective quality, and proposes a saliency-guided training strategy considering the differences in human eye sensitivity, thereby comprehensively improving visual quality;

[0039] 2. The present invention improves the network structure, specifically: designing a coding layer based on a multi-scale feature grid and a decoder module SNeRV block based on a lightweight convolutional neural network, which effectively improves the representation ability of the network and improves parameter efficiency, thereby improving the objective quality of reconstruction under limited parameter quantity;

[0040] 3. The present invention proposes a saliency-guided training strategy, which uses saliency detection preprocessing and saliency mask to guide training, giving priority to the reconstruction quality of salient areas and more efficient parameter allocation to obtain higher visual quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a basic flow chart of a video coding method based on implicit neural representation considering saliency of the present invention;

[0042] Figure 2 It is a structural diagram of a video implicit neural representation network based on a multi-scale feature grid of the present invention;

[0043] Figure 3 is a structural diagram of a multi-scale feature grid of the present invention;

[0044] Figure 4 It is a block diagram of the decoder SNeRV of the present invention;

[0045] Figure 5 is a schematic diagram of the saliency-guided training strategy of the present invention;

[0046] Figure 6 1. It is a comparison result diagram of video representation between the present invention and the existing video implicit neural representation method;

[0047] Figure 7 is a comparison diagram of the visualization results of the present invention and the existing video implicit neural representation method;

[0048] Figure 8 This is a comparison diagram of video compression results between the present invention and the prior art method;

[0049] Fig. 9 It is a diagram of ablation experiment results of each module proposed in the present invention. DETAILED DESCRIPTION

[0050] The technical solution provided by the present application will be further described below in conjunction with specific embodiments and accompanying drawings. The advantages and features of the present application will become more apparent with the following description.

[0051] Step 1: Preprocess the original video to be encoded, including splitting the video into images frame by frame and constructing a data set;

[0052] Step 2: Construct a video implicit neural representation network based on a multi-scale feature grid, wherein the video implicit neural representation network includes a multi-scale feature grid as an embedding layer and a decoder ( Figure 2 shown);

[0053] The multi-scale feature grid (such as Figure 3 As shown in Figure 2, the embedding layer encodes the position of the input frame index t and outputs the feature embedding of the frame. The feature grid G ​​is a tensor of S×C×H×W, where S represents the number of frames, C represents the number of channels, H represents the height, and W represents the width. Input the frame index t, find two adjacent feature maps in the time dimension of the feature grid, and perform linear interpolation along the time dimension to obtain the feature embedding of the frame corresponding to index t in the feature grid G:

[0054]

[0055] Where φ(G,t) represents the embedding of frame index t in the feature grid G, t is the index of the input frame, s is the total number of frames in the feature grid, and T is the total number of frames in the video. is the normalized index of t in the feature grid G, m and n are the indices of two adjacent reference embeddings in the grid, using the base function and top function Compute, G[x] is the xth feature embedding of G. Construct multiple feature grids with different spatiotemporal resolutions and encode the index t of the input frame. Upsample all the obtained feature embeddings to match the spatial resolution of the largest embedding and concatenate them in the channel dimension to form the final feature embedding E t :

[0056] E t =Concat(Upsample(φ(G1,t)……,φ(G K ,t)))

[0057] Where K is the total number of feature grids. Subsequently, a convolutional layer is used to project the embedding to the number of input channels of the decoder, outputting the initial feature X0:

[0058] X0=Conv(E t )

[0059] The decoder performs several upsampling on the initial feature X0 and finally outputs a reconstructed frame. The decoder includes N SNeRV blocks that gradually upsample and process the features and an output stage that maps the final features to the output reconstructed frame. Figure 4 As shown in Figure 2, it mainly includes two stages: upsampling stage and feature processing stage F n .

[0060] In the upsampling stage, we first use bilinear interpolation to upsample the output X of the previous block. n-1 Then, the position index of the block is encoded through a local feature grid, and then passed through a linear layer to adjust the number of channels and added to the upsampled feature map.

[0061] The feature processing stage includes a convnext module for adjusting the number of channels and two RepConv modules with residual links.

[0062] The RepConv module consists of a reparameterized depthwise separable convolution (repDWconv) and a feedforward network (FFN). RepDWconv applies reparameterization

[18] to deep convolution (DWConv). During training, RepDWconv adopts a multi-branch structure with multiple convolution paths, which helps better learning and optimization. During inference, these multiple paths are reparameterized into a single path DWConv, resulting in a simple and efficient architecture. This approach can improve training results without introducing additional parameters. The feedforward network (FFN) mainly consists of two point-wise convolutions (PWConv) connected by residuals.

[0063] The processing of SNeRV is shown in the formula:

[0064] X n =F n (Upsample(X n-1 )+localgrid(t))

[0065] The output of the last SNeRV block Xn is mapped to the final output frame through a convolutional layer and a sigmoid activation function

[0066]

[0067] Step 3: Input the training set images obtained in step 1 into the video implicit neural representation network constructed in step 2, and output the reconstructed image;

[0068] Step 4: Perform saliency prediction on each frame obtained in step 1 to obtain a saliency map; after masking the reconstructed image obtained in step 3, the input image and the saliency map, calculate the salient area reconstruction loss; compare the reconstructed image obtained in step 3 with the input image, and calculate the global reconstruction loss; weighted sum the global reconstruction loss and the salient area reconstruction loss to obtain the total loss;

[0069] The saliency-guided training (e.g. Figure 5 The process includes saliency detection preprocessing and training based on saliency loss. In the saliency detection preprocessing stage, the original video frames are preprocessed using the pre-trained saliency detection model to generate a saliency map S for each frame, reflecting the visual importance of each region. According to the saliency of the pixels, the pixels are binarized according to a certain threshold to generate a mask. This step is performed only once before the training starts:

[0070] mask t =SaliencyDetection(y t )

[0071] Based on the training of saliency loss, the total loss function consists of two parts, global reconstruction loss and salient area reconstruction loss.

[0072] Global reconstruction loss L globle It is used to guide network training to make the reconstructed frame image as close to the original image as possible. Specifically, it uses a combination of MSE loss and SSIM loss. The calculation formula is as follows:

[0073]

[0074] in To reconstruct the frame, y t is the original frame.

[0075] The salient area reconstruction loss takes into account the different sensitivities of the human eye to different areas. The areas that are more sensitive to the human eye, that is, more salient areas, are given additional weights to guide the network to prioritize the reconstruction quality of more salient areas, so as to improve the overall visual quality. Specifically, the reconstructed frame and the original frame are multiplied point by point with the saliency mask, and then a combination of MSE loss and SSIM loss is used for the salient areas. The calculation formula is as follows:

[0076]

[0077] Finally, the weighted sum of the global reconstruction loss and the salient area reconstruction loss is used to obtain the total loss function:

[0078] L=λ1*L globle +λ2*L saliency

[0079] The specific parameter settings are λ to 0.7, λ1 to 0.8, and λ2 to 0.2;

[0080] Step 5: According to the total loss obtained in step 4, the Adam optimizer is used to perform saliency-guided training to optimize the video implicit neural representation network end-to-end. The learning rate uses a cosine annealing strategy with a maximum value of 0.002. 10% of the total training rounds are used for warm-up, and a total of 300 rounds of training are performed.

[0081] Step 6: Loop through steps 3 to 5, train for a specified number of rounds, save the trained video implicit neural representation network model parameters, and compress the multi-scale feature grids and decoder network parameters in the trained video implicit neural representation network model as compressed data. Specifically, first perform 30 rounds of unstructured pruning retraining, then perform 8-bit quantization on the parameters, and finally further compress through lossless entropy coding to obtain a video bitstream.

[0082] Step 7: Send and decompress the video bitstream, decompress to obtain a multi-scale feature grid and a decoder; input the frame index of each frame into the multi-scale feature grid, output the feature embedding corresponding to the frame, input the feature embedding into the decoder, output the corresponding reconstructed image, arrange the reconstructed images in sequence, and obtain the decoded video.

[0083] Note: The feature grid and decoder are themselves compressed data for transmission.

[0084] Effect verification example

[0085] The video coding method based on implicit neural representation considering saliency proposed in the present invention includes a video implicit neural representation network based on a multi-scale feature grid and a saliency-guided training strategy. A comparative experiment was conducted with the closest prior art. The data set used in the experiment is the UVG data set constructed by the Ultra Video Group, an academic video coding group in Finland, which includes 7 video sequences with a resolution of 1920*1080, a frame rate of 120 frames / s, a length ranging from 2.5s to 5s, and a total of 3900 frames.

[0086] The method of the present invention is compared with existing NeRV methods including NeRV, ENeRV, HNeRV, and HiNeRV, and the peak signal-to-noise ratio (PSNR) is used to evaluate the reconstruction quality. The comparison is made under models with the same or similar number of parameters, which are divided into small size (0.5M), medium size (1.5M), and large size (3M). All models are trained for the same number of 300 epochs. For all NeRV methods, this embodiment maintains the same structure and training settings as described in the original method, and only adjusts the network width to match the total number of parameters. The comparison results on the UVG dataset are shown in Figure 2. Figure 6 The experimental results show that the proposed method achieves PSNR values ​​of 30.19, 33.92 and 35.57 for the S, M and L models respectively on the UVG dataset.

[0087] Compared with the most advanced model HiNeRV, the improvements are 0.52, 0.46 and 0.3 respectively. Figure 7 As shown in the visualization results, it can be clearly seen that in the model with the same number of parameters, the video reconstructed by the present invention is superior to the most advanced existing method HiNeRV in terms of visual quality, and the detail area is clearer; compared with the video compression results of the existing video implicit neural representation method and the video coding standard, the rate-distortion curve is shown in Figure 8 shown.

[0088] Finally, ablation experiments were conducted on the multi-scale feature grid module, SNeRV block, and saliency-based training strategy proposed in this paper to prove their effectiveness. On the basis of the baseline model, the multi-scale feature grid module, SNeRV block, and saliency-based training strategy were added in sequence. Finally, the reconstruction quality on the UVG video dataset was as follows: Fig. 9 shown.

[0089] The above description is only a description of the preferred embodiments of the present application, and is not intended to limit the scope of the present application. Any changes or modifications made by any person skilled in the art based on the above disclosed technical contents shall be deemed as equivalent effective embodiments and shall fall within the scope of protection of the technical solution of the present application.

Claims

1. A video coding method based on implicit neural representation considering saliency, characterized in that: The steps include: Step 1: Preprocess the original video to be encoded, including splitting the video into images frame by frame and constructing a data set; Step 2: constructing a video implicit neural representation network model based on a multi-scale feature grid, wherein the video implicit neural representation network includes a multi-scale feature grid and a decoder; Step 3: Input the training set images obtained in step 1 into the video implicit neural representation network constructed in step 2, and output the reconstructed image; Step 4: Perform saliency prediction on each frame obtained in step 1 to obtain a saliency map; after masking the reconstructed image obtained in step 3, the input image and the saliency map, calculate the salient area reconstruction loss; compare the reconstructed image obtained in step 3 with the input image, and calculate the global reconstruction loss; weighted sum the global reconstruction loss and the salient area reconstruction loss to obtain the total loss; Step 5: Based on the total loss obtained in step 4, use the Adam optimizer to perform saliency-guided training to optimize the video implicit neural representation network end-to-end; Step 6: loop through steps 3 to 5, train for a specified number of rounds, save the trained video implicit neural representation network model, and compress the multi-scale feature grids and decoder network parameters in the trained video implicit neural representation network model as compressed data to obtain a video bitstream; Step 7: Send and decompress the video bitstream to obtain a multi-scale feature grid and a decoder; input the frame index of each frame into the multi-scale feature grid, output the feature embedding of the frame, input the feature embedding into the decoder, output the corresponding reconstructed image, arrange the reconstructed images in sequence, and obtain the decoded video.

2. The video coding method based on implicit neural representation considering saliency as claimed in claim 1, characterized in that: Step 2: The video implicit neural representation network model based on multi-scale feature grid includes a multi-scale feature grid and a decoder; The multi-scale feature grid acts as an embedding layer to positionally encode the input frame index t and output a feature embedding of the frame; The feature grid G ​​is a tensor of S×C×H×W, where S represents the number of frames, C represents the number of channels, H represents the height, and W represents the width. Input the frame index t, find two adjacent feature maps in the time dimension of the feature grid, and perform linear interpolation along the time dimension to obtain the feature embedding of the frame corresponding to index t in the feature grid G: Where φ(G,t) represents the embedding of frame index t in the feature grid G, t is the input index, s is the total number of frames in the feature grid, and T is the total number of frames in the video; is the normalized index of t in the feature grid G, m and n are the indices of two adjacent reference embeddings in the grid, using the base function and top function Compute, G[x] is the xth feature embedding of G; construct multiple feature grids with different spatiotemporal resolutions and encode the index t; upsample all the obtained feature embeddings to match the spatial resolution of the largest embedding and concatenate them in the channel dimension to form the final feature embedding E t : E t =Concat(Upsample(φ(G1,t)……,φ(G K ,t))) Where K is the total number of feature grids; then, a convolutional layer is used to project the embedding to the number of input channels of the decoder, outputting the initial feature X0: X0=Conv(E t ) The decoder upsamples the initial feature X0 several times and finally outputs a reconstructed frame. The decoder includes N SNeRV blocks that gradually upsample and process the features and an output stage that maps the final features to the output reconstructed frame.

3. The video coding method based on implicit neural representation considering saliency as claimed in claim 2, characterized in that: In step 2: Each SNeRV Block consists of two stages: upsampling stage and feature processing stage F n ; In the upsampling stage, we first use bilinear interpolation to upsample the output X of the previous block. n-1 Upsampling; then, the position index of the block is encoded through a local feature grid, and then the number of channels is adjusted through a linear layer and added to the upsampled feature map; The feature processing stage includes a convnext module for adjusting the number of channels and two RepConv modules with residual links; the RepConv module consists of a reparameterized depthwise separable convolution (repDWconv) and a feedforward network (FFN); RepDWconv applies reparameterization [18] to deep convolution (DWConv). During training, RepDWconv adopts a multi-branch structure with multiple convolution paths, which helps to better learn and optimize; during inference, these multiple paths are reparameterized into a single path DWConv, which is a simple and efficient architecture; the feedforward network (FFN) is mainly composed of two point-wise convolutions (PWConv) connected by residuals.

4. The video coding method based on implicit neural representation considering saliency as claimed in claim 3, characterized in that: In step 2: The processing process of SNeRV Block is shown in the formula: X n =F n (Upsample(X n-1 )+localgrid(t)) The output of the last SNeRV block Xn is mapped to the final output frame through a convolutional layer and a sigmoid activation function 5. The video coding method based on implicit neural representation considering saliency as claimed in claim 1, characterized in that: Step 4: The training based on saliency guidance includes two parts: saliency detection preprocessing and training based on saliency loss; In the saliency detection preprocessing stage, the original video frames are preprocessed using the pre-trained saliency detection model to generate a saliency map S for each frame, reflecting the visual importance of each region; based on the saliency of the pixel, it is binarized according to a certain threshold to generate a mask Mask. This step is performed only once before the training starts: mask t =SaliencyDetection(y t ) Based on the training of saliency loss, the total loss function consists of two parts: global reconstruction loss and salient area reconstruction loss; the global reconstruction loss L globle It is used to guide network training to make the reconstructed frame image close to the original image. Specifically, it uses a combination of MSE loss and SSIM loss. The calculation formula is as follows: in To reconstruct the frame, y t is the original frame.

6. The video coding method based on implicit neural representation considering saliency as claimed in claim 5, characterized in that: In step 4: The salient area reconstruction loss takes into account the different sensitivities of the human eye to different areas. The areas that are more sensitive to the human eye, that is, more salient areas, are given additional weights to guide the network to prioritize the reconstruction quality of more salient areas, so as to improve the overall visual quality. Specifically, the reconstructed frame, the original frame, and the saliency mask are multiplied point by point, and then a combination of MSE loss and SSIM loss is used for the salient areas. The calculation formula is as follows: Finally, the weighted sum of the global reconstruction loss and the salient area reconstruction loss is used to obtain the total loss function: L=λ1*L globle +λ2*L saliency 。

Images