A method for video coding based on implicit neural representation considering saliency

Abstract

The application provides a video coding method based on implicit neural representation considering saliency, comprising: original video preprocessing; constructing a video implicit neural representation network based on a multi-scale feature grid, comprising a multi-scale feature grid and a decoder; optimizing the model through a saliency-guided training strategy; compressing the multi-scale feature grid and the decoder as compressed data to obtain a video code stream; sending and decompressing the video code stream, generating feature embedding through the feature grid according to the frame index of each frame, inputting the feature embedding into the decoder to output a corresponding reconstructed image, arranging the reconstructed images in sequence to obtain a decoded video. The application codes the video in an implicit neural network, proposes a multi-scale feature grid and a decoder based on a light-weight convolutional neural network, significantly improves the objective quality of video reconstruction, and introduces saliency preprocessing and a saliency-guided training method to comprehensively improve the visual quality of video reconstruction.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video coding and deep learning, and particularly relates to a video coding method based on implicit neural representation considering saliency. BACKGROUND

[0002] With the rapid growth of video data, video storage and transmission face great challenges. In order to alleviate this burden, video compression technology has become an important research direction. Traditional video coding standards, such as H.264 and HEVC, rely on hand-designed coding modules and have achieved good compression effect. However, these methods have been difficult to meet the increasing resolution and frame rate requirements when facing high-definition videos, 4K or even higher resolution scenarios. Therefore, new coding technology has become a research hotspot. In recent years, with the rapid development of deep learning, video coding methods based on neural networks have made significant improvements in reconstruction quality. These methods usually use end-to-end network models to replace traditional hand-designed modules, providing a better balance between compression ratio and video quality. However, due to the complexity of these methods involving network architecture and computing process, the amount of calculation and decoding complexity increases, especially in low latency or device computing power limited scenarios, it is difficult to achieve efficient decoding.

[0003] Implicit Neural Representations (INR) is a new type of signal representation, which is gradually applied to image and video compression tasks. Unlike traditional coding methods, INR-based video coding fits the video content by training a special neural network, rather than using pixel-level coding. This method compresses the video into the weight representation of the network by learning the mapping function from the frame index to the image, achieving an efficient coding process. The network weights are compressed and stored as the bitstream. During decoding, the frame index is input into the network for forward propagation, thus reconstructing the video. Compared with deep learning end-to-end methods, this method has lower parameter quantity and faster decoding speed. Some existing INR-based video coding methods have achieved preliminary results. For example, Dupont et al. (Dupont E, Goliński A, Alizadeh M, et al. Coin: Compression with implicit neural representations [J]. arXiv preprint arXiv:2103.03123, 2021.) proposed an image compression method based on INR, which trains a multi-layer perceptron (MLP) to fit the mapping relationship between pixel position and RGB value, and quantizes the network weights to store as the bitstream. Rho et al. (Rho D, Cho J, Ko J H, et al. Neural residual flow fields for efficient video representations [C] / / Proceedings of the Asian Conference on Computer Vision. 2022:3447-3463.) further proposed a pixel-level INR video compression method, which trains the mapping relationship between pixel position, motion vector and RGB residual as an implicit function, and encodes the quantized network weights.Srinivasan et al. (Srinivasan P P, Deng B, Zhang X, et al. Nerv: Neural reflectance and visibility fields for relighting and view synthesis [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 7495-7504.) adopted convolutional neural network (CNN) as the implicit neural representation based video coding method, which introduced the relationship between frame index and image mapping in video coding. Li et al. (Li Z, Wang M, Pi H, et al. E-nerv: Expedite neural video representation with disentangled spatial-temporal context [C] / / European Conference on Computer Vision. Cham: Springer Nature Switzerland. 2022: 267-284.) decomposed the spatial and temporal context of video implicit neural representation, which improved the representation ability of the model. Chen et al. (Chen H, Gwilliam M, Lim S N, et al. Hnerv: A hybrid neural representation for videos [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023: 10270-10279.) proposed a hybrid implicit neural representation method, which further improved the reconstruction quality by introducing feature embedding related to video content.

[0004] Although these methods have made certain progress in improving video compression efficiency and reconstruction quality, most of them still face the problem of single structure, limited representation ability and insufficient parameter efficiency. The human eye shows different degrees of visual sensitivity in different areas, for example, objects moving in the foreground often attract more attention than background areas, and preferentially improving the reconstruction quality of these areas can significantly improve the visual experience. However, existing implicit neural representation based video coding methods use a uniform weight allocation strategy for video reconstruction, ignoring this. SUMMARY

[0005] To solve the above problems, the application provides a video coding method based on implicit neural representation considering saliency.

[0006] The application designs a video implicit neural representation network based on a multi-scale feature grid and a saliency-guided training method, and realizes a video coding method based on implicit neural representation considering saliency. The video implicit neural representation network based on the multi-scale feature grid comprises a multi-scale feature grid and a decoder based on light-weight convolution. The multi-scale feature grid utilizes the inter-frame correlation of the video, provides content-related rich feature embedding, and improves the representation capability of the network. The decoder part of the application is designed as an SNeRV block based on light-weight convolution, and combines the reparameterization technology to train a single structure deployment in a branch structure, improves the parameter efficiency, and greatly improves the reconstruction quality under the same parameter amount. The saliency-guided training method firstly pre-processes each frame of the video through saliency detection, outputs a saliency map, and then guides the training process of the network with the saliency map, preferentially fits the area more sensitive to the human eye, improves the subjective visual quality, and realizes the effect of parameter allocation, and improves the parameter efficiency.

[0007] The technical scheme of the application is as follows:

[0008] A video coding method based on implicit neural representation considering saliency, comprising the following steps:

[0009] Step 1: pre-processing the original video to be coded, including splitting the video into images frame by frame, and constructing a data set;

[0010] Step 2: constructing a video implicit neural representation network model based on a multi-scale feature grid, wherein the video implicit neural representation network comprises a multi-scale feature grid and a decoder;

[0011] Step 3: inputting the images of the training set obtained in step 1 into the video implicit neural representation network constructed in step 2, and outputting reconstructed images;

[0012] Step 4: predicting the saliency of each frame obtained in step 1 to obtain a saliency map; after masking the reconstructed images obtained in step 3, the input images and the saliency map, calculating the saliency area reconstruction loss; comparing the reconstructed images obtained in step 3 with the input images, and calculating the global reconstruction loss; and weighting and summing the global reconstruction loss and the saliency area reconstruction loss to obtain a total loss;

[0013] Step 5: according to the total loss obtained in step 4, using an Adam optimizer to perform saliency-guided training, and optimizing the video implicit neural representation network in an end-to-end manner;

[0014] Step 6: Steps 3-5 are executed in a loop for a specified number of rounds, the trained video implicit neural representation network model is saved, the multi-scale feature grid and the decoder network parameters in the trained video implicit neural representation network model are compressed as compressed data to obtain a video bitstream;

[0015] Step 7: The video bitstream is sent and decompressed to obtain a multi-scale feature grid and a decoder; the frame index of each frame is input into the multi-scale feature grid, and the feature embedding of the frame is output; the feature embedding is input into the decoder, and the corresponding reconstructed image is output; the reconstructed images are arranged in order to obtain a decoded video.

[0016] Further, the step 2:

[0017] The video implicit neural representation network comprises a multi-scale feature grid and a decoder.

[0018] The multi-scale feature grid encodes the input frame index t as a position, and outputs the feature embedding of the frame. The feature grid G is a SxCxHxW tensor, where S represents the number of frames, C represents the number of channels, H represents the height, and W represents the width. The input frame index t finds two adjacent feature maps in the time dimension of the feature grid, and linearly interpolates in the time dimension to obtain the feature embedding of the frame corresponding to the index t in the feature grid G:

[0019]

[0020] Where φ(G, t) represents the embedding of frame index t in feature grid G, t is the input index, s is the total number of frames in the feature grid, and T is the total number of frames of the video. is the normalized index of t in the feature grid G, and m and n are the indices of the two adjacent reference embeddings in the grid, respectively using the bottom function and the top function G[x] is the xth feature embedding of G. We construct multiple feature grids with different temporal and spatial resolutions to encode the time index t. All the obtained feature embeddings are upsampled to match the spatial resolution of the largest embedding, and are concatenated in the channel dimension to form the final feature embedding E t :

[0021] E t =Concat(Upsample(φ(G1,t)……,φ(G K ,t)))

[0022] Where K is the total number of feature grids. Then, the embedding is projected to the input channel number of the decoder using a convolutional layer, and the initial feature X0 is output:

[0023] X0=Conv(E t)

[0024] The decoder outputs the reconstructed frame finally by upsampling the initial feature X0several times, including N SNeRV blocks to gradually upsample and process the feature and an output stage to map the final feature to the output reconstructed frame. The SNeRV block mainly includes two stages: an upsampling stage and a feature processing stage F n In the upsampling stage, first, the output X n-1 is upsampled using bilinear interpolation. Then, the position index of the block is encoded by a local feature grid, and a linear layer is used to adjust the channel number and add it to the upsampled feature mapping. The feature processing stage includes a convnext module to adjust the channel number and two RepConv modules with residual links. The RepConv module consists of a reparameterization deep separable convolution (repDWconv) and a feed-forward network (FFN). RepDWconv applies reparameterization to deep convolution (DWConv), which has multiple branches with multiple convolution paths during training, which helps better learning and optimization. During inference, these multiple paths are reparameterized into a single path DWConv, resulting in a simple and efficient architecture. This method can improve training results without introducing additional parameters. The feed-forward network (FFN) mainly consists of two pointwise convolutions (PWConv) connected by residual links. The processing of SNeRV is shown in the formula:

[0025] X n = F n (Upsample(X n-1 )+localgrid(t))

[0026] The output of the last SNeRV block Xn is mapped to the final output frame by a convolution layer and a sigmoid activation function

[0027]

[0028] Further, the step 4:

[0029] The saliency-guided training includes two parts: saliency detection preprocessing and saliency loss-based training. In the saliency detection preprocessing stage, a pre-trained saliency detection model is used to preprocess the original video frames to generate a saliency map S for each frame, reflecting the visual importance of each region. According to the saliency of the pixels, binarization is performed according to a certain threshold to generate a mask Mask, and this step is only performed once before training begins:

[0030] mask t = SaliencyDetection (y t )

[0031] Based on the training of the saliency loss, the total loss function includes two parts, the global reconstruction loss and the salient region reconstruction loss. The global reconstruction loss L globle is used to guide the network training to make the reconstructed frame image as close to the original image as possible, specifically using the combination of MSE loss and SSIM loss, the calculation formula is as follows:

[0032]

[0033] Wherein is the reconstructed frame, y t is the original frame. The salient region reconstruction loss takes into account the different sensitivities of the human eye to different regions, and the region that is more sensitive to the human eye is additionally weighted to guide the network to preferentially improve the reconstruction quality of the more salient region, so as to improve the overall visual quality, specifically first multiplying the reconstructed frame and the original frame with the saliency mask point by point, and then using the combination of MSE loss and SSIM loss for the salient region, the calculation formula is as follows:

[0034]

[0035] Finally, the global reconstruction loss and the salient region reconstruction loss are weighted and summed to obtain the total loss function:

[0036] L = λ1*L globle + λ2*L saliency

[0037] By adopting the above scheme, the application has the beneficial effects that:

[0038] 1. The application proposes a video coding method based on implicit neural representation considering saliency, designs a model structure with higher parameter efficiency to improve objective quality, and proposes a saliency-guided training strategy considering the difference in human eye sensitivity, thereby comprehensively improving the visual quality.

[0039] 2. The application improves the network structure, specifically: designing an encoding layer based on a multi-scale feature grid and a decoder module SNeRV block based on a lightweight convolutional neural network, effectively improving the representation ability of the network, improving the parameter efficiency, and thereby improving the reconstruction objective quality under limited parameter amount.

[0040] 3. The application proposes a saliency-guided training strategy, which is preprocessed by saliency detection and guided by saliency mask for training, preferentially considering the reconstruction quality of the salient region, and more effectively allocating parameters to obtain higher visual quality. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is the basic flow chart of the video coding method based on implicit neural representation of saliency considered by the present application;

[0042] Figure 2 is the structure diagram of the video implicit neural representation network based on multi-scale feature grid of the present application;

[0043] Figure 3 is the structure diagram of the multi-scale feature grid of the present application;

[0044] Figure 4 is the structure diagram of the decoder SNeRV block of the present application;

[0045] Figure 5 is the schematic diagram of the training strategy guided by saliency of the present application;

[0046] Figure 6 is the video representation comparison result diagram of the present application and the existing video implicit neural representation method;

[0047] Figure 7 is the visualization result comparison diagram of the present application and the existing video implicit neural representation method;

[0048] Figure 8 is the video compression comparison result diagram of the present application and the existing method;

[0049] Figure 9 is the result diagram of the ablation experiment of each module proposed by the present application. DETAILED DESCRIPTION

[0050] The technical solutions provided by the present application will be further described below in combination with specific embodiments and their accompanying drawings. The advantages and features of the present application will be clearer in combination with the following description.

[0051] Step 1: Preprocess the original video to be encoded, including splitting the video into images frame by frame, and constructing a dataset;

[0052] Step 2: Build a video implicit neural representation network based on a multi-scale feature grid, which includes a multi-scale feature grid as an embedding layer and a decoder (as shown in Figure 2 );

[0053] The multi-scale feature grid (as shown in Figure 3 ) as an embedding layer encodes the position of the input frame index t, and outputs the feature embedding of the frame. The feature grid G is a SxCxHxW tensor, where S represents the number of frames, C represents the number of channels, H represents the height, and W represents the width. The input frame index t finds two adjacent feature maps in the time dimension of the feature grid, and performs linear interpolation in the time dimension to obtain the feature embedding of the frame corresponding to the index t in the feature grid G:

[0054]

[0055] where φ(G, t) denotes the embedding of frame index t in feature grid G, t is the index of input frame, s is the total number of frames in feature grid, and T is the total number of frames of the video. is the normalized index of t in feature grid G, and m and n are the indices of two adjacent reference embeddings in the grid, respectively using the bottom function and the top function is calculated, and G[x] is the xth feature embedding of G. Multiple feature grids with different spatio-temporal resolutions are constructed to encode the index t of the input frame. All the obtained feature embeddings are upsampled to match the spatial resolution of the largest embedding and concatenated in the channel dimension to form the final feature embedding E t :

[0056] E t =Concat(Upsample(φ(G1,t)……,φ(G K ,t)))

[0057] where K is the total number of feature grids. Subsequently, the embedding is projected to the input channel number of the decoder using a convolutional layer, and the initial feature X0 is output:

[0058] X0=Conv(E t )

[0059] The decoder outputs the reconstructed frame finally by several times of upsampling on the initial feature X0. The decoder includes N SNeRV blocks to gradually upsample and process the features and an output stage to map the final feature to the output reconstructed frame. The SNeRV Block (as shown in Figure 4 ) mainly includes two stages: the upsampling stage and the feature processing stage F n .

[0060] In the upsampling stage, the output X n-1 of the previous block is first upsampled using bilinear interpolation. Then, the position index of the block is encoded by a local feature grid, and the channel number is adjusted by a linear layer adjuster and added to the upsampled feature mapping.

[0061] The feature processing stage includes a convnext module for adjusting the channel number and two RepConv modules with residual links.

[0062] The RepConv module consists of a reparameterized depthwise separable convolution (RepDWconv) and a feed-forward network (FFN). RepDWconv applies reparameterization to a depthwise convolution (DWConv), which adopts a multi-branch structure with multiple convolution paths during training, which helps better learning and optimization. During inference, these multiple paths are reparameterized into a single path DWConv, resulting in a simple and efficient architecture. This method can improve training results without introducing additional parameters. The feed-forward network (FFN) mainly consists of two pointwise convolutions (PWConv) connected by a residual connection.

[0063] The processing process of SNeRV is shown in the formula:

[0064] X n =F n (Upsample(X n-1 )+localgrid(t))

[0065] The output of the last SNeRV block Xn is mapped to the final output frame by a convolution layer and a sigmoid activation function

[0066]

[0067] Step 3: input the image of the training set obtained in step 1 into the video implicit neural representation network constructed in step 2, and output the reconstructed image;

[0068] Step 4: perform saliency prediction on each frame obtained in step 1 to obtain a saliency map; mask the reconstructed image obtained in step 3 with the input image and the saliency map, and calculate the salient region reconstruction loss; compare the reconstructed image obtained in step 3 with the input image, and calculate the global reconstruction loss; and weight the sum of the global reconstruction loss and the salient region reconstruction loss to obtain the total loss;

[0069] The saliency-guided training (as shown in Figure 5 ) includes saliency detection preprocessing and training based on saliency loss. In the saliency detection preprocessing stage, a pre-trained saliency detection model is used to preprocess the original video frames to generate a saliency map S for each frame, reflecting the visual importance of each region. According to the saliency of the pixels, binarization is performed according to a certain threshold to generate a mask Mask, and this step is only performed once before training begins:

[0070] mask t =SaliencyDetection(y t )

[0071] Based on the training of saliency loss, the total loss function includes two parts, global reconstruction loss and salient region reconstruction loss.

[0072] Global reconstruction loss L globle To guide the network training to make the reconstructed frame image as close to the original image as possible, a combination of MSE loss and SSIM loss is used, and the calculation formula is as follows:

[0073]

[0074] Wherein is the reconstructed frame, y t is the original frame.

[0075] The salient region reconstruction loss takes into account the different sensitivities of the human eye to different regions. The region more sensitive to the human eye is additionally weighted to guide the network to preferentially improve the reconstruction quality of the more salient region, so as to improve the overall visual quality. Specifically, first multiply the reconstructed frame and the original frame with the saliency mask point by point, and then use a combination of MSE loss and SSIM loss for the salient region, and the calculation formula is as follows:

[0076]

[0077] Finally, the global reconstruction loss and the salient region reconstruction loss are weighted and summed to obtain the total loss function:

[0078] L = λ1*L globle + λ2*L saliency

[0079] The specific parameter setting λ is set to 0.7, λ1 is 0.8, and λ2 is 0.2.

[0080] Step 5: According to the total loss obtained in step 4, use the Adam optimizer to perform saliency-guided training, and optimize the video implicit neural representation network end-to-end. The learning rate uses a cosine annealing strategy, the maximum value is set to 0.002, and 10% of the total training rounds are used for preheating, and a total of 300 rounds are trained.

[0081] Step 6: Loop steps 3-5 to train a specified number of rounds, save the trained video implicit neural representation network model parameters, and compress the multi-scale feature grid and decoder network parameters in the trained video implicit neural representation network model as compression data. Specifically, first perform 30 rounds of unstructured pruning retraining, then quantize the parameters to 8 bits, and finally further compress through lossless entropy coding to obtain the video bitstream.

[0082] Step 7: Send and decompress the video code stream, and decompress to obtain a multi-scale feature grid and a decoder; input the frame index of each frame into the multi-scale feature grid, output the feature embedding corresponding to the frame, input the feature embedding into the decoder, output the corresponding reconstructed image, arrange the reconstructed images in order, and obtain the decoded video.

[0083] Note: The feature grid and the decoder are data compressed and transmitted.

[0084] Effect verification embodiment

[0085] The video coding method based on implicit neural representation considering saliency proposed in the application includes a multi-scale feature grid-based video implicit neural representation network and a saliency-guided training strategy. A comparison experiment is made with the closest prior art. The data set used in the experiment is the UVG data set constructed by the academic video coding group Ultra Video Group in Finland, which contains 7 video sequences with a resolution of 1920*1080, a frame rate of 120 frames / s, and a length of 2.5s to 5s, a total of 3900 frames.

[0086] The method of the application is compared with the existing NeRV methods including NeRV, ENeRV, HNeRV and HiNeRV, and the peak signal-to-noise ratio (PSNR) is used to evaluate the reconstruction quality. Comparison is made under the condition that the number of parameters is the same or similar, which is divided into small size (0.5M), medium size (1.5M) and large size (3M), and all models are trained for the same number of 300 epochs. For all NeRV methods, this embodiment maintains the same structure and training settings as described in the original method, and only adjusts the network width to match the total number of parameters. The comparison results on the UVG data set are as shown in Figure 6 The experimental results show that the method of the application realizes PSNR values of 30.19, 33.92 and 35.57 for S, M and L models respectively on the UVG data set.

[0087] Compared with the most advanced existing model HiNeRV, it is increased by 0.52, 0.46 and 0.3 respectively. As shown in Figure 7 From the visualization results, it can be seen that among the models with the same number of parameters, the reconstructed video of the application is better than the most advanced existing method HiNeRV in visual quality, and the detail area is clearer; and the comparison results of the existing video implicit neural representation method and the video compression of the video coding standard, the rate-distortion curve is as shown in Figure 8 .

[0088] Finally, the multi-scale feature grid module, the SNeRV block and the saliency-based training strategy proposed in the present application are subjected to an ablation experiment to prove their effectiveness. The multi-scale feature grid module, the SNeRV block and the saliency-based training strategy are added to the baseline model in turn, and finally the reconstruction quality on the UVG video dataset is as shown in Table 5. Figure 9

[0089] The above description is only a description of the preferred embodiments of the present application, and is not any limitation on the scope of the present application. Any modification or modification made by any person skilled in the art according to the above disclosed technical content shall be regarded as an equivalent effective embodiment, and shall fall within the scope of protection of the technical scheme of the present application.​

Claims

1. A video encoding and decoding method based on implicit neural representation considering saliency, characterized in that: The steps include: Step 1: Preprocess the original video to be encoded, including splitting the video into images frame by frame and constructing a dataset; Step 2: Construct a video implicit neural representation network model based on a multi-scale feature grid, wherein the video implicit neural representation network includes a multi-scale feature grid and a decoder; Step 3: Input the training set images obtained in step 1 into the video implicit neural representation network constructed in step 2, and output the reconstructed image; Step 4: Perform saliency prediction on each frame obtained in step 1 to obtain a saliency map. Mask the reconstructed image obtained in step 3 with the input image and the saliency map, and calculate the salient region reconstruction loss. Compare the reconstructed image obtained in step 3 with the input image to calculate the global reconstruction loss. The total loss is obtained by weighted summing the global reconstruction loss and the salient region reconstruction loss. Step 5: Based on the total loss obtained in step 4, use the Adam optimizer to perform saliency-guided training to optimize the video implicit neural representation network end-to-end; Step 6: Loop through steps 3 to 5, train for a specified number of rounds, save the trained video implicit neural representation network model, and compress the multi-scale feature grids and decoder network parameters in the trained video implicit neural representation network model as compressed data to obtain a video bitstream. Step 7: Send and decompress the video stream to obtain a multi-scale feature grid and decoder; input the frame index of each frame into the multi-scale feature grid, output the feature embedding of the frame, input the feature embedding into the decoder, output the corresponding reconstructed image, and arrange the reconstructed images in sequence to obtain the decoded video.

2. The video encoding and decoding method based on implicit neural representation considering saliency according to claim 1, characterized in that: Step 2: The video implicit neural representation network model based on multi-scale feature grid includes a multi-scale feature grid and a decoder; The multi-scale feature grid acts as an embedding layer to position-encode the input frame index t and output the feature embedding of the frame; The feature grid G ​​is a tensor of size S×C×H×W, where S represents the number of frames, C represents the number of channels, H represents the height, and W represents the width. Given a frame index t, we find two adjacent feature maps in the time dimension of the feature grid and perform linear interpolation along the time dimension to obtain the feature embedding of the frame corresponding to index t in the feature grid G: in represents the embedding of frame index t in the feature grid G, where t is the input index, s is the total number of frames in the feature grid, and T is the total number of frames in the video; is the normalized index of t in the feature grid G, m and n are the indices of two adjacent reference embeddings in the grid, using the bottom function and top function Compute, G[x] is the x-th feature embedding of G; construct multiple feature grids with different spatiotemporal resolutions and encode the index t; upsample all the obtained feature embeddings to match the spatial resolution of the largest embedding and concatenate them in the channel dimension to form the final feature embedding : Where K is the total number of feature grids; then, a convolutional layer is used to project the embedding to the number of input channels of the decoder, outputting the initial feature : The decoder performs the initial feature The decoder includes N SNeRV blocks that gradually upsample and process the features, and an output stage that maps the final features to the output reconstructed frame.

3. The video encoding and decoding method based on implicit neural representation considering saliency according to claim 2, characterized in that: In step 2: Each SNeRV Block consists of two stages: upsampling stage and feature processing stage ; In the upsampling stage, the output of the previous block is first interpolated using bilinear interpolation. Upsampling; then, the position index of the block is encoded through a local feature grid, and then the number of channels is adjusted through a linear layer and added to the upsampled feature map; The feature processing stage includes a convnext module for adjusting the number of channels and two RepConv modules with residual links; the RepConv module consists of a reparameterized depth-wise separable convolution repDWconv and a feedforward network FFN; RepDWconv applies reparameterization to the deep convolution DWConv. During training, RepDWconv adopts a multi-branch structure with multiple convolution paths, which helps to better learn and optimize; during inference, these multiple paths are reparameterized into a single path DWConv, which is a simple and efficient architecture; the feedforward network FFN is mainly composed of two point-to-point convolutions PWConv connected by residuals.

4. The video encoding and decoding method based on implicit neural representation considering saliency according to claim 3, characterized in that: In step 2: The processing of SNeRV Block is shown in the formula: The output of the last SNeRV block Xn is mapped to the final output frame through a convolutional layer and a sigmoid activation function : 。 5. The video encoding and decoding method based on implicit neural representation considering saliency according to claim 1, characterized in that: Step 4: The saliency-guided training consists of two parts: saliency detection preprocessing and saliency loss-based training. In the saliency detection preprocessing stage, the original video frames are preprocessed using a pre-trained saliency detection model to generate a saliency map S for each frame, reflecting the visual importance of each region. Based on the saliency of the pixels, the pixels are binarized according to a certain threshold to generate a mask. This step is performed only once before training begins: Based on the training of saliency loss, the total loss function consists of two parts: global reconstruction loss and salient area reconstruction loss; global reconstruction loss It is used to guide network training to make the reconstructed frame image close to the original image. Specifically, it uses a combination of MSE loss and SSIM loss. The calculation formula is as follows: in To reconstruct the frame, is the original frame.

6. The video encoding and decoding method based on implicit neural representation considering saliency according to claim 5, characterized in that: In step 4: The salient area reconstruction loss takes into account the different sensitivities of the human eye to different areas. The areas that are more sensitive to the human eye, that is, more salient areas, are given additional weights to guide the network to prioritize improving the reconstruction quality of more salient areas, thereby improving the overall visual quality. Specifically, the reconstructed frame, the original frame, and the saliency mask are multiplied point by point, and then a combination of MSE loss and SSIM loss is used for the salient areas. The calculation formula is as follows: Finally, the weighted sum of the global reconstruction loss and the salient area reconstruction loss is used to obtain the total loss function: 。

Images