A method for video coding based on saliency

Abstract

The application provides a video coding method based on saliency, wherein the coding method comprises the following steps: judging whether a current video image is the first frame in an image group; if yes, using an intra-frame coding method to code the current video image into a code stream BS; if no, obtaining a reference image from a decoded image buffer, and using an inter-frame coding method to code the current video image into a motion vector code stream BS mv and a prediction residual code stream BS r The application has the advantages that, under the same code rate, the video coded by the method of the application has better quality in the salient region, so that the reconstructed video has better subjective quality, or under the same subjective quality, the video code rate is lower, and resources are saved.

Description

Technical Field

[0001] This invention relates to the field of video encoding and decoding technology, and specifically to a saliency-based video encoding and decoding method. Background Technology

[0002] In recent years, with the development of live streaming and short video services, video streams have accounted for a large portion of network transmission bandwidth. Traditional video coding technologies, from H.264 to H.266, have seen significant improvements in coding efficiency. However, traditional video coding schemes rely on manually designed modules, including intra / inter-frame prediction, loop filtering, and DCT transform. In addition, researchers have turned their attention to the field of deep learning, attempting to leverage the powerful nonlinear expressive capabilities of deep neural networks to optimize video coding end-to-end.

[0003] A common neural network-based video coding framework employs a hybrid coding structure similar to traditional video coding schemes, replacing various required modules with corresponding neural network models, such as... Figure 1 As shown. Neural network-based video coding models achieve end-to-end optimization of video coding. However, they can only encode images predicted between frames, not keyframes predicted intra-frame. Therefore, video coding models are usually used in conjunction with image encoders. A common neural network-based image coding model employs an autoencoder structure, such as... Figure 2 As shown, this video coding model can utilize different loss functions to optimize corresponding objective metrics. For example, it can use the Mean Square Error (MSE) loss function to optimize the Peak Signal-to-Noise Ratio (PSNR) of the reconstructed video, or use the structural similarity-related loss function to optimize the structural similarity of the reconstructed video. However, higher objective quality does not necessarily mean higher subjective quality. Considering that most video receivers are human visual systems, optimizing the subjective quality of the video based on the characteristics of human vision can improve coding efficiency.

[0004] Attention mechanisms are the cognitive processes by which the human visual system allocates resources. Specifically, when observing the environment, people focus on only a certain part of it while ignoring others. In computer vision, one task is to predict or estimate the parts of an image or video that are likely to attract human attention, called saliency prediction. The result is the saliency weight for each pixel; the larger the weight, the more likely the pixel is to attract human attention. Typically, the weight ranges from 0 to 1, and the set of weights is called a saliency map. Optimizing video coding perceptually based on this human visual characteristic of attention mechanisms is a relatively effective approach. The core idea of ​​this method is to adjust the bit allocation within a frame, increasing the number of bits encoding salient regions while decreasing the number of bits encoding non-salient regions.

[0005] Most existing technical solutions are optimization methods designed for traditional video coding schemes (H.264, H.265, and H.266, etc.), aiming to improve subjective quality through perceptual optimization from the perspectives of quantization, bitrate control, or rate-distortion optimization. For example, in the quantization step, the quantization parameters (or quantization step size) generated by the original bitrate control strategy are fine-tuned based on the saliency map. The adjustment strategy is to increase the quantization parameters in non-saliency areas and decrease the quantization parameters in saliency areas to directly adjust the intra-frame bit allocation. One specific method is... Q istep and Q step These are the adjusted quantization step size and the original quantization step size, W and w, respectively. i S and s are the average saliency weights of the current frame and the current block, respectively. i The current encoded image size and the current encoded block size are considered respectively. When allocating the number of encoded bits for each macroblock or tree coding unit in rate control, a salience weight is added as a determining factor. A specific rate control strategy is as follows: in and T pic SW represents the target bit count of the current tree coding unit and the target bit count of the current encoded image, respectively. CurCTU and SW notCodedCTU These represent the saliency weights of the current CTU and the saliency weights of the uncoded CTUs in the current encoded image, respectively. However, these methods cannot be directly applied to deep learning-based video coding schemes because deep learning-based video coding schemes differ significantly from traditional video coding standards, with each step being different. One existing method for optimizing deep learning video coding models divides the video content into foreground and background regions. During training, the loss function is obtained by weighting the foreground and background loss functions, specifically L = αL. FG+(1-α)L BG L FG and L BG These are the loss functions for the foreground region and the background region, respectively, with α being a user-defined parameter. Under the same bitrate, the optimized encoding model produces videos with better foreground quality. However, this method only optimizes videos containing a specific type of object, such as portraits, and does not extend to all videos, thus limiting its practical application. Summary of the Invention

[0006] Against the above background, this invention discloses a saliency-based video encoding and decoding method, which optimizes a neural network-based video coding model. Specifically, the following technical solution is adopted:

[0007] The first aspect of the present invention provides a saliency-based video coding method, comprising:

[0008] Determine whether the current video image is the first frame in the image group;

[0009] If so, the current video image is encoded into a bitstream BS using intra-frame coding.

[0010] If not, then obtain the reference image from the decoded image buffer and use inter-frame coding to encode the current video image into a motion vector bitstream (BS). mv And predicting residual bitstream BS r In the coding prediction residual bitstream BS r Previously, the predicted residuals were filtered based on the saliency map.

[0011] Furthermore, the intra-frame coding method includes the following steps:

[0012] For the current video image to be encoded x t The latent vector z is obtained by inputting it into the image encoder. t ;

[0013] The latent vector z t Quantization is performed to obtain the signal to be encoded.

[0014] Signal to be encoded The input entropy encoder encodes the binary bitstream BS.

[0015] Furthermore, the inter-frame coding method includes the following steps:

[0016] For the current video image to be encoded x t This is combined with the reference image obtained from the decoded image cache. The motion vector v is obtained by inputting the motion estimation network. t ;

[0017] The motion vector v t The input motion vector encoder yields the implicit motion vector m. t Then m t Quantify to obtain

[0018] Will The input entropy encoder encodes the motion vector bitstream BS. mv ;

[0019] Will The input motion vector decoder obtains the reconstructed motion vectors. Reconstructed motion vectors and reference image The predicted image is obtained by inputting the motion compensation network.

[0020] x t and predicted images The predicted residual r is obtained by subtraction. t ;

[0021] According to the implicit vector z t and motion vector implicit vector m t The saliency map s of the current video image is predicted. t According to the saliency map s t For the predicted residual r t The final prediction residual r′ is obtained by filtering. t ;

[0022] The filtered prediction residual r′ t The prediction residual encoder is input to obtain the prediction residual latent vector y. t And quantify it to obtain

[0023] Will The input entropy encoder encodes the predicted residual bitstream BS. r .

[0024] Furthermore, based on the latent vector z t and motion vector implicit vector m t The saliency map s of the current video image is predicted. t Specifically, it includes:

[0025] The latent vector z t and motion vector implicit vector m t As image features and motion vector features, respectively representing spatial and temporal information, z t and m t Stacking along the channel dimension yields the fused feature f;

[0026] The fused features f are input into the saliency map network, which predicts the saliency map s of the current video image to be encoded. t .

[0027] Furthermore, according to the saliency map s t For the predicted residual r t Perform filtering r′ t Specifically, it includes:

[0028] According to the saliency map s t For the predicted residual r t Adjust the amplitude:

[0029] r′ t =r t -r t ×(1-s t )×σ

[0030] Where σ represents the strength of the filter.

[0031] A second aspect of the present invention provides a saliency-based video decoding method, comprising:

[0032] For the binary bitstream BS obtained by intra-frame coding, decode it to obtain the reconstructed video image and store it in the decoded image buffer;

[0033] For the motion vector bitstream BS obtained by inter-frame coding mv And predicting residual bitstream BS r By combining the reference image extracted from the decoded image buffer, the reconstructed video image is decoded and stored in the decoded image buffer. This process involves decoding the predicted residual bitstream (BS). r Then, the prediction residuals are compensated based on the saliency map.

[0034] Furthermore, for the binary bitstream BS obtained by intra-frame coding, decoding it to obtain the reconstructed video image and storing it in the decoded image buffer specifically includes:

[0035] For the bitstream BS, its input entropy decoder is used to decode the reconstructed latent vector.

[0036] Reconstructed latent vectors Input the image decoder to obtain the reconstructed video image.

[0037] Furthermore, for the motion vector bitstream BS obtained by inter-frame coding mv And predicting residual bitstream BS r Combining the reference image extracted from the decoded image buffer, the decoded reconstructed video image specifically includes:

[0038] For motion vector bitstream BS mv The input to the entropy decoder is used to decode and obtain the reconstructed motion vector latent vector. And input it into the motion vector decoder to obtain the reconstructed motion vector.

[0039] Reconstructed motion vectors and reference image extracted from the decoded image buffer The predicted image is obtained by inputting the motion compensation network.

[0040] For predicting residual bitstream BS r The input entropy decoder decodes the reconstructed prediction residual latent vector. Then it is input into the prediction residual decoder to obtain the reconstructed prediction residual.

[0041] The saliency map corresponding to the current decoded image reconstructed based on the information in the bitstream. And based on the filtering at the encoding end, the prediction residual is... The final predicted residual is obtained by performing compensation.

[0042] The final predicted residual The reconstructed video image is obtained by adding it to the predicted image.

[0043] Furthermore, the saliency map corresponding to the current decoded image is reconstructed based on the information in the bitstream. Specifically, it includes:

[0044] The reconstruction latent vector of the nearest intra-coded image to the current reconstructed image. As image features, the reconstructed motion vector latent vector As a feature of motion vectors, and Stacking along the channel dimension yields fused features.

[0045] Fusion features The input saliency map network predicts and obtains the reconstructed saliency map.

[0046] Furthermore, the prediction residual is predicted based on the filtering at the encoding end. The final predicted residual is obtained by performing compensation. Specifically, it includes:

[0047] Based on the reconstructed saliency map Compensation for forecast residuals: Where σ represents the strength of the filter.

[0048] The method of this invention predicts the saliency map of a video using the spatial features of an image and the temporal features of motion vectors. Instead of directly using the image and motion vectors as input to the saliency map network, this invention uses the corresponding latent vectors from the image coding network and the video coding network as input. The saliency map network shares some shallow convolutional layers from both the image coding network and the video coding network. This method reduces the number of model parameters and saves computational resources. In the video coding scheme, at the encoding end, the prediction residual signal during the video coding process is filtered based on the saliency map, and the bit allocation of the encoded residual signal is adjusted. At the decoding end, the saliency map is reconstructed using information from the bitstream, and the filtering process at the encoding end is compensated accordingly based on the reconstructed saliency map. The advantage of this invention is that, at the same bitrate, the video encoded by the optimized encoder has better quality in salient regions, resulting in better subjective quality of the reconstructed video, or, at the same subjective quality, the video bitrate is lower, saving resources. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of a neural network-based video coding framework that employs a hybrid coding structure.

[0050] Figure 2 This is a schematic diagram of a neural network-based image coding model that employs an autoencoder structure.

[0051] Figure 3 This is a schematic diagram of the complete video encoding / decoding and encoding-aware optimization model constructed in an embodiment of the present invention.

[0052] Figure 4 This is a schematic diagram of a saliency graph network in an embodiment of the present invention.

[0053] Figure 5 This is a schematic diagram of the complete video encoding / decoding and encoding / decoding perception optimization model constructed in an embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0055] Example 1

[0056] This invention proposes a method for optimizing existing deep learning-based video coding models. An embodiment of using this invention to optimize an existing video coding model is described below, specifically comprising the following steps:

[0057] First, a video coding model is constructed. Specifically, this includes the following steps:

[0058] First step, such as Figure 2 As shown, an image compression model is constructed according to existing technologies.

[0059] The second step, as Figure 1 As shown, a video compression model is constructed according to existing technologies.

[0060] The third step, as Figure 3 As shown, based on the first and second steps, a complete video coding and perception optimization model is constructed. Specifically, the video coding process is as follows:

[0061] Input image to be encoded x t It determines whether it is intra-frame coding or inter-frame coding. If it is intra-frame coding, an image coding model is used for encoding. Specifically, the encoding process consists of the following steps:

[0062] 1) Image x to be encoded t The input image encoder obtains the latent vector z. t , i.e. z t =Enc(x t ), where Enc is the image encoder;

[0063] 2) The implicit vector z t The input quantizer obtains the signal to be encoded. Right now Where Q is the quantizer, specifically, a uniform quantizer with an interval of 1 is used here. This invention does not impose any particular limitation on the quantizer; other uniform quantizers with different intervals or other non-uniform quantizers can also be used.

[0064] 3) Input the signal to be encoded into the entropy encoder to obtain the binary code stream BS, i.e. EC stands for entropy encoder.

[0065] The overall encoding process is BS = EC(Q(Enc(x)). t ))).

[0066] If the current image to be encoded is an inter-frame coded image, then a video coding model is used for encoding. Specifically, the encoding process is as follows:

[0067] 1) Set the current image to be encoded, x t and reference image The motion vector v is obtained by inputting the motion estimation network. t ,Right now Reference image Derived from the decoded image cache, ME is the motion estimation network;

[0068] 2) Transfer the motion vector v t Inputting the motion vector into the motion vector encoding network yields the hidden motion vector m. t That is, mt =Enc mv (v t ), where Enc mv It is a motion vector encoder;

[0069] 3) Implicit vector m of motion vector t The input quantizer obtains the signal to be encoded. Then the signal to be encoded The input entropy encoder encodes the motion vector bitstream BS. mv BS mv =EC(Q(m) t In this paper, EC and Q are the entropy encoder and quantizer, respectively, as in image encoding step 2). Here, a uniform quantizer with an interval of 1 is used, but the quantizer is not limited in this invention.

[0070] 4) The signal to be encoded obtained in step 3) The input motion vector is decoded by the motion vector decoder to obtain the reconstructed motion vector. Right now Dec mv For motion vector decoders;

[0071] 5) Reconstruct the motion vector and reference image The image is obtained by inputting the motion compensation network to perform motion compensation. Right now Where MC stands for motion compensation network;

[0072] 6) Set the current image to be encoded, x t With predicted image The predicted residual r is obtained by subtraction. t ,Right now

[0073] 7) The implicit vector z t and motion vector implicit vector m t As image features and motion vector features, respectively representing spatial and temporal information, z t and m t The fused feature f is obtained by stacking along the channel dimension. Specifically, the latent vector z t Shape Motion vector implicit vector m t Shape Where b is the batch size, h and w are the height and width of the image to be encoded, c1 and c2 are the corresponding number of channels, and ε is the downsampling factor of the convolutional layer, which is determined by the specific network structure. In this embodiment, the downsampling factor ε is 8, so the shape of f is... Common feature extraction methods use several consecutive convolutional layers as feature extractors to extract image (motion vector) features. Here, for video coding schemes, the corresponding intermediate signals are directly extracted as image (motion vector) features. The subsequent saliency map network shares the shallow convolutional layers in the video coding model, saving the resources required to build additional convolutional layers for feature extraction and the computational cost.

[0074] 8) Input the fused feature f into the saliency map network to obtain the saliency map s of the current image. t =Sal(f)=Sal(cat(z) t ,m t ), where cat is a stacking operation, and the Sal saliency map network has the following structure: Figure 4 As shown, CONV(i,j,k) and DECONV(i,j,k) represent convolutional and deconvolutional layers with kernel size i×i, number of kernels j, and stride k, respectively. Except for the last layer, the activation function after each (de)convolutional layer is ReLU, and the activation function after the last convolutional layer is Sigmoid. It should be noted that the specific scheme described in this embodiment is for clearly explaining the technical solution in this invention and does not limit the invention. The specific structure of the saliency graph network here is designed after balancing computational complexity and performance, and is only used as an example. Other structures or schemes can be used. For example, the number of inputs to the saliency graph network model can be increased, using multiple motion vectors as inputs to allow the model to acquire more temporal information; or the accumulated motion vectors between multiple images can be used as inputs to increase the accuracy of temporal information; the number of convolutional layers or the number of convolutional kernels in the saliency graph network can be increased to make saliency graph prediction more accurate.

[0075] 9) Based on the saliency map s in step 8), t The predicted residual r obtained in step 6) t Filtering is performed to obtain r′ t The purpose of filtering is to adjust the bit allocation of the encoded prediction residual. A specific operation is r′. t =r t -r t ×(1-s t )×σ, where σ represents the filter strength. In this embodiment, σ=0.2, but this invention does not specifically limit this parameter and it can be set according to the usage scenario. For the unnormalized saliency map s t The operation can be changed to Max() is the function to find the maximum value. Other available filtering methods include r′. t =r t -r t ×(1-s t )2 ×σ,r′ t =(r t +r t ×s t ×σ) / 2 and r′ t =(r t +r t ×s t 2 ×σ) / 2 etc.

[0076] 10) The filtered prediction residual r′ t The input residual encoder encodes the predicted residual latent vector y. t Then, the predicted residual is input to the quantizer and quantized to obtain... The bitstream BS of the prediction residual is obtained by inputting entropy encoding. r ,Right now Where EC is the entropy encoder, and Enc r For residual encoders, Q is the quantizer; the motion vector bitstream BS... mv And predicting residual bitstream BS r Integrate into the complete bitstream of the current image to be encoded.

[0077] A complete video stream includes intra-frame coded bitstream (BS) and inter-frame coded bitstream (BS). mv and BS r The specific decoding steps are as follows:

[0078] The decoding steps for an intra-frame coded bitstream (BS) are as follows:

[0079] 1) Input the bitstream BS into the corresponding entropy decoder to obtain the reconstructed latent vector. Right now Where ED stands for entropy decoder;

[0080] 2) Reconstruct the latent vectors The input image decoder decodes to obtain the reconstructed image. Right now Where Dec is the image decoder; the reconstructed image is stored in the decoded image buffer;

[0081] The overall decoding process is as follows:

[0082] The decoding process of inter-frame coded video streams is as follows:

[0083] 1) Convert the motion vector bitstream BS mv Entropy decoding is performed to obtain the reconstructed motion vector latent vector. Then the motion vector implicit vector The motion vector decoder obtains the reconstructed motion vectors. Right now Dec mv It is a motion vector decoder, and ED is an entropy decoder;

[0084] 2) Reconstruct the motion vector and reference image The image is obtained by inputting the motion compensation network to perform motion compensation. Right now Where MC stands for motion compensation network;

[0085] 3) Predict the residual bitstream BS r Entropy decoding is performed to obtain the reconstructed prediction residual latent vector. Then the reconstructed prediction residual latent vector The input residual decoder decodes to obtain the reconstructed prediction residual. Right now Dec r ED is a residual decoder;

[0086] 4) Add the predicted image to the reconstructed prediction residual to obtain the reconstructed image. Right now The reconstructed image is stored in the decoded image buffer.

[0087] The fourth step is to train the constructed model, which can be specifically divided into the following steps:

[0088] 1) Prepare the dataset for training the video coding model and the saliency map prediction model. The dataset should contain several high-definition video clips (compressed high-definition video or original YUV video). Decode or convert the video clips and save each frame of the video as a lossless RGB format image, such as PNG format. The dataset should contain annotations for training the saliency map prediction model. If the dataset does not contain corresponding annotations, existing advanced video saliency map prediction models can be used to process the video clips to obtain estimated saliency map labels for model training. For ease of explanation, this embodiment uses the Vimeo-90k dataset and uses existing video saliency map prediction techniques to obtain the saliency maps corresponding to the training data as labels during training.

[0089] 2) Pre-train the image coding model and video coding model using existing techniques or load existing pre-trained model parameters, and randomly initialize the parameters of the saliency map network;

[0090] 3) Freeze the model parameters except for the saliency map network, and train the saliency map network. The loss function during training is: Where h and w are the height and width of the image to be encoded, respectively, and s t and These are the predicted saliency map and saliency map labels, respectively; the training learning rate is set to 10. -4 The batch size is 4, and the Adam optimization algorithm is used to minimize the loss function. The training is conducted for three rounds.

[0091] 4) Freeze the image coding model, fine-tune the video coding model and saliency map network, and use the loss function during training as follows: Where L rd The rate-distortion loss function is used, where R represents the number of bits consumed during encoding, including the number of bits encoding the motion vector and the number of bits for the prediction residual; the mean square error loss is used to measure the distortion of the reconstructed image, where the hyperparameter λ balances the number of encoded bits and the quality of the reconstructed image, and is set to 256,512,1024,2048 according to the original technique; L s This refers to the loss function used in step two for training the saliency map network: α (balance rate distortion loss function) and α (saliency map loss function). When λ = 256,512, α is set to 0.05; when λ = 1024,2048, α is set to 0.1. The training learning rate is set to 10. -5 The batch size is 4, and the Adam optimization algorithm is used to minimize the loss function. The training is conducted for three rounds.

[0092] This invention does not impose restrictions on the loss function during training; it can be replaced according to the application scenario or requirements. For example, the absolute mean error can be used to measure the distortion of the reconstructed image. Alternatively, you can use structural similarity (SSIM) related loss functions, such as... The loss function for training saliency maps can use other correlation functions that measure probabilistic similarity, such as KL divergence.

[0093] The fifth step is to encode the video segment to be encoded using the trained model. Specifically, the video segment to be encoded is divided into several Groups of Pictures (GOPs). The first image in each GOP is encoded intra-frame, and the remaining images in the GOP are encoded inter-frame. If the saliency map of the video segment to be encoded has been predicted or the relevant annotations of the saliency map already exist, the saliency map prediction step can be omitted during encoding, and the existing saliency map can be directly used to filter the prediction residuals.

[0094] In this embodiment, the bit allocation when encoding video images is changed by introducing a filtering operation on the prediction residual. Compared with the video coding model without filtering, salient regions consume more bits and have higher reconstruction quality, thereby improving the subjective quality of the reconstructed video. The closer the saliency map used in the filtering step is to the subjective characteristics of the human eye, the higher the subjective quality of the reconstructed video.

[0095] Example 2

[0096] The following describes another implementation of the video coding perception optimization method of the present invention to optimize the existing video coding model. Its main difference from the previous embodiment is that the prediction residual is compensated for by the saliency map at the decoding end, which corresponds to the filtering operation at the coding end. The advantage of this operation is that it can save the bit rate in non-saliency areas while retaining the quality of saliency areas, so as to save bit rate while retaining subjective quality.

[0097] Specifically, see Figure 5 In this embodiment, only the video stream decoding and model training steps are different from the previous embodiment. The other identical steps will not be repeated. Only the different steps will be described.

[0098] The video stream decoding process is as follows:

[0099] 1) Convert the motion vector bitstream BS mv Entropy decoding is performed to obtain the reconstructed motion vector latent vector. Then the motion vector implicit vector The input motion vector decoder obtains the reconstructed motion vectors. Right now Dec mv It is a motion vector decoder, and ED is an entropy decoder;

[0100] 2) Reconstruct the motion vector and reference image The image is obtained by inputting the motion compensation network to perform motion compensation. Right now Where MC stands for motion compensation network;

[0101] 3) Predict the residual bitstream BS r Entropy decoding is performed to obtain the reconstructed prediction residual latent vector. Then the reconstructed prediction residual latent vector The input residual decoder decodes to obtain the reconstructed prediction residual. Right now Dec r ED is a residual decoder;

[0102] 4) Obtain the reconstruction latent vector of the intra-coded image closest to the current reconstructed image. and reconstructed motion vectors and implicit vectors As image features and motion vector features, where k is the index of the nearest reconstructed intra-coded image to the current reconstructed image, and The reconstructed fusion features are obtained by stacking along the channel dimension. Specifically, Shape Shape Where b is the batch size, h and w are the height and width of the image to be encoded, c1 and c2 are the corresponding number of channels, and ε is the downsampling factor of the convolutional layer, which is determined by the specific network structure. In this scheme, the downsampling factor ε is 8. Shape

[0103] 5) The fusion features of the reconstruction Inputting a saliency map into a saliency map network yields a reconstructed saliency map. The saliency map network here is the same as the saliency map network in step 8) of encoding inter-frame images in Example 1, to ensure the reconstructed saliency map. Saliency graph s with the encoding end t Consistency;

[0104] 6) Based on the reconstructed saliency map Predicted residuals for reconstruction Compensation is performed to obtain the final predicted residual. The compensation here should be the inverse operation of the filtering operation during encoding. Specifically, if the filtering operation during encoding is the first specific scheme described in Example 1, then the compensation method here is: Where σ is the filtering strength corresponding to the filtering operation in the encoding process. Here, it is set to be consistent with the encoding process, and is set to σ = 0.2.

[0105] 7) Predict the image Compared with the predicted residual Adding them together yields the reconstructed image. Right now The reconstructed image is stored in the decoded image buffer;

[0106] The constructed model is trained. Specifically, the training process includes the following steps:

[0107] 1) Training data preparation, this step is the same as the corresponding step in Example 1.

[0108] 2) Pre-train the image coding model and video coding model using existing techniques or load existing pre-trained models, and randomly initialize the parameters of the saliency map network;

[0109] 3) Freeze the model parameters except for the saliency map network, and train the saliency map network. The loss function during training is: Where h and w are the height and width of the image to be encoded, respectively, and s t and These are the predicted saliency map and saliency map labels, respectively. For the reconstructed saliency map, β is a hyperparameter balancing the two loss functions, set to 0.5 here. This invention does not impose specific limitations on this parameter and it can be changed according to the application scenario; the training learning rate is set to 10. -4 With a batch size of 4, the Adam optimization algorithm is used to minimize the loss function, and the training is performed for three epochs; then the learning rate is changed to 10. -5 With all other settings unchanged, continue training for 3 more rounds.

[0110] In this embodiment, by introducing filtering operations on the prediction residual at the encoding end and compensation operations on the prediction residual at the decoding end, compared with the original video coding model, the optimized method consumes fewer bits when encoding non-salient regions while retaining the reconstruction quality of salient regions. This allows the reconstructed video to save bitrate while maintaining the same subjective quality, thus reducing bandwidth pressure during transmission.

Claims

1. A saliency-based video coding method, characterized in that, include: Determine whether the current video image is the first frame in the image group; If so, then intra-frame coding methods are used to encode the current video image into a bitstream. ; If not, then for the current video image to be encoded This is combined with the reference image obtained from the decoded image cache. Input the motion estimation network to obtain motion vectors ; motion vector Input motion vector encoder to obtain motion vector latent vector Then Quantify to obtain ; Will The input entropy encoder encodes the motion vector bitstream. ; Will The input motion vector decoder obtains the reconstructed motion vectors. The reconstructed motion vector and reference image The predicted image is obtained by inputting the motion compensation network. ; The current video image to be encoded and predicted images The predicted residuals are obtained by subtraction. ; Based on the reconstructed latent vector of the nearest intra-coded image to the current image to be encoded. and reconstructing motion vectors and latent vectors Predict the saliency map of the current video image According to the saliency map For prediction residuals Filtering is performed to obtain the final prediction residual. ; The filtered prediction residual The prediction residual latent vector is obtained by inputting the prediction residual encoder. And quantify it to obtain ; Will The input entropy encoder encodes the predicted residual bitstream. .

2. The method according to claim 1, characterized in that, The intra-frame coding method includes the following steps: For the current video image to be encoded The latent vector is obtained by inputting it into the image encoder. ; latent vectors Quantization is performed to obtain the signal to be encoded. ; Signal to be encoded The input entropy encoder encodes the binary bitstream. .

3. The method according to claim 1, characterized in that, According to the implicit vector and motion vectors and implicit vectors Predict the saliency map of the current video image Specifically, it includes: latent vectors and motion vectors and implicit vectors As image features and motion vector features, respectively representing spatial and temporal information, they will... and Stacking along the channel dimension yields fused features. ; Fusion features The input saliency map network predicts the saliency map of the current video image to be encoded. .

4. The method according to claim 1, characterized in that, According to the saliency map For prediction residuals Perform filtering Specifically, it includes: According to the saliency map For prediction residuals Adjust the amplitude: in This indicates the strength of the filter.

5. A saliency-based video decoding method, characterized in that, include: For the binary bitstream obtained by intra-frame coding The video image is decoded to obtain the reconstructed video image and stored in the decoded image buffer; For the motion vector bitstream obtained by inter-frame coding and predicting residual bitstream ,Will The input entropy decoder decodes to obtain the reconstructed motion vector latent vector. And input it into the motion vector decoder to obtain the reconstructed motion vector. ; Reconstructed motion vectors and reference image extracted from the decoded image buffer The predicted image is obtained by inputting the motion compensation network. ; Will The input entropy decoder decodes to obtain the reconstructed prediction residual latent vector. Then it is input into the prediction residual decoder to obtain the reconstructed prediction residual. ; The saliency map corresponding to the current decoded image reconstructed based on the information in the bitstream. And based on the filtering at the encoding end, the prediction residual is... The final predicted residual is obtained by performing compensation. ; The final predicted residual The reconstructed video image is obtained by adding it to the predicted image. .

6. The method according to claim 5, characterized in that, For the binary bitstream obtained by intra-frame coding Decoding the video to obtain the reconstructed image and storing it in the decoded image buffer specifically includes: For bitstream The input entropy decoder is used to decode the reconstructed latent vector. ; Reconstructed latent vectors Input the image decoder to obtain the reconstructed video image. .

7. The method according to claim 5, characterized in that, The saliency map corresponding to the current decoded image reconstructed based on the information in the bitstream. Specifically, it includes: The reconstruction latent vector of the nearest intra-coded image to the current reconstructed image. As image features, the reconstructed motion vector latent vector As a feature of motion vectors, and Stacking along the channel dimension yields fused features. ; Fusion features The input saliency map network predicts and obtains the reconstructed saliency map. .

8. The method according to claim 5, characterized in that, Based on the filtering at the encoding end, predict the residual. The final predicted residual is obtained by performing compensation. Specifically, it includes: Based on the reconstructed saliency map Compensation for forecast residuals: ;in This indicates the strength of the filter.

Images