Video encoding and decoding method

Abstract

The invention relates to the technical field of computers, and provides a video encoding and decoding method. The method comprises the following steps: inputting a potential representation reconstruction value of a target image frame of a video to be encoded into an inter-frame prediction model in an encoder to obtain a potential representation prediction value of the current image frame at the encoder side; obtaining a potential representation residual value of the current image frame according to a difference value between the potential representation predicted value and the potential representation actual value of the current image frame on the encoder side; reconstructing the potential representation of the current image frame according to the potential representation residual value of the current image frame to obtain a potential representation reconstruction value of the current image frame at the decoder side; and decoding the potential representation reconstruction value of the current image frame at the decoder side to obtain a reconstructed image frame of the current image frame at the decoder side. According to the video encoding and decoding method, the data volume and the calculation amount of the model can be reduced, the prediction efficiency of the model is improved, the limitation of a fixed mode of motion estimation and compensation is relieved, and the prediction accuracy of the model is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video encoding and decoding method. Background Art

[0002] Currently, video content accounts for over 80% of internet traffic, and this proportion is expected to increase further. The amount of raw, uncompressed video data is enormous. For example, storing an uncompressed 90-minute, 1080p movie requires 750GB of memory. Therefore, during video transmission, the video must be encoded at the sender to compress the video size and decoded at the receiver to retrieve the original video.

[0003] With the development of deep learning, the learning and representation capabilities of deep models are becoming more and more powerful. They are widely used in various fields and have achieved remarkable results. However, when using deep models to optimize traditional video coding and decoding methods, since traditional video coding and decoding methods generally adopt inter-frame prediction modes based on motion estimation and compensation, on the one hand, neural network-based motion estimation is generally achieved by estimating optical flow information using optical flow network models, but the current common video resolutions are relatively high, which will make the input and output dimensions of the optical flow network model very large, resulting in a large amount of data and calculation of the optical flow network model, reducing the prediction efficiency of the model; on the other hand, this prediction mode adds a priori restriction to the learning of the deep model, which will to a certain extent restrict the learning ability of the deep model and reduce the prediction accuracy of the model. Summary of the Invention

[0004] An embodiment of the present application provides a video coding and decoding method to solve the technical problem that the inter-frame prediction mode of the traditional video coding and decoding method causes a large amount of data and calculation of the depth model, restricts the learning ability of the depth model, and thus reduces the prediction efficiency and accuracy of the model.

[0005] In a first aspect, an embodiment of the present application provides a video encoding and decoding method, including:

[0006] Inputting a latent representation reconstruction value of a target image frame of the video to be encoded into an inter-frame prediction model in an encoder, and obtaining a latent representation prediction value of a current image frame of the video to be encoded output by the inter-frame prediction model at the encoder side; the target image frame is a plurality of image frames in a video sequence of the video to be encoded that are located before the current image frame;

[0007] Controlling the encoder to obtain a potential representation residual value of the current image frame according to a difference between a potential representation prediction value and an actual potential representation value of the current image frame at the encoder side;

[0008] reconstructing a potential representation of the current image frame according to a potential representation residual value of the current image frame to obtain a potential representation reconstructed value of the current image frame at a decoder side;

[0009] Controlling the decoder to decode the potential representation reconstruction value of the current image frame at the decoder side to obtain a reconstructed image frame of the current image frame at the decoder side;

[0010] The inter-frame prediction model is a spatiotemporal prediction model based on a sliding window video transformer.

[0011] In one embodiment, reconstructing the potential representation of the current image frame according to the potential representation residual value of the current image frame to obtain the potential representation reconstructed value of the current image frame at the decoder side includes:

[0012] Inputting the latent representation reconstruction value of the target image frame into the inter-frame prediction model in the decoder to obtain the latent representation prediction value of the current image frame output by the inter-frame prediction model at the decoder side;

[0013] Controlling the encoder to quantize and entropy encode the potential representation residual value of the current image frame to obtain a residual code stream;

[0014] If the decoder receives the residual code stream transmitted by the encoder, controlling the decoder to perform entropy decoding and inverse quantization on the residual code stream to obtain a residual reconstruction value on the decoder side;

[0015] The decoder is controlled to obtain a potential representation reconstruction value of the current image frame at the decoder side according to a sum of a potential representation prediction value of the current image frame at the decoder side and a residual reconstruction value at the decoder side.

[0016] In one embodiment, after obtaining the residual bitstream, the method includes:

[0017] Controlling the encoder to perform entropy decoding and inverse quantization on the residual bit stream to obtain a residual reconstruction value on the encoder side;

[0018] Controlling the encoder to obtain a potential representation reconstruction value of the current image frame at the encoder side according to a sum of a potential representation prediction value of the current image frame at the encoder side and a residual reconstruction value at the encoder side;

[0019] Controlling the encoder to decode the potential representation reconstruction value of the current image frame at the encoder side to obtain a reconstructed image frame of the current image frame at the encoder side;

[0020] Whether to control the encoder to transmit the residual code stream to the decoder is determined according to a distortion value of a reconstructed image frame of the current image frame at the encoder side relative to the current image frame.

[0021] In one embodiment, determining whether to control the encoder to transmit the residual code stream to the decoder based on a distortion value of a reconstructed image frame of the current image frame at the encoder side relative to the current image frame includes:

[0022] If the distortion value is greater than the distortion threshold, not controlling the encoder to transmit the residual code stream to the decoder;

[0023] If the distortion value is less than or equal to the distortion threshold, controlling the encoder to transmit the residual code stream to the decoder.

[0024] In one embodiment, if the distortion value is greater than a distortion threshold, the following steps may be performed:

[0025] Controlling the encoder to perform BPG encoding on the current image frame to obtain an encoded image frame of the current image frame;

[0026] The decoder is controlled to perform BPG decoding on the encoded image frame to obtain a decoded image frame of the current image frame.

[0027] In one embodiment, before inputting the latent representation reconstruction value of the target image frame of the video to be encoded into the inter-frame prediction model in the encoder, the method includes:

[0028] Compressing a video sequence of a video to be encoded to obtain a latent representation of an image frame in the video sequence;

[0029] Taking the first multiple image frames in the video sequence as target image frames at an initial moment, controlling the encoder to quantize and entropy encode the latent representation of the target image frames at the initial moment, and obtaining a latent representation encoding value of the target image frames at the initial moment;

[0030] The decoder is controlled to perform entropy decoding and inverse quantization on the latent representation coding value of the target image frame at the initial moment to obtain the latent representation reconstruction value of the target image frame at the initial moment.

[0031] In one embodiment, obtaining the potential representation reconstruction value of the current image frame at the decoder side includes:

[0032] Adding the latent representation reconstruction value of the current image frame at the decoder side to the set of latent representation reconstruction values ​​of the target image frame to obtain a first set;

[0033] The first set is used as the updated latent representation reconstruction value of the target image frame, and the step of inputting the latent representation reconstruction value of the target image frame into the inter-frame prediction model in the decoder is returned.

[0034] In one embodiment, obtaining the potential representation reconstruction value of the current image frame at the encoder side includes:

[0035] Adding the latent representation reconstruction value of the current image frame at the encoder side to the set of latent representation reconstruction values ​​of the target image frame to obtain a second set;

[0036] The second set is used as the updated latent representation reconstruction value of the target image frame, and the step of inputting the latent representation reconstruction value of the target image frame of the to-be-encoded video into the inter-frame prediction model in the encoder is returned.

[0037] In a second aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory storing a computer program, wherein when the processor executes the program, the steps of the video encoding and decoding method described in the first aspect are implemented.

[0038] In a third aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the video encoding and decoding method described in the first aspect.

[0039] The video encoding and decoding method provided by the present application inputs the potential representation reconstruction value of the target image frame of the video to be encoded into the inter-frame prediction model in the encoder, obtains the potential representation prediction value of the current image frame of the video to be encoded on the encoder side output by the inter-frame prediction model, the target image frame is a plurality of image frames located before the current image frame in the video sequence of the video to be encoded, controls the encoder to obtain the potential representation residual value of the current image frame based on the difference between the potential representation prediction value and the potential representation actual value of the current image frame on the encoder side, reconstructs the potential representation of the current image frame based on the potential representation residual value of the current image frame, obtains the potential representation reconstruction value of the current image frame on the decoder side, controls the decoder to decode the potential representation reconstruction value of the current image frame on the decoder side, and obtains the reconstructed image frame of the current image frame on the decoder side. On the one hand, using the potential representation of image frames for encoding and decoding, that is, using the low-dimensional representation of image frames for encoding and decoding, can effectively reduce the data volume of image frames, thereby effectively reducing the input and output data dimensions of the inter-frame prediction model, reducing the data volume and computational complexity of the model, and improving the prediction efficiency of the model; on the other hand, since the inter-frame prediction model is a spatiotemporal prediction model based on a sliding window video transformer, the model introduces an attention mechanism and has good parallel computing capabilities, which can improve coding efficiency, thereby further improving the prediction efficiency of the model. In addition, the model removes the limitations of the fixed mode of motion estimation and compensation, improves the learning freedom of the model, enables the model to better learn spatiotemporal correlations, more effectively remove spatiotemporal information redundancy, and improve the prediction accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0041] Figure 1 This is one of the flow charts of the video encoding and decoding method provided in the embodiment of the present application;

[0042] Figure 2 This is a diagram of the overall architecture of the video encoding and decoding method provided in the embodiment of the present application;

[0043] Figure 3 This is an architecture diagram of an inter-frame prediction model in a video encoding and decoding method provided in an embodiment of the present application;

[0044] Figure 4 This is a network structure diagram of a generator in the video encoding and decoding method provided in an embodiment of the present application;

[0045] Figure 5This is the second flow chart of the video encoding and decoding method provided in the embodiment of the present application;

[0046] Figure 6 This is the third flow chart of the video encoding and decoding method provided in the embodiment of the present application;

[0047] Figure 7 This is the fourth flow chart of the video encoding and decoding method provided in the embodiment of the present application;

[0048] Figure 8 This is a network structure diagram of the inter-frame prediction model in the video encoding and decoding method provided in an embodiment of the present application;

[0049] Figure 9 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions in this application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0051] Figure 1 This is one of the flow charts of the video encoding and decoding method provided in the embodiment of the present application;

[0052] Figure 2 This is a diagram of the overall architecture of the video encoding and decoding method provided in the embodiment of the present application;

[0053] Figure 3 This is an architecture diagram of an inter-frame prediction model in a video encoding and decoding method provided in an embodiment of the present application;

[0054] Figure 4 This is a network structure diagram of the generator in the video encoding and decoding method provided in an embodiment of the present application.

[0055] Reference Figure 1 , an embodiment of the present application provides a video encoding and decoding method, which may include:

[0056] 101. Inputting a latent representation reconstruction value of a target image frame of the video to be encoded into an inter-frame prediction model in an encoder to obtain a latent representation prediction value of a current image frame of the video to be encoded output by the inter-frame prediction model at the encoder side;

[0057] The target image frames are multiple image frames that are located before the current image frame in the video sequence of the video to be encoded;

[0058] 102. Control the encoder to obtain a potential representation residual value of the current image frame according to a difference between a potential representation prediction value and an actual potential representation value of the current image frame at the encoder side;

[0059] 103. Reconstruct the latent representation of the current image frame according to the latent representation residual value of the current image frame to obtain a latent representation reconstructed value of the current image frame at the decoder side;

[0060] 104. Control the decoder to decode the potential representation reconstruction value of the current image frame at the decoder side to obtain a reconstructed image frame of the current image frame at the decoder side.

[0061] The inter-frame prediction model is a spatiotemporal prediction model based on a sliding window video transformer.

[0062] Reference Figure 2 :

[0063] In step 101, the latent representation reconstruction value of the target image frame of the inter-frame prediction model P input into the encoder is The inter-frame prediction model P predicts the current image frame x based on the potential representation reconstruction value of the k target image frames. t The latent representation prediction value at the encoder side Among them, t is the current time, then the current image frame x t is the image frame at time t.

[0064] The inter-frame prediction model P is a spatiotemporal prediction model based on a sliding window video transformer. This model inherits the attention mechanism of the transformer and extends the calculation scope to the spatiotemporal domain. It also introduces a hierarchical design and a sliding window, which enables it to process spatiotemporal information, realize parallel computing, and improve the computing speed.

[0065] Reference Figure 3 The inter-frame prediction model P consists of two networks: an encoding network and a prediction network. The encoding network encodes and compresses the potential representation reconstruction values ​​of multiple image frames before the input current image frame into a hidden state tensor, and the prediction network decodes this hidden state tensor to give the potential representation prediction value of the current image frame. Among them, the encoding network is mainly composed of multiple sliding window video transformation blocks. With the help of these transformation blocks, the temporal and spatial correlations between consecutive image frames can be well extracted, and then this information is stored in the hidden state tensor. The prediction network consists of some deconvolution layers and convolution layers, whose function is to parse the hidden information, adjust the spatial resolution and number of channels to produce the final prediction data.

[0066] In step 102, the current image frame x t The latent representation prediction value at the encoder side and potential represents the actual value z tThe difference between the two is taken as the current image frame x t The potential residual value r t .

[0067] In step 104, the current image frame x can be generated by controlling the generator G in the decoder. t The latent representation reconstruction value at the decoder side Decode and get the current image frame x t Reconstructed image frame at the decoder side

[0068] Reference Figure 4 The generator consists of a deconvolution and activation layer, a multi-residual block layer, and a residual block layer. The deconvolution and activation layer is composed of a deconvolution layer, a batch normalization layer, and a leaky linear rectification layer connected in sequence. The multi-residual block layer is composed of multiple residual block layers connected in sequence. The residual block layer is composed of a deconvolution layer, a batch normalization layer, a leaky linear rectification layer, a deconvolution layer, and a batch normalization layer connected in sequence.

[0069] It should be noted that the number and connection method of the deconvolution and activation layer, multi-residual block layer and residual block layer in the generator can be set according to actual needs. The number and connection method of each layer in the deconvolution and activation layer, multi-residual block layer and residual block layer can also be set according to actual needs, which is not limited here. In this embodiment, the generator is composed of 1 deconvolution and activation layer, 6 multi-residual block layers and 5 residual block layers connected in sequence, wherein the deconvolution and activation layer is composed of 1 deconvolution layer, 1 batch normalization layer and 1 leaky linear rectification layer connected in sequence, the multi-residual block layer is composed of 3 residual block layers connected in sequence, and the residual block layer is composed of 1 deconvolution layer, 1 batch normalization layer, 1 leaky linear rectification layer, 1 deconvolution layer and 1 batch normalization layer connected in sequence.

[0070] The video encoding and decoding method provided by this embodiment inputs the potential representation reconstruction value of the target image frame of the video to be encoded into the inter-frame prediction model in the encoder, obtains the potential representation prediction value of the current image frame of the video to be encoded on the encoder side output by the inter-frame prediction model, the target image frame is a plurality of image frames located before the current image frame in the video sequence of the video to be encoded, controls the encoder to obtain the potential representation residual value of the current image frame based on the difference between the potential representation prediction value and the potential representation actual value of the current image frame on the encoder side, reconstructs the potential representation of the current image frame based on the potential representation residual value of the current image frame, obtains the potential representation reconstruction value of the current image frame on the decoder side, controls the decoder to decode the potential representation reconstruction value of the current image frame on the decoder side, and obtains the reconstructed image frame of the current image frame on the decoder side. On the one hand, using the potential representation of image frames for encoding and decoding, that is, using the low-dimensional representation of image frames for encoding and decoding, can effectively reduce the data volume of image frames, thereby effectively reducing the input and output data dimensions of the inter-frame prediction model, reducing the data volume and computational complexity of the model, and improving the prediction efficiency of the model; on the other hand, since the inter-frame prediction model is a spatiotemporal prediction model based on a sliding window video transformer, the model introduces an attention mechanism and has good parallel computing capabilities, which can improve coding efficiency, thereby further improving the prediction efficiency of the model. In addition, the model removes the limitations of the fixed mode of motion estimation and compensation, improves the learning freedom of the model, enables the model to better learn spatiotemporal correlations, more effectively remove spatiotemporal information redundancy, and improve the prediction accuracy of the model.

[0071] Figure 5 This is a second flow chart of the video encoding and decoding method provided in the embodiment of the present application; Figure 5 In one embodiment, reconstructing the latent representation of the current image frame according to the latent representation residual value of the current image frame to obtain the latent representation reconstructed value of the current image frame at the decoder side may include:

[0072] 501. Input the latent representation reconstruction value of the target image frame into the inter-frame prediction model in the decoder to obtain the latent representation prediction value of the current image frame output by the inter-frame prediction model at the decoder side;

[0073] 502. Control the encoder to quantize and entropy encode the potential representation residual value of the current image frame to obtain a residual code stream;

[0074] 503. If the decoder receives the residual code stream transmitted by the encoder, the decoder is controlled to perform entropy decoding and inverse quantization on the residual code stream to obtain a residual reconstruction value on the decoder side;

[0075] 504. Control the decoder to obtain a potential representation reconstruction value of the current image frame at the decoder side according to the sum of the potential representation prediction value of the current image frame at the decoder side and the residual reconstruction value at the decoder side.

[0076] Reference Figure 2 :

[0077] In step 501, the latent representation reconstruction value of the target image frame of the inter-frame prediction model P input into the decoder is also The inter-frame prediction model P predicts the current image frame x based on the potential representation reconstruction value of the k target image frames. t Latent representation prediction value at the decoder side

[0078] In step 502, the current image frame x t The potential residual value r t After quantization and entropy coding process Q, the residual code stream q is obtained t The quantization and entropy coding methods are not limited here. In this embodiment, the alternating direction multiplier method is used to quantize r t Quantize and use adaptive arithmetic coding to quantize r t Entropy coding is performed. Through quantization and entropy coding, the spatiotemporal redundancy between consecutive image frames can be reduced, and the residual code stream q can be reduced. t bit rate.

[0079] In step 503, the residual code stream q t After decoding and dequantization, the residual reconstruction value on the decoder side is obtained

[0080] In step 504, the current image frame x t Latent representation prediction value at the decoder side Residual reconstruction value at the decoder side The sum of the values ​​is used as the current image frame x t The latent representation reconstruction value at the decoder side

[0081] It should be noted that in actual applications, there is no strict time sequence relationship between step 501, step 502, and step 503; that is, they can be executed simultaneously, or any step can be executed first, depending on actual needs and is not limited here. However, step 502 must be executed before step 503.

[0082] This embodiment quantizes and entropy encodes the potential representation residual value of the current image frame through the encoder, which can reduce the temporal and spatial redundancy between consecutive image frames, thereby reducing the bit rate of the residual code stream and improving the encoding and decoding efficiency and accuracy. At the same time, the same prediction as the encoder side is performed on the decoder side, and the predicted potential representation prediction value of the decoder side is combined with the residual reconstruction value on the decoder side to obtain the potential representation reconstruction value of the current image frame on the decoder side. This can make the prediction and reconstruction on the decoder side independent of the prediction and reconstruction on the encoder side, avoid mutual influence, and ensure the accuracy of the decoder side's prediction and reconstruction of the potential representation of the current image frame.

[0083] Figure 6 This is the third flow chart of the video encoding and decoding method provided in the embodiment of the present application; Figure 6 In one embodiment, after obtaining the residual code stream, the following steps may be included:

[0084] 601. Control the encoder to perform entropy decoding and inverse quantization on the residual bit stream to obtain a residual reconstruction value on the encoder side;

[0085] 602. Control the encoder to obtain a potential representation reconstruction value of the current image frame at the encoder side according to a sum of a potential representation prediction value of the current image frame at the encoder side and a residual reconstruction value at the encoder side;

[0086] 603. Control the encoder to decode the latent representation reconstruction value of the current image frame on the encoder side to obtain a reconstructed image frame of the current image frame on the encoder side;

[0087] 604 : Determine whether to control the encoder to transmit a residual code stream to the decoder based on a distortion value of the reconstructed image frame of the current image frame at the encoder side relative to the current image frame.

[0088] Reference Figure 2 :

[0089] In step 601, the residual code stream q t After decoding and dequantization, the residual reconstruction value on the encoder side is obtained

[0090] In step 602, the current image frame x t The latent representation prediction value at the encoder side Residual reconstruction value from the encoder side The sum of the values ​​is used as the current image frame x t The latent representation reconstruction value at the encoder side

[0091] In step 603, the current image frame x can be generated by controlling the generator G in the encoder. t The latent representation reconstruction value at the encoder side Decode and get the current image frame x t Reconstructed image frame at the encoder side

[0092] In step 604, the current image frame x t Reconstructed image frame at the encoder side Relative to the current image frame x t The distortion value is recorded as

[0093] like If the distortion threshold Thresh is greater than the value of the encoder, the encoder is not controlled to transmit the residual code stream to the decoder, and the encoder is controlled to process the current image frame x. t Perform BPG (Better Portable Graphics) encoding to reduce distortion and prevent error propagation from causing greater coding distortion in subsequent image frames. Then control the decoder to perform BPG decoding on the BPG-encoded image frame to obtain the current image frame x t The decoded image frame using BPG is equivalent to the intra-frame coded frame in the traditional video codec, that is, the I frame.

[0094] like If the value is less than or equal to the distortion threshold Thresh, the encoder is controlled to transmit the residual code stream to the decoder.

[0095] Most traditional video encoding and decoding methods have a fixed I-frame interval. This approach cannot adaptively adjust the I-frame interval according to changes in video content, which to a certain extent limits the improvement of rate-distortion performance.

[0096] This embodiment performs the same latent representation reconstruction and image frame reconstruction on the encoder side as on the decoder side, which can make the reconstruction on the encoder side and the reconstruction on the decoder side independent of each other and avoid mutual influence, thereby ensuring the accuracy of the encoder side's reconstruction of the latent representation of the current image frame, and further ensuring the accuracy of the encoder side's reconstruction of the current image frame. At the same time, the encoder side adopts a mechanism of setting a distortion threshold to monitor the predicted coding distortion in real time and adjust the coding method to achieve adaptive setting of the I frame interval. Compared with other methods of fixing the I frame interval, this makes the setting of the I frame interval more in line with the characteristics of the video content, which is beneficial to improving the coding rate-distortion performance, and the rate-distortion performance can be controlled by changing the distortion threshold.

[0097] In addition, since the inter-frame prediction model can produce more accurate prediction values, the potential representation residual values ​​are smaller and sparser, further improving the coding rate-distortion performance.

[0098] Figure 7 This is the fourth flow chart of the video encoding and decoding method provided in the embodiment of the present application;

[0099] Figure 8 This is a network structure diagram of the inter-frame prediction model in the video encoding and decoding method provided in an embodiment of the present application.

[0100] Reference Figure 7 In one embodiment, before inputting the latent representation reconstruction value of the target image frame of the video to be encoded into the inter-frame prediction model in the encoder, the method may include:

[0101] 701. Compress a video sequence of a video to be encoded to obtain a potential representation of an image frame in the video sequence;

[0102] 702. Use the first multiple image frames in the video sequence as target image frames at an initial moment, and control the encoder to quantize and entropy encode the latent representation of the target image frames at the initial moment to obtain a latent representation coding value of the target image frames at the initial moment.

[0103] 703. Control the decoder to perform entropy decoding and inverse quantization on the latent representation coding value of the target image frame at the initial moment to obtain a latent representation reconstruction value of the target image frame at the initial moment.

[0104] Reference Figure 2 :

[0105] In step 701, the video sequence x1, ..., x T-1 ,x T Compress and obtain the potential representation z1,…,z of the image frames in the video sequence T-1 ,z T , T is the number of image frames, and the data format of the potential representation of each image frame is recorded as H×W×C, where H is the height of the image frame, W is the width of the image frame, and C is the number of channels of the image frame.

[0106] It should be noted that a suitable compression method can be selected to compress the video sequence according to actual needs, and this is not limited here. In this embodiment, the BPGAN (Back Pro pagation Generative Adversarial Networks) image compression algorithm is used to compress the video sequence. This algorithm can compress an image signal into an optimal latent representation. This latent representation is input into a trained generator network to reconstruct a high-quality, low-distortion image signal. This embodiment uses the BPGAN image compression algorithm to compress all image frames of the video sequence to be encoded to obtain the corresponding low-dimensional latent representation. In this way, we map the operation space from the original pixel space to a lower-dimensional latent space, thereby reducing the amount of data for subsequent modules and reducing the computational complexity. After performing a series of encoding and decoding operations in the latent space, the corresponding trained generator network is used to map the data in the latent space back to the pixel space.

[0107] In step 702, the number of the first multiple image frames obtained can be set according to actual needs and is not limited here. In this embodiment, frames 1-8 can be selected as the target image frames at the initial moment, and then z1, z2…, z8 are quantized and entropy encoded to obtain the potential representation coding value of the target image frame at the initial moment.

[0108] In step 703, the latent representation coding value of the target image frame at the initial moment is entropy decoded and dequantized to obtain the latent representation reconstruction value of the target image frame at the initial moment These 8 latent representation reconstruction values ​​are used to initialize the input of the inter-frame prediction models in the encoder and decoder.

[0109] Reference Figure 8 The encoding network of the inter-frame prediction model consists of three stages. The first stage consists of a module consisting of a linear embedding layer and a sliding window video transform block. Two modules can be set. The second stage consists of a module consisting of a 3D block merging layer and a sliding window video transform block. Six modules can be set. The third stage consists of a module consisting of a 3D block merging layer and a sliding window video transform block. Two modules can be set. The prediction network consists of one stage, namely the fourth stage. The fourth stage consists of a module consisting of a deconvolution and activation layer and a 1×1 convolution layer. Three modules can be set.

[0110] It should be noted that the number and structure of each component in the inter-frame prediction model can be set according to actual needs and are not limited here.

[0111] Will When inputting the inter-frame prediction model to predict the potential representation of the 9th frame, the input sequence is first divided into 3D blocks using the layer The data format 8×H×W×C is divided into blocks with a size of 2×4×4×C. 3D blocks, after the 3D block division layer, 8×H×W×C is transformed into

[0112] After the input sequence enters the first stage, the linear embedding layer is used to embed each 3D block and convert it into the input data format required by the sliding window video transformation block. The sliding window video transformation block integrates the self-attention and cross-attention mechanisms, introduces a hierarchical design and a sliding window attention calculation design, and is used to extract the temporal and spatial features of the data. After the first stage, the data format of the input sequence is changed to Transformed into Where N is an arbitrary feature dimension.

[0113] After the input sequence enters the second stage, it is downsampled using the 3D block merging layer to reduce the resolution and adjust the number of channels. The downsampling factor can be doubled each time. Therefore, for each 3D block, a pixel is selected every 2 dimensional units in its time dimension, height dimension, and width dimension to form a new 3D block. All new 3D blocks are then connected as a whole tensor and finally expanded. The data format of the input sequence is becomes That is, the time dimension, height dimension and width dimension are each reduced by two times. At this time, the channel dimension changes from N to 8N, which is 8 times the original. At this time, the channel dimension can be adjusted to twice the original through a fully connected layer, that is, adjusted to

[0114] After the input sequence enters the third stage, it undergoes a downsampling process similar to the second stage, and its data format is changed from Transformed into

[0115] After the input sequence enters the fourth stage, it is upsampled using the deconvolution and activation layers to increase the resolution, and then the 1×1 convolution layer is used to adjust the number of channels. The output is a data format that meets the prediction data. After the fourth stage, the data format of the input sequence is changed from Transformed to H×W×C, it is the potential representation prediction value of the 9th frame The data format.

[0116] As can be seen from the upper right, when predicting the potential representation of the current image frame, the potential representation reconstruction values ​​of the previous 8 frames of the current image frame can be taken as the input sequence of the inter-frame prediction model. After the three stages of the encoding network, the hidden state tensor is obtained, and its data format is That is, the time series length is reduced to 1 and the spatial resolution is reduced to the original The number of channels is 4N; the hidden state tensor is then upsampled three times (deconvolution layer) and the number of channels is adjusted (1×1 convolution layer) to obtain the final prediction data, that is, the potential representation prediction value of the current image frame, and its data format is H×w×c.

[0117] This embodiment uses the first multiple image frames in the video sequence as the target image frames at the initial moment, and directly quantizes and entropy encodes the potential representation of the target image frames at the initial moment to obtain the potential representation coding value of the target image frame at the initial moment. The potential representation coding value is then entropy decoded and dequantized to obtain the potential representation reconstruction value of the target image frame at the initial moment. The potential representation reconstruction value is used to initialize the input of the encoder and decoder, so that the encoding and decoding processes in the encoder and decoder can be started and run smoothly, thereby realizing the reconstruction of the current image frame.

[0118] Reference Figure 2 In one embodiment, after obtaining the potential representation reconstruction value of the current image frame at the decoder side, the following steps may be included:

[0119] Add the potential representation reconstruction value of the current image frame on the decoder side to the set of potential representation reconstruction values ​​of the target image frame to obtain a first set, use the first set as the updated potential representation reconstruction value of the target image frame, and return to the step of inputting the potential representation reconstruction value of the target image frame into the inter-frame prediction model in the decoder.

[0120] Similarly, after obtaining the potential representation reconstruction value of the current image frame on the encoder side, it can include:

[0121] Add the potential representation reconstruction value of the current image frame on the encoder side to the set of potential representation reconstruction values ​​of the target image frame to obtain a second set, use the second set as the updated potential representation reconstruction value of the target image frame, and return to the step of inputting the potential representation reconstruction value of the target image frame of the video to be encoded into the inter-frame prediction model in the encoder.

[0122] In this embodiment, after obtaining the potential representation reconstruction value of the current image frame on the encoder side and the decoder side, it is added to the input sequence of the inter-frame prediction model corresponding to the value, which can ensure the smooth prediction and reconstruction of the potential representation of the next image frame of the current image frame, and further ensure the smooth prediction and reconstruction of the potential representation of all image frames in the video sequence of the video to be encoded.

[0123] Figure 9 The following is a schematic diagram of the structure of an electronic device, such as Figure 9As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other via the communication bus 940. The processor 910 may call a computer program in the memory 930 to execute the steps of the video encoding and decoding method, for example, including:

[0124] Inputting a latent representation reconstruction value of a target image frame of the video to be encoded into an inter-frame prediction model in an encoder, and obtaining a latent representation prediction value of a current image frame of the video to be encoded output by the inter-frame prediction model at the encoder side; the target image frame is a plurality of image frames in a video sequence of the video to be encoded that are located before the current image frame;

[0125] Controlling the encoder to obtain a potential representation residual value of the current image frame according to a difference between a potential representation prediction value and an actual potential representation value of the current image frame at the encoder side;

[0126] reconstructing a potential representation of the current image frame according to a potential representation residual value of the current image frame to obtain a potential representation reconstructed value of the current image frame at a decoder side;

[0127] Controlling the decoder to decode the potential representation reconstruction value of the current image frame at the decoder side to obtain a reconstructed image frame of the current image frame at the decoder side;

[0128] The inter-frame prediction model is a spatiotemporal prediction model based on a sliding window video converter. In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program code.

[0129] On the other hand, embodiments of the present application further provide a computer program product, comprising a computer program. The computer program may be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer may perform the steps of the video encoding and decoding methods provided in the above embodiments, for example, including:

[0130] Inputting a latent representation reconstruction value of a target image frame of the video to be encoded into an inter-frame prediction model in an encoder, and obtaining a latent representation prediction value of a current image frame of the video to be encoded output by the inter-frame prediction model at the encoder side; the target image frame is a plurality of image frames in a video sequence of the video to be encoded that are located before the current image frame;

[0131] Controlling the encoder to obtain a potential representation residual value of the current image frame according to a difference between a potential representation prediction value and an actual potential representation value of the current image frame at the encoder side;

[0132] reconstructing a potential representation of the current image frame according to a potential representation residual value of the current image frame to obtain a potential representation reconstructed value of the current image frame at a decoder side;

[0133] Controlling the decoder to decode the potential representation reconstruction value of the current image frame at the decoder side to obtain a reconstructed image frame of the current image frame at the decoder side;

[0134] The inter-frame prediction model is a spatiotemporal prediction model based on a sliding window video transformer.

[0135] On the other hand, an embodiment of the present application further provides a processor-readable storage medium, wherein the processor-readable storage medium stores a computer program, wherein the computer program is configured to cause a processor to execute the steps of the methods provided in the above embodiments, for example, including:

[0136] Inputting a latent representation reconstruction value of a target image frame of the video to be encoded into an inter-frame prediction model in an encoder, and obtaining a latent representation prediction value of a current image frame of the video to be encoded output by the inter-frame prediction model at the encoder side; the target image frame is a plurality of image frames in a video sequence of the video to be encoded that are located before the current image frame;

[0137] Controlling the encoder to obtain a potential representation residual value of the current image frame according to a difference between a potential representation prediction value and an actual potential representation value of the current image frame at the encoder side;

[0138] reconstructing a potential representation of the current image frame according to a potential representation residual value of the current image frame to obtain a potential representation reconstructed value of the current image frame at a decoder side;

[0139] Controlling the decoder to decode the potential representation reconstruction value of the current image frame at the decoder side to obtain a reconstructed image frame of the current image frame at the decoder side;

[0140] The inter-frame prediction model is a spatiotemporal prediction model based on a sliding window video transformer.

[0141] The processor-readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO)), optical storage (such as CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (such as ROMs, EPROMs, EEPROMs, non-volatile memories (NANDFLASH), solid-state drives (SSDs)), etc.

[0142] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A video encoding and decoding method, characterized in that: include: Inputting a latent representation reconstruction value of a target image frame of the video to be encoded into an inter-frame prediction model in an encoder, and obtaining a latent representation prediction value of a current image frame of the video to be encoded output by the inter-frame prediction model at the encoder side; The target image frames are a plurality of image frames located before the current image frame in the video sequence of the video to be encoded; Controlling the encoder to obtain a potential representation residual value of the current image frame according to a difference between a potential representation prediction value and an actual potential representation value of the current image frame at the encoder side; reconstructing a potential representation of the current image frame according to a potential representation residual value of the current image frame to obtain a potential representation reconstructed value of the current image frame at a decoder side; Controlling the decoder to decode the potential representation reconstruction value of the current image frame at the decoder side to obtain a reconstructed image frame of the current image frame at the decoder side; The inter-frame prediction model is a spatiotemporal prediction model based on a sliding window video transformer.

2. The video encoding and decoding method according to claim 1, wherein: The reconstructing the potential representation of the current image frame according to the potential representation residual value of the current image frame to obtain the potential representation reconstructed value of the current image frame at the decoder side includes: Inputting the latent representation reconstruction value of the target image frame into the inter-frame prediction model in the decoder to obtain the latent representation prediction value of the current image frame output by the inter-frame prediction model at the decoder side; Controlling the encoder to quantize and entropy encode the potential representation residual value of the current image frame to obtain a residual code stream; If the decoder receives the residual code stream transmitted by the encoder, controlling the decoder to perform entropy decoding and inverse quantization on the residual code stream to obtain a residual reconstruction value on the decoder side; The decoder is controlled to obtain a potential representation reconstruction value of the current image frame at the decoder side according to a sum of a potential representation prediction value of the current image frame at the decoder side and a residual reconstruction value at the decoder side.

3. The video encoding and decoding method according to claim 2, wherein: After obtaining the residual code stream, the method includes: Controlling the encoder to perform entropy decoding and inverse quantization on the residual bit stream to obtain a residual reconstruction value on the encoder side; Controlling the encoder to obtain a potential representation reconstruction value of the current image frame at the encoder side according to a sum of a potential representation prediction value of the current image frame at the encoder side and a residual reconstruction value at the encoder side; Controlling the encoder to decode the potential representation reconstruction value of the current image frame at the encoder side to obtain a reconstructed image frame of the current image frame at the encoder side; Whether to control the encoder to transmit the residual code stream to the decoder is determined according to a distortion value of a reconstructed image frame of the current image frame at the encoder side relative to the current image frame.

4. The video encoding and decoding method according to claim 3, wherein: The determining, based on a distortion value of a reconstructed image frame of the current image frame at the encoder side relative to the current image frame, whether to control the encoder to transmit the residual code stream to the decoder includes: If the distortion value is greater than the distortion threshold, not controlling the encoder to transmit the residual code stream to the decoder; If the distortion value is less than or equal to the distortion threshold, controlling the encoder to transmit the residual code stream to the decoder.

5. The video encoding and decoding method according to claim 4, wherein: If the distortion value is greater than the distortion threshold, the method further includes: Controlling the encoder to perform BPG encoding on the current image frame to obtain an encoded image frame of the current image frame; The decoder is controlled to perform BPG decoding on the encoded image frame to obtain a decoded image frame of the current image frame.

6. The video encoding and decoding method according to claim 1, wherein: Before inputting the latent representation reconstruction value of the target image frame of the video to be encoded into the inter-frame prediction model in the encoder, the method includes: Compressing a video sequence of a video to be encoded to obtain a latent representation of an image frame in the video sequence; Taking the first multiple image frames in the video sequence as target image frames at an initial moment, controlling the encoder to quantize and entropy encode the latent representation of the target image frames at the initial moment, and obtaining a latent representation encoding value of the target image frames at the initial moment; The decoder is controlled to perform entropy decoding and inverse quantization on the latent representation coding value of the target image frame at the initial moment to obtain the latent representation reconstruction value of the target image frame at the initial moment.

7. The video encoding and decoding method according to claim 2, wherein: The obtaining of the potential representation reconstruction value of the current image frame at the decoder side includes: Adding the latent representation reconstruction value of the current image frame at the decoder side to the set of latent representation reconstruction values ​​of the target image frame to obtain a first set; The first set is used as the updated latent representation reconstruction value of the target image frame, and the step of inputting the latent representation reconstruction value of the target image frame into the inter-frame prediction model in the decoder is returned.

8. The video encoding and decoding method according to claim 3, wherein: The obtaining of the potential representation reconstruction value of the current image frame at the encoder side includes: Adding the latent representation reconstruction value of the current image frame at the encoder side to the set of latent representation reconstruction values ​​of the target image frame to obtain a second set; The second set is used as the updated latent representation reconstruction value of the target image frame, and the step of inputting the latent representation reconstruction value of the target image frame of the to-be-encoded video into the inter-frame prediction model in the encoder is returned.

9. An electronic device comprising a processor and a memory storing a computer program, characterized in that: When the processor executes the computer program, the steps of the video encoding and decoding method according to any one of claims 1 to 8 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the video encoding and decoding method according to any one of claims 1 to 8 are implemented.

Images