End-to-end video coding and decoding method, system, equipment and storage medium

Abstract

The invention provides an end-to-end video coding and decoding method, system and device and a storage medium, and belongs to the technical field of information processing. The coding method comprises the following steps: acquiring an original motion vector of a current frame image in a video; inputting motion vectors of multiple reconstructed frame images before the current frame image in the video into a VVT-based motion vector prediction network to obtain a predicted motion vector corresponding to the current frame image, and determining a motion vector residual compression code of the current frame image based on the original motion vector and the predicted motion vector; determining a decoding motion vector based on the motion vector residual compression coding of the current frame image and the prediction motion vector, and inputting the decoding motion vector and a plurality of reconstructed frame images before the current frame image into a VVT-based motion compensation network to obtain a current prediction frame image, and determining an image prediction residual error compression code based on the current frame image and the current prediction frame image. The method is used for improving the compression ratio while ensuring the video quality.

Description

Technical Field

[0001] The present disclosure relates to the field of information processing technology, and in particular to an end-to-end video encoding and decoding method, system, device, and storage medium. Background Art

[0002] The essence of video coding is to eliminate various redundant information (such as spatial, temporal, visual, statistical, etc.) in image and video signals through algorithms to achieve the purpose of signal compression.

[0003] In recent years, the introduction of video compression standards such as H.264, H.265, and AV1 has further improved video data compression ratios, but this has also gradually increased the complexity of video encoding and decoding algorithms. Furthermore, with the continuous improvement of people's material and spiritual living standards, the demand for high-definition and even ultra-high-definition video is becoming increasingly urgent. The core issue of how to ensure video quality while reducing bandwidth consumption has gradually become a research focus. Summary of the Invention

[0004] The present disclosure provides an end-to-end video encoding and decoding method, system, device and storage medium.

[0005] A first aspect of the present disclosure provides an end-to-end video encoding method, comprising:

[0006] Get the original motion vector of the current frame image in the video;

[0007] Inputting motion vectors of multiple reconstructed frame images preceding the current frame image in the video into a motion vector prediction network based on a video visual attention mechanism, obtaining a predicted motion vector corresponding to the current frame image, and determining a motion vector residual compression code for the current frame image based on the original motion vector and the predicted motion vector;

[0008] Based on the motion vector residual compression coding of the current frame image and the predicted motion vector, a decoded motion vector is determined, the decoded motion vector and the multiple frames of reconstructed frame images before the current frame image are input into a motion compensation network based on a video visual attention mechanism to obtain a current predicted frame image, and based on the current frame image and the current predicted frame image, an image prediction residual compression coding is determined.

[0009] A second aspect of the present disclosure provides an end-to-end video decoding method, comprising:

[0010] Obtaining motion vector residual compression coding and image prediction residual compression coding corresponding to the current frame image from the video code stream;

[0011] Obtaining motion vectors of a plurality of consecutive reconstructed frame images before a current frame image and inputting them into a motion vector prediction network based on a video visual attention mechanism to obtain a predicted motion vector of the current frame image;

[0012] Determine a decoded motion vector based on the predicted motion vector and the motion vector residual compression coding, input the decoded motion vector and multiple reconstructed frame images before the current frame image into a motion compensation network based on a video visual attention mechanism to obtain a current predicted frame image;

[0013] Based on the image prediction residual compression coding and the current predicted frame image, a reconstructed frame image corresponding to the current frame image is determined.

[0014] A third aspect of the present disclosure provides an end-to-end video encoding and decoding system, including:

[0015] A motion vector prediction network based on a video visual attention mechanism is used to obtain a predicted motion vector corresponding to the current frame image based on motion vectors of multiple reconstructed frame images preceding the current frame image in the video;

[0016] A motion vector residual coding module, configured to determine a motion vector residual compression coding of the current frame image based on the original motion vector of the current frame image and the predicted motion vector;

[0017] A motion vector residual decoding network, configured to determine a decoded motion vector based on the motion vector residual compression coding of the current frame image and the predicted motion vector;

[0018] A motion compensation network based on a video visual attention mechanism is used to obtain a current predicted frame image based on the decoded motion vector and multiple reconstructed frame images before the current frame image;

[0019] An image prediction residual coding module, configured to determine image prediction residual compression coding based on the current frame image and the current predicted frame image;

[0020] The image prediction residual decoding module is used to determine the reconstructed frame image corresponding to the current frame image based on the image prediction residual compression coding and the current predicted frame image.

[0021] A fourth aspect of the present disclosure provides an electronic device, including:

[0022] at least one processor;

[0023] a memory having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to the first aspect or the second aspect;

[0024] At least one I / O interface is connected between the processor and the memory and is configured to implement information interaction between the processor and the memory.

[0025] A fifth aspect of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method according to the first aspect or the second aspect.

[0026] The present disclosure has the following advantages:

[0027] During the end-to-end video encoding process, the motion vectors of multiple reconstructed frames preceding the current frame in the video are input into a motion vector prediction network based on a video visual attention mechanism to obtain a predicted motion vector corresponding to the current frame. Based on the original motion vector and the predicted motion vector, a motion vector residual compression coding of the current frame is determined. A decoded motion vector is determined based on the motion vector residual compression coding of the current frame and the predicted motion vector. The decoded motion vector and multiple reconstructed frames preceding the current frame are input into a motion compensation network based on a video visual attention mechanism to obtain a current predicted frame. Finally, an image prediction residual compression coding is determined based on the current frame and the current predicted frame. By introducing two machine learning models, a motion vector prediction network based on a video visual attention mechanism and a motion compensation network based on a video visual attention mechanism, into the end-to-end video encoding process, the inherent characteristics of the machine learning models are combined to ensure the image quality of video compression. Furthermore, the video image compression ratio is improved by combining the characteristics of multiple consecutive reconstructed frames based on the video visual attention mechanism.

[0028] Similarly, during the end-to-end video decoding process, the motion vectors of the multiple consecutive reconstructed frames preceding the current frame image are obtained and input into a motion vector prediction network based on a video visual attention mechanism to obtain the predicted motion vector of the current frame image. The decoded motion vector is determined based on the predicted motion vector and the motion vector residual compression coding. The decoded motion vector and the multiple reconstructed frames preceding the current frame image are input into a motion compensation network based on a video visual attention mechanism to obtain the current predicted frame image, and then the reconstructed frame image corresponding to the current frame image is determined. By introducing two machine learning models, a motion vector prediction network based on a video visual attention mechanism and a motion compensation network based on a video visual attention mechanism, into the end-to-end video decoding process, the image quality of the video decompression is guaranteed by combining the characteristics of the machine learning models themselves. At the same time, the video image compression ratio is improved by combining the characteristics of the video visual attention mechanism that aggregates the characteristics of multiple consecutive reconstructed frames. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic flow chart of an end-to-end video encoding method provided in an embodiment of the present disclosure;

[0030] Figure 2 A coding flow chart provided in an embodiment of the present disclosure;

[0031] Figure 3 A schematic flow chart of an end-to-end video decoding method provided in an embodiment of the present disclosure;

[0032] Figure 4 A decoding flow chart provided in an embodiment of the present disclosure;

[0033] Figure 5 A schematic diagram of a framework of an end-to-end video encoding and decoding system provided in an embodiment of the present disclosure;

[0034] Figure 6 A schematic diagram of constructing a VVT input sequence provided in an embodiment of the present disclosure;

[0035] Figure 7 This is a schematic structural diagram of an end-to-end video encoding device provided in an embodiment of the present disclosure;

[0036] Figure 8 This is a schematic diagram of the structure of an end-to-end video decoding device provided in an embodiment of the present disclosure;

[0037] Figure 9 A schematic structural diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0038] The following describes the specific embodiments of the present disclosure in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present disclosure and are not intended to limit the present disclosure.

[0039] As used in this disclosure, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0040] The terms used in the present disclosure are only used to describe specific embodiments and are not intended to limit the present disclosure.As used in the present disclosure, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0041] When the terms “comprising” and / or “made of…” are used in the present disclosure, it specifies the existence of the stated features, integers, steps, operations, elements and / or components, but does not preclude the existence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0042] Unless otherwise defined, all terms (including technical and scientific terms) used in this disclosure have the same meanings as those commonly understood by those skilled in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning unless expressly defined in this disclosure.

[0043] In the disclosed embodiment, an end-to-end video encoding and decoding method based on VVT (Video Vision Transformer, video visual attention mechanism) is proposed. In recent years, neural networks have been widely used in the field of artificial intelligence, and have triggered a widespread upsurge in in-depth research on neural networks. End-to-end video encoding and decoding based on neural networks has also become one of the hot research issues. Since the proposal of VVT, the application of VVT in video sequences has been more effective than traditional convolutional neural networks, and has received widespread attention in the field of image and video applications. Here, by introducing VVT in the end-to-end video encoding and decoding process, it is possible to further improve the video image compression ratio while ensuring the quality of the video image.

[0044] The present disclosure provides an end-to-end video encoding method. Figure 1 FIG. 1 is a flow chart of an end-to-end video encoding method. The end-to-end encoding method mainly includes:

[0045] Step 101: Obtain the original motion vector of the current frame image in the video.

[0046] In some embodiments, obtaining the original motion vector of the current frame image in the video includes: inputting the current frame image and a reconstructed frame image before the current frame image into an optical flow motion estimation network to obtain the original motion vector of the current frame image.

[0047] Step 102: Input the motion vectors of multiple reconstructed frame images in the video that are located before the current frame image into a motion vector prediction network based on a video visual attention mechanism to obtain the predicted motion vector corresponding to the current frame image, and determine the motion vector residual compression coding of the current frame image based on the original motion vector and the predicted motion vector.

[0048] In some embodiments, based on the original motion vector and the predicted motion vector, the motion vector residual compression coding of the current frame image is determined, including: obtaining the motion vector residual based on the original motion vector and the predicted motion vector; using a motion vector residual coding network to encode the motion vector residual and then quantize it to obtain the motion vector residual compression coding of the current frame image.

[0049] In some embodiments, the 0th frame (ie, the intra-frame) of the video is encoded using an intra-frame algorithm, and the first reconstructed frame image is reconstructed by decoding based on the 0th frame (ie, the intra-frame).

[0050] Step 103: Determine the decoded motion vector based on the motion vector residual compression coding of the current frame image and the predicted motion vector, input the decoded motion vector and the multiple frames of reconstructed frame images before the current frame image into a motion compensation network based on the video visual attention mechanism to obtain the current predicted frame image, and determine the image prediction residual compression coding based on the current frame image and the current predicted frame image.

[0051] In some embodiments, it also includes: using a prediction residual decoding network to decode the image prediction residual compression code to obtain a final image prediction residual; adding the final image prediction residual to the current prediction frame image to obtain a current frame image prediction result; using a filtering adjustment network to filter the current frame image prediction result to obtain a reconstructed frame image corresponding to the current frame image for encoding the next frame image.

[0052] In some embodiments, the method further includes: jointly training the motion vector prediction network based on the video visual attention mechanism and the motion compensation network based on the video visual attention mechanism, wherein one training process includes:

[0053] Obtain the distortion between the original frame image corresponding to the current frame image and the reconstructed frame image, and obtain a first bit rate for encoding the motion vector residual and a second bit rate for encoding the image prediction residual; determine a joint network loss function based on the distortion, the first bit rate and the second bit rate; terminate the training process when the loss value of the joint network loss function meets the training termination condition, and adjust the model parameters of the motion vector prediction network based on the video visual attention mechanism and the motion compensation network based on the video visual attention mechanism to optimize the loss value of the joint network loss function and execute the next training process when the training termination condition is not met.

[0054] By jointly learning the image distortion and the number of coding bits before and after video image encoding and decoding, the trained motion vector prediction network based on the video visual attention mechanism and the motion compensation network based on the video visual attention mechanism can ensure the image quality of video image encoding and decoding, while further improving the video image compression ratio.

[0055] In an exemplary embodiment, in order to jointly optimize the distortion and the number of coding bits between the original frame and the reconstructed frame, the rate-distortion under the traditional hybrid coding framework is used as the joint network loss function, which is expressed as:

[0056]

[0057] in Represents the original frame x t and reconstructed frames The distortion between R and R is measured by the mean square error (MSE). mvd represents the first code rate used to encode the motion vector residual (mvd), R res Indicates the second code rate used to encode the image prediction residual (res).

[0058] In some embodiments, the decoded motion vector is determined based on the motion vector residual compression coding of the current frame image and the predicted motion vector, including: using a motion vector residual decoding network to decode the motion vector residual compression coding to obtain a final motion vector residual; based on the final motion vector residual and the predicted motion vector, obtaining the decoded motion vector of the current frame image.

[0059] Exemplarily, the final motion vector residual and the predicted motion vector are added to obtain a decoded motion vector of the current frame image.

[0060] In some embodiments, the image prediction residual compression coding is determined based on the current frame image and the current predicted frame image, including: subtracting the current predicted frame image from the current frame image to obtain the image prediction residual; using a prediction residual coding network to encode the image prediction residual and then quantize it to obtain the image prediction residual compression coding of the current frame image.

[0061] In an exemplary embodiment, Figure 2 The encoding flow diagram shown here illustrates the end-to-end video compression process as follows:

[0062] The first step is to encode the 0th frame (intra-frame) using the intra-frame algorithm, and reconstruct the first reconstructed frame based on the 0th frame (intra-frame) decoding.

[0063] The second step is to reconstruct the previous frame image of the current frame image. The original frame image of the current frame image (x t ) is input into the optical flow motion estimation network (Flow-ME) to obtain the original motion vector (v t );

[0064] The third step is to use the motion vector mv of multiple decoded reconstructed frame images before the current frame image. Input to the VVT-based motion vector prediction network (VVT-MVP) to obtain the predicted motion vector mv of the current frame image

[0065] The fourth step is to calculate the original motion vector of the current frame image (v t ) minus the predicted motion vector mv Get the motion vector residual mvd(d t ), mvd(d t ) is input to the motion vector residual coding network (MVD-Enc), and the encoding (m t ), after quantization, the quantized compression code is obtained That is, the motion vector residual compression coding of the current frame image;

[0066] Step 5: motion vector residual mvd(d t ) quantization compression coding The final motion vector residual mvd is obtained by decompression through the motion vector residual decoding network (MVD-Dec) Final motion vector residual mvd and VVT-MVP predicted motion vector mv Add to get the decoded motion vector mv And cache it into the decoded motion vector buffer (Decoded MV Buffer);

[0067] Step 6: Decode the motion vector mv of the current frame image Multiple reconstructed frame images before the current frame image Input to the VVT-based motion compensation network (VVT-MC) to obtain the current predicted frame image

[0068] Step 7: The current original frame image (x t ) minus the current predicted frame image Get the image prediction residual (r t ), the image prediction residual (r t ) is input into the prediction residual coding network (Res-Enc), and the coding (y t ), after quantization, the quantized compression code is obtained That is, the image prediction residual compression coding of the current frame image is obtained;

[0069] Step 8: Quantization compression coding The final image prediction residual is obtained through the prediction residual decoding network (Res-Dec) Final image prediction residual With the current predicted frame image Add up to get the current frame image prediction result

[0070] Step 9: Prediction result of current frame image After filtering and adjusting the network (Frame-Filter), the final reconstructed frame image of the current frame image is obtained And cached in the decoded frame buffer (Decoded Frame Buffer), used for the next frame image of the current frame image (x t+1 )coding.

[0071] In an embodiment of the present disclosure, during an end-to-end video encoding process, motion vectors of multiple reconstructed frame images preceding the current frame image in the video are input into a motion vector prediction network based on a video visual attention mechanism to obtain a predicted motion vector corresponding to the current frame image. Based on the original motion vector and the predicted motion vector, a motion vector residual compression coding of the current frame image is determined. A decoded motion vector is determined based on the motion vector residual compression coding of the current frame image and the predicted motion vector. The decoded motion vector and multiple reconstructed frame images preceding the current frame image are input into a motion compensation network based on a video visual attention mechanism to obtain a current predicted frame image. Based on the current frame image and the current predicted frame image, an image prediction residual compression coding is determined. By introducing two machine learning models, a motion vector prediction network based on a video visual attention mechanism and a motion compensation network based on a video visual attention mechanism, into the end-to-end video encoding process, the image quality of the video compression is guaranteed by combining the characteristics of the machine learning models themselves. At the same time, by combining the characteristics of the video visual attention mechanism that aggregates the characteristics of multiple consecutive reconstructed frame images, the video image compression ratio is improved.

[0072] Correspondingly, the embodiment of the present disclosure also provides an end-to-end video decoding method, Figure 3 FIG. 1 is a flow chart of an end-to-end video decoding method. The end-to-end decoding method mainly includes:

[0073] Step 301: Obtain the motion vector residual compression code and the image prediction residual compression code corresponding to the current frame image from the video code stream.

[0074] In some embodiments, the encoding of the first frame image of the video code stream adopts an intra-frame algorithm to decode the intra-frame to obtain the first reconstructed frame image.

[0075] Step 302: Obtain motion vectors of a plurality of consecutive reconstructed frame images before the current frame image and input them into a motion vector prediction network based on a video visual attention mechanism to obtain a predicted motion vector of the current frame image.

[0076] Step 303: Determine a decoded motion vector based on the predicted motion vector and the motion vector residual compression coding, input the decoded motion vector and multiple reconstructed frame images before the current frame image into a motion compensation network based on a video visual attention mechanism to obtain a current predicted frame image.

[0077] In some embodiments, a plurality of consecutive reconstructed frame images preceding the current frame image are obtained from a decoded frame buffer.

[0078] In some embodiments, the decoded motion vector is determined based on the predicted motion vector and the motion vector residual compression coding, including: using a motion vector residual decoding network to decode the motion vector residual compression coding to obtain a final motion vector residual; based on the predicted motion vector and the final motion vector residual, obtaining the decoded motion vector of the current frame image.

[0079] In some embodiments, the decoded motion vectors are cached in a motion vector cache.

[0080] Step 304 : Determine a reconstructed frame image corresponding to the current frame image based on the image prediction residual compression coding and the current predicted frame image.

[0081] In some embodiments, based on the image prediction residual compression code and the current prediction frame image, the reconstructed frame image corresponding to the current frame image is determined, including: using a prediction residual decoding network to decode the image prediction residual compression code to obtain a final image prediction residual; based on the final image prediction residual and the current prediction frame image, determining the current frame image prediction result; using a filtering adjustment network to filter the current frame image prediction result to obtain the reconstructed frame image corresponding to the current frame image.

[0082] In an exemplary embodiment, Figure 4 The decoding flow diagram shown illustrates the end-to-end video decoding process as follows:

[0083] The first step is to use the intra-frame algorithm to decode the intra-frame and obtain the first reconstructed frame image.

[0084] The second step is to use the motion vector mv of multiple decoded reconstructed frame images before the current frame image. Input to the VVT-based motion vector prediction network (VVT-MVP) to obtain the predicted motion vector mv of the current frame image

[0085] The third step is to decode the motion vector residual MVD compression code from the code stream The final motion vector residual mvd is obtained by the nonlinear inverse transformation of the motion vector residual decoding network (MVD-Dec) and the predicted motion vector mv Add to get the decoded motion vector mv

[0086] The fourth step is to decode the motion vector mv Multiple reconstructed frame images before the current frame image Input to the VVT-based motion compensation network (VVT-MC) to obtain the current predicted frame image

[0087] The fifth step is to decode the code stream to obtain the image prediction residual compression code The final image prediction residual is obtained by decoding the prediction residual decoding network (Res-Dec) Final image prediction residual With the current predicted frame image Add up to get the current frame image prediction result

[0088] Step 6: Current frame image prediction results After filtering and adjusting the network (Frame-Filter), the final reconstructed frame image of the current frame is obtained And cached in the decoded frame buffer (Decoded Frame Buffer), used for the next frame image of the current frame image (x t+1 )coding.

[0089] In the disclosed embodiment, during the end-to-end video decoding process, the motion vectors of the consecutive multiple reconstructed frame images preceding the current frame image are obtained and input into a motion vector prediction network based on a video visual attention mechanism to obtain the predicted motion vector of the current frame image. The decoded motion vector is determined based on the predicted motion vector and the motion vector residual compression coding. The decoded motion vector and the multiple reconstructed frame images preceding the current frame image are input into a motion compensation network based on a video visual attention mechanism to obtain the current predicted frame image, and then the reconstructed frame image corresponding to the current frame image is determined. By introducing two machine learning models, a motion vector prediction network based on a video visual attention mechanism and a motion compensation network based on a video visual attention mechanism, into the end-to-end video decoding process, the image quality of the video decompression is guaranteed by combining the characteristics of the machine learning models themselves. At the same time, the video image compression ratio is improved by combining the characteristics of the video visual attention mechanism that aggregates the characteristics of multiple consecutive reconstructed frame images.

[0090] The steps of the various methods above are divided only for clarity of description. During implementation, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this disclosure. Adding insignificant modifications or introducing insignificant designs to the algorithm or process without changing the core design of the algorithm and process are all within the scope of protection of this disclosure.

[0091] Based on the same concept, the embodiment of the present disclosure also provides an end-to-end video encoding and decoding system. Figure 5 The figure shows a schematic diagram of the framework of the end-to-end video encoding and decoding system, which mainly includes:

[0092] A video visual attention mechanism-based motion vector prediction network (VVT-MVP) is used to obtain a predicted motion vector corresponding to the current frame image based on the motion vectors of multiple reconstructed frames in the video that precede the current frame image;

[0093] A motion vector residual coding module, configured to determine a motion vector residual compression coding of the current frame image based on the original motion vector of the current frame image and the predicted motion vector;

[0094] A motion vector residual decoding network (MVD-Dec), configured to determine a decoded motion vector based on the motion vector residual compression coding of the current frame image and the predicted motion vector;

[0095] A video visual attention-based motion compensation network (VVT-MC) is configured to obtain a current predicted frame image based on the decoded motion vector and multiple reconstructed frame images preceding the current frame image;

[0096] An image prediction residual coding module, configured to determine image prediction residual compression coding based on the current frame image and the current predicted frame image;

[0097] The image prediction residual decoding module is used to determine the reconstructed frame image corresponding to the current frame image based on the image prediction residual compression coding and the current predicted frame image.

[0098] In some embodiments, the motion vector residual encoding module includes:

[0099] a motion vector residual coding network (MVD-Enc), configured to encode the motion vector residual;

[0100] The first quantization submodule is used to quantize the motion vector residual after the motion vector residual coding network coding to obtain the motion vector residual compression code of the current frame image.

[0101] In some embodiments, the image prediction residual coding module includes:

[0102] A prediction residual coding network (Res-Enc), configured to encode the image prediction residual;

[0103] The second quantization submodule is used to quantize the image prediction residual after the prediction residual coding network is coded to obtain the image prediction residual compression code of the current frame image.

[0104] In some embodiments, the image prediction residual decoding module includes:

[0105] A prediction residual decoding network (Res-Dec), configured to decode the image prediction residual compression coding to obtain a final image prediction residual;

[0106] A filter adjustment network (Frame-Filter) is used to filter the current frame image prediction result obtained by adding the final image prediction residual to the current prediction frame image to obtain a reconstructed frame image corresponding to the current frame image.

[0107] In some embodiments, the system further includes an optical flow motion estimation network (Flow-ME) for obtaining an original motion vector of the current frame image based on a previous reconstructed frame image of the current frame image (Current Frame) and an original frame image of the current frame image.

[0108] In some embodiments, the system further includes a decoded frame buffer (Decoded Frame Buffer) for caching the reconstructed frame image for encoding and decoding of the next frame image.

[0109] In some embodiments, the system further includes a decoded motion vector buffer (Decoded MV Buffer) for caching decoded motion vectors.

[0110] The end-to-end video encoding and decoding system provided by the embodiments of the present disclosure introduces two machine learning models, a motion vector prediction network based on VVT and a motion compensation network, under the traditional hybrid coding framework. The image quality of video decompression is guaranteed by combining the characteristics of the machine learning models themselves. At the same time, the video image compression ratio is improved by combining the characteristics of multiple continuous reconstructed frame images based on the video visual attention mechanism.

[0111] In addition to the two machine learning models of the VVT-based motion vector prediction network and the motion compensation network, the filter adjustment network can also be implemented using a machine learning model. In the process of jointly training the motion vector prediction network based on the video visual attention mechanism and the motion compensation network based on the video visual attention mechanism, the filter adjustment network is also jointly trained.

[0112] In the above embodiments, the platform on which the system runs can support basic deep learning operations, such as supporting deep learning model implementations such as VVT-MVP, VVT-MC, FlowME, MVD-Enc / Dec, and Res-Enc / Dec.

[0113] In the above embodiments, Figure 6 The figure shows the schematic diagram of constructing the VVT ​​input sequence, where W, H, and T represent the input video sequence frame width, frame height, and number of frames, respectively. t ,...,x t+i represents the input video frame t to frame t+i, p1, p2, ..., p j 、p j+1 Represents the corresponding VVT building block (also called patch block) in video frame t to frame t+i. In the video sequence, n 2D frame data (reconstructed frame image or MV frame data) are sampled each time, and a 16X16 patch block is sampled within each frame. Then, all the patch blocks in n frames are horizontally linked and input into the transformer.

[0114] In the above embodiment, VVT-MVP is used to predict the motion vector of the current frame to be reconstructed, and the input is the motion vector of the reconstructed frame image sequence from frame t-1 to frame ti (assuming the current frame is t), with dimensions of ix W x H; the output is the motion vector of the current frame image, with dimensions of W x H, where W and H are the width and height of the frame sequence image, i is the number of frames of the input reconstructed frame image, and L is the input and output block length.

[0115] In the above embodiment, VVT-MC is used to predict the current frame to be reconstructed. The input consists of two parts: one is the reconstruction value of the sequence of images from frame t-1 to frame ti in the reconstructed frame image, and the other is the motion vector of the sequence of frames t-1 to frame ti in the reconstructed frame image, with dimensions of ix W x H and (i+1)x W x H respectively; the output is the reconstruction value of the current frame image, with dimensions of W x H, where W and H are the width and height of the frame sequence image, i is the number of input reconstructed frames, and L is the input and output block lengths.

[0116] In the above embodiment, the filter adjustment network is used to reduce the block edge effect of the current reconstructed frame (prediction value plus residual value). The input is the reconstruction value of the current reconstructed frame, with a dimension of W x H. After passing through the frame filter adjustment network, the output dimension is W x H. The filter adjustment network can adopt a traditional CNN convolutional network or a vision transformer network.

[0117] An embodiment of the present disclosure provides an end-to-end video encoding device. The specific implementation of the device can be found in the relevant description of the method embodiment, which will not be repeated here. Figure 7 The figure shows the structure of the device, which mainly includes:

[0118] An acquisition module 701 is used to acquire the original motion vector of the current frame image in the video;

[0119] A motion vector encoding module 702 is configured to input motion vectors of multiple reconstructed frames preceding the current frame in the video into a motion vector prediction network based on a video visual attention mechanism, obtain a predicted motion vector corresponding to the current frame, and determine a motion vector residual compression code for the current frame based on the original motion vector and the predicted motion vector;

[0120] The prediction residual coding module 703 determines the decoded motion vector based on the motion vector residual compression coding of the current frame image and the predicted motion vector, inputs the decoded motion vector and the multiple frames of reconstructed frame images before the current frame image into the motion compensation network based on the video visual attention mechanism, obtains the current predicted frame image, and determines the image prediction residual compression coding based on the current frame image and the current predicted frame image.

[0121] An end-to-end video decoding device is provided in an embodiment of the present disclosure. The specific implementation of the device can be found in the relevant description of the method embodiment, which will not be repeated here. Figure 8 The figure shows the structure of the device, which mainly includes:

[0122] An acquisition module 801 is configured to obtain a motion vector residual compression code and an image prediction residual compression code corresponding to a current frame image from a video bitstream;

[0123] A motion vector prediction module 802 is configured to obtain motion vectors of a plurality of consecutive reconstructed frames preceding a current frame image and input the obtained motion vectors into a motion vector prediction network based on a video visual attention mechanism to obtain a predicted motion vector of the current frame image; wherein N is an integer greater than or equal to 1;

[0124] A decoding module 803 is configured to determine a decoded motion vector based on the predicted motion vector and the motion vector residual compression coding, and input the decoded motion vector and multiple reconstructed frame images before the current frame image into a motion compensation network based on a video visual attention mechanism to obtain a current predicted frame image;

[0125] The compensation module 804 is configured to determine a reconstructed frame image corresponding to the current frame image based on the image prediction residual compression coding and the current predicted frame image.

[0126] The functions or modules included in the apparatus provided in the embodiments of the present disclosure can be used to execute the method described in the method embodiments. The specific implementation and technical effects thereof can be referred to the description of the above method embodiments, and will not be repeated here for the sake of brevity.

[0127] It should be noted that all modules involved in this embodiment are logical modules. In actual applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, to highlight the innovations of this disclosure, this embodiment does not include units that are not closely related to solving the technical problems proposed by this disclosure. However, this does not mean that other units do not exist in this embodiment.

[0128] Reference Figure 9 , an embodiment of the present disclosure provides an electronic device, comprising:

[0129] at least one processor 901;

[0130] a memory 902 storing at least one program, which, when executed by the at least one processor, causes the at least one processor to implement the above method;

[0131] At least one I / O interface 903 is connected between the processor and the memory and is configured to implement information exchange between the processor and the memory.

[0132] Among them, the processor 901 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 902 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read-write interface) 903 is connected between the processor 901 and the memory 902, and can realize information interaction between the processor 901 and the memory 902, including but not limited to a data bus (Bus), etc.

[0133] In some embodiments, the processor 901 , the memory 902 , and the I / O interface 903 are connected to each other via a bus, and further connected to other components of the computing device.

[0134] This embodiment further provides a computer-readable medium on which a computer program is stored. When the program is executed by a processor, the method provided in this embodiment is implemented. To avoid repeated description, the specific steps of the method are not repeated here.

[0135] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods applied for above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable, and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0136] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0137] Those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present embodiment and to form different embodiments.

[0138] It is understood that the above embodiments are merely exemplary embodiments for illustrating the principles of the present disclosure, and the present disclosure is not limited thereto. Those skilled in the art may make various modifications and improvements without departing from the spirit and substance of the present disclosure, and such modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. An end-to-end video encoding method, characterized in that: include: Get the original motion vector of the current frame image in the video; Inputting motion vectors of multiple reconstructed frame images in the video before the current frame image into a motion vector prediction network based on a video visual attention mechanism, obtaining a predicted motion vector corresponding to the current frame image, and determining a motion vector residual compression coding of the current frame image based on the original motion vector and the predicted motion vector; Based on the motion vector residual compression coding of the current frame image and the predicted motion vector, a decoded motion vector is determined, the decoded motion vector and multiple reconstructed frame images before the current frame image are input into a motion compensation network based on a video visual attention mechanism to obtain a current predicted frame image, and based on the current frame image and the current predicted frame image, an image prediction residual compression coding is determined.

2. The end-to-end video encoding method according to claim 1, characterized in that The method further comprises: Decoding the image prediction residual compression code using a prediction residual decoding network to obtain a final image prediction residual; Adding the final image prediction residual to the current prediction frame image to obtain a current frame image prediction result; A filter adjustment network is used to filter the prediction result of the current frame image to obtain a reconstructed frame image corresponding to the current frame image for encoding the next frame image.

3. The end-to-end video encoding method according to claim 2, characterized in that: The method further comprises: The motion vector prediction network based on the video visual attention mechanism and the motion compensation network based on the video visual attention mechanism are jointly trained, and a training process includes: Obtaining a degree of distortion between an original frame image corresponding to the current frame image and the reconstructed frame image, and obtaining a first code rate for encoding the motion vector residual and a second code rate for encoding the image prediction residual; Determining a joint network loss function based on the distortion, the first bit rate, and the second bit rate; When the loss value of the joint network loss function meets the training termination condition, the training process is terminated. When the training termination condition is not met, the model parameters of the motion vector prediction network based on the video visual attention mechanism and the motion compensation network based on the video visual attention mechanism are adjusted to optimize the loss value of the joint network loss function and perform the next training process.

4. The end-to-end video encoding method according to claim 1, characterized in that: Determining the motion vector residual compression coding of the current frame image based on the original motion vector and the predicted motion vector includes: Obtaining a motion vector residual based on the original motion vector and the predicted motion vector; The motion vector residual is encoded by a motion vector residual coding network and then quantized to obtain the motion vector residual compression coding of the current frame image.

5. The end-to-end video encoding method according to claim 4, characterized in that: Determining a decoded motion vector based on the motion vector residual compression coding of the current frame image and the predicted motion vector includes: Using a motion vector residual decoding network to decode the motion vector residual compression code to obtain a final motion vector residual; Based on the final motion vector residual and the predicted motion vector, a decoded motion vector of the current frame image is obtained.

6. The end-to-end video encoding method according to claim 1, characterized in that: Determining image prediction residual compression coding based on the current frame image and the current predicted frame image includes: Subtracting the current predicted frame image from the current frame image to obtain an image prediction residual; The image prediction residual is encoded by using a prediction residual coding network and then quantized to obtain the image prediction residual compression coding of the current frame image.

7. An end-to-end video decoding method, characterized in that: include: Obtaining motion vector residual compression coding and image prediction residual compression coding corresponding to the current frame image from the video bit stream; Obtaining motion vectors of a plurality of consecutive reconstructed frame images before a current frame image and inputting them into a motion vector prediction network based on a video visual attention mechanism to obtain a predicted motion vector of the current frame image; Determine a decoded motion vector based on the predicted motion vector and the motion vector residual compression coding, input the decoded motion vector and multiple reconstructed frame images before the current frame image into a motion compensation network based on a video visual attention mechanism to obtain a current predicted frame image; Based on the image prediction residual compression coding and the current prediction frame image, a reconstructed frame image corresponding to the current frame image is determined.

8. The end-to-end video decoding method according to claim 7, characterized in that: The method of compressing and encoding the predicted motion vector and the motion vector residual to determine the decoded motion vector comprises: Using a motion vector residual decoding network to decode the motion vector residual compression code to obtain a final motion vector residual; Based on the predicted motion vector and the final motion vector residual, a decoded motion vector of the current frame image is obtained.

9. The end-to-end video decoding method according to claim 7, characterized in that: Determining a reconstructed frame image corresponding to the current frame image based on the image prediction residual compression coding and the current prediction frame image includes: Decoding the image prediction residual compression code using a prediction residual decoding network to obtain a final image prediction residual; Determine a current frame image prediction result based on the final image prediction residual and the current prediction frame image; A filter adjustment network is used to filter the prediction result of the current frame image to obtain a reconstructed frame image corresponding to the current frame image.

10. An end-to-end video encoding and decoding system, characterized in that: include: A motion vector prediction network based on a video visual attention mechanism is used to obtain a predicted motion vector corresponding to the current frame image based on motion vectors of multiple reconstructed frame images before the current frame image in the video; A motion vector residual coding module, used for determining the motion vector residual compression coding of the current frame image based on the original motion vector of the current frame image and the predicted motion vector; A motion vector residual decoding network, used for determining a decoded motion vector based on the motion vector residual compression coding of the current frame image and the predicted motion vector; A motion compensation network based on a video visual attention mechanism is used to obtain a current predicted frame image based on the decoded motion vector and multiple reconstructed frame images before the current frame image; An image prediction residual coding module, used for determining image prediction residual compression coding based on the current frame image and the current prediction frame image; The image prediction residual decoding module is used to determine the reconstructed frame image corresponding to the current frame image based on the image prediction residual compression coding and the current prediction frame image.

11. The system according to claim 10, characterized in that The motion vector residual encoding module comprises: A motion vector residual coding network, used for encoding the motion vector residual; The first quantization submodule is used to quantize the motion vector residual after the motion vector residual coding network coding to obtain the motion vector residual compression coding of the current frame image.

12. The system according to claim 10, characterized in that The image prediction residual coding module comprises: A prediction residual coding network, used for encoding the image prediction residual; The second quantization submodule is used to quantize the image prediction residual after the prediction residual coding network is encoded to obtain the image prediction residual compression coding of the current frame image.

13. The system according to claim 10, characterized in that The image prediction residual decoding module comprises: A prediction residual decoding network, used for decoding the image prediction residual compression coding to obtain a final image prediction residual; The filter adjustment network is used to filter the current frame image prediction result obtained by adding the final image prediction residual to the current prediction frame image to obtain a reconstructed frame image corresponding to the current frame image.

14. An electronic device, characterized in that: include: at least one processor; A memory having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6 or the method according to any one of claims 7 to 9; At least one I / O interface is connected between the processor and the memory and is configured to implement information interaction between the processor and the memory.

15. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 6 or the method according to any one of claims 7 to 9 is implemented.

Images