End-to-end video encoding and decoding method, system, device and storage medium

Abstract

The present invention discloses an end-to-end video encoding and decoding method, system, device and storage medium, which are corresponding solutions. In the solution, additional spatial references are introduced for end-to-end video encoding, and a new motion information encoding and decoding and motion alignment method are designed to solve the problems encountered by end-to-end video in representing large motion information and describing newly appeared objects; specifically, a low-resolution video encoder is introduced, and decoded low-resolution reconstructed motion information and low-resolution reconstructed frames are used to perform iterative enhancement at the decoding end to obtain higher-quality motion information. At the same time, the low-resolution reconstructed frames are also continuously enhanced in the iterative enhancement to provide more additional spatial reference frames. Based on the higher-quality motion information and richer reference frames, the quality of the predicted context is enhanced, thereby improving the encoding and decoding performance.

Description

Technical Field

[0001] The present invention relates to the field of video encoding and decoding technology, and in particular to an end-to-end video encoding and decoding method, system, device and storage medium. Background Art

[0002] During video storage and transmission, it is usually necessary to encode the video to reduce storage capacity and transmission bandwidth. The video encoding and decoding process usually includes motion information encoding and decoding, context prediction, and frame encoding and decoding. During the encoding process, the motion information is first encoded into a binary code stream. Then, the reconstructed motion information is used to align the reference frame to obtain the predicted context. Finally, the predicted context is used to encode the current frame into a binary code stream. During the decoding process, the binary code stream of the motion information is first decoded into reconstructed motion information. Then, the reconstructed motion information is used to align the reference frame to obtain the predicted context. Finally, the predicted context is used to decode the binary code stream of the current frame into a reconstructed frame. Among them, the higher the quality of the predicted context in the prediction process, the higher the compression ratio that can be achieved at the same reconstruction quality.

[0003] In recent years, with the rapid development of deep learning and deep learning-based end-to-end image coding, end-to-end video coding has also seen significant advancements. End-to-end video coding replaces all modules in the video coding system with learnable modules based on neural networks, optimized using a rate-distortion loss function. Better module combinations and utilization methods are key to improving the compression ratio of end-to-end video coding. In particular, enhancing the quality of reconstructed motion information and optimizing motion alignment techniques can significantly improve the quality of predicted context, thereby achieving higher coding compression ratios.

[0004] Prior art 1: a motion information encoding and decoding method for enhancing the motion information representation capability.

[0005] Motion information encoding and decoding will obtain reconstructed motion information at the decoding end. The quality of reconstructed motion information affects the quality of context prediction in motion alignment. Generally speaking, the reference frame is used at the encoding end. and the current frame x t Estimate the motion information m t The motion information will be encoded into a binary stream by the motion information encoder, and the decoder will decode the binary stream to obtain the reconstructed motion information. The representation capability and quality of the reconstructed motion information will affect the subsequent motion alignment process. The following describes existing motion information encoding and decoding methods that enhance the representation capability of motion information.

[0006] One implementation method is to enhance the motion information representation capability at the decoding end. Decoding obtains reconstructed motion information, which is further fused with reference frames and reference features to generate multiple motion information offset maps. The motion information offset map is added to the reconstructed motion information to obtain multiple enhanced motion information. This enhancement method cannot perceive the content of the current frame and has limited enhancement effect; the encoded motion information is simple and has large distortion, which cannot effectively process large motion information.

[0007] Another implementation method is to enhance the motion information representation capability during the encoding and decoding process. When the encoder performs motion information estimation, it estimates multiple motion information offset maps. And directly encode the motion information offset map instead of encoding a motion information m t The decoding end obtains multiple reconstructed motion information offset maps This method uses multiple motion information offset maps estimated at the encoder to enhance motion information representation, enabling perception of the current frame's content and reducing motion information encoding distortion. However, this method requires encoding multiple motion information offset maps, increasing the bitrate required for motion information encoding. Furthermore, motion information offsets are limited by the local receptive field of convolutional networks, making it difficult to effectively model large motion information and limiting their representation capabilities.

[0008] The related articles of prior art 1 are as follows:

[0009] Article 1: Li J, Li B, Lu Y.Neural video compression with diverse contexts[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition.2023:22616-22626.

[0010] Article 2: Hu Z, Lu G, Xu D.FVC: A new framework towards deep videocompression in feature space[C] / / Proceedings of the IEEE / CVF conference oncomputer vision and pattern recognition.2021:1502-1511.

[0011] Prior art 2: motion alignment method for enhanced context prediction.

[0012] After obtaining the reconstructed motion information, effective motion alignment can obtain high-quality prediction context. Motion alignment aligns the content of the reference frame to the current temporal position through motion. The accuracy of this operation is affected by the quality of the motion information and the quality of the reference frame. The higher the quality and the stronger the representation ability of the motion information, the better the prediction context. and higher quality, more diverse reference frames Both can effectively improve the quality of the predicted context. Higher-quality predicted context directly affects the encoding compression ratio of the current frame. The following describes how to perform context prediction with enhanced motion alignment after obtaining reconstructed motion information.

[0013] In one implementation, the reference frame Extract multiple scale features and will reconstruct motion information Also downsampled to reconstruct motion information at multiple scales Then multiple motion alignments are performed on the corresponding scales to obtain context predictions at multiple scales. This multi-scale motion alignment enhances the utilization of reference frame information and can alleviate the problem of inaccurate reconstructed motion information caused by large motion information at small scales. However, the prediction context generated using only the temporal reference frame is difficult to handle occlusions and newly appeared objects.

[0014] In another implementation, multiple motion information offset maps obtained by motion information encoding are obtained. This method extracts the reference frame as a feature And use multiple motion information offset maps to perform multiple alignment operations on different channels of the feature to obtain the predicted context The predicted context is expressed by the feature F of the current frame. t The subtraction is performed to obtain the residual, and the bitrate of the current frame is reduced by encoding and decoding the residual. The compression efficiency of residual coding has been shown to be inferior to that of conditional coding methods; although different motion information offset maps are used for motion alignment to make better use of the reference frame content, its reference content is still limited and it is difficult to handle occlusion and newly appeared objects.

[0015] The related articles of prior art 2 are as follows:

[0016] Article 3: Sheng X, Li J, Li B, et al. Temporal context mining for learned video compression[J]. IEEE Transactions on Multimedia, 2022, 25: 7311-7322.

[0017] Article 4: Hu Z, Xu D, Lu G, et al. Fvc: An end-to-end framework towards deep video compression in feature space [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(4): 4569-4585. Summary of the Invention

[0018] The purpose of the present invention is to provide an end-to-end video encoding and decoding method, system, device and storage medium, which can effectively model large motion information and effectively describe newly appearing objects in the current frame.

[0019] The purpose of the present invention is achieved through the following technical solutions:

[0020] An end-to-end video encoding and decoding method, comprising:

[0021] Downsampling processing: The current frame is spatially downsampled, and the obtained video frame is called the low-resolution current frame;

[0022] Low-resolution encoding and decoding: encoding and decoding the low-resolution current frame to obtain a low-resolution reconstructed frame and reconstructed motion information;

[0023] Iteratively enhanced context prediction: Combining a full-resolution temporal reference frame, a low-resolution reconstructed frame, and reconstructed motion information, iteratively enhances the reconstructed motion information, thereby enhancing the context information. The final output context information is used for full-resolution encoding and decoding of the current frame. In addition, upsampling is performed between iterative enhancements so that the resolution of the final output context information is the same as that of the current frame. The full-resolution temporal reference frame is the reconstructed frame obtained through full-resolution encoding and decoding when encoding and decoding the previous frame.

[0024] Full-resolution encoding and decoding: Combine the context information obtained by iterative enhanced context prediction to encode and decode the current frame to obtain a reconstructed frame.

[0025] An end-to-end video encoding and decoding system, comprising:

[0026] The downsampling module is used to perform spatial downsampling on the current frame. The obtained video frame is called the low-resolution current frame.

[0027] A low-resolution encoding and decoding model, used to encode and decode the low-resolution current frame to obtain a low-resolution reconstructed frame and reconstructed motion information;

[0028] An iteratively enhanced context prediction model is configured to combine a full-resolution temporal reference frame, a low-resolution reconstructed frame, and reconstructed motion information, iteratively enhance the reconstructed motion information, and thereby enhance the context information. The context information ultimately output is used for full-resolution encoding and decoding of the current frame, and upsampling is performed between iterative enhancements so that the resolution of the ultimately output context information is the same as the resolution of the current frame. The full-resolution temporal reference frame is a reconstructed frame obtained by full-resolution encoding and decoding when encoding and decoding the previous frame.

[0029] The full-resolution codec model is used to encode and decode the current frame in combination with the context information obtained by iterative enhanced context prediction to obtain a reconstructed frame.

[0030] A processing device comprising: one or more processors; a memory for storing one or more programs;

[0031] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0032] A readable storage medium stores a computer program, which implements the aforementioned method when the computer program is executed by a processor.

[0033] It can be seen from the technical solution provided by the present invention that an additional spatial reference is introduced for end-to-end video coding, and a new motion information encoding and decoding and motion alignment method is designed (corresponding to the iterative enhanced context prediction solution provided by the present invention) to solve the problems encountered by end-to-end video in large motion information representation and description of newly appeared objects; specifically, a low-resolution video encoder is introduced, and the decoded low-resolution reconstructed motion information and low-resolution reconstructed frames are iteratively enhanced at the decoding end to obtain higher-quality motion information. At the same time, the low-resolution reconstructed frames will also be continuously enhanced in the iterative enhancement to provide higher-quality additional spatial reference information. Based on higher-quality motion information and richer reference frames, the quality of the predicted context is enhanced, which can improve the encoding and decoding performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 A flowchart of an end-to-end video encoding and decoding method provided by an embodiment of the present invention;

[0036] Figure 2A schematic diagram of the overall framework of an end-to-end video encoding and decoding method provided by an embodiment of the present invention;

[0037] Figure 3 A schematic diagram of low-resolution video encoding and decoding provided by an embodiment of the present invention;

[0038] Figure 4 A schematic diagram of iteratively enhanced context prediction provided by an embodiment of the present invention;

[0039] Figure 5 A schematic diagram showing a quality comparison between the reconstructed motion information after iterative enhancement provided by an embodiment of the present invention and the existing model DCVC-DC;

[0040] Figure 6 A schematic diagram showing a quality comparison between the iteratively enhanced context information provided by the present invention and the existing model DCVC-DC;

[0041] Figure 7 A schematic diagram of an end-to-end video encoding and decoding system provided by an embodiment of the present invention;

[0042] Figure 8 A schematic diagram of a processing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0044] First, the following terms may be used in this article:

[0045] The terms "include," "comprises," "contains," "has," or other similar expressions should be interpreted as non-exclusive. For example, "including certain technical features (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products, or manufactured articles, etc.) should be interpreted as including not only the technical features explicitly listed, but also other technical features known in the art that are not explicitly listed.

[0046] The term "consisting of" excludes any technical features not explicitly listed. If used in a claim, this term renders the claim closed, excluding any technical features other than those explicitly listed, except for conventional impurities associated with them. If this term appears only in a clause of a claim, it limits only the elements explicitly listed in that clause; elements listed in other clauses are not excluded from the claim as a whole.

[0047] The following describes in detail an end-to-end video encoding and decoding method, system, device, and storage medium provided by the present invention. Any content not described in detail in the embodiments of the present invention belongs to the prior art known to professionals in the field. Where specific conditions are not specified in the embodiments of the present invention, the process is carried out in accordance with conventional conditions in the art or the conditions recommended by the manufacturer. Reagents or instruments used in the embodiments of the present invention, where the manufacturer is not specified, are all commercially available conventional products.

[0048] Example 1

[0049] The embodiment of the present invention adopts an end-to-end video encoding and decoding method, such as Figure 1 As shown, it mainly includes the following steps:

[0050] Step 1: Downsampling processing.

[0051] In this step, the current frame is spatially downsampled, and the obtained video frame is called a low-resolution current frame.

[0052] Step 2: Low-resolution encoding and decoding.

[0053] In this step, the low-resolution current frame is encoded and decoded to obtain a low-resolution reconstructed frame and reconstructed motion information.

[0054] The preferred implementation of this step is as follows: Combine the low-resolution temporal reference frame and Perform motion coding to obtain motion information, and then decode to obtain reconstructed motion information Among them, the low-resolution temporal reference frame is the low-resolution reconstructed frame obtained when encoding and decoding the previous frame (i.e., when the encoding and decoding method of the present invention is executed on the previous frame, obtained by step 2), t is the index of the current frame; after combining the motion information with the low-resolution time domain reference frame for context prediction, the low-resolution current frame is encoded, and the low-resolution reconstructed frame is obtained by decoding

[0055] Step 3: Iteratively enhance context prediction.

[0056] In this step, the full-resolution time domain reference frame, the low-resolution reconstructed frame and the reconstructed motion information are combined to iteratively enhance the reconstructed motion information, thereby enhancing the context information. The context information finally output is used for full-resolution encoding and decoding of the current frame, and upsampling is performed in the gaps of iterative enhancement so that the resolution of the context information finally output is the same as the resolution of the current frame; wherein the full-resolution time domain reference frame is the reconstructed frame obtained by full-resolution encoding and decoding when encoding and decoding the previous frame (that is, when the encoding and decoding method of the present invention is executed on the previous frame, it is obtained by step 4).

[0057] The preferred implementation of this step is as follows:

[0058] In the i-th enhancement stage, combined with the full-resolution temporal reference frame Enhanced context information obtained in the previous enhancement stage and the enhanced motion information obtained in the previous enhancement stage Perform motion enhancement to obtain enhanced motion information of the i-th enhancement stage Where i is the index of the enhancement stage. When i=1, the enhanced context information obtained in the previous enhancement stage is Reconstructing frames for low resolution Enhanced motion information obtained in the previous enhancement stage To reconstruct motion information t is the index of the current frame.

[0059] Utilizing enhanced motion information Full resolution temporal reference frame Perform motion information compensation to obtain a time-domain aligned reference frame, and use the time-domain aligned reference frame to enhance the context information obtained in the previous enhancement stage Perform context prediction enhancement to obtain enhanced context information for the i-th enhancement stage

[0060] Repeat N times to obtain the enhanced context information of the Nth enhancement stage That is the context information of the final output; and, in the gap between iterative enhancements, that is, between adjacent enhancement stages, the enhanced context information is upsampled, and the total upsampling ratio is the same as the spatial domain downsampling ratio.

[0061] Step 4: Full resolution encoding and decoding.

[0062] In this step, the current frame is encoded and decoded in combination with the context information obtained by iteratively enhancing the context prediction to obtain a reconstructed frame.

[0063] In this embodiment of the present invention, if the first frame is encoded and decoded at full resolution, an image encoder is used in conjunction with existing schemes. Starting from the second frame, encoding and decoding are performed using the scheme provided by the present invention. In other words, the current frame range described in the present invention is from the second frame to the last frame. The above description uses the encoding and decoding process of the current frame as an example. All frames in the video are encoded and decoded using the above method, ultimately completing the encoding and decoding of the entire video.

[0064] Preferably, full-resolution encoding and decoding is implemented by a full-resolution encoding and decoding model, low-resolution encoding and decoding is implemented by a low-resolution encoding and decoding model, and iterative enhanced context prediction is implemented by an iterative enhanced context prediction model. The full-resolution encoding and decoding model, the low-resolution encoding and decoding model, and the iterative enhanced context prediction model are trained in the following manner:

[0065] Construct the following loss function:

[0066]

[0067] Where t is the index of the current frame, t = 2,…,T, T is the total number of frames; L1, L2 and L3 are three loss functions, D(·,·) is the distortion loss, R(.) is the bit rate requirement; x t is the current frame, To reconstruct the frame; is the low-resolution current frame, is the low-resolution reconstructed frame; λ is the parameter for controlling the bit rate, and w is the parameter for regulating the low-resolution reconstruction quality (i.e., the distortion loss of the low-resolution codec model) and the full-resolution reconstruction quality (the distortion loss of the full-resolution codec model);

[0068] Use the loss function L1 to optimize the low-resolution video encoding and decoding model;

[0069] Use the L2 loss function to optimize the full-resolution encoding and decoding model and the iterative enhanced context prediction model;

[0070] Afterwards, the loss function L3 is used to optimize the low-resolution video codec model, the full-resolution codec model and the iterative enhanced context prediction model.

[0071] In the above solution provided by the embodiment of the present invention, the downsampling method can be arbitrarily selected, the low-resolution downsampling ratio can be arbitrarily specified, and the number of iterative enhancements N is not limited.

[0072] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the method provided by the embodiment of the present invention is described in detail below with reference to specific embodiments.

[0073] 1. Overall introduction of the plan.

[0074] Considering that the existing motion information encoding and decoding and motion alignment methods encode a single motion alignment with limited temporal reference frames, on the one hand, it is difficult to effectively model large motion information, and on the other hand, the limited temporal reference frames are difficult to effectively describe newly appearing objects in the current frame.

[0075] The present invention provides an end-to-end video encoding and decoding method to solve the above problems. The present invention introduces an additional spatial reference (i.e., encoding full resolution frame x t When low-resolution reconstructed frames are introduced ), a new motion information encoding and decoding and motion alignment method (i.e., the iterative enhanced context prediction scheme provided above) is designed to solve the problems encountered by end-to-end video in large motion information representation and description of newly appeared objects. Specifically, the full-resolution motion information encoding and decoding is improved to a low-resolution video codec, and the decoded low-resolution reconstructed motion information and low-resolution reconstructed frames are used for iterative enhancement at the decoding end to obtain higher-quality motion information. At the same time, the low-resolution reconstructed frames will also be continuously enhanced in the iterative enhancement to provide more additional spatial reference frames. Based on higher-quality motion information and richer reference frames, the quality of the predicted context is enhanced.

[0076] The above method provided by the present invention only improves some links, and therefore can be applied to any end-to-end video coding system.

[0077] Figure 2 The overall framework of the end-to-end video encoding and decoding method provided by the present invention is presented. The current frame encoding and decoding process can be implemented with reference to conventional technologies, so it is not described in detail. The following mainly focuses on low-resolution encoding and decoding (low-resolution video encoding and decoding) and iterative enhanced context prediction.

[0078] 1. Low-resolution codec.

[0079] like Figure 3 The figure shows a schematic diagram of low-resolution encoding and decoding. This process is similar to the commonly used video encoding / decoding system, which requires relying on the temporal reference frame for motion information encoding, context prediction, and current frame encoding and decoding. The difference is that all encoding is performed on the low-resolution spatial domain, that is, the low-resolution current frame to be encoded. is the current frame x t The low-resolution frame is obtained by spatial downsampling, and the low-resolution temporal reference frame is the encoding of the previous low-resolution frame x t-1 The resulting low-resolution reconstructed frame

[0080] The main process is as follows: Combine the low-resolution temporal reference frame and the low-resolution current frame Perform motion coding to obtain motion information, and then decode to obtain reconstructed motion information Combining motion information with low-resolution temporal reference frames for context prediction, the low-resolution current frame is encoded and decoded to obtain a low-resolution reconstructed frame.

[0081] 2. Iteratively enhance context prediction.

[0082] Through the aforementioned low-resolution encoding and decoding, low-resolution reconstructed motion information can be obtained and low-resolution reconstructed frames With the temporal reference frame Together we enter the iterative enhanced context prediction process. In the iterative enhanced context prediction, multiple enhancement stages are performed to continuously enhance the quality of motion information and context prediction. and low-resolution reconstructed frames Provides motion information and spatial reference for encoding the current frame.

[0083] like Figure 4 As shown, it is a schematic diagram of iteratively enhanced context prediction. The initial input and In the equivalent of and Final output Equivalent to the final output context information C t In each enhancement stage, the temporal reference frame Enhanced context information obtained in the previous enhancement stage and the enhanced motion information obtained in the previous enhancement stage A motion enhancement will be performed together to obtain the enhanced motion information of the i-th enhancement stage (i represents the index of the enhancement stage). Get the enhanced motion information of the i-th enhancement stage The temporal reference frame Motion information compensation is performed to obtain a time-domain aligned reference frame, which is aligned with the enhanced context information obtained in the previous enhancement stage. Perform a context prediction enhancement together to obtain the enhanced context information of the i-th enhancement stage

[0084] The enhanced context needs to be added between multiple enhancement stages and enhanced motion information Continuously upsample. The final enhanced context This is the final output context information C tThere is no additional restriction on the upsampling interval here. Users can set it based on actual conditions or experience. They only need to ensure that the upsampling ratio is the same as the aforementioned spatial domain downsampling ratio, so that the resolution of the final output context information is the same as the resolution of the current frame image.

[0085] In addition, each model involved in the above method provided in the embodiment of the present invention needs to be trained.

[0086] The distortion loss is recorded as D(·,·), and the reconstructed frame is obtained by encoding The estimated bit rate requirement is Define the following three loss functions:

[0087]

[0088] Here, λ is a parameter that controls the bitrate and is used to train models with different compression ratios. w is a parameter that controls the quality of low-resolution and full-resolution reconstruction. A smaller w means less bitrate is allocated to low-resolution reconstruction, while a larger w means more bitrate is allocated to low-resolution reconstruction.

[0089] The specific training process is as follows:

[0090] Low-resolution video codec model training: using low-resolution video frame data and combining it with L1 loss function optimization;

[0091] Full-resolution codec model and iterative enhanced context prediction model training: using low-resolution video frame data and full-resolution video frame data, combined with loss function L2 optimization;

[0092] Overall model training: Use low-resolution video frame data, full-resolution video frame data, and combine with loss function L3 optimization.

[0093] Among the above three optimization processes, the first two optimization processes can be executed in parallel or in any order. After the current two optimization processes are completed, the third optimization process (i.e., overall model training) is entered.

[0094] 2. Example introduction.

[0095] Example 1: A method based on 4x low-resolution video encoding and decoding and 6 enhancement stages.

[0096] The downsampling ratio is set to 4 times, bicubic (bilinear interpolation algorithm) is used for downsampling, the number of enhancement stages N=6, and upsampling is performed every 2 enhancements (a total of 2 upsamplings, and no upsampling is performed in the last 2).

[0097] Example 2: A method based on 4x low-resolution video encoding and decoding and 9 enhancement stages.

[0098] The downsampling ratio is set to 4 times, bilinear downsampling is used, the number of enhancement stages N=9, and upsampling is performed every 3 enhancements (a total of 2 upsamplings, and no upsampling is performed in the last 2).

[0099] Example 3: A method based on 8x low-resolution video encoding and decoding and 8 enhancement stages.

[0100] The downsampling ratio is set to 8 times, bicubic downsampling is used, the number of enhancement stages N=8, and upsampling is performed every 2 enhancements (a total of 3 upsamplings, and no upsampling is performed in the last 2).

[0101] The sampling ratio, sampling method, and number of enhancement stages involved in the above process are all examples. In actual applications, they can be set by the user according to actual conditions or experience, and the present invention does not impose any restrictions.

[0102] 3. Description of the effect.

[0103] Here, the effect of the present invention is mainly illustrated by experiments. In the experiments, the end-to-end video coding model DCVC-DC is used as a low-resolution encoder.

[0104] 1. Large motion information modeling effect.

[0105] On large motion sequences, it can be observed that the quality of reconstructed motion information after iterative enhancement has been significantly improved, such as Figure 5 shown.

[0106] Taking a large motion sequence as an example, Figure 5 The motion information of RAFT GT (pseudo motion information) is used as a pseudo label to measure the quality of motion information; AEPE (Average End Point Error) is used as an indicator to measure the quality of motion information. The lower the AEPE, the more accurate the motion information representation; SSIM (Structural Similarity Index) is used as another indicator to measure the quality of motion information. It reflects the effect of alignment using motion information. The higher the SSIM, the better the alignment effect. Compared with DCVC-DC, the present invention has better motion information representation and motion alignment effects, which means it has better reconstructed motion information.

[0107] 2. Describe the effects of the newly appeared objects.

[0108] In the preceding and following frames with obvious new objects, it can be observed that the iteratively enhanced context prediction accurately describes the new objects, e.g. Figure 6 shown.

[0109] exist Figure 6In part (c), the object (hammer) that appears in the current frame is Figure 6 It does not appear in the temporal reference frame shown in part (a) of , so existing methods only use the context prediction generated by the temporal reference, i.e. Figure 6 Part (b) of the present invention is difficult to describe the object, and the prediction is inaccurate, which leads to a decrease in coding efficiency. In the present invention, due to the introduction of low-resolution spatial domain reference and the continuous enhancement of its description in the context prediction process during the iterative enhancement context prediction, it can be seen that the prediction context of the present invention, i.e. Figure 6 Part (d) effectively describes Figure 6 The new objects appear in part (c).

[0110] 3. Positive effect on coding performance.

[0111] The present invention achieves excellent results in these two areas on common test datasets such as HEVC, UVG, MCL-JCV, and USTC-TD. As shown in Table 1, encoding performance gains are measured using BD-rate (RGB-BDBR), with negative values ​​representing the percentage of bitrate savings and positive values ​​representing the percentage of bitrate gains. The existing technology is used as a baseline for comparison. Applying low-resolution video encoding and decoding and iteratively enhanced context prediction significantly improves compression ratios.

[0112] Table 1: Performance comparison between the proposed method and existing end-to-end video coding models

[0113]

[0114] The first column in Table 1 is the name of each scheme, the first 6 rows are the performance of the models corresponding to the existing schemes on different data, and the last row SEVC (ours) is the performance of the scheme of the present invention on different data. The scheme described in Example 1 above is used in the experiment. The smaller the value in Table 1, the better the encoding performance (higher bit rate saving). The anchor for comparison is VTM (H.266 standard encoder). The values ​​of VTM are all 0.0, so if the value is negative, it means bit rate saving relative to VTM.

[0115] Through the description of the above embodiments, those skilled in the art will clearly understand that the above embodiments can be implemented through software or by using software plus a necessary general-purpose hardware platform. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, a USB flash drive, a mobile hard disk, etc.) and includes a number of instructions for causing a computer device (such as a personal computer, a server, or a network device) to execute the methods described in the various embodiments of the present invention.

[0116] Example 2

[0117] The present invention also provides an end-to-end video encoding and decoding system, which is mainly used to implement the method provided in the above embodiment, such as Figure 7 As shown, the system mainly includes:

[0118] The downsampling module is used to perform spatial downsampling on the current frame. The obtained video frame is called the low-resolution current frame.

[0119] A low-resolution encoding and decoding model, used to encode and decode the low-resolution current frame to obtain a low-resolution reconstructed frame and reconstructed motion information;

[0120] An iteratively enhanced context prediction model is configured to combine a full-resolution temporal reference frame, a low-resolution reconstructed frame, and reconstructed motion information, iteratively enhance the reconstructed motion information, and thereby enhance the context information. The context information ultimately output is used for full-resolution encoding and decoding of the current frame, and upsampling is performed between iterative enhancements so that the resolution of the ultimately output context information is the same as the resolution of the current frame. The full-resolution temporal reference frame is a reconstructed frame obtained by full-resolution encoding and decoding when encoding and decoding the previous frame.

[0121] The full-resolution codec model is used to encode and decode the current frame in combination with the context information obtained by iterative enhanced context prediction to obtain a reconstructed frame.

[0122] Furthermore, encoding and decoding the low-resolution current frame to obtain a low-resolution reconstructed frame and reconstructed motion information includes:

[0123] Combined with the low-resolution temporal reference frame, the low-resolution current frame Perform motion coding to obtain motion information, and then decode to obtain reconstructed motion information Among them, the low-resolution temporal reference frame is the low-resolution reconstructed frame obtained when encoding and decoding the previous frame t is the index of the current frame;

[0124] Combining motion information with low-resolution temporal reference frames for context prediction, the low-resolution current frame is encoded and decoded to obtain a low-resolution reconstructed frame.

[0125] Furthermore, the combining of the full-resolution temporal reference frame, the low-resolution reconstructed frame and the reconstructed motion information, iteratively enhancing the reconstructed motion information, and thereby enhancing the context information includes:

[0126] In the i-th enhancement stage, combined with the full-resolution temporal reference frame Enhanced context information obtained in the previous enhancement stage and the enhanced motion information obtained in the previous enhancement stage Perform motion enhancement to obtain enhanced motion information of the i-th enhancement stage Where i is the index of the enhancement stage. When i=1, the enhanced context information obtained in the previous enhancement stage is Reconstructing frames for low resolution Enhanced motion information obtained in the previous enhancement stage To reconstruct motion information t is the index of the current frame;

[0127] Utilizing enhanced motion information Full resolution temporal reference frame Perform motion information compensation to obtain a time-domain aligned reference frame, and use the time-domain aligned reference frame to enhance the context information obtained in the previous enhancement stage Perform context prediction enhancement to obtain enhanced context information for the i-th enhancement stage

[0128] Repeat N times to obtain the enhanced context information of the Nth enhancement stage That is the context information of the final output; and, in the gap between iterative enhancements, that is, between adjacent enhancement stages, the enhanced context information is upsampled, and the total upsampling ratio is the same as the spatial domain downsampling ratio.

[0129] Furthermore, the full-resolution codec model, the low-resolution codec model and the iterative enhanced context prediction model are trained in the following manner:

[0130] Construct the following loss function:

[0131]

[0132] Where t is the index of the current frame; L1, L2 and L3 are three loss functions, D(·,·) is the distortion loss, R(.) is the bit rate requirement; x t is the current frame, To reconstruct the frame; is the low-resolution current frame, is the low-resolution reconstructed frame; λ is the parameter for controlling the bit rate, and w is the parameter for regulating the low-resolution reconstruction quality and the full-resolution reconstruction quality;

[0133] Use the loss function L1 to optimize the low-resolution video encoding and decoding model;

[0134] Use the L2 loss function to optimize the full-resolution encoding and decoding model and the iterative enhanced context prediction model;

[0135] Afterwards, the loss function L3 is used to optimize the low-resolution video codec model, the full-resolution codec model and the iterative enhanced context prediction model.

[0136] Those skilled in the art will clearly understand that for the convenience and brevity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0137] Example 3

[0138] The present invention also provides a processing device, such as Figure 8 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the aforementioned embodiment.

[0139] Furthermore, the processing device further includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0140] In the embodiment of the present invention, the specific types of the memory, input device, and output device are not limited; for example:

[0141] The input device can be a touch screen, image acquisition device, physical button or mouse;

[0142] The output device may be a display terminal;

[0143] The memory may be a random access memory (RAM) or a non-volatile memory, such as a disk memory.

[0144] Example 4

[0145] The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0146] In the embodiments of the present invention, the computer-readable storage medium may be provided in the aforementioned processing device, for example, as a memory in the processing device. Alternatively, the computer-readable storage medium may be a USB flash drive, a removable hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk, among other media capable of storing program code.

[0147] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or any form of implication that the information constitutes prior art already known to those skilled in the art.

Claims

1. An end-to-end video encoding and decoding method, characterized in that: include: Downsampling processing: The current frame is spatially downsampled, and the obtained video frame is called the low-resolution current frame; Low-resolution encoding and decoding: encoding and decoding the low-resolution current frame to obtain a low-resolution reconstructed frame and reconstructed motion information; Iteratively enhanced context prediction: Combining a full-resolution temporal reference frame, a low-resolution reconstructed frame, and reconstructed motion information, iteratively enhances the reconstructed motion information, thereby enhancing the context information. The final output context information is used for full-resolution encoding and decoding of the current frame. In addition, upsampling is performed between iterative enhancements so that the resolution of the final output context information is the same as that of the current frame. The full-resolution temporal reference frame is the reconstructed frame obtained through full-resolution encoding and decoding when encoding and decoding the previous frame. Full-resolution encoding and decoding: Combine the context information obtained by iterative enhanced context prediction to encode and decode the current frame to obtain a reconstructed frame.

2. The end-to-end video encoding and decoding method according to claim 1, wherein: The encoding and decoding of the low-resolution current frame to obtain a low-resolution reconstructed frame and reconstructed motion information includes: Combined with the low-resolution temporal reference frame, the low-resolution current frame Perform motion coding to obtain motion information, and then decode to obtain reconstructed motion information Among them, the low-resolution temporal reference frame is the low-resolution reconstructed frame obtained when encoding and decoding the previous frame t is the index of the current frame; Combining motion information with low-resolution temporal reference frames for context prediction, the low-resolution current frame is encoded and decoded to obtain a low-resolution reconstructed frame.

3. The end-to-end video encoding and decoding method according to claim 1, wherein: The combining of the full-resolution temporal reference frame, the low-resolution reconstructed frame and the reconstructed motion information, iteratively enhancing the reconstructed motion information, and thereby enhancing the context information includes: In the i-th enhancement stage, combined with the full-resolution temporal reference frame Enhanced context information obtained in the previous enhancement stage and the enhanced motion information obtained in the previous enhancement stage Perform motion enhancement to obtain enhanced motion information of the i-th enhancement stage Where i is the index of the enhancement stage. When i=1, the enhanced context information obtained in the previous enhancement stage is Reconstructing frames for low resolution Enhanced motion information obtained in the previous enhancement stage To reconstruct motion information t is the index of the current frame; Utilizing enhanced motion information Full resolution temporal reference frame Perform motion information compensation to obtain a time-domain aligned reference frame, and use the time-domain aligned reference frame to enhance the context information obtained in the previous enhancement stage Perform context prediction enhancement to obtain enhanced context information for the i-th enhancement stage Repeat N times to obtain the enhanced context information of the Nth enhancement stage That is the context information of the final output; and, in the gap between iterative enhancements, that is, between adjacent enhancement stages, the enhanced context information is upsampled, and the total upsampling ratio is the same as the spatial domain downsampling ratio.

4. The end-to-end video encoding and decoding method according to claim 1, wherein: Full-resolution encoding and decoding is implemented by a full-resolution encoding and decoding model, low-resolution encoding and decoding is implemented by a low-resolution encoding and decoding model, and iterative enhanced context prediction is implemented by an iterative enhanced context prediction model; the full-resolution encoding and decoding model, the low-resolution encoding and decoding model, and the iterative enhanced context prediction model are trained in the following manner: Construct the following loss function: Where t is the index of the current frame; L1, L2 and L3 are three loss functions, D(·,·) is the distortion loss, R(.) is the bit rate requirement; x t is the current frame, To reconstruct the frame; is the low-resolution current frame, is the low-resolution reconstructed frame; λ is the parameter for controlling the bit rate, and w is the parameter for regulating the low-resolution reconstruction quality and the full-resolution reconstruction quality; Use the loss function L1 to optimize the low-resolution video encoding and decoding model; Use the L2 loss function to optimize the full-resolution encoding and decoding model and the iterative enhanced context prediction model; Afterwards, the loss function L3 is used to optimize the low-resolution video codec model, the full-resolution codec model and the iterative enhanced context prediction model.

5. An end-to-end video encoding and decoding system, characterized in that: include: The downsampling module is used to perform spatial downsampling on the current frame. The obtained video frame is called the low-resolution current frame. A low-resolution encoding and decoding model, used to encode and decode the low-resolution current frame to obtain a low-resolution reconstructed frame and reconstructed motion information; An iteratively enhanced context prediction model is configured to combine a full-resolution temporal reference frame, a low-resolution reconstructed frame, and reconstructed motion information, iteratively enhance the reconstructed motion information, and thereby enhance the context information. The context information ultimately output is used for full-resolution encoding and decoding of the current frame, and upsampling is performed between iterative enhancements so that the resolution of the ultimately output context information is the same as the resolution of the current frame. The full-resolution temporal reference frame is a reconstructed frame obtained by full-resolution encoding and decoding when encoding and decoding the previous frame. The full-resolution codec model is used to encode and decode the current frame in combination with the context information obtained by iterative enhanced context prediction to obtain a reconstructed frame.

6. The end-to-end video encoding and decoding system according to claim 5, wherein: The encoding and decoding of the low-resolution current frame to obtain a low-resolution reconstructed frame and reconstructed motion information includes: Combined with the low-resolution temporal reference frame, the low-resolution current frame Perform motion coding to obtain motion information, and then decode to obtain reconstructed motion information Among them, the low-resolution temporal reference frame is the low-resolution reconstructed frame obtained when encoding and decoding the previous frame t is the index of the current frame; Combining motion information with low-resolution temporal reference frames for context prediction, the low-resolution current frame is encoded and decoded to obtain a low-resolution reconstructed frame.

7. The end-to-end video encoding and decoding system according to claim 5, wherein: The combining of the full-resolution temporal reference frame, the low-resolution reconstructed frame and the reconstructed motion information, iteratively enhancing the reconstructed motion information, and thereby enhancing the context information includes: In the i-th enhancement stage, combined with the full-resolution temporal reference frame Enhanced context information obtained in the previous enhancement stage and the enhanced motion information obtained in the previous enhancement stage Perform motion enhancement to obtain enhanced motion information of the i-th enhancement stage Where i is the index of the enhancement stage. When i=1, the enhanced context information obtained in the previous enhancement stage is Reconstructing frames for low resolution Enhanced motion information obtained in the previous enhancement stage To reconstruct motion information t is the index of the current frame; Utilizing enhanced motion information Full resolution temporal reference frame Perform motion information compensation to obtain a time-domain aligned reference frame, and use the time-domain aligned reference frame to enhance the context information obtained in the previous enhancement stage Perform context prediction enhancement to obtain enhanced context information for the i-th enhancement stage Repeat N times to obtain the enhanced context information of the Nth enhancement stage That is the context information of the final output; and, in the gap between iterative enhancements, that is, between adjacent enhancement stages, the enhanced context information is upsampled, and the total upsampling ratio is the same as the spatial domain downsampling ratio.

8. The end-to-end video encoding and decoding system according to claim 5, wherein: The full-resolution codec model, the low-resolution codec model, and the iterative enhanced context prediction model are trained in the following manner: Construct the following loss function: Where t is the index of the current frame; L1, L2 and L3 are three loss functions, D(·,·) is the distortion loss, R(.) is the bit rate requirement; x t is the current frame, To reconstruct the frame; is the low-resolution current frame, is the low-resolution reconstructed frame; λ is the parameter for controlling the bit rate, and w is the parameter for regulating the low-resolution reconstruction quality and the full-resolution reconstruction quality; Use the loss function L1 to optimize the low-resolution video encoding and decoding model; Use the L2 loss function to optimize the full-resolution encoding and decoding model and the iterative enhanced context prediction model; Afterwards, the loss function L3 is used to optimize the low-resolution video codec model, the full-resolution codec model and the iterative enhanced context prediction model.

9. A processing device, characterized in that: include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images