Video coding method and device, electronic equipment and non-transient computer readable storage medium

Abstract

The present disclosure provides a video encoding method and apparatus, an electronic device, and a non-transitory computer readable storage medium, the video encoding method comprising: for each adjacent frame of a current frame, performing motion estimation on a current block in the current frame, searching for a plurality of motion vectors from the adjacent frames, and encoding a video on the basis of the plurality of motion vectors searched from the adjacent frames, performing motion compensation on the current block to obtain a motion compensation result for each adjacent frame; performing time domain filtering on the current block based on the motion compensation result of each adjacent frame in the at least one adjacent frame; and encoding the current block based on the time domain filtering result of the current block. In the present disclosure, each adjacent frame is no longer limited to only one motion compensation prediction, but a plurality of motion compensation predictions can be used, that is, each adjacent frame is allowed to correspond to a plurality of motion compensation results. Therefore, the prediction precision of motion compensation can be remarkably improved, and the filtering effect and the coding performance of the MCTF can be further improved.

Description

Technical Field

[0001] The present disclosure relates to video coding and compression, and more particularly to a video coding method and apparatus thereof, an electronic device and a non-transitory computer-readable storage medium. Background Art

[0002] Various electronic devices (e.g., digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smart phones, video teleconferencing devices, video streaming devices, etc.) support digital video. Electronic devices send and receive or otherwise transmit digital video data through a communication network, and / or store digital video data on a storage device. Due to the limited bandwidth capacity of the communication network and the limited storage resources of the storage device, before the video data is transmitted or stored, the video data can be compressed using a video codec according to one or more video codec standards. For example, video codec standards include general video codec (VVC), joint exploration test model (JEM), high efficiency video codec (HEVC / H.265), advanced video codec (AVC / H.264), moving picture expert group (MPEG) codec, etc. Video codecs generally use prediction methods (e.g., inter-frame prediction, intra-frame prediction, etc.) that utilize the inherent redundancy in video data. Video codecs are intended to compress video data into a form that uses a lower bit rate while avoiding or minimizing the degradation of video quality.

[0003] Motion Compensated Temporal Filter (MCTF) is an encoder-only video filtering technology that uses the temporal correlation of the video and the inter-block reference relationship to perform temporal filtering on the reference frame to reduce the temporal redundant information generated in the video block reference process, thereby improving the overall coding efficiency.

[0004] In related technologies, when using MCTF for time domain filtering, each frame uses the classic block-based motion compensation technology, that is, only one motion compensation block is searched for each frame, and then multiple motion compensation blocks corresponding to all adjacent frames of the current frame and the current block are subjected to bilateral weighted filtering. However, using only a single hypothesis compensation for each frame may result in low prediction quality, especially unable to meet the prediction requirements for some complex scenes, resulting in poor time domain filtering effect and low coding efficiency. Summary of the invention

[0005] Examples of the present disclosure provide a video encoding method and apparatus thereof, an electronic device, and a non-transitory computer-readable storage medium.

[0006] According to a first aspect of the present disclosure, a video encoding method is provided, comprising: for each adjacent frame of at least one adjacent frame of a current frame, performing the following operations: performing motion estimation on a current block in the current frame, searching for multiple motion vectors from adjacent frames, and performing motion compensation on the current block based on the multiple motion vectors searched from the adjacent frames to obtain a motion compensation result, thereby obtaining a motion compensation result for each adjacent frame in the at least one adjacent frame; performing time domain filtering on the current block based on the motion compensation result for each adjacent frame in the at least one adjacent frame to obtain a time domain filtering result of the current block; and encoding the current block based on the time domain filtering result of the current block.

[0007] Optionally, performing motion compensation on the current block based on the multiple motion vectors searched from the adjacent frames includes: performing weighted summation on the multiple motion vectors searched from the adjacent frames to obtain a final motion vector; and performing motion compensation on the current block based on the final motion vector.

[0008] Optionally, the performing motion compensation on the current block based on the multiple motion vectors searched from the adjacent frames includes: performing motion compensation on the current block respectively based on the multiple motion vectors searched from the adjacent frames to obtain multiple motion compensation results; performing weighted summation on the multiple motion compensation results to obtain a final motion compensation result; wherein, the performing time domain filtering on the current block based on the motion compensation result for each adjacent frame in the at least one adjacent frame includes: performing time domain filtering on the current block based on the final motion compensation result obtained for each adjacent frame in the at least one adjacent frame.

[0009] Optionally, the performing motion compensation on the current block based on the multiple motion vectors searched from the adjacent frames includes: performing motion compensation on the current block respectively based on the multiple motion vectors searched from the adjacent frames to obtain multiple motion compensation results; wherein, the performing time domain filtering on the current block based on the motion compensation result for each adjacent frame in the at least one adjacent frame includes: performing time domain filtering on the current block based on the multiple motion compensation results obtained for each adjacent frame in the at least one adjacent frame.

[0010] Optionally, searching for multiple motion vectors from adjacent frames includes: searching for multiple motion vector candidates from the adjacent frames using the same motion vector search method; and selecting a predetermined number of optimal motion vector candidates from the multiple motion vector candidates according to a predetermined rule as the multiple motion vectors.

[0011] Optionally, searching for multiple motion vectors from adjacent frames includes: searching for multiple motion vector candidates from the adjacent frames by using different motion vector search methods to determine the multiple motion vectors.

[0012] Optionally, searching for multiple motion vectors from adjacent frames includes: searching for a first motion vector candidate from the adjacent frames by a first motion vector search method, and searching for at least one second motion vector candidate from the adjacent frames by at least one second motion vector search method, and selecting a predetermined number of optimal second motion vector candidates from the at least one second motion vector candidate according to a predetermined rule; and determining the first motion vector candidate and the predetermined number of optimal second motion vector candidates as the multiple motion vectors.

[0013] Optionally, the weights used for the weighted summation may be determined according to a prediction characteristic corresponding to each motion vector, or may be determined according to a machine learning or deep learning approach.

[0014] Optionally, the weights used for the weighted summation may be determined according to a prediction characteristic corresponding to each motion compensation result, or may be determined according to a machine learning or deep learning approach.

[0015] Optionally, the filtering weights used for the time domain filtering may be determined according to the prediction features corresponding to each motion compensation result.

[0016] According to a second aspect of the present disclosure, a video encoding device is provided, comprising: a motion estimation module and a motion compensation module, configured to perform the following operations for each adjacent frame in at least one adjacent frame of a current frame: perform motion estimation on a current block in the current frame, search for multiple motion vectors from adjacent frames, and perform motion compensation on the current block based on the multiple motion vectors searched from the adjacent frames to obtain a motion compensation result, thereby obtaining a motion compensation result for each adjacent frame in the at least one adjacent frame; a time domain filtering module, configured to perform time domain filtering on the current block based on the motion compensation result for each adjacent frame in the at least one adjacent frame to obtain a time domain filtering result for the current block; and an encoding module, configured to encode the current block based on the time domain filtering result of the current block.

[0017] Optionally, the motion estimation module and the motion compensation module are configured to: perform weighted summation on the multiple motion vectors searched from the adjacent frames to obtain a final motion vector; and perform motion compensation on the current block based on the final motion vector.

[0018] Optionally, the motion estimation module and the motion compensation module are configured to: perform motion compensation on the current block based on the multiple motion vectors searched from the adjacent frames to obtain multiple motion compensation results; perform weighted summation on the multiple motion compensation results to obtain a final motion compensation result; and the time domain filtering module is configured to: perform time domain filtering on the current block based on the final motion compensation result obtained for each adjacent frame in the at least one adjacent frame.

[0019] Optionally, the motion estimation module and the motion compensation module are configured to: perform motion compensation on the current block based on the multiple motion vectors searched from the adjacent frames, and obtain multiple motion compensation results; the time domain filtering module is configured to: perform time domain filtering on the current block based on the multiple motion compensation results obtained for each adjacent frame in the at least one adjacent frame.

[0020] Optionally, the motion estimation module and the motion compensation module are configured to: search for multiple motion vector candidates from the adjacent frames by using the same motion vector search method; and select a predetermined number of optimal motion vector candidates from the multiple motion vector candidates as the multiple motion vectors according to a predetermined rule.

[0021] Optionally, the motion estimation module and the motion compensation module are configured to: search for a plurality of motion vector candidates from the adjacent frames by using different motion vector search methods to determine the plurality of motion vectors.

[0022] Optionally, the motion estimation module and the motion compensation module are configured to: search for a first motion vector candidate from the adjacent frames by a first motion vector search method, and search for at least one second motion vector candidate from the adjacent frames by at least one second motion vector search method, and select a predetermined number of optimal second motion vector candidates from the at least one second motion vector candidate according to a predetermined rule; and determine the first motion vector candidate and the predetermined number of optimal second motion vector candidates as the multiple motion vectors.

[0023] Optionally, the weights used for the weighted summation may be determined according to a prediction characteristic corresponding to each motion vector, or may be determined according to a machine learning or deep learning approach.

[0024] Optionally, the weights used for the weighted summation may be determined according to a prediction characteristic corresponding to each motion compensation result, or may be determined according to a machine learning or deep learning approach.

[0025] Optionally, the filtering weights used for the time domain filtering may be determined according to the prediction features corresponding to each motion compensation result.

[0026] According to a third aspect of the present disclosure, a device for video encoding is provided, comprising: one or more processors; and a memory, coupled to the one or more processors and configured to store instructions that can be executed by the one or more processors, wherein the one or more processors are configured to execute the video encoding method according to the present disclosure when executing the instructions.

[0027] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium for storing computer-executable instructions is provided, wherein the computer-executable instructions, when executed by one or more computer processors, cause the one or more computer processors to perform the video encoding method according to the present disclosure.

[0028] According to a fifth aspect of the present disclosure, a computer program product is provided, having instructions for storing a bit stream, wherein the bit stream comprises: video data generated by the video encoding method according to the present disclosure.

[0029] According to a sixth aspect of the present disclosure, a computer program product is provided, comprising computer instructions, wherein the computer instructions implement the video encoding method according to the present disclosure when executed by at least one processor.

[0030] According to a seventh aspect of the present disclosure, a method for generating a bit stream is provided, comprising: generating a bit stream according to the video encoding method of the present disclosure.

[0031] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0032] In the present disclosure, a coding time domain filtering method based on multi-hypothesis weighted prediction is proposed using the idea of ​​multi-hypothesis motion compensation. When performing motion-compensated time domain filtering (MCTF), not only one motion vector (MV) is searched for each adjacent frame, but multiple MVs are allowed to be searched. Based on the searched multiple MVs, multi-hypothesis weighted prediction time domain filtering is performed, for example, motion estimation based on multiple hypotheses, motion compensation based on multiple hypotheses, and time domain filtering based on multiple hypotheses. In this way, the prediction accuracy of motion estimation, motion compensation or time domain filtering can be significantly improved, and the filtering effect of MCTF can be improved. In particular, in complex scenes and high-dynamic scenes, higher coding efficiency can be obtained, coding performance can be significantly improved, and error propagation distortion caused by channel errors can also be reduced.

[0033] It is to be understood that both the foregoing general description and the following detailed description are examples only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate examples according to the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0035] Figure 1 is a block diagram illustrating an exemplary system for encoding and decoding video blocks in parallel according to some embodiments of the present disclosure.

[0036] Figure 2 is a block diagram illustrating an exemplary video encoder according to some embodiments described herein.

[0037] Figure 3 is a block diagram illustrating an exemplary video decoder according to some embodiments of the present application.

[0038] Figure 4 is a flowchart illustrating a video encoding method according to an exemplary embodiment of the present disclosure.

[0039] Figure 5 is a schematic diagram illustrating motion estimation and motion compensation according to an exemplary embodiment of the present disclosure.

[0040] Figure 6 is a schematic diagram illustrating weighted fusion of multiple hypotheses in a motion estimation stage according to an exemplary embodiment of the present disclosure.

[0041] Figure 7 is a schematic diagram illustrating multi-hypothesis weighted fusion in a motion compensation stage according to an exemplary embodiment of the present disclosure.

[0042] Figure 8 is a schematic diagram showing multi-hypothesis weighted fusion in the time domain filtering stage according to an exemplary embodiment of the present disclosure.

[0043] Fig. 9 is a block diagram illustrating a video encoding apparatus according to an exemplary embodiment of the present disclosure.

[0044] Fig.10 is a diagram illustrating a computing environment coupled with a user interface according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0045] Reference will now be made in detail to specific embodiments, examples of which are shown in the accompanying drawings. In the following detailed description, a number of non-limiting specific details are set forth to aid in understanding the subject matter presented herein. However, various alternatives may be used without departing from the scope of the claims, and the subject matter may be practiced without these specific details. For example, the subject matter presented herein may be implemented on many types of electronic devices having digital video capabilities.

[0046] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the drawings are used to distinguish objects, but not to describe any specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those shown in the drawings or described in the present disclosure.

[0047] Figure 1 is a block diagram illustrating an exemplary system 10 for encoding and decoding video blocks in parallel according to some embodiments of the present disclosure. Figure 1 As shown in , system 10 includes a source device 12 that generates and encodes video data to be later decoded by a target device 14. Source device 12 and target device 14 may include any of a wide variety of electronic devices, including cloud servers, server computers, desktop or laptop computers, tablet computers, smart phones, set-top boxes, digital televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some implementations, source device 12 and target device 14 are equipped with wireless communication capabilities.

[0048] In some implementations, target device 14 may receive the encoded video data to be decoded via link 16. Link 16 may include any type of communication medium or device capable of moving the encoded video data from source device 12 to target device 14.

[0049] In some other embodiments, the encoded video data may be sent from the output interface 22 to the storage device 32. The encoded video data in the storage device 32 may then be accessed by the target device 14 via the input interface 28.

[0050] like Figure 1 , source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Video source 18 may include sources such as, or a combination of such sources: a video capture device (e.g., a camera), a video archive containing previously captured video, a video feed interface for receiving video from a video content provider, and / or a computer graphics system for generating computer graphics data as source video.

[0051] The captured, pre-captured, or computer-generated video may be encoded by video encoder 20. The encoded video data may be sent directly to target device 14 via output interface 22 of source device 12. The encoded video data may also (or alternatively) be stored on storage device 32 for later access by target device 14 or other devices for decoding and / or playback.

[0052] Target device 14 includes input interface 28, video decoder 30, and display device 34. Input interface 28 may include a receiver and / or a modem and receives encoded video data via link 16. The encoded video data transmitted via link 16 or provided on storage device 32 may include various syntax elements generated by video encoder 20 for use by video decoder 30 in decoding the video data. Such syntax elements may be included within the encoded video data sent over a communication medium, stored on a storage medium, or stored on a file server.

[0053] The video encoder 20 and the video decoder 30 can operate according to a proprietary standard or an industry standard (e.g., VVC, HEVC, Part 10 of MPEG-4, AVC) or an extension of such a standard. It should be understood that the present application is not limited to a specific video encoding / decoding standard and can be applied to other video encoding / decoding standards. It is generally believed that the video encoder 20 of the source device 12 can be configured to encode the video data according to any of these current standards or future standards. Similarly, it is also generally believed that the video decoder 30 of the target device 14 can be configured to decode the video data according to any of these current standards or future standards.

[0054] The video encoder 20 and the video decoder 30 may be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic devices, software, hardware, firmware, or any combination thereof. When partially implemented in software, the electronic device may store instructions for the software in a suitable non-transitory computer-readable medium, and use one or more processors to execute the instructions in the hardware to perform the video encoding / decoding operations disclosed in the present disclosure. Each of the video encoder 20 and the video decoder 30 may be included in one or more encoders or decoders, and either of the encoders or decoders may be integrated as part of a combined encoder / decoder (CODEC) in the corresponding device.

[0055] Figure 2 is a block diagram illustrating an exemplary video encoder 20 according to some embodiments described in the present application. The video encoder 20 can perform intra-frame prediction coding and inter-frame prediction coding on video blocks within a video frame. Intra-frame prediction coding relies on spatial prediction to reduce or remove spatial redundancy in video data within a given video frame or picture. Inter-frame prediction coding relies on temporal prediction to reduce or remove temporal redundancy in video data within adjacent video frames or pictures of a video sequence. It should be noted that in the field of video coding and decoding, the term "frame" can be used as a synonym for the term "image" or "picture".

[0056] like Figure 2 As shown in , the video encoder 20 includes a video data memory 40, a prediction processing unit 41, a decoded picture buffer (DPB) 64, an adder 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The prediction processing unit 41 further includes a motion estimation unit 42, a motion compensation unit 44, a segmentation unit 45, an intra-frame prediction processing unit 46, and an intra-frame block copy (BC) unit 48. In some embodiments, the video encoder 20 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and an adder 62 for video block reconstruction. A loop filter 63 such as a deblocking filter can be located between the adder 62 and the DPB 64 to filter the block boundary to remove the block effect from the reconstructed video. In addition to the deblocking filter, another loop filter (e.g., a sample adaptive offset (SAO) filter, a cross-component sample adaptive offset (CCSAO) filter, and / or an adaptive loop filter (ALF)) can also be used to filter the output of the adder 62. In some examples, the loop filter may be omitted, and the decoded video blocks may be provided directly by adder 62 to DPB 64. Video encoder 20 may take the form of a fixed or programmable hardware unit, or may be distributed among one or more of the illustrated fixed or programmable hardware units.

[0057] Video data memory 40 may store video data to be encoded by components of video encoder 20. Figure 1 Video source 18 is shown obtaining video data in video data memory 40. DPB 64 is a buffer that stores reference video data (eg, reference frames or pictures) for use by video encoder 20 in encoding the video data (eg, in intra or inter prediction coding modes).

[0058] like Figure 2 As shown in , after receiving the video data, the segmentation unit 45 within the prediction processing unit 41 segments the video data into video blocks. This segmentation may also include segmenting the video frame into strips, tiles (e.g., a set of video blocks), or other larger coding units (CUs) according to a predefined splitting structure associated with the video data, such as a quadtree (QT) structure. It should be noted that the terms "block" or "video block" used herein may be a portion of a frame or picture, in particular a rectangular (square or non-square) portion. With reference to, for example, HEVC and VVC, a block or video block may be or correspond to a coding tree unit (CTU), a CU, a prediction unit (PU), or a transform unit (TU) and / or may be or correspond to a corresponding block (e.g., a coding tree block (CTB), a coding block (CB), a prediction block (PB), or a transform block (TB)) and / or a sub-block.

[0059] The prediction processing unit 41 may select one of a plurality of feasible prediction coding modes for the current video block based on the error results (e.g., coding rate and distortion level), such as one of one or more inter-frame prediction coding modes in a plurality of intra-frame prediction coding modes. The prediction processing unit 41 may provide the resulting intra-frame prediction coding block or inter-frame prediction coding block to the adder 50 to generate a residual block, and to the adder 62 to reconstruct the coding block for subsequent use as part of a reference frame. The prediction processing unit 41 also provides syntax elements (e.g., motion vectors, intra-frame mode indicators, partition information, and other such syntax information) to the entropy coding unit 56.

[0060] To select an appropriate intra-prediction coding mode for the current video block, intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-prediction coding of the current video block in relation to one or more neighboring blocks in the same frame as the current block to be encoded to provide spatial prediction. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 may perform inter-prediction coding of the current video block in relation to one or more prediction blocks in one or more reference frames to provide temporal prediction. Video encoder 20 may perform multiple encoding passes, for example, to select an appropriate coding mode for each block of video data.

[0061] In some embodiments, motion estimation unit 42 determines an inter-prediction mode for a current video frame by generating a motion vector according to a predetermined pattern within a sequence of video frames, the motion vector indicating the displacement of a video block within the current video frame relative to a prediction block within a reference video frame. Motion estimation performed by motion estimation unit 42 is the process of generating a motion vector that estimates the motion of a video block. For example, a motion vector may indicate the displacement of a video block within a current video frame or picture relative to a prediction block within a reference frame associated with a current block being encoded within the current frame. The predetermined pattern may designate a video frame in a sequence as a P frame or a B frame. Intra BC unit 48 may determine a vector (e.g., a block vector) for intra BC coding in a manner similar to the motion vector determined by motion estimation unit 42 for inter prediction, or may utilize motion estimation unit 42 to determine a block vector.

[0062] Regardless of whether the prediction block is from the same frame according to intra-frame prediction or from a different frame according to inter-frame prediction, the video encoder 20 can form a residual video block by subtracting the pixel values ​​of the prediction block from the pixel values ​​of the current video block being encoded. The pixel difference values ​​forming the residual video block may include both luma component differences and chroma component differences.

[0063] Intra-prediction processing unit 46 may encode the current block using various intra-prediction modes, for example, during separate encoding passes, and intra-prediction processing unit 46 (or, in some examples, mode selection unit) may select an appropriate intra-prediction mode to use from the tested intra-prediction modes. Intra-prediction processing unit 46 may provide information indicating the intra-prediction mode selected for the block to entropy encoding unit 56. Entropy encoding unit 56 may encode the information indicating the selected intra-prediction mode into the bitstream.

[0064] After prediction processing unit 41 determines a prediction block for the current video block via inter-prediction or intra-prediction, adder 50 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block may be included in one or more TUs and provided to transform processing unit 52. Transform processing unit 52 transforms the residual video data into residual transform coefficients using a transform (e.g., a discrete cosine transform (DCT) or a conceptually similar transform).

[0065] The transform processing unit 52 may send the resulting transform coefficients to a quantization unit 54. The quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may also reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, the quantization unit 54 may then perform a scan on the matrix including the quantized transform coefficients. Alternatively, the entropy coding unit 56 may perform the scan.

[0066] After quantization, entropy coding unit 56 entropy encodes the quantized transform coefficients into a video bitstream using, for example, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding method or technique. The encoded bitstream may then be sent to a video bitstream such as a Figure 1 The video decoder 30 shown, or archived as Figure 1 The video frame may be stored in storage device 32 as shown for later transmission to or retrieval by video decoder 30. Entropy encoding unit 56 may also entropy encode motion vectors and other syntax elements for the current video frame being encoded.

[0067] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual video block in the pixel domain for use in generating reference blocks for predicting other video blocks. As noted above, motion compensation unit 44 may generate a motion compensated prediction block from one or more reference blocks of a frame stored in DPB 64. Motion compensation unit 44 may also apply one or more interpolation filters to the prediction block to calculate sub-integer pixel values ​​for use in motion estimation.

[0068] Adder 62 adds the reconstructed residual block to the motion compensated prediction block generated by motion compensation unit 44 to generate a reference block for storage in DPB 64. The reference block may then be used as a prediction block by intra BC unit 48, motion estimation unit 42, and motion compensation unit 44 to inter-predict another video block in a subsequent video frame.

[0069] Figure 3 1 is a block diagram showing an exemplary video decoder 30 according to some embodiments of the present application. The video decoder 30 includes a video data memory 79, an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, an adder 90, and a DPB 92. The prediction processing unit 81 further includes a motion compensation unit 82, an intra-frame prediction unit 84, and an intra-frame BC unit 85. The video decoder 30 may perform the above-mentioned Figure 2 The encoding process is substantially the inverse of the decoding process described with respect to video encoder 20. For example, motion compensation unit 82 may generate prediction data based on motion vectors received from entropy decoding unit 80, and intra-prediction unit 84 may generate prediction data based on intra-prediction mode indicators received from entropy decoding unit 80.

[0070] In some examples, embodiments of the present disclosure may be dispersed in one or more of the units of the video decoder 30. For example, the intra BC unit 85 may perform embodiments of the present application alone or in combination with other units of the video decoder 30, such as the motion compensation unit 82, the intra prediction unit 84, and the entropy decoding unit 80. In some examples, the video decoder 30 may not include the intra BC unit 85, and the functions of the intra BC unit 85 may be performed by other components of the prediction processing unit 81, such as the motion compensation unit 82.

[0071] Video data memory 79 may store video data, such as an encoded video bitstream, to be decoded by other components of video decoder 30. Video data stored in video data memory 79 may be obtained, for example, from storage device 32, from a local video source (e.g., a camera), via wired or wireless network communication of video data, or by accessing a physical data storage medium (e.g., a flash drive or hard disk).

[0072] During the decoding process, the video decoder 30 receives an encoded video bitstream representing video blocks of an encoded video frame and associated syntax elements. The entropy decoding unit 80 of the video decoder 30 entropy decodes the bitstream to generate quantization coefficients, motion vectors or intra-frame prediction mode indicators, and other syntax elements. The entropy decoding unit 80 then forwards the motion vectors or intra-frame prediction mode indicators, and other syntax elements to the prediction processing unit 81.

[0073] When a video frame is encoded as an intra-prediction coded (I) frame or for intra-coded prediction blocks in other types of frames, the intra-prediction unit 84 of the prediction processing unit 81 can generate prediction data for a video block of the current video frame based on the intra-prediction mode transmitted by the signal and reference data from a previously decoded block of the current frame.

[0074] When the video frame is encoded as an inter-frame prediction coding (i.e., B or P) frame, the motion compensation unit 82 of the prediction processing unit 81 generates one or more prediction blocks for the video block of the current video frame based on the motion vector and other syntax elements received from the entropy decoding unit 80. Each of the prediction blocks can be generated from a reference frame in one of the reference frame lists. The video decoder 30 can construct the reference frame lists, i.e., List 0 and List 1, based on the reference frames stored in the DPB 92 using a default construction technique.

[0075] In some examples, when a video block is encoded according to the intra BC mode described herein, intra BC unit 85 of prediction processing unit 81 generates a prediction block for the current video block based on the block vector and other syntax elements received from entropy decoding unit 80. The prediction block may be within a reconstructed region of the same picture as the current video block as defined by video encoder 20.

[0076] Motion compensation unit 82 and / or intra BC unit 85 determine prediction information for a video block of the current video frame by parsing motion vectors and other syntax elements, and then use the prediction information to generate a prediction block for the current video block being decoded.

[0077] Motion compensation unit 82 may also perform interpolation using interpolation filters as used by video encoder 20 during encoding of the video blocks to calculate interpolated values ​​for sub-integer pixels of reference blocks. In this case, motion compensation unit 82 may determine the interpolation filters used by video encoder 20 based on the received syntax elements and use these interpolation filters to produce the prediction blocks.

[0078] Inverse quantization unit 86 inverse quantizes the quantized transform coefficients provided in the bitstream and entropy decoded by entropy decoding unit 80, using the same quantization parameters calculated by video encoder 20 for each video block in the video frame to determine the degree of quantization. Inverse transform processing unit 88 applies an inverse transform (e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients to reconstruct the residual block in the pixel domain.

[0079] After the motion compensation unit 82 or the intra BC unit 85 generates a prediction block for the current video block based on the vector and other syntax elements, the adder 90 reconstructs the decoded video block for the current video block by adding the residual block from the inverse transform processing unit 88 to the corresponding prediction block generated by the motion compensation unit 82 and the intra BC unit 85. A loop filter 91 (e.g., a deblocking filter, an SAO filter, a CCSAO filter, and / or an ALF) may be located between the adder 90 and the DPB 92 to further process the decoded video block. In some examples, the loop filter 91 may be omitted, and the decoded video block may be provided directly to the DPB 92 by the adder 90. The decoded video blocks in a given frame are then stored in the DPB 92, which stores reference frames for subsequent motion compensation of the next video block. The DPB 92 or a memory device separate from the DPB 92 may also store the decoded video for later presentation on a display device (e.g., Figure 1 on a display device 34).

[0080] As mentioned above, the motion estimation compensated temporal filter (MCTF) is an encoder-only video filtering technology that uses the temporal correlation of the video and the inter-block reference relationship to perform temporal filtering on the reference frame to reduce the temporal redundant information generated in the video block reference process, thereby improving the overall coding efficiency.

[0081] MCTF technology can be mainly divided into two parts, namely the "motion estimation and compensation part" and the "filtering part". In the "motion estimation and compensation part", MCTF will first perform block-level motion estimation (Motion Estimation, ME). Specifically, the current image can be evenly divided into multiple blocks of m×m size, for example, multiple blocks of 8×8, and each block can be searched to obtain a motion vector (Motion Vector, MV). Then, MV can be used to perform classic block-based motion compensation on each 8×8 block, and then all 8×8 motion compensated image blocks can be directly spliced ​​to obtain a motion frame. In this way, this motion compensation process can be performed for each adjacent frame of the current frame, and then multiple motion compensated frames corresponding to multiple adjacent frames of the current frame can be obtained. In the "filtering part", each block of the current frame and the multiple motion compensated blocks corresponding to the adjacent frames are weighted according to a certain proportion and bilateral filtering is performed to generate the final time domain filtering result.

[0082] However, in the related art, when using MCTF for temporal filtering, each frame uses the classic block-based motion compensation technology, that is, only one motion compensation block is obtained for each frame, and then multiple motion compensation blocks corresponding to all adjacent frames of the current frame are subjected to bilateral weighted filtering with the current block. However, using only a single hypothesis compensation for each frame may result in low prediction quality, which will lead to poor temporal filtering effect and low coding efficiency.

[0083] In order to solve the above-mentioned problems existing in the related art, the video encoding method and its device, electronic device and non-transient computer-readable storage medium provided by the present disclosure use the idea of ​​multi-hypothesis motion compensation, and propose a coding time domain filtering method based on multi-hypothesis weighted prediction. When performing time domain filtering (MCTF) based on motion compensation, not only one motion vector (MV) is searched for each adjacent frame, but multiple MVs are allowed to be searched out. Based on the searched multiple MVs, time domain filtering with multi-hypothesis weighted prediction is performed, for example, motion estimation based on multiple hypotheses, motion compensation based on multiple hypotheses, and time domain filtering based on multiple hypotheses. In this way, the prediction accuracy of motion estimation, motion compensation or time domain filtering can be significantly improved, and the filtering effect of MCTF can be improved. Especially in complex scenes and high dynamic scenes, higher coding efficiency can be obtained, coding performance can be significantly improved, and error propagation distortion caused by channel errors can also be reduced.

[0084] Next, we will refer to Figures 4 to 10 The video encoding method and its device, electronic device and non-transitory computer-readable storage medium in the present disclosure are described in detail.

[0085] Figure 4 is a flowchart illustrating a video encoding method according to an exemplary embodiment of the present disclosure.

[0086] Reference Figure 4 In step 401, for each adjacent frame in at least one adjacent frame of the current frame, the following operations may be performed: motion estimation may be performed on the current block in the current frame to search for multiple motion vectors from the adjacent frames, and motion compensation may be performed on the current block based on the multiple motion vectors searched from the adjacent frames to obtain a motion compensation result, thereby obtaining a motion compensation result for each adjacent frame in at least one adjacent frame. It should be noted that the number of motion compensation results corresponding to each adjacent frame may be the same or different, and the present disclosure does not impose any specific limitation on this.

[0087] The existing MCTF technology can be mainly divided into two parts, namely the "motion estimation and compensation part" and the "filtering part":

[0088] For the "motion estimation" part: Assume that the current frame is I t , that is, for the current frame I t To filter, several frames before and after the current frame are needed. For example, four frames before and after the current frame are filtered in sequence. t Perform block-level motion estimation. Assuming the number of adjacent frames N = 8, it can be recorded as: I t-4 ,I t-3 ,I t-2 ,I t-1 ,I t+1 ,I t+2 ,I t+3 ,I t+4 .like Figure 5 As shown, Figure 5 is a schematic diagram illustrating motion estimation and motion compensation according to an exemplary embodiment of the present disclosure, wherein the current frame I y The first four frames are: t-4 ,I y-3 ,I t-2 ,I y-1 , current frame I t The last four frames are: t+1 ,I t+2 ,I t+3 ,I t+4 In this way, the current frame I can be searched tEach N×N (e.g., 8×8) block in the image corresponds to a most similar block in each of the adjacent 8 frames, and the position offset of the similar block relative to the current block can be represented by a motion vector MV. Exemplarily, in order to speed up the motion estimation process, a pyramid motion estimation form can be adopted, that is, a fast estimation can be performed on the downsampled image first, and then the original resolution ME can be performed based on the downsampled motion estimation ME result. Specifically, integer pixel ME can be performed first for the original resolution, and then sub-pixel ME can be performed based on the integer pixel result, and finally a MV corresponding to each block can be obtained.

[0089] Regarding the "motion compensation" part: for the current 8×8 block, in each adjacent frame, the corresponding MV searched can be directly used to perform classic block-level motion compensation to obtain the motion compensated image block corresponding to the adjacent frame. In this way, by performing motion compensation in 8 adjacent frames respectively, 8 image blocks with the best match corresponding to 8 adjacent frames can be obtained, that is, 8 motion compensated blocks.

[0090] For the "filtering part", each block of the current frame and multiple motion compensation blocks corresponding to adjacent frames are weighted according to a certain ratio to perform bilateral filtering, and generate the final time domain filtering result.

[0091] However, in the above-mentioned existing MCTF technology, using only a single hypothesis compensation for each frame may result in low prediction quality, which will lead to poor time domain filtering effect and low coding efficiency. In order to solve this problem, the present disclosure uses the idea of ​​multi-hypothesis motion compensation and proposes a coding time domain filtering method based on multi-hypothesis weighted prediction. When performing motion compensation-based time domain filtering (MCTF), for each adjacent frame, not only one motion vector (MV) is searched, but multiple MVs are allowed to be searched out. Based on the searched multiple MVs, time domain filtering of multi-hypothesis weighted prediction is performed.

[0092] Various implementations of searching for multiple motion vectors MV of adjacent frames of the current frame will be introduced below.

[0093] According to an exemplary embodiment of the present disclosure, multiple motion vector candidates can be searched from adjacent frames by the same motion vector search method. Then, a predetermined number of optimal motion vector candidates can be selected from the multiple motion vector candidates according to a predetermined rule as multiple motion vectors corresponding to the adjacent frames. The number of "multiple motion vector candidates" can be set based on actual conditions, for example, it can be set to: 2, 3, 5, etc., and the present disclosure does not make specific restrictions on this.

[0094] Exemplarily, several MV candidates with smaller distortion or rate distortion may be searched from adjacent frames using the same motion vector search method, and then a predetermined number of optimal motion vector candidates may be selected from the searched MV candidates as multiple motion vectors corresponding to the adjacent frames.

[0095] In this way, by searching multiple motion vectors, a more comprehensive and accurate description of the motion characteristics of the current block can be ensured, avoiding the phenomenon that the motion compensation effect is poor due to motion compensation based on only a single motion vector.

[0096] According to an exemplary embodiment of the present disclosure, multiple motion vector candidates can also be searched from adjacent frames by different motion vector search methods to determine multiple motion vectors corresponding to the adjacent frames. Exemplarily, similar to the bidirectional prediction method in the coding inter-frame prediction, other MVs can be searched based on the searched optimal MV to make the weighted prediction result of motion compensation optimal; or, the optimal MV can be searched using multiple pyramid levels of MCTF; or, multiple MV candidates can be searched using a method based on overlapping block motion estimation ME, and then several better MV candidates can be selected from the multiple searched MV candidates as multiple motion vectors of adjacent frames, etc. The present disclosure does not specifically limit the motion vector search method, and the aforementioned implementation is merely an exemplary description.

[0097] In this way, since each motion vector search method has its own search focus, using a variety of different motion vector search methods for motion vector search can obtain multiple motion vectors with different characteristics, which can ensure a more complete and comprehensive characterization of the motion characteristics of the image block to obtain a better motion compensation effect, thereby further improving the time domain filtering effect and coding performance.

[0098] According to an exemplary embodiment of the present disclosure, a first motion vector candidate may be searched from adjacent frames by a first motion vector search method, and at least one second motion vector candidate may be searched from adjacent frames by at least one second motion vector search method. Then, a predetermined number of optimal second motion vector candidates may be selected from the at least one second motion vector candidate according to a predetermined rule. Next, the first motion vector candidate and the predetermined number of optimal second motion vector candidates may be determined as a plurality of motion vectors.

[0099] Exemplarily, a first motion vector search method can be used to search for an optimal motion vector candidate from adjacent frames, and other motion vector search methods other than the first motion vector search method can also be used to obtain n suboptimal MV candidates for the adjacent frames. Next, the distortion (for example, Error or SSIM) of each of the n suboptimal MV candidates can be calculated, and then the distortion difference between the distortion of each suboptimal MV candidate and the distortion of the aforementioned optimal motion vector candidate can be calculated. Then, a preset number of suboptimal MV candidates with smaller corresponding distortion differences can be selected from the above n suboptimal MV candidates for retention. Among them, when n=1, it is to determine whether the distortion difference between this suboptimal MV candidate and the aforementioned optimal motion vector candidate is less than a preset threshold. If the distortion difference corresponding to the suboptimal MV candidate is less than the preset threshold, the suboptimal MV candidate can be retained; otherwise, the suboptimal MV candidate can be discarded, that is, at this time, only the aforementioned optimal motion vector candidate can be used for motion compensation.

[0100] In this way, through the cross-application of a variety of different motion vector search methods, it is possible to ensure a more complete and accurate representation of the motion characteristics of the image block to obtain a better motion compensation effect, thereby further improving the time domain filtering effect and coding performance.

[0101] It should be noted that in the present disclosure, in order to speed up the motion estimation process, a pyramid motion estimation form can be used, that is, a fast estimation can be performed on the downsampled image first, and then the original resolution ME can be performed based on the downsampled motion estimation ME result. Specifically, integer pixel ME can be performed first for the original resolution, and then sub-pixel ME can be performed based on the integer pixel result, and finally multiple MVs corresponding to each adjacent frame can be obtained.

[0102] In step 402, the current block may be subjected to time domain filtering based on the motion compensation result for each adjacent frame in at least one adjacent frame to obtain a time domain filtering result of the current block. Exemplarily, as described above, assuming that the number of adjacent frames of the current frame is 8, the current block may be subjected to time domain filtering based on multiple motion compensation blocks obtained for each adjacent frame in the 8 adjacent frames to obtain a time domain filtering block of the current block.

[0103] In step 403, the current block may be encoded based on the time domain filtering result of the current block.

[0104] The multi-hypothesis weighting technology proposed in the present disclosure can be applied to any stage in MCTF. Below, examples are given to introduce the implementation process of using the multi-hypothesis weighting technology proposed in the present disclosure in various stages of MCTF (such as, but not limited to, motion estimation, motion compensation, and temporal filtering).

[0105] Motion Estimation Based on Multiple Hypotheses

[0106] According to an exemplary embodiment of the present disclosure, a weighted summation may be performed on multiple motion vectors searched from adjacent frames to obtain a final motion vector. Then, motion compensation may be performed on the current block based on the final motion vector. Subsequently, temporal filtering may be performed on the current block based on the motion compensation result of the current block.

[0107] Figure 6 is a schematic diagram showing the application of a multi-hypothesis weighting technique in a motion estimation stage according to an exemplary embodiment of the present disclosure. Figure 6 , assuming that there are N adjacent frames in total for the current frame, multiple motion vectors MV can be searched out for each adjacent frame. Then, for each adjacent frame, a weighted sum can be performed on the multiple motion vectors searched from the adjacent frame to obtain the final motion vector corresponding to the adjacent frame. In this way, N adjacent frames can obtain N final motion vectors corresponding to each other. Next, for each adjacent frame, motion compensation can be performed on the current block of the current frame based on the final motion vector corresponding to the adjacent frame to obtain the motion compensation result corresponding to the adjacent frame, that is, the motion compensation block. In this way, N adjacent frames can obtain N motion compensation blocks corresponding to each other. Then, based on the N motion compensation blocks corresponding to each other in the N adjacent frames, time domain filtering can be performed on the current block to obtain the time domain filtering result of the current block, that is, the time domain filtering block.

[0108] Thus, in the present disclosure, a multi-vector hypothesis motion compensation prediction algorithm can be used, that is, multi-hypothesis weighted fusion can be performed in the motion estimation stage, which can significantly improve the prediction accuracy of motion compensation and further enhance the filtering effect of MCTF.

[0109] According to an exemplary embodiment of the present disclosure, the weight used for weighted summation of motion vectors may be determined according to a prediction characteristic corresponding to each motion vector, or may also be determined according to a machine learning or deep learning approach.

[0110] Exemplarily, the weight of the motion vector may be determined according to the distortion (e.g., error or structural similarity (SSIM)) of each motion vector among multiple motion vectors corresponding to the same adjacent frame. For example, the smaller the distortion of the motion vector, the higher the weight corresponding to the motion vector may be.

[0111] Motion compensation based on multiple hypotheses

[0112] According to an exemplary embodiment of the present disclosure, motion compensation can be performed on the current block based on multiple motion vectors searched from adjacent frames to obtain multiple motion compensation results, that is, multiple motion compensation blocks can be obtained. Then, the multiple motion compensation results can be weighted summed to obtain a final motion compensation result, that is, a final motion compensation block can be obtained. Next, the current block can be time-domain filtered based on the final motion compensation result obtained for each adjacent frame in at least one adjacent frame. Exemplarily, as mentioned above, assuming that the number of adjacent frames is 8, the current block can be time-domain filtered based on the final motion compensation block corresponding to each adjacent frame in the 8 adjacent frames.

[0113] Figure 7 is a schematic diagram showing multi-hypothesis weighted fusion in the motion compensation stage according to an exemplary embodiment of the present disclosure. Figure 7 , assuming that there are N adjacent frames in total for the current frame, then each adjacent frame can search out multiple motion vectors MVs, and then each adjacent frame can obtain multiple motion compensation blocks. Then, for each adjacent frame, the multiple motion compensation blocks corresponding to the adjacent frame can be weighted fused to obtain the only final motion compensation block corresponding to the adjacent frame. In this way, by performing weighted fusion of motion compensation blocks for N adjacent frames respectively, N final motion compensation blocks corresponding to N adjacent frames can be obtained. Next, the N final motion compensation blocks can be used to perform time domain filtering on the current block to obtain the time domain filtering result of the current block, that is, the time domain filtering block.

[0114] Specifically, first, motion compensation can be performed on multiple MVs (denoted as M MVs) of each adjacent frame to obtain multiple motion compensation prediction results (MCPs) corresponding to the adjacent frames. m ), i.e., multiple motion compensation blocks. Then, weighted fusion can be performed on multiple motion compensation results corresponding to each adjacent frame, i.e., multiple motion compensation blocks, to obtain the final motion compensation result MCP corresponding to each adjacent frame, i.e., the final motion compensation block:

[0115]

[0116] Among them, w m Represents the weight of the mth motion compensation block.

[0117] Next, in the subsequent time domain filtering stage, the current block of the current frame may be time-domain filtered using the final motion compensation result corresponding to each adjacent frame of the multiple adjacent frames of the current frame to obtain the time domain filtering result of the current block.

[0118] Thus, in the present disclosure, a multi-hypothesis motion compensation improved algorithm can be used, that is, multi-hypothesis weighted fusion can be performed in the motion compensation stage, thereby generating a more accurate prediction block for each adjacent frame for subsequent time domain filtering. The prediction accuracy of motion compensation can be significantly improved, thereby improving the filtering effect of MCTF.

[0119] According to an exemplary embodiment of the present disclosure, the weight used for weighted summation may be determined according to the prediction characteristics corresponding to each motion compensation result, or may be determined according to a machine learning or deep learning approach.

[0120] Exemplarily, the weight corresponding to the motion compensation block can be determined according to the distortion (e.g., Error or SSIM) of each motion compensation block in multiple motion compensation blocks corresponding to the same adjacent frame. For example, the smaller the distortion of the motion compensation block, the higher the weight of the motion compensation block.

[0121] Time Domain Filtering Based on Multiple Hypotheses

[0122] According to an exemplary embodiment of the present disclosure, the current block can also be motion compensated based on multiple motion vectors searched from adjacent frames to obtain multiple motion compensation results, that is, multiple motion compensation blocks can be obtained. Then, the current block can be temporally filtered based on the multiple motion compensation results obtained for each adjacent frame in at least one adjacent frame. Exemplarily, as mentioned above, assuming that the number of adjacent frames of the current frame is 8, the current block can be temporally filtered based on the multiple motion compensation blocks obtained for each adjacent frame in the 8 adjacent frames.

[0123] Figure 8 is a schematic diagram showing multi-hypothesis weighted fusion in the time domain filtering stage according to an exemplary embodiment of the present disclosure. Figure 8 , assuming that there are N adjacent frames in total for the current frame, multiple motion vectors MVs can be searched out for each adjacent frame, and multiple motion compensation blocks can be obtained for each adjacent frame. Then, in the temporal filtering stage, the current block of the current frame can be temporally filtered using the multiple motion compensation blocks corresponding to each adjacent frame in the multiple adjacent frames of the current frame to obtain the temporal filtering result of the current block, i.e., the temporal filtering block.

[0124] Specifically, for each adjacent frame, multiple MVs (denoted as M MVs) corresponding to the adjacent frame can be used to perform classical motion compensation respectively, and multiple motion compensation prediction results (MCP) corresponding to multiple MVs can be obtained. m), i.e., multiple motion compensation blocks. Then, the multiple motion compensation blocks corresponding to each adjacent frame in the multiple adjacent frames will be used in the subsequent time domain filtering stage. Exemplarily, assuming that there are 8 adjacent frames in the current frame, and each adjacent frame corresponds to 2 motion compensation blocks, the current block of the current frame can be time-domain filtered based on 2×8=16 motion compensation blocks to obtain the time domain filter block of the current block.

[0125] It should be noted that in the "temporal filtering part", in the related art, only N (or less than N) motion compensation prediction blocks corresponding to the N adjacent frames of the current frame are used for temporal filtering. In the present disclosure, the number of motion compensation blocks corresponding to each adjacent frame can be expanded from one to multiple, and then temporal filtering can be performed based on more than N motion compensation prediction blocks to obtain the final filtering result of the current block, that is, the final filtering block.

[0126] In this way, in the present disclosure, an improved multi-hypothesis time domain filtering algorithm can be used, that is, for N adjacent frames, more than N motion compensation prediction blocks can be generated for time domain weighted filtering, that is, multi-hypothesis weighted fusion can be performed in the time domain filtering stage, which can significantly improve the prediction accuracy of motion compensation, and thus improve the filtering effect of MCTF.

[0127] According to an exemplary embodiment of the present disclosure, the filtering weight for time domain filtering may be determined according to the prediction feature corresponding to each motion compensation result.

[0128] Exemplarily, the weight corresponding to the motion compensation block can be determined according to the distortion (e.g., Error or SSIM) of each motion compensation block in multiple motion compensation blocks corresponding to the same adjacent frame. For example, the smaller the distortion of the motion compensation block, the higher the weight of the motion compensation block.

[0129] It should be noted that in the present disclosure, three multi-hypothesis weighted time domain filtering methods are proposed, namely: "multi-hypothesis weighted fusion in the motion estimation stage", "multi-hypothesis weighted fusion in the motion compensation stage" and "multi-hypothesis weighted fusion in the time domain filtering stage". In addition, in addition to the multi-hypothesis weighted fusion in the above three stages, the multiple motion compensation results of each adjacent frame can also be multi-hypothesis weighted fusion in other stages of MCTF. The present disclosure does not make specific restrictions on this, and the above implementation is only an exemplary description.

[0130] Fig. 9 is a block diagram illustrating a video encoding apparatus 900 according to an exemplary embodiment of the present disclosure.

[0131] Reference Fig. 9The video encoding device 900 may include a motion estimation module 901, a motion compensation module 902, a time domain filtering module 903 and an encoding module 904.

[0132] For each adjacent frame in at least one adjacent frame of the current frame, the following operations can be performed: the motion estimation module 901 performs motion estimation on the current block in the current frame to search for multiple motion vectors from the adjacent frames, and the motion compensation module 902 can perform motion compensation on the current block based on the multiple motion vectors searched from the adjacent frames to obtain a motion compensation result, thereby obtaining a motion compensation result for each adjacent frame in at least one adjacent frame.

[0133] According to an exemplary embodiment of the present disclosure, the motion estimation module 901 can search for multiple motion vector candidates from adjacent frames by the same motion vector search method. Then, the motion estimation module 901 selects a predetermined number of optimal motion vector candidates from the multiple motion vector candidates according to a predetermined rule as multiple motion vectors corresponding to the adjacent frames. The number of the multiple motion vector candidates can be set based on actual conditions, for example, it can be set to: 2, 3, 5, etc., and the present disclosure does not make specific restrictions on this.

[0134] In this way, by searching multiple motion vectors, a more comprehensive and accurate description of the motion characteristics of the current block can be ensured, avoiding the phenomenon that the motion compensation effect is poor due to motion compensation based on only a single motion vector.

[0135] According to an exemplary embodiment of the present disclosure, the motion estimation module 901 can also search for multiple motion vector candidates from adjacent frames through different motion vector search methods to determine multiple motion vectors corresponding to the adjacent frames. In this way, since each motion vector search method corresponds to its own search focus, multiple motion vectors with different characteristics can be obtained by using a variety of different motion vector search methods to perform motion vector search, which can ensure a more comprehensive characterization of the motion features of the image block to obtain a better motion compensation effect, thereby further improving the temporal filtering effect and encoding performance.

[0136] According to an exemplary embodiment of the present disclosure, the motion estimation module 901 may also search for a first motion vector candidate from an adjacent frame by a first motion vector search method, and may search for at least one second motion vector candidate from an adjacent frame by at least one second motion vector search method. Then, the motion estimation module 901 may select a predetermined number of optimal second motion vector candidates from at least one second motion vector candidate according to a predetermined rule. Next, the motion estimation module 901 may determine the first motion vector candidate and the predetermined number of optimal second motion vector candidates as multiple motion vectors.

[0137] In this way, through the cross-application of a variety of different motion vector search methods, it is possible to ensure a more complete and accurate representation of the motion characteristics of the image block to obtain a better motion compensation effect, thereby further improving the time domain filtering effect and coding performance.

[0138] It should be noted that in the present disclosure, in addition to searching for motion vectors from adjacent frames in the above-mentioned manner, other methods may also be used for searching, and the present disclosure does not impose any specific limitation on this, and the above-mentioned motion vector search method is merely an exemplary description.

[0139] According to an exemplary embodiment of the present disclosure, the motion compensation module 902 may perform weighted summation on multiple motion vectors searched from adjacent frames to obtain a final motion vector. Then, the motion compensation module 902 may perform motion compensation on the current block based on the final motion vector.

[0140] Thus, in the present disclosure, a multi-vector hypothesis motion compensation prediction algorithm can be used, that is, multi-hypothesis weighted fusion can be performed in the motion estimation stage, which can significantly improve the prediction accuracy of motion compensation and further enhance the filtering effect of MCTF.

[0141] According to an exemplary embodiment of the present disclosure, the weight used for weighted summation of motion vectors may be determined according to a prediction characteristic corresponding to each motion vector, or may also be determined according to a machine learning or deep learning approach.

[0142] According to an exemplary embodiment of the present disclosure, the motion compensation module 902 can perform motion compensation on the current block based on multiple motion vectors searched from adjacent frames, and obtain multiple motion compensation results, that is, multiple motion compensation blocks can be obtained. Then, the motion compensation module 902 can perform weighted summation on the multiple motion compensation results to obtain the final motion compensation result, that is, a final motion compensation block can be obtained. Next, the time domain filtering module 903 can perform time domain filtering on the current block based on the final motion compensation result obtained for each adjacent frame in at least one adjacent frame. Exemplarily, as mentioned above, assuming that the number of adjacent frames is 8, the current block can be time-domain filtered based on the final motion compensation block corresponding to each adjacent frame in the 8 adjacent frames.

[0143] Thus, in the present disclosure, a multi-hypothesis motion compensation improved algorithm can be used, that is, multi-hypothesis weighted fusion can be performed in the motion compensation stage, thereby generating a more accurate prediction block for each adjacent frame for subsequent time domain filtering. The prediction accuracy of motion compensation can be significantly improved, thereby improving the filtering effect of MCTF.

[0144] According to an exemplary embodiment of the present disclosure, the weight used for weighted summation may be determined according to the prediction characteristics corresponding to each motion compensation result, or may be determined according to a machine learning or deep learning approach.

[0145] According to an exemplary embodiment of the present disclosure, the motion compensation module 902 can also perform motion compensation on the current block based on multiple motion vectors searched from adjacent frames, and obtain multiple motion compensation results, that is, multiple motion compensation blocks can be obtained. Then, the time domain filtering module 903 can perform time domain filtering on the current block based on the multiple motion compensation results obtained for each adjacent frame in at least one adjacent frame. Exemplarily, as mentioned above, assuming that the number of adjacent frames of the current frame is 8, the current block can be subjected to time domain filtering based on the multiple motion compensation blocks obtained for each adjacent frame in the 8 adjacent frames.

[0146] In this way, in the present disclosure, an improved multi-hypothesis time domain filtering algorithm can be used, that is, for N adjacent frames, more than N motion compensation prediction blocks can be generated for time domain weighted filtering, that is, multi-hypothesis weighted fusion can be performed in the time domain filtering stage, which can significantly improve the prediction accuracy of motion compensation, and thus improve the filtering effect of MCTF.

[0147] According to an exemplary embodiment of the present disclosure, the filtering weight for time domain filtering may be determined according to the prediction feature corresponding to each motion compensation result.

[0148] It should be noted that in the present disclosure, three multi-hypothesis weighted time domain filtering methods are proposed, namely: "multi-hypothesis weighted fusion in the motion estimation stage", "multi-hypothesis weighted fusion in the motion compensation stage" and "multi-hypothesis weighted fusion in the time domain filtering stage". In addition, in addition to the multi-hypothesis weighted fusion in the above three stages, the multiple motion compensation results of each adjacent frame can also be multi-hypothesis weighted fusion in other stages of MCTF. The present disclosure does not make specific restrictions on this, and the above implementation is only an exemplary description.

[0149] The time domain filtering module 903 can perform time domain filtering on the current block based on the motion compensation result of each adjacent frame in at least one adjacent frame to obtain the time domain filtering result of the current block. Exemplarily, as described above, assuming that the number of adjacent frames of the current frame is 8, the time domain filtering module 903 can perform time domain filtering on the current block based on multiple motion compensation blocks obtained from each adjacent frame in the 8 adjacent frames to obtain the time domain filtering block of the current block.

[0150] The encoding module 904 may encode the current block based on the time domain filtering result of the current block.

[0151] Fig.10A computing environment 1010 is shown coupled to a user interface 1050. The computing environment 1010 may be part of a data processing server. The computing environment 1010 includes a processor 1020, a memory 1030, and an input / output (I / O) interface 1040.

[0152] The processor 1020 generally controls the overall operation of the computing environment 1010, such as operations associated with display, data acquisition, data communication, and image processing. The processor 1020 may include one or more processors for executing instructions to perform all or some of the steps in the above method. In addition, the processor 1020 may include one or more modules that facilitate the interaction between the processor 1020 and other components. The processor may be a central processing unit (CPU), a microprocessor, a single-chip microcomputer, a graphics processing unit (GPU), etc.

[0153] The memory 1030 is configured to store various types of data to support the operation of the computing environment 1010. The memory 1030 may include predetermined software 1032. Examples of such data include instructions for any application or method operating on the computing environment 1010, video data sets, image data, etc. The memory 1030 may be implemented by using any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0154] The I / O interface 1040 provides an interface between the processor 1020 and peripheral interface modules (e.g., a keyboard, a click wheel, buttons, etc.). The buttons may include, but are not limited to, a home button, a start scan button, and a stop scan button. The I / O interface 1040 may be coupled to an encoder and a decoder.

[0155] In an embodiment, a non-transitory computer-readable storage medium including, for example, a plurality of programs in a memory 1030 and / or storing a bit stream generated by the above encoding method or a bit stream to be decoded by the above decoding method is also provided, wherein the plurality of programs can be executed by a processor 1020 in a computing environment 1010 to perform the above method. In one example, the plurality of programs can be executed by a processor 1020 in a computing environment 1010 to (for example, from Figure 2The video encoder 20 in the computing environment 1010 receives a bit stream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or one or more associated syntax elements, etc.), and can also be executed by the processor 1020 in the computing environment 1010 to perform the above-mentioned decoding method according to the received bit stream or data stream. In another example, the multiple programs can be executed by the processor 1020 in the computing environment 1010 to perform the above-mentioned encoding method to encode the video information (e.g., video blocks representing video frames, and / or one or more associated syntax elements, etc.) into a bit stream or data stream, and can also be executed by the processor 1020 in the computing environment 1010 to (e.g., to Figure 3 Alternatively, a non-transitory computer-readable storage medium may store the bit stream or data stream generated by the encoder (e.g., Figure 2 The video encoder 20 in FIG. 1 generates a video signal for a decoder (eg, Figure 3 A bitstream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or associated one or more syntax elements, etc.) used by the video decoder 30 in the video decoder 30 when decoding video data. The non-transitory computer-readable storage medium can be, for example, a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0156] In an embodiment, a bit stream generated by the above encoding method or a bit stream to be decoded by the above decoding method is provided. In an embodiment, a bit stream including encoded video information generated by the above encoding method or encoded video information to be decoded by the above decoding method is provided.

[0157] In an embodiment, a computing device is also provided, which includes: one or more processors (e.g., processor 1020); and a non-transitory computer-readable storage medium or memory 1030 in which multiple programs that can be executed by the one or more processors are stored, wherein the one or more processors are configured to perform the above-mentioned method when executing the multiple programs.

[0158] In an embodiment, a computer program product having instructions for storing or transmitting a bitstream is also provided, wherein the bitstream includes the encoded video information generated by the above encoding method or the encoded video information to be decoded by the above decoding method. In an embodiment, a computer program product including, for example, a plurality of programs in a memory 1030 is also provided, wherein the plurality of programs can be executed by a processor 1020 in a computing environment 1010 for performing the above method. For example, the computer program product may include a non-transitory computer-readable storage medium.

[0159] In an embodiment, the computing environment 1010 may be implemented by one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), FPGAs, GPUs, controllers, microcontrollers, microprocessors, or other electronic components for performing the above methods.

[0160] In an embodiment, a method for storing a bit stream is also provided, comprising: storing the bit stream on a digital storage medium, wherein the bit stream comprises encoded video information generated by the above encoding method or encoded video information to be decoded by the above decoding method.

[0161] In an embodiment, a method for transmitting a bit stream generated by the above encoder is also provided. In an embodiment, a method for receiving a bit stream to be decoded by the above decoder is also provided.

[0162] The description of the present disclosure has been presented for purposes of illustration and is not intended to be exhaustive or limited to the present disclosure. Many modifications, variations, and alternative embodiments will be apparent to one of ordinary skill in the art having the benefit of the teachings presented in the foregoing description and the associated drawings.

[0163] Unless otherwise specifically stated, the order of steps of the method according to the present disclosure is intended to be illustrative only, and the steps of the method according to the present disclosure are not limited to the order specifically described above, but can be changed according to actual conditions. In addition, at least one of the steps of the method according to the present disclosure can be adjusted, combined or deleted according to actual needs.

[0164] The examples are chosen and described in order to explain the principles of the present disclosure and to enable other persons skilled in the art to understand the various embodiments of the present disclosure and to best utilize the basic principles and various embodiments with various modifications suitable for the specific use contemplated. Therefore, it will be understood that the scope of the present disclosure is not limited to the specific examples of the disclosed embodiments, and modifications and other embodiments are intended to be included within the scope of the present disclosure.

Claims

1. A video encoding method, characterized in that: include: For each adjacent frame in at least one adjacent frame of the current frame, the following operations are performed: performing motion estimation on a current block in the current frame, searching for a plurality of motion vectors from adjacent frames, and performing motion compensation on the current block based on the plurality of motion vectors searched from the adjacent frames to obtain a motion compensation result, thereby obtaining a motion compensation result for each adjacent frame in the at least one adjacent frame; Based on the motion compensation result of each adjacent frame in the at least one adjacent frame, performing time domain filtering on the current block to obtain a time domain filtering result of the current block; The current block is encoded based on a time domain filtering result of the current block.

2. The video encoding method according to claim 1, characterized in that: The performing motion compensation on the current block based on the multiple motion vectors searched from the adjacent frames comprises: Performing weighted summation on the multiple motion vectors searched from the adjacent frames to obtain a final motion vector; Based on the final motion vector, motion compensation is performed on the current block.

3. The video encoding method according to claim 1, characterized in that: The performing motion compensation on the current block based on the multiple motion vectors searched from the adjacent frames comprises: Performing motion compensation on the current block based on the multiple motion vectors searched from the adjacent frames to obtain multiple motion compensation results; Performing weighted summation on the multiple motion compensation results to obtain a final motion compensation result; The step of performing time domain filtering on the current block based on the motion compensation result of each adjacent frame in the at least one adjacent frame includes: Based on the final motion compensation result obtained for each adjacent frame of the at least one adjacent frame, time domain filtering is performed on the current block.

4. The video encoding method according to claim 1, characterized in that: The performing motion compensation on the current block based on the multiple motion vectors searched from the adjacent frames comprises: Performing motion compensation on the current block based on the multiple motion vectors searched from the adjacent frames to obtain multiple motion compensation results; The step of performing time domain filtering on the current block based on the motion compensation result of each adjacent frame in the at least one adjacent frame includes: Based on the multiple motion compensation results obtained for each adjacent frame in the at least one adjacent frame, time domain filtering is performed on the current block.

5. The video encoding method according to claim 1, characterized in that: The step of searching for a plurality of motion vectors from adjacent frames comprises: Searching for a plurality of motion vector candidates from the adjacent frames by using the same motion vector search method; A predetermined number of optimal motion vector candidates are selected from the multiple motion vector candidates according to a predetermined rule as the multiple motion vectors.

6. The video encoding method according to claim 1, characterized in that: The step of searching for a plurality of motion vectors from adjacent frames comprises: A plurality of motion vector candidates are searched from the adjacent frames by using different motion vector search methods to determine the plurality of motion vectors.

7. The video encoding method according to claim 1, characterized in that: The step of searching for a plurality of motion vectors from adjacent frames comprises: Searching for a first motion vector candidate from the adjacent frames by a first motion vector search method, searching for at least one second motion vector candidate from the adjacent frames by at least one second motion vector search method, and selecting a predetermined number of optimal second motion vector candidates from the at least one second motion vector candidate according to a predetermined rule; The first motion vector candidate and the predetermined number of optimal second motion vector candidates are determined as the plurality of motion vectors.

8. The video encoding method according to claim 2, characterized in that: The weight used for the weighted summation may be determined according to the prediction characteristics corresponding to each motion vector, or may be determined according to a machine learning or deep learning approach.

9. The video encoding method according to claim 3, characterized in that: The weights used for the weighted summation may be determined according to the prediction characteristics corresponding to each motion compensation result, or may be determined according to a machine learning or deep learning approach.

10. The video encoding method according to claim 4, characterized in that: The filtering weight used for the time domain filtering may be determined according to the prediction feature corresponding to each motion compensation result.

11. A video encoding device, characterized in that: include: The motion estimation module and the motion compensation module are configured to perform the following operations for each adjacent frame in at least one adjacent frame of the current frame: perform motion estimation on a current block in the current frame, search for multiple motion vectors from adjacent frames, and perform motion compensation on the current block based on the multiple motion vectors searched from the adjacent frames to obtain a motion compensation result, thereby obtaining a motion compensation result for each adjacent frame in the at least one adjacent frame; A time domain filtering module, configured to perform time domain filtering on the current block based on a motion compensation result of each adjacent frame in the at least one adjacent frame, to obtain a time domain filtering result of the current block; The encoding module is configured to encode the current block based on the time domain filtering result of the current block.

12. An apparatus for video encoding, comprising: one or more processors; as well as a memory coupled to the one or more processors and configured to store instructions executable by the one or more processors, Wherein, when executing the instructions, the one or more processors are configured to perform the video encoding method according to any one of claims 1-10.

13. A non-transitory computer-readable storage medium for storing computer-executable instructions, which, when executed by one or more computer processors, cause the one or more computer processors to perform the video encoding method according to any one of claims 1-10.

14. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by at least one processor, the video encoding method according to any one of claims 1 to 10 is implemented.

15. A method for generating a bitstream, comprising: A bit stream is generated according to the video encoding method according to any one of claims 1 to 10.

Images