Video coding method and device, equipment, medium, product, and method for storing and sending bit stream

Abstract

The invention relates to a video coding method and device, equipment, a medium, a product and a bit stream storing and sending method, and the video coding method comprises the steps: respectively executing integer pixel search for at least one reference frame of a current block, and obtaining the rate distortion cost of each reference frame in the at least one reference frame; and respectively executing sub-pixel search on the at least one reference frame, and for each reference frame in the at least one reference frame, skipping the sub-pixel search for the reference frame when the rate distortion cost of the reference frame is greater than a rate distortion cost threshold.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of video coding, and in particular, to a video coding method and device, equipment, medium, product, a method of storing and sending a bitstream. BACKGROUND

[0002] Various electronic devices (e.g., digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video gaming consoles, smartphones, video teleconferencing devices, video streaming devices, etc.) support digital video. Electronic devices transmit and receive, or otherwise communicate, digital video data over communication networks and / or store digital video data on storage devices. Because the bandwidth of communication networks and the storage capacity of storage devices are limited resources, video data can be compressed according to one or more video coding standards prior to being communicated or stored. For example, video coding standards include Versatile Video Coding (VVC), Joint Exploration Test Model (JEM), High Efficiency Video Coding (HEVC / H.265), Advanced Video Coding (AVC / H.264), Moving Picture Experts Group (MPEG) codecs, etc. Video coding often employs predictive methods (e.g., inter-prediction, intra-prediction, etc.) that take advantage of redundancy inherent in video data. Video coding aims to compress video data into a form that uses a lower bit rate, while avoiding or minimizing degradations to video quality.

[0003] However, in the process of video coding, because the integer pixel search and the fractional pixel search are sequentially performed, complete integer pixel search and fractional pixel search are performed on all reference frames, which brings a large amount of calculation, and it is difficult to improve the calculation speed. SUMMARY

[0004] The present disclosure provides a video coding method and device, equipment, medium, product, a method of storing and sending a bitstream to at least solve the problem of large amount of calculation caused by performing complete integer pixel search and fractional pixel search on all reference frames. The technical solutions of the present disclosure are as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a video coding method is provided, the video coding method comprising: performing integer pixel search on at least one reference frame of a current block respectively to obtain a rate-distortion cost of each reference frame in the at least one reference frame; performing fractional pixel search on the at least one reference frame respectively, wherein for each reference frame in the at least one reference frame, when the rate-distortion cost of the reference frame is greater than a rate-distortion cost threshold, fractional pixel search for the reference frame is skipped.

[0006] Optionally, the rate-distortion cost threshold is determined based on a predetermined multiple of a current optimal rate-distortion cost.

[0007] Optionally, the current optimal rate-distortion cost is an optimal rate-distortion cost among rate-distortion costs of reference frames for which the sub-pixel search has been performed.

[0008] Optionally, the predetermined multiple is determined according to a coding level to which a current reference frame belongs and / or a type of the current reference frame.

[0009] Optionally, the predetermined multiple is positively correlated with a level height of a coding level to which a video frame in which the current block is located belongs.

[0010] Optionally, the predetermined multiple is determined according to a motion vector derived in the integer-pixel search.

[0011] Optionally, the predetermined multiple is determined according to a motion vector corresponding to an optimal rate-distortion cost in the integer-pixel search.

[0012] Optionally, performing the sub-pixel search on the at least one reference frame respectively comprises: performing the sub-pixel search on the at least one reference frame respectively in an order from small to large of rate-distortion costs of the at least one reference frame in the integer-pixel search.

[0013] Optionally, performing the sub-pixel search on the at least one reference frame respectively further comprises: for each of the at least one reference frame, skipping the sub-pixel search on the reference frame and subsequent reference frames when a rate-distortion cost of the reference frame is greater than the rate-distortion cost threshold.

[0014] According to a second aspect of embodiments of the present disclosure, a video encoding apparatus is provided, which comprises: an integer-pixel search unit configured to perform an integer-pixel search on at least one reference frame of a current block respectively to obtain a rate-distortion cost of each of the at least one reference frame; and a sub-pixel search unit configured to perform a sub-pixel search on the at least one reference frame respectively, wherein for each of the at least one reference frame, the sub-pixel search on the reference frame is skipped when a rate-distortion cost of the reference frame is greater than a rate-distortion cost threshold.

[0015] According to a third aspect of embodiments of the present disclosure, an electronic device is provided, which comprises: a processor; and a memory for storing processor-executable instructions, wherein the processor-executable instructions, when executed by the processor, cause the processor to perform the video encoding method according to the example embodiments of the present disclosure.

[0016] According to a fourth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, which, when instructions in the computer readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the video encoding method according to the exemplary embodiments of the present disclosure.

[0017] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, which has instructions for storing a bitstream, wherein the bitstream comprises video data generated by the video encoding method according to the exemplary embodiments of the present disclosure.

[0018] According to a sixth aspect of the embodiments of the present disclosure, a method for storing a bitstream is provided, which comprises: generating a bitstream according to the video encoding method according to the exemplary embodiments of the present disclosure; and storing the bitstream.

[0019] According to a seventh aspect of the embodiments of the present disclosure, a method for transmitting a bitstream is provided, which comprises: generating a bitstream according to the video encoding method according to the exemplary embodiments of the present disclosure; and transmitting the bitstream.

[0020] The technical solutions provided by the embodiments of the present disclosure at least bring the following beneficial effects:

[0021] According to the video encoding scheme of the present disclosure, the integer pixel search process and the fractional pixel search process can be split, the integer pixel search is performed on all reference frames first, and then according to the result of the integer pixel search, the rate-distortion cost of each reference frame is compared with the rate-distortion cost threshold, and the fractional pixel search of the reference frame is adaptively skipped when the rate-distortion cost of the reference frame meets the condition, thereby reducing the calculation amount of the fractional pixel search and greatly improving the encoding speed.

[0022] In addition, according to the video encoding scheme of the present disclosure, the reference frames can be sorted according to the result of the integer pixel search, and in the case that the rate-distortion cost of the integer pixel search of the current reference frame is greater than the current optimal rate-distortion cost, since the rate-distortion cost of the integer pixel search of the subsequent reference frame is worse than the rate-distortion cost of the integer pixel search of the current frame, the fractional pixel search of the current reference frame and all subsequent reference frames can be directly skipped, further improving the encoding speed.

[0023] In addition, according to the video encoding scheme of the present disclosure, the predetermined multiple for adaptively skipping the fractional pixel search can be determined according to the encoding level and / or frame type of the video frame where the current block is located or according to the motion complexity of the current block, so that the degree of skipping the fractional pixel search can be adjusted according to the characteristics of the current encoding block and / or the encoding frame, and for the frames in important positions or the blocks with high motion complexity, the fractional pixel search can be skipped less, so that the encoding speed is improved while the encoding quality is balanced, and a higher encoding cost performance is achieved.

[0024] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0025] The accompanying drawings, which are incorporated in and form a part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure without imposing on the disclosure limitations that are inconsistent with the patentable scope of the disclosure.

[0026] Figure 1 FIG. 1 is a block diagram illustrating an exemplary system for encoding and decoding video blocks, in accordance with some embodiments of the present disclosure.

[0027] Figure 2 FIG. 2 is a block diagram illustrating an exemplary video encoder, in accordance with some embodiments of the present disclosure.

[0028] Figure 3 FIG. 3 is a block diagram illustrating an exemplary video decoder, in accordance with some embodiments of the present disclosure.

[0029] Figure 4 FIG. 4 illustrates a diagram showing the position of spatial candidates;

[0030] Figure 5 FIG. 5 illustrates a diagram showing candidate pairs considered for redundancy check for spatial candidates;

[0031] Figure 6 FIG. 6 illustrates a diagram showing scaling of motion vectors for temporal candidates;

[0032] Figure 7 FIG. 7 illustrates a diagram showing candidate positions of temporal candidates;

[0033] Figure 8 FIG. 8 illustrates a diagram showing a motion vector difference (MMVD) search point for merge mode;

[0034] Figure 9 FIG. 9 illustrates a diagram showing a multi-reference frame motion search architecture.

[0035] Figure 10 FIG. 10 illustrates a diagram showing a motion search flow in a video encoding process.

[0036] Figure 11 FIG. 11 illustrates a schematic flow chart diagram of one example of a video encoding method, in accordance with some embodiments of the present disclosure.

[0037] Figure 12 FIG. 12 illustrates a schematic flow chart diagram of another example of a video encoding method, in accordance with some embodiments of the present disclosure.

[0038] Figure 13A diagram illustrating a video encoding device in accordance with some embodiments of the disclosure is shown.

[0039] Figure 14 A diagram illustrating a computing environment coupled with a user interface in accordance with some embodiments of the disclosure is shown. DETAILED DESCRIPTION

[0040] Reference will now be made in detail to specific implementations, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous non-limiting specific details are set forth in order to help provide an understanding of the subject matter presented herein. But the subject matter presented can be practiced without the

[0041] It should be noted that the terms "first", "second", and the like, used in the description and the claims of the present disclosure as well as the drawings, are used to distinguish between similar objects and are not necessarily used to describe a particular sequential or chronological order. It is to be understood that such terms so used are interchangeable under appropriate circumstances such that the embodiments of the present disclosure described herein are capable of operation in other sequences than described or otherwise illustrated herein.

[0042] Figure 1 A block diagram illustrating an exemplary system 10 for encoding and decoding video blocks in parallel in accordance with some embodiments of the disclosure is shown. As shown in Figure 1 The system 10 includes a source device 12 that generates and encodes video data to be decoded at a later time by a destination device 14, as shown in

[0043] In some embodiments, the destination device 14 can receive the encoded video data to be decoded via a link 16. The link 16 can comprise any type of communication medium or device capable of moving the encoded video data from the source device 12 to the destination device 14.

[0044] In other embodiments, the encoded video data can be transmitted from the output interface 22 to a storage device 32. Subsequently, the encoded video data in the storage device 32 can be accessed by the destination device 14 via an input interface 28.

[0045] As Figure 1As shown in FIG. 1, source device 12 includes video source 18, video encoder 20, and output interface 22. Video source 18 can include a source such as a video capture device, e.g., a video camera, a video archive containing previously captured video, a video feed interface to receive video from a video content provider, and / or a computer graphics system for generating computer graphics data as the source video, or a combination of such sources. Output interface 22 can include a transmitter for transmitting encoded video data to target device 14 or a storage device 32 for storage of the encoded video data.

[0046] The captured, pre-captured, or computer-generated video can be encoded by video encoder 20. The encoded video data can be transmitted directly to target device 14 via output interface 22 of source device 12. The encoded video data can also (or alternatively) be stored onto storage device 32 for later access by target device 14 or other devices, e.g., for decoding and / or playback.

[0047] Target device 14 includes input interface 28, video decoder 30, and display device 34. Input interface 28 can include a receiver and / or a modem and receives encoded video data over link 16. The encoded video data communicated over link 16, or provided on storage device 32, can include a variety of syntax elements generated by video encoder 20 for use by video decoder 30 in decoding video data. Such syntax elements can be included within the encoded video data transmitted on a communication medium, stored on a storage medium, or stored on a file server.

[0048] Video encoder 20 and video decoder 30 can operate according to proprietary or industry standards, such as VVC, HEVC, MPEG-4, Part 10, AVC, or extensions of such standards. As should be appreciated, the present application is not limited to a specific video encoding / decoding standard and can be applicable to other video encoding / decoding standards. In general, video encoder 20 of source device 12 can be configured to encode video data according to any of these current or future standards. Similarly, it is also contemplated that video decoder 30 of target device 14 can be configured to decode video data according to any of these current or future standards.

[0049] Video encoder 20 and video decoder 30 can each be implemented as any of a variety of suitable encoder and / or decoder circuitry, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic circuitry, software, hardware, firmware or any combinations thereof. When implemented partially in software, an electronic device can store instructions for the software in a suitable, non- transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the video coding / decoding operations disclosed in the present disclosure. Each of video encoder 20 and video decoder 30 can be included in one or more encoders or decoders, either of which can be integrated as part of a combined encoder / decoder (CODEC) in a respective device.

[0050] Figure 2 is a block diagram illustrating an example video encoder 20, in accordance with some embodiments described in the present disclosure. Video encoder 20 can perform intra-prediction encoding and inter-prediction encoding on video blocks within a video frame. Intra-prediction encoding relies on spatial prediction to reduce or remove spatial redundancy in the video data within a given video frame or picture. Inter-prediction encoding relies on temporal prediction to reduce or remove temporal redundancy in the video data within neighboring video frames or pictures of a video sequence. It should be noted that in the field of video coding, the term "frame" can be used as a synonym for the term "image" or "picture."

[0051] As Figure 2As shown in FIG. 1, video encoder 20 includes video data memory 40, prediction processing unit 41, decoded picture buffer (DPB) 64, summer 50, transform processing unit 52, quantization unit 54, and entropy encoding unit 56. Prediction processing unit 41 further includes motion estimation unit 42, motion compensation unit 44, partition unit 45, intra-prediction processing unit 46, and intra block copy (BC) unit 48. In some implementations, video encoder 20 also includes inverse quantization unit 58, inverse transform processing unit 60, and summer 62 for video block reconstruction. A loop filter 63, such as a deblocking filter, can be located between summer 62 and DPB 64 to filter block boundaries to remove blockiness artifacts from reconstructed video. In addition to the deblocking filter, another loop filter (e.g., a sample adaptive offset (SAO) filter, a cross component sample adaptive offset (CCSAO) filter, and / or an adaptive loop filter (ALF)) can be used to filter the output of summer 62. In some examples, the loop filters can be omitted, and the decoded video blocks can be provided directly from summer 62 to DPB 64. Video encoder 20 can take the form of a fixed or programmable hardware encoder, or can be dispersed into one or more of the illustrated fixed or programmable hardware encoders.

[0052] Video data memory 40 can store video data to be encoded by the components of video encoder 20. The video data in video data memory 40 can be obtained, for example, from video source 18 as shown in FIG. 1. Video data memory 40 can store code streams generated by entropy encoding unit 56, or other intermediate results generated by other components of video encoder 20, including motion vectors, residual data, and so on. Figure 1

[0053] As shown in FIG. 1, video encoder 20 includes video data memory 40, prediction processing unit 41, decoded picture buffer (DPB) 64, summer 50, transform processing unit 52, quantization unit 54, and entropy encoding unit 56. Prediction processing unit 41 further includes motion estimation unit 42, motion compensation unit 44, partition unit 45, intra-prediction processing unit 46, and intra block copy (BC) unit 48. In some implementations, video encoder 20 also includes inverse quantization unit 58, inverse transform processing unit 60, and summer 62 for video block reconstruction. A loop filter 63, such as a deblocking filter, can be located between summer 62 and DPB 64 to filter block boundaries to remove blockiness artifacts from reconstructed video. In addition to the deblocking filter, another loop filter (e.g., a sample adaptive offset (SAO) filter, a cross component sample adaptive offset (CCSAO) filter, and / or an adaptive loop filter (ALF)) can be used to filter the output of summer 62. In some examples, the loop filters can be omitted, and the decoded video blocks can be provided directly from summer 62 to DPB 64. Video encoder 20 can take the form of a fixed or programmable hardware encoder, or can be dispersed into one or more of the illustrated fixed or programmable hardware encoders. Figure 2

[0054] ​​Prediction processing unit 41 can select one of a plurality of possible predictive encoding modes, e.g., one of a plurality of intra-predictive encoding modes or one of a plurality of inter-predictive encoding modes, for the current video block based on the error results (e.g., coding rate and level of distortion). Prediction processing unit 41 can provide the resulting intra- or inter-predicted block to summer 50 to generate a residual block, and to summer 62 to reconstruct the encoded block for use as part of a reference frame at a later time. Prediction processing unit 41 also provides syntax elements, e.g., motion vectors, intra-mode indicators, partitioning information, and other such syntax information, to entropy encoding unit 56.

[0055] To select an appropriate intra-predictive encoding mode for the current video block, intra-prediction processing unit 46 within prediction processing unit 41 can perform intra-predictive encoding of the current video block in relation to one or more neighboring blocks in the same frame as the current block being encoded to provide spatial prediction. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 perform inter-predictive encoding of the current video block in relation to one or more predictive blocks in one or more reference frames to provide temporal prediction. Video encoder 20 can perform multiple encoding passes, e.g., to select a suitable encoding mode for each block of video data.

[0056] In some implementations, motion estimation unit 42 determines an inter-prediction mode for a current video frame by generating motion vectors according to a predetermined pattern within a sequence of video frames, the motion vectors indicating displacement of video blocks within the current video frame relative to predictive blocks within a reference video frame. Motion estimation performed by motion estimation unit 42 is a process of generating motion vectors that estimate motion for video blocks. For example, a motion vector can indicate displacement of a video block within a current video frame or picture relative to a predictive block within a reference frame that is related to a current block being encoded within the current frame. The predetermined pattern can designate video frames in the sequence as P-frames or B-frames. Intra-BC unit 48 can determine vectors for intra-BC encoding (e.g., block vectors) in a similar manner as motion vectors determined by motion estimation unit 42 for inter-prediction, or can utilize block vectors determined by motion estimation unit 42.

[0057] Whether the predictive block is from the same frame according to intra-prediction or a different frame according to inter-prediction, video encoder 20 can form pixel difference values by subtracting pixel values of the predictive block from pixel values of the current video block being encoded, thereby forming a residual video block. The pixel difference values forming the residual video block can include both luma component differences and chroma component differences.

[0058] Intra prediction processing unit 46 can encode the current block using various intra prediction modes, e.g., during a separate encoding pass, and can select a suitable intra prediction mode to use from the tested intra prediction modes (or, in some examples, the mode selection unit). Intra prediction processing unit 46 can provide information indicating the selected intra prediction mode for the block to entropy encoding unit 56. Entropy encoding unit 56 can encode information indicating the selected intra prediction mode in the bitstream.

[0059] After prediction processing unit 41 determines a prediction block for the current video block via either inter prediction or intra prediction, summer 50 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block can be included in one or more TUs and is provided to transform processing unit 52. Transform processing unit 52 transforms the residual video data into residual transform coefficients using a transform, e.g., a discrete cosine transform (DCT) or a conceptually similar transform.

[0060] Transform processing unit 52 can send the resulting transform coefficients to quantization unit 54. Quantization unit 54 quantizes the transform coefficients to further reduce bit rate. The quantization process can also reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be modified by adjusting a quantization parameter. In some examples, quantization unit 54 can then perform a scan of the matrix including the quantized transform coefficients. Alternatively, entropy encoding unit 56 can perform the scan.

[0061] After quantization, entropy encoding unit 56 entropy encodes the quantized transform coefficients using, e.g., context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy encoding methodology or technique, into a video bitstream. The encoded bitstream can then be transmitted to video decoder 30 as shown in FIG. 3, or archived on a storage device 32 as shown in FIG. 3 for later transmission to or retrieval by video decoder 30. Entropy encoding unit 56 can also entropy encode motion vectors and other syntax elements for the current video frame being encoded. Figure 1 Figure 1

[0062] ​​The inverse quantization unit 58 and the inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual video block in the pixel domain for generating a reference block used to predict other video blocks. As noted above, the motion compensation unit 44 can generate a motion compensated prediction block from one or more reference blocks of a frame stored in the DPB 64. The motion compensation unit 44 can also apply one or more interpolation filters to the prediction block to calculate sub-integer pixel values for use in motion estimation.

[0063] The summer 62 adds the reconstructed residual block to the motion compensated prediction block produced by the motion compensation unit 44 to produce a reference block for storage in the DPB 64. The reference block can then be used by the intra BC unit 48, the motion estimation unit 42, and the motion compensation unit 44 as a prediction block to inter predict another video block in a subsequent video frame.

[0064] Figure 3 FIG. 1 shows a block diagram of an example video decoder 30 in accordance with some embodiments of the present application. The video decoder 30 includes a video data memory 79, an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, a summer 90, and a DPB 92. The prediction processing unit 81 further includes a motion compensation unit 82, an intra prediction unit 84, and an intra BC unit 85. The video decoder 30 can perform decoding processes substantially reciprocal to those described above in connection with the video encoder 20. Figure 2 The decoding processes described in relation to the video encoder 20 are substantially reciprocal. For example, the motion compensation unit 82 can generate prediction data based on motion vectors received from the entropy decoding unit 80, while the intra prediction unit 84 can generate prediction data based on intra prediction mode indicators received from the entropy decoding unit 80.

[0065] In some examples, embodiments of the present disclosure can be dispersed among one or more of the units of the video decoder 30. For example, the intra BC unit 85 can perform embodiments of the present application alone or in combination with other units of the video decoder 30, such as the motion compensation unit 82, the intra prediction unit 84, and the entropy decoding unit 80. In some examples, the video decoder 30 can not include the intra BC unit 85, and the functionality of the intra BC unit 85 can be performed by other components of the prediction processing unit 81, such as the motion compensation unit 82.

[0066] The video data memory 79 can store video data, such as an encoded video bitstream, to be decoded by the other components of the video decoder 30. The video data stored in the video data memory 79 can be obtained, for example, from the storage device 32, from a local video source, such as a camera, via wired or wireless network communication of video data, or by accessing a physical data storage medium, such as a flash drive or hard disk.

[0067] During the decoding process, video decoder 30 receives an encoded video bitstream that represents encoded video frames and associated syntax elements of video blocks. Entropy decoding unit 80 of video decoder 30 entropy decodes the bitstream to generate quantized coefficients, motion vectors or intra prediction mode indicators, and other syntax elements. Entropy decoding unit 80 then forwards the motion vectors or intra prediction mode indicators, and other syntax elements to prediction processing unit 81.

[0068] When a video frame is coded as an intra-predicted coded (I) frame or an intra coded prediction block in other types of frames, intra-prediction unit 84 of prediction processing unit 81 can generate prediction data for a video block of the current video frame based on the intra-prediction mode signaled and reference data from previously decoded blocks of the current frame.

[0069] When a video frame is coded as an inter-predicted coded (i.e., B or P) frame, motion compensation unit 82 of prediction processing unit 81 produces one or more prediction blocks for a video block of the current video frame based on the motion vectors and other syntax elements received from entropy decoding unit 80. Each of the prediction blocks can be produced from a reference frame within one of the reference frame lists. Video decoder 30 can construct the reference frame lists, i.e., List 0 and List 1, using default construction techniques based on reference frames stored in DPB 92.

[0070] In some examples, when a video block is encoded according to the intra BC mode described herein, intra BC unit 85 of prediction processing unit 81 produces a prediction block for the current video block based on the block vector and other syntax elements received from entropy decoding unit 80. The prediction block can be within a reconstructed region of the same picture as the current video block, as defined by video encoder 20.

[0071] Motion compensation unit 82 and / or intra BC unit 85 determine the prediction information for a video block of the current video frame by parsing the motion vectors and other syntax elements, and then use the prediction information to produce a prediction block for the current video block being decoded.

[0072] Motion compensation unit 82 can also perform interpolation to calculate interpolated values for sub-integer pixels of reference blocks using interpolation filters as used by video encoder 20 during encoding of the video blocks. In this case, motion compensation unit 82 can determine the interpolation filters used by video encoder 20 from the syntax elements received, and use these interpolation filters to produce the prediction blocks.

[0073] Inverse quantization unit 86 inverse quantizes quantized transform coefficients provided in the bitstream and entropy decoded by entropy decoding unit 80 using the same quantization parameter calculated by video encoder 20 for each video block in the video frame to determine a degree of quantization. Inverse transform processing unit 88 applies an inverse transform, e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process, to the transform coefficients in order to reconstruct the residual blocks in the pixel domain.

[0074] After motion compensation unit 82 or intra BC unit 85 generates the prediction block for the current video block based on the vectors and other syntax elements, adder 90 reconstructs the decoded video block for the current video block by adding the residual block from inverse transform processing unit 88 to the corresponding prediction block generated by motion compensation unit 82 and intra BC unit 85. In-loop filter 91, e.g., a de-blocking filter, a SAO filter, a CCSAO filter, and / or an ALF, can be located between adder 90 and DPB 92 to further process the decoded video block. In some examples, in-loop filter 91 can be omitted, and the decoded video block can be directly provided by adder 90 to DPB 92. The decoded video blocks in a given frame are then stored in DPB 92, which stores reference frames for subsequent motion compensation of video blocks that follow. DPB 92 or a memory device separate from DPB 92 can also store decoded video for later presentation on a display device (e.g., display device 34 of FIG. 1). Figure 1

[0075] In general, the basic inter prediction scheme applied in VVC remains almost the same as that of HEVC, except that some prediction tools are further extended, added, and / or improved, such as extended merge prediction, MMVD, symmetric motion vector difference (MVD) coding, affine motion compensation prediction, SbTMVP, adaptive motion vector resolution (AMVR), bi-prediction with CU-level weights (BCW), BDOF, PROF, DMVR, GPM, and CIIP.

[0076] Extended merge prediction

[0077] As video data acquisition technology continues to improve and video block sizes for preserving details of video data become finer, the amount of data needed to represent motion vectors of a current picture also substantially increases. One way to overcome this challenge is to use motion information (e.g., motion vectors) of spatial neighboring CUs, temporal collocated CUs, etc. of a current CU as an approximation (e.g., prediction) of the motion information of the current CU, which is also referred to as a “motion vector predictor (MVP)” of the current CU.

[0078] ​Like the process of selecting a prediction block in a reference picture during inter prediction of a coding block, both video encoder 20 and video decoder 30 need to employ a set of rules to construct a MVP candidate list for a current CU, and then select one MVP candidate from the MVP candidate list as the MVP for the current CU. By doing so, the MVP candidate list itself does not need to be transmitted between video encoder 20 and video decoder 30, and the index of the selected MVP candidate from the MVP candidate list is enough for video encoder 20 and video decoder 30 to use the same selected MVP candidate from the MVP candidate list to encode and decode the current CU.

[0079] In VVC, the MVP candidate list is constructed by including the following five kinds of MVPs in order:

[0080] — spatial MVPs from spatial neighboring CUs (i.e., spatial candidates);

[0081] — temporal MVPs from temporal collocated CUs (i.e., temporal candidates);

[0082] — history-based MVPs (HMVPs) from a first-in-first-out (FIFO) table;

[0083] — pair-wise average MVPs; and

[0084] — zero MVP.

[0085] The size of the MVP candidate list is signaled in the sequence parameter set header, and the maximum allowed size of the MVP candidate list is 6. For each CU coded in merge mode, the index of the best MVP candidate is coded using truncated unary binarization. The first bin of the index is coded using a context, and bypass coding is used for the other bins of the index.

[0086] The process of deriving each type of MVP is provided below. Like HEVC, VVC also supports deriving the MVP candidate list for all CUs within a certain size region in parallel.

[0087] Deriving MVP from spatial candidates

[0088] Deriving MVP from spatial candidates (e.g., CUs neighboring the current CU 101) in VVC is the same as deriving MVP from spatial candidates in HEVC, except that the positions of the first two spatial candidates are swapped. From the left of the current CU 101, Figure 4 Figure 4 ​Up to four spatial candidates are selected from the indicated spatial candidates (i.e., top position B0, left position A0, top right position B1, bottom left position A1, and top left position B2). The process is performed in the order of the CUs at positions B0, A0, B1, A1, and B2. The CU at position B2 is considered only if one or more CUs at positions B0, A0, B1, and A1 are unavailable (e.g., because the one or more CUs belong to other stripes or tiles) or are intra-coded.

[0089] After adding the CU at position B0 as a candidate to the merged candidate list, redundancy checks are performed on the addition of the remaining candidates to the merged candidate list. This ensures that candidates with the same motion information are excluded from the merged candidate list, thereby improving encoding and decoding efficiency. To reduce computational complexity, not all possible candidate pairs are considered in the redundancy check. Instead, only... Figure 5 Pairs linked by arrowed lines are considered, and a candidate is added to the merged candidate list only if the motion information of the candidate in the pair used for redundancy checking differs from the motion information of the candidate to be added. The spatial MVP obtained from the candidates in the merged candidate list is added to the MVP candidate list.

[0090] MVP is selected based on time candidate.

[0091] During the process of obtaining the MVP based on the time candidate, only one time candidate is added to the merge candidate list. Specifically, when obtaining the MVP based on this time candidate, for the current CU (e.g., Figure 6 The curr_CU 303 in the image is based on the image belonging to the same position (e.g., Figure 6 The corresponding CU of col_pic 302 in (e.g., Figure 6 The scaled motion vector (col_CU 301) is used as a time candidate to obtain the time MVP candidate, and this scaled motion vector is added to the MVP candidate list as a time MVP candidate. The list of reference images and their indices for obtaining the corresponding CU are explicitly signaled in the strip header. Figure 6 As shown, the scaled motion vector is obtained (i.e., scaled) based on the motion vector of the co-located CU using the image sequence count (POC) distance (i.e., tb and td), where tb is defined as the current image (e.g., ...). Figure 6 Reference image for curr_pic 304 (e.g., Figure 6 The difference between curr_ref305 and the current image in the POC, while td is defined as the reference image of the co-located image (e.g., Figure 6 The POC difference between col_ref 306 and the corresponding image. The reference image index for the time candidate is set to zero.

[0092] As Figure 7 illustrated, the position in the current CU 401 for the temporal candidate (i.e., collocated CU) is selected from positions CO and CI. If the CU at position CO in the collocated picture is unavailable, intra coded or located outside the current CTU row, the CU at position CI is used as the collocated CU for deriving the temporal MVP candidate. Otherwise, the CU at position CO is used as the collocated CU for deriving the temporal MVP candidate.

[0093] Deriving HMVP candidate

[0094] After the spatial and temporal MVPs, the HMVP candidate is added to the MVP candidate list. The motion information of previously coded blocks is stored in the HMVP table and used as the MVP for the current CU. The table with multiple HMVP candidates is maintained during the encoding / decoding process. The table is reset (emptied) when a new CTU row is encountered. Whenever there is a CU that is non-subblock inter coded, the associated motion information is added as a new HMVP candidate to the last entry of the HMVP table.

[0095] The size of the HMVP table is set to 6. When a new HMVP candidate is inserted into the HMVP table, the constrained FIFO rule is utilized, where a redundancy check is first applied to find if there is a same HMVP in the HMVP table. If found, the same HMVP is removed from the HMVP table and all the HMVP candidates after it are moved forward, and the same HMVP is added to the last entry of the HMVP table.

[0096] The HMVP candidate can be used in the MVP candidate list construction process. The latest few HMVP candidates in the HMVP table are checked in turn and inserted into the MVP candidate list after the temporal MVP candidates. The redundancy check is applied to the HMVP candidate with respect to the spatial and / or temporal MVP candidates.

[0097] To reduce the number of redundancy check operations, the following simplifications are introduced:

[0098] - redundancy check is applied to the last two entries in the HMVP table with respect to the spatial MVP candidates derived from the spatial candidates at positions Al and Bl, respectively; and

[0099] - the process of constructing the MVP candidate list from the HMVP candidates is terminated once the total number of available MVP candidates reaches the maximum allowed size of the MVP candidate list minus 1.

[0100] Deriving pairwise average MVP candidate

[0101] A pairwise average MVP candidate is generated by averaging the MVPs resulting from the predefined pair of the first two merge candidates in the existing merge candidate list. The first merge candidate in the predefined pair can be defined as p0Cand, and the second merge candidate in the predefined pair can be defined as piCand. Separately for each reference picture list, the average motion vector is calculated according to the availability of the motion vectors of p0Cand and piCand. If both motion vectors are available for one reference picture list, the two motion vectors are averaged even if they point to different reference pictures, and the reference picture of the averaged motion vector is set to the reference picture of p0Cand; if only one motion vector is available for one reference picture list, the motion vector is used directly; if no motion vector is available for one reference picture list, the motion vector and the reference picture index for the reference picture list remain invalid.

[0102] Zero MVP

[0103] When the MVP candidate list is not full after adding the pairwise average MVP candidate, a zero MVP is inserted at the end of the MVP candidate list until the maximum allowed size of the MVP candidate list is reached.

[0104] MMVD

[0105] As mentioned above, in the merge mode, the motion information (i.e., the MVP candidate) is implicitly derived from the MVP candidate list constructed for the current CU and directly used as the MV of the current CU for generating the predicted samples of the current CU, which can cause a certain error between the actual MV of the current CU and the implicitly derived MVP. To improve the accuracy of the MV of the current CU, MMVD is introduced in VVC, in which the motion vector difference (MVD) of the current CU is added to the implicitly derived MVP to obtain the MV of the current CU. After signaling the regular merge flag, an MMVD flag is signaled to specify whether the MMVD mode is used for the current CU.

[0106] In the MMVD mode, after selecting the MVP candidate from the first two MVP candidates in the MVP candidate list, MMVD information is signaled, in which the MMVD information includes an MMVD candidate flag for specifying which one of the first two MVP candidates is selected as the MV basis, a distance index for indicating the motion magnitude information of the MVD, and a direction index for indicating the motion direction information of the MVD.

[0107] The distance index for specifying the motion magnitude information of the MVD indicates the distance between the current CU and the position in the reference picture (e.g., the L0 reference picture 501 or the L1 reference picture 503) pointed to by the selected MVP candidate. Figure 8 The direction index for specifying the motion direction information of the MVD indicates the direction of the MVD, i.e., the direction of the motion of the current CU relative to the position in the reference picture (e.g., the L0 reference picture 501 or the L1 reference picture 503) pointed to by the selected MVP candidate.Figure 8 The predefined offset from the origin (indicated by the dashed circle in FIG. 7) can be used to derive the MVD, and the MVD can be added to the selected MVP candidate. The relationship between the distance index and the predefined offset is specified in Table 1 below.

[0108]

[0109] Table 1

[0110] The direction index specifies the sign of the MVD, which indicates the direction of the MVD relative to the origin. Table 2 specifies the relationship between the direction index and the predefined sign. Note that the meaning of the sign of the MVD can change depending on the information of the selected MVP candidate. When the selected MVP candidate is a uni-prediction MV or a bi-prediction MV in which both MVs point to the same side of the current picture (i.e., both POCs of the two reference pictures (e.g., the reference pictures of List 0 and List 1, also referred to as the L0 reference picture and the L1 reference picture, respectively) of the current picture are greater than the POC of the current picture or are less than the POC of the current picture), the sign in Table 2 specifies the sign of the MVD added to the selected MVP candidate. When the selected MVP candidate is a bi-prediction MV in which the two MVs point to different sides of the current picture (i.e., one reference picture of the current picture has a POC greater than the POC of the current picture, while the other reference picture of the current picture has a POC less than the POC of the current picture), if the POC distance for the L0 reference picture (i.e., the POC distance between the L0 reference picture and the current picture) is greater than the POC distance for the L1 reference picture (i.e., the POC distance between the L1 reference picture and the current picture), the sign in Table 2 specifies the sign of the MVD for List 0 (MVD0) added to the MVP for List 0 (MVP0) of the selected MVP candidate, and the sign of the MVD for List 1 (MVD1) added to the MVP for List 1 (MVP1) of the selected MVP candidate is opposite to the sign in Table 2; otherwise, if the POC distance for the L1 reference picture is greater than the POC distance for the L0 reference picture, the sign in Table 2 specifies the sign of MVD1 added to MVP1, and the sign of MVD0 added to MVP0 is opposite to the sign in Table 2.

[0111]

[0112] Table 2

[0113] MVD is scaled according to the POC distance. If the POC distance for L0 reference pictures and L1 reference pictures is the same, no scaling of MVD is needed. Otherwise, if the POC distance for L0 reference pictures is greater than the POC distance for L1 reference pictures, MVD1 is scaled. If the POC distance for L1 reference pictures is greater than the POC distance for L0 reference pictures, MVD0 is scaled.

[0114] Symmetric MVD coding

[0115] In VVC, in addition to normal MVD signaling in uni-prediction mode and bi-prediction mode, a symmetric MVD mode for MVD signaling in bi-prediction mode is applied. In symmetric MVD mode, the motion information including the indices of reference pictures in list 0 and list 1 and the MVD associated with list 1 is not signaled from the encoder 20 but derived at the decoder 30.

[0116] The decoding process of symmetric MVD mode is as follows.

[0117] At slice level, the variable BiDirPredFlag and the reference picture indices RefIdxSymL0 and RefIdxSymL1 for the current picture are derived as follows:

[0118] - If the flag mvd_l1_zero_flag is equal to 1, the variable BiDirPredFlag is set equal to 0, where the flag mvd_l1_zero_flag indicates whether the MVD of each CU associated with list 1 in the slice of the current picture is zero;

[0119] - Otherwise, if the closest reference picture in list 0 and the closest reference picture in list 1 to the current picture form a forward-backward pair of reference pictures or a backward-forward pair of reference pictures, the variable BiDirPredFlag is set to 1; otherwise, BiDirPredFlag is set to 0.

[0120] At CU level, if the current CU is bi-predictively coded and the variable BiDirPredFlag is equal to 1, a symmetric MVD mode flag indicating whether the symmetric MVD mode is used is explicitly signaled.

[0121] RefIdxSymL0 and RefIdxSymL1 are set to the indices of the reference pictures in list 0 and list 1, respectively.

[0122] When the symmetric MVD mode flag is true, only the MVP index for list 0 mvp_l0_flag, the MVP index for list 1 mvp_l1_flag and the MVD for list 0 MVD0(mvdx0, mvpy0) are explicitly signaled. The MVP for list 0 MVP0(mvpx0, mvpy0) is derived based on the MVP index for list 0 mvp_l0_flag and the above MVP candidate list, and the MVP for list 1 MVP1(mvpx1, mvpy1) is derived based on the MVP index for list 1 mvp_l1_flag and the above MVP candidate list. The MVD for list 1 MVD1(mvdx1, mvdy1) is set equal to (-mvdx0, -mvdy0). The final motion vector MV for list 0 MV0(mvx0, mvy0) and the final motion vector MV for list 1 MV1(mvx1, mvy1) are derived using the following equations, respectively:

[0123]

[0124] As described in the foregoing, in the process of video encoding, because full integer pixel search and fractional pixel search are performed on all reference frames, a large amount of calculation is brought, which leads to difficulty in improving the calculation speed.

[0125] Specifically, in video coding standards such as H.264 / High Efficiency Video Coding (HEVC), a current frame can use a frame that has been encoded as a reference frame to perform inter-frame prediction encoding to greatly reduce the transmission cost of a residual signal. Existing video coding standards often allow the current frame to use multiple encoded frames for inter-frame prediction encoding. The introduction of multiple reference frames can improve the probability of each coding unit (CU) in the current frame searching for a suitable matching block, thereby greatly improving the coding efficiency.

[0126] However, in the case of an increasing number of reference frames, motion search is performed in each reference frame to find the optimal matching block for all CUs of the current frame, which means that the coding complexity will increase exponentially with the number of reference frames. For real-time transmission scenarios such as video live streaming, which have high requirements for coding speed, efficient multi-reference frame architecture and acceleration algorithms need to be developed to reduce the coding complexity while ensuring compression efficiency.

[0127] For example, an example of a multi-reference frame motion search architecture adopted in open source software x264 / x265 [2] and HEVC reference software HM is shown in FIG. 1. Figure 9 Figure 9 ​In the prior art, the maximum number of reference frames set by a user is denoted as N, and the current frame needs to traverse N reference frames, such as reference frames Ref 0, Ref 1 to Ref N-1, in the forward and backward reference frame lists (for example, List 0 and List 1) respectively, and motion estimation is performed in each reference frame to search for a matching block with the minimum rate-distortion cost (RDcost).

[0128] RDcost = SATD + λ * R (2)

[0129] wherein the sum of absolute transformed difference (SATD) measures the distortion of the reference block and the current block, and R is the bits of the MV of the encoded current motion vector. λ is a Lagrange multiplier, which is related to the quantization parameter (QP) of the current block. After traversing all the frames, the reference frame index and the MV with the minimum RDcost are selected for encoding.

[0130] Here, the motion search includes both integer pixel search and fractional pixel search. An example of the conventional motion search process of an encoder is shown in Figure 10 .

[0131] As shown in Figure 10 , when encoding the current CU, the available encoded frames are derived from the encoded pictures, and the available encoded frames are sequentially used as reference frames. However, integer pixel search is performed first, and after the optimal integer pixel MV of the current reference frame is obtained, fractional pixel search is performed around the integer pixel MV. Since interpolation of the integer pixel is required in the fractional pixel search, the fractional pixel search is often a module with a high time consumption ratio of the encoder. After the fractional pixel search is completed, the obtained RDcost is compared with the current optimal RDcost. If the RDcost is less than the optimal RDcost, the current reference frame index and the MV are recorded and marked as the optimal reference frame and the MV. However, the search of the next reference frame can be continued.

[0132] As can be seen from the above process, in the conventional search scheme, the integer pixel search and the fractional pixel search are sequentially performed, and the complete integer pixel search process and the fractional pixel search process need to be performed for all the reference frames, and finally only one optimal reference frame is selected. In terms of the forward or backward reference frame list in Figure 9 , it means that N-1 calculation results are finally not adopted, and there is a large pruning and acceleration space.

[0133] To address or at least mitigate some of the aforementioned problems, exemplary embodiments of this disclosure provide a video encoding method, a video encoding apparatus, an electronic device, a computer-readable storage medium, a computer program product, a method for storing bitstreams, and a method for transmitting bitstreams. Reference will be made below. Figures 11 to 14 Provide a detailed description.

[0134] According to a first aspect of the embodiments of this disclosure, a video coding method is provided. This method can separate the integer-pixel search process from the sub-pixel search process. First, an integer-pixel search is performed on all reference frames. Then, based on the results of the integer-pixel search, the rate-distortion cost of each reference frame is compared with a rate-distortion cost threshold. When the rate-distortion cost of a reference frame meets a condition, the sub-pixel search of that reference frame is adaptively skipped, thereby reducing the computational load of the sub-pixel search and significantly improving the coding speed.

[0135] Figure 11 An illustrative flow diagram of a video encoding method according to an exemplary embodiment of the present disclosure is shown. For example... Figure 11 As shown, the video encoding method may include:

[0136] In step S1110, an integer pixel search can be performed on at least one reference frame of the current block to obtain the rate-distortion cost of each reference frame in the at least one reference frame.

[0137] Here, the current block can be any video block to be encoded in any video frame, such as, but not limited to, a CU.

[0138] As an example, the aforementioned reference frame can be any of the reference frames in the forward or backward reference frame list. For instance, when encoding the current CU, an integer-pixel search can be performed on all reference frames in the order of the default reference frame list.

[0139] Here, the search method of the embodiments of this disclosure can be applied to the forward reference frame list and / or the backward reference frame list of a video block. When applied to both, the optimal reference frame can be obtained for the forward reference frame list and the backward reference frame list respectively. Then, the two optimal reference frames are compared to determine the final reference frame used to encode the current block.

[0140] The integer pixel search method can be the same as existing integer pixel search methods, for example, it can be used with... Figure 9 The integer pixel search method shown is the same, and this disclosure does not impose any particular limitation on the specific method of integer pixel search. The rate-distortion cost can be calculated in, for example, but not limited to, the way shown in equation (2) above.

[0141] At step S1120, the sub-pixel search can be performed on the at least one reference frame respectively, wherein for each of the at least one reference frame, when the rate-distortion cost of the reference frame is greater than the rate-distortion cost threshold, the sub-pixel search for the reference frame is skipped.

[0142] In an embodiment of the present disclosure, an adaptive skip algorithm can be employed in the sub-pixel search of the reference frame, by comparing the RDcost of the integer-pixel search of the current frame with the RDcost of the optimal sub-pixel search, whether to skip the sub-pixel search of the current frame is adaptively selected, so as to greatly improve the encoding speed while ensuring the compression performance.

[0143] In step S1110 above, the integer-pixel search is performed on all the reference frames respectively, and the rate-distortion cost of each reference frame in the integer-pixel search is obtained. In step S1120, according to the rate-distortion cost of each reference frame in the integer-pixel search, it can be determined whether the sub-pixel search of the reference frame needs to be skipped, if the result of the determination is not to skip, the sub-pixel search of the reference frame can be performed based on the result of the integer-pixel search of the reference frame, around the pixel point searched in the integer-pixel search (for example, the neighboring pixel points); if the result of the determination is to skip, the sub-pixel search of the reference frame is not performed, so that the calculation amount can be reduced and the encoding speed can be improved.

[0144] Here, the manner of the sub-pixel search for a single reference frame can be the same as the existing sub-pixel search manner, for example, can be the same as the sub-pixel search manner shown in Figure 9 The present disclosure does not particularly limit the specific manner of the sub-pixel search.

[0145] In the above process of adaptively skipping the sub-pixel search, the rate-distortion cost threshold can be determined based on a predetermined multiple of the current optimal rate-distortion cost.

[0146] Here, the predetermined multiple of the current optimal rate-distortion cost can be used to measure the degree that the rate-distortion cost of the reference frame in the integer-pixel search is greater than the current optimal rate-distortion cost. When the rate-distortion cost of the reference frame in the integer-pixel search is greater than the predetermined multiple of the current optimal rate-distortion cost, it can be considered that the rate-distortion cost of the reference frame in the integer-pixel search is much greater than the current optimal rate-distortion cost, and the rate-distortion cost of the sub-pixel search of the reference frame will also be greater than the current optimal rate-distortion cost, therefore, the process of the sub-pixel search of the reference frame can be skipped.

[0147] The current optimal rate-distortion cost could be, for example, the optimal rate-distortion cost among the rate-distortion costs of the reference frames for which a pixel-wise search has been performed. Here, before a pixel-wise search has been performed (e.g., when performing a pixel-wise search on the first reference frame), the current optimal rate-distortion cost could be, for example, a preset value. This preset value could be, for example, the maximum value of the data type of the program executing the method; for example, if the program uses the int data type, the preset value could be the maximum value of the int data type, 0x7ffffff.

[0148] For example, before performing a pixel-by-pixel search, the integer-pixel search rate-distortion cost of the current reference frame can be calculated. Compared to the current optimal rate-distortion cost (RDcost) best The comparison is performed with a predetermined multiple k, and skips steps based on the comparison result. This comparison process can be illustrated as shown in equation (3):

[0149]

[0150] in, This indicates the comparison process. The predetermined multiplier k can be set according to actual needs, for example, it can be set to 1.28 based on the experiment.

[0151] As can be seen from equation (3) above, if the current reference frame corresponds to an integer pixel search Compared to the optimal pixel-wise search RDcost best If the value is much larger, then the current reference frame can be considered unsuitable as the reference frame for the current block, and the pixel-by-pixel search of the current reference frame can be skipped in advance to improve encoding speed.

[0152] In embodiments of this disclosure, the predetermined multiplier can be a fixed value, for example, and the predetermined multiplier can be the same for different encoded frames and encoded blocks. However, it is not limited to this. The predetermined multiplier can also be determined based on the current block and / or the video frame in which the current block is located. The predetermined multiplier can be different for different encoded frames and encoded blocks.

[0153] In one example, the predetermined multiplier can be determined based on the coding level of the video frame to which the current block belongs and / or the type of the current reference frame.

[0154] For example, the predetermined multiplier can be positively correlated with the level of the coding hierarchy of the video frame to which the current block belongs. Specifically, in a hierarchical coding structure, the coding quality of frames at higher levels may affect the coding quality of subsequent frames. Therefore, the probability of skipping can be adjusted by adjusting the predetermined multiplier, using a larger predetermined multiplier for higher-level frames and a smaller predetermined multiplier for lower-level frames.

[0155] For example, since P frames tend to be referenced by more frames than B frames, the quality of P frames greatly influences the quality of subsequent frames, the predetermined multiples for P frames and B frames can be set differently. Specifically, the predetermined multiple for P frames can be greater than that for B frames, so that P frames tend to skip fewer search processes and B frames tend to skip more search processes.

[0156] In embodiments of the present disclosure, since frames that are important in the encoding process have a greater influence on the quality of subsequent frames and / or the overall encoding quality, setting the predetermined multiples in the above manner can skip fewer important frames, thereby balancing the encoding speed and the encoding quality while improving the encoding speed and ensuring the encoding quality.

[0157] In another example based on or instead of the above example, the predetermined multiple can be determined according to a motion vector derived in the integer-pixel search.

[0158] Specifically, the predetermined multiple can be set according to the motion degree of a block. For example, the predetermined multiple can be determined according to a motion vector (MV) corresponding to the optimal rate-distortion cost in the integer-pixel search, specifically, the motion of the current block can be determined according to the amplitude of the optimal MV obtained in the integer-pixel search.

[0159] Here, for blocks with intense motion, the search process can be skipped as little as possible, and for relatively stationary blocks, the search process can be skipped as much as possible, so that the probability of skipping the sub-pixel search can be adjusted considering the complexity of the current block, so that the encoding quality of blocks with different complexities can be ensured while improving the encoding speed.

[0160] As an example, the amplitude of the optimal MV of the current block in the integer-pixel search can be compared with a preset amplitude threshold, in the case where the amplitude is greater than the amplitude threshold, it can be considered that the motion of the block is intense, and a preset first multiple can be used; in the case where the amplitude is less than the amplitude threshold, it can be considered that the motion of the block is gentle or in a stationary state, and a preset second multiple can be used, here, the first multiple can be greater than the second multiple.

[0161] In addition, as an example, in the above sub-pixel search, the sub-pixel search can be performed on the at least one reference frame in a preset order or randomly. For example, the reference frames can be sequentially subjected to sub-pixel search skipping judgment in the order of the initial reference frame list (such as the forward or backward reference frame list) to skip or perform sub-pixel search; or the reference frames can be sequentially subjected to sub-pixel search skipping judgment in a random order.

[0162] Here, according to the example embodiments of the present disclosure, the fractional pixel search can be performed on the at least one reference frame respectively in the order of the rate-distortion cost of the at least one reference frame in the integer pixel search from small to large.

[0163] Specifically, the reference frame list can be reordered according to the RDcost of the integer pixel search from small to large. According to the reordered order, it is determined whether to skip the fractional pixel search for the reference frames in sequence. In the case of determining not to skip, the fractional pixel search can be performed around the optimal integer pixel of the reference frame to obtain the rate-distortion cost RDcost of the current reference frame in the fractional pixel search. j and compared with the current optimal rate-distortion cost RDcost best . If the rate-distortion cost RDcost of the current fractional pixel search is smaller than the current optimal rate-distortion cost RDcost j , the MV and the reference frame index of the current frame are marked as optimal, and the rate-distortion cost RDcost of the current fractional pixel search is assigned to the optimal rate-distortion cost RDcost best , and the subsequent fractional pixel search is continued until all the reference frames are traversed, and the search is ended. j best

[0164] Through the above-mentioned reordering, the optimal reference frame in the integer pixel search can be searched first in the fractional pixel search, thereby facilitating the fast search of the optimal reference frame in the fractional pixel search.

[0165] As an example, the step of performing the fractional pixel search on the at least one reference frame respectively can further include: for each of the at least one reference frame, when the rate-distortion cost of the reference frame is greater than a rate-distortion cost threshold, skipping the fractional pixel search of the reference frame and subsequent reference frames.

[0166] Specifically, in the case of reordering the reference frames according to the results of the integer pixel search, if the rate-distortion cost of the integer pixel search of the current reference frame is greater than the current optimal rate-distortion cost, since the rate-distortion cost of the integer pixel search of the subsequent reference frame is greater or worse than the rate-distortion cost of the integer pixel search of the current frame, the fractional pixel search of the current reference frame and all the subsequent reference frames can be directly skipped.

[0167] Through the above-mentioned example, the fractional pixel search can be performed in sequence based on the reordered reference frame order, thereby the fractional pixel search of the subsequent reference frames can be directly skipped in the case that the current reference frame meets certain conditions, and the encoding speed is further improved.

[0168] An example flow of a video encoding method according to an embodiment of the present disclosure will be described below with reference to Figure 12 . As shown in FIG. 1, the video encoding method according to an embodiment of the present disclosure can include the following steps. Figure 12 ​​As shown, in step S1201, the current CU can be encoded, and in step S1202, the available reference frames can be loaded.

[0169] In step S1203, the i-th frame can be taken as the current reference frame, where i is an integer from 1 to N, and N is the number of reference frames loaded in step S1202. In step S1204, a full-pel search can be performed on the current reference frame to select the optimal full-pel MV.

[0170] In step S1205, it can be determined whether the current i is less than N. In response to i being less than N, the value of i can be incremented by 1, and the process returns to step S1203; in response to i not being less than N, step S1206 can be performed.

[0171] In step S1206, all reference frame indexes can be sorted according to the rate-distortion cost of the full-pel search to form a new reference frame list. In step S1207, the j-th frame in the list can be taken as the current reference frame, where j is an integer from 1 to N.

[0172] In step S1208, it can be determined whether to skip the sub-pel search for the current reference frame. Specifically, it can be determined whether the optimal full-pel of the current reference frame is greater than a rate-distortion cost threshold. In response to not being greater than the threshold, in step S1209, a sub-pel search can be performed around its optimal full-pel to obtain the rate-distortion cost RDcostjof the sub-pel search. ； In response to being greater than the threshold, in step S1210, the sub-pel search for the current reference frame and subsequent reference frames can be skipped.

[0173] In step S1211, it can be determined whether the rate-distortion cost RDcostj j is less than the current optimal rate-distortion cost RDcost best . In response to being less than the current optimal rate-distortion cost RDcost best , in step S1212, the current reference frame index and MV can be marked as optimal, and RDcostj= RDcost best , and then step S1213 can be performed. In response to not being less than the current optimal rate-distortion cost RDcost best , step S1213 can be performed.

[0174] In step S1213, it can be determined whether j is less than N. In response to j being less than N, the value of j can be incremented by 1, and the process returns to step S1208; in response to j not being less than N, the search process can be ended, and the current optimal reference frame index and MV can be output.

[0175] In embodiments of the present disclosure, comparative experiments are conducted on a video encoding process employing the integer-pel and fractional-pel search method according to embodiments of the present disclosure and a video encoding process not employing the method, and the experimental results prove that the method according to embodiments of the present disclosure can save 6.32% of encoding time of a K265 mobile terminal encoder, with a BD-SSIM loss of only 0.064% and a BD-PSNR loss of only 0.067%. Here, BD- -Delta rate) refers to the difference in code rate of two encoding algorithms under the same quality evaluation index, a negative number indicates code rate saving, SSIM (Structural Similarity) refers to structural similarity measurement, and PSNR (Peak signal-to-noise ratio) refers to peak signal-to-noise ratio.

[0176] The video encoding method according to embodiments of the present disclosure proposes a multi-reference frame motion search architecture for real-time transmission scenarios. The architecture is more flexible and is different from the motion search architecture in open-source encoders and reference software. The integer-pel and fractional-pel search in the multi-reference frame can be split or decoupled. After integer-pel search is performed on all reference frames, the candidate reference frames are sorted in ascending order according to the obtained integer-pel rate-distortion cost. Then, fractional-pel search is performed on the sorted reference frames one by one. Based on the integer-pel search, more intermediate information can be obtained, so that the fractional-pel search adaptive jump-out algorithm can be executed. In the multi-reference frame fractional-pel search, whether to skip the fractional-pel search of the current frame is adaptively selected, which greatly improves the encoding speed under the premise of ensuring compression performance.

[0177] According to a second aspect of embodiments of the present disclosure, a video encoding apparatus is provided, as shown in Figure 13 The video encoding apparatus can include an integer-pel search unit 1310 and a fractional-pel search unit 1320.

[0178] The integer-pel search unit 1310 is configured to perform integer-pel search on at least one reference frame of a current block respectively, to obtain a rate-distortion cost of each reference frame in the at least one reference frame.

[0179] The fractional-pel search unit 1320 is configured to perform fractional-pel search on the at least one reference frame respectively, wherein for each reference frame in the at least one reference frame, when the rate-distortion cost of the reference frame is greater than a rate-distortion cost threshold, the fractional-pel search for the reference frame is skipped.

[0180] As an example, the rate-distortion cost threshold is determined based on a predetermined multiple of the current optimal rate-distortion cost.

[0181] As an example, the current optimal rate-distortion cost is the optimal rate-distortion cost among the rate-distortion costs of the reference frames for which the sub-pixel search has been performed.

[0182] As an example, the predetermined multiple is determined according to the coding level to which the current reference frame belongs and / or the type of the current reference frame.

[0183] As an example, the predetermined multiple is positively correlated with the level height of the coding level to which the video frame in which the current block is located belongs.

[0184] As an example, the predetermined multiple is determined according to the motion vector derived in the integer-pixel search.

[0185] As an example, the predetermined multiple is determined according to the motion vector corresponding to the optimal rate-distortion cost in the integer-pixel search.

[0186] As an example, the sub-pixel search unit 1320 is configured to perform the sub-pixel search for each of the at least one reference frame in the order of the rate-distortion costs of the at least one reference frame from small to large.

[0187] As an example, the sub-pixel search unit 1320 is configured to, for each of the at least one reference frame, skip the sub-pixel search for the reference frame and subsequent reference frames when the rate-distortion cost of the reference frame is greater than a rate-distortion cost threshold.

[0188] As to the apparatus in the above-described embodiments, the specific manners in which the respective units perform operations have been described in detail in the embodiments of the method, and thus will not be elaborated here.

[0189] According to a third aspect of embodiments of the present disclosure, an electronic device is provided, which includes a processor and a memory for storing processor-executable instructions, wherein the processor-executable instructions, when executed by the processor, cause the processor to perform the video encoding method according to the example embodiments of the present disclosure.

[0190] As an example, the electronic device is not necessarily a single device, but can also be any collection of apparatuses or circuits that can individually or jointly execute the above-described instructions (or instruction sets). The electronic device can also be part of an integrated control system or system manager, or can be configured as a server that interfaces with a local or remote server (e.g., via wireless transmission).

[0191] In the electronic device, the processor can include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a special-purpose processor system, a microcontroller, or a microprocessor. As an example but not limitation, the processor can also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0192] The processor can execute instructions or code stored in the memory, where the memory can also store data. The instructions and data can also be sent and received over a network via the network interface device, where the network interface device can employ any known transmission protocol.

[0193] The memory can be integral to the processor, such as RAM or flash memory disposed within an integrated circuit microprocessor or the like. Further, the memory can include a separate device, such as an external disk drive, memory array, or other storage device usable by any database system. The memory and processor can be operatively coupled, or can communicate with each other, such as through I / O ports, network connections, or the like, so that the processor can read files stored in the memory.

[0194] Further, the electronic device can also include a video display, such as a liquid crystal display, and a user interface, such as a keyboard, mouse, touch input device, or the like. All of the components of the electronic device can be connected via a bus and / or network.

[0195] In an exemplary embodiment, a computer readable storage medium can also be provided, which, when instructions stored therein are executed by a processor of an electronic device, enables the electronic device to perform a video encoding method according to exemplary embodiments of the present disclosure.

[0196] Computer-readable storage media may be, for example, a memory that includes instructions. Optionally, computer-readable storage media may be: read-only memory (ROM), random access memory (RAM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0197] In an exemplary embodiment, a computer program product is also provided, having instructions for storing a bitstream, wherein the bitstream includes video data generated by a video encoding method as described in an exemplary embodiment of this disclosure.

[0198] In an exemplary embodiment, a method for storing a bitstream may also be provided, the method comprising: generating a bitstream according to the video encoding method described in the exemplary embodiment of the present disclosure; and storing the bitstream.

[0199] In an exemplary embodiment, a method for transmitting a bitstream may also be provided, the method comprising: generating a bitstream according to a video encoding method as described in an exemplary embodiment of the present disclosure; and transmitting the bitstream.

[0200] Figure 14A computing environment 1410 is shown coupled with a user interface 1450. The computing environment 1410 can be part of a data processing server. The computing environment 1410 includes a processor 1420, a memory 1430, and an input / output (I / O) interface 1440.

[0201] The processor 1420 generally controls the overall operation of the computing environment 1410, such as operations associated with displaying, data acquisition, data communication, and image processing. The processor 1420 can include one or more processors for executing instructions to perform all or some of the steps in the above-described methods. In addition, the processor 1420 can include one or more modules that facilitate interaction with other components of the computing environment 1410. The processor can be a central processing unit (CPU), a microprocessor, a microcontroller, a graphics processing unit (GPU), etc.

[0202] The memory 1430 is configured to store various types of data to support the operation of the computing environment 1410. The memory 1430 can include predetermined software 1432. Examples of such data include instructions for any application or method operating on the computing environment 1410, video data sets, image data, etc. The memory 1430 can be implemented by using any type of volatile or non-volatile memory devices, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.

[0203] The I / O interface 1440 provides an interface between the processor 1420 and peripheral interface modules (e.g., a keyboard, a click wheel, a button, etc.). The button can include, but is not limited to, a home button, a start scanning button, and a stop scanning button. The I / O interface 1440 can be coupled with an encoder and a decoder.

[0204] In an embodiment, a non-transitory computer-readable storage medium including, for example, a plurality of programs in the memory 1430 and / or storing a bitstream generated by the above-described encoding method is also provided, which can be executed by the processor 1420 in the computing environment 1410 for performing the above-described method. In one example, the plurality of programs can be executed by the processor 1420 in the computing environment 1410 to (e.g., from Figure 2A bitstream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or associated one or more syntax elements, etc.) can be received by a video encoder 20 in a video encoding apparatus 100, for example. In another example, the plurality of programs can be executed by a processor 1420 in a computing environment 1410 for performing the above-mentioned encoding method to encode video information (e.g., video blocks representing video frames, and / or associated one or more syntax elements, etc.) into a bitstream or data stream, and can also be executed by the processor 1420 in the computing environment 1410 for (e.g., to a video decoder 30 in a video decoding apparatus 200, etc.). Figure 3 A bitstream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or associated one or more syntax elements, etc.) can be transmitted by a video decoder 30 in a video decoding apparatus 200, for example. Alternatively, a non-transitory computer-readable storage medium can have stored therein a bitstream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or associated one or more syntax elements, etc.) generated by an encoder (e.g., a video encoder 20 in a video encoding apparatus 100, etc.) using, for example, the above-mentioned encoding method for use by a decoder (e.g., a video decoder 30 in a video decoding apparatus 200, etc.) in decoding video data. Figure 2 A bitstream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or associated one or more syntax elements, etc.) can be transmitted by a video decoder 30 in a video decoding apparatus 200, for example. Alternatively, a non-transitory computer-readable storage medium can have stored therein a bitstream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or associated one or more syntax elements, etc.) generated by an encoder (e.g., a video encoder 20 in a video encoding apparatus 100, etc.) using, for example, the above-mentioned encoding method for use by a decoder (e.g., a video decoder 30 in a video decoding apparatus 200, etc.) in decoding video data. Figure 3 A bitstream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or associated one or more syntax elements, etc.) can be transmitted by a video decoder 30 in a video decoding apparatus 200, for example. Alternatively, a non-transitory computer-readable storage medium can have stored therein a bitstream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or associated one or more syntax elements, etc.) generated by an encoder (e.g., a video encoder 20 in a video encoding apparatus 100, etc.) using, for example, the above-mentioned encoding method for use by a decoder (e.g., a video decoder 30 in a video decoding apparatus 200, etc.) in decoding video data.

[0205] In an embodiment, a bitstream generated by the above-mentioned encoding method is provided. In an embodiment, a computer program product having instructions for storing or transmitting a bitstream including encoded video information generated by the above-mentioned encoding method is provided. In an embodiment, a computer program product including a plurality of programs in, for example, the memory 1430 can be executed by the processor 1420 in the computing environment 1410 for performing the above-mentioned method.

[0206] In an embodiment, a computing device is also provided, including one or more processors (e.g., the processor 1420), and a non-transitory computer-readable storage medium or memory 1430 having stored therein a plurality of programs that can be executed by the one or more processors, wherein the one or more processors, when executing the plurality of programs, are configured to perform the above-mentioned method.

[0207] In an embodiment, a computer program product having instructions for storing or transmitting a bitstream including encoded video information generated by the above-mentioned encoding method is provided. In an embodiment, a computer program product including a plurality of programs in, for example, the memory 1430 can be executed by the processor 1420 in the computing environment 1410 for performing the above-mentioned method. For example, the computer program product can include a non-transitory computer-readable storage medium.

[0208] In an embodiment, the computing environment 1410 can be implemented by one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), FPGAs, GPUs, controllers, microcontrollers, microprocessors, or other electronic components, for performing the above-mentioned method.

[0209] The description of the present disclosure has been presented for purposes of illustration and is not intended to be exhaustive or limited to the disclosure as set forth in the foregoing description and the associated drawings are presented for purposes of illustration and are not intended to be exhaustive or to limit the disclosure. Many modifications, variations, and alternative embodiments will be apparent to those of ordinary skill in the art, in view of the foregoing description, from the teaching presented in the foregoing description and associated drawings.

[0210] Unless otherwise specifically noted, the order of steps in the methods according to the present disclosure is merely illustrative and the steps in the methods according to the present disclosure are not limited to the specific order presented above, but can be varied according to actual circumstances. In addition, at least one of the steps in the methods according to the present disclosure can be adjusted, combined, or deleted according to actual needs.

[0211] Examples are chosen and described in order to explain the principles of the present disclosure and to enable others skilled in the art to best utilize the various embodiments of the present disclosure, and to best utilize the various embodiments for the intended purposes and with various modifications as are suited to the particular use or implementation. Accordingly, it is to be understood that the scope of the present disclosure is not to be limited to the specific examples disclosed and that modifications and other embodiments are intended to be included within the scope of the present disclosure.

[0212] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure following, in general, the principles of the present disclosure and including such departures from the present disclosure that come within known or customary practice in the art to which the present disclosure pertains. The specification and examples are to be regarded as illustrative only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0213] It is to be understood that the present disclosure is not limited to the precise construction described and as shown in the accompanying drawings, and that changes and modifications can be effected therein by those skilled in the art without departing from the scope of the disclosure. The scope of the present disclosure is limited only by the claims appended hereto.

Claims

1. A video encoding method, characterized in that, The video encoding method includes: Perform an integer pixel search for at least one reference frame in the current block to obtain the rate-distortion cost of each reference frame in the at least one reference frame; A pixel-wise search is performed on each of the at least one reference frame, wherein, for each of the at least one reference frame, if the rate-distortion cost of the reference frame is greater than the rate-distortion cost threshold, the pixel-wise search for that reference frame is skipped.

2. The video encoding method according to claim 1, characterized in that, The rate-distortion cost threshold is determined based on a predetermined multiple of the current optimal rate-distortion cost.

3. The video encoding method according to claim 2, characterized in that, The current optimal rate-distortion cost is the optimal rate-distortion cost among the rate-distortion costs of the reference frames in which the pixel-by-pixel search has been performed.

4. The video encoding method according to claim 2, characterized in that, The predetermined multiple is determined based on the coding level to which the current reference frame belongs and / or the type of the current reference frame.

5. The video encoding method according to claim 4, characterized in that, The predetermined multiple is positively correlated with the level height of the coding layer to which the video frame containing the current block belongs.

6. The video encoding method according to claim 2, characterized in that, The predetermined multiple is determined based on the motion vector derived from the integer pixel search.

7. The video encoding method according to claim 6, characterized in that, The predetermined multiple is determined based on the motion vector corresponding to the optimal rate-distortion cost in the integer pixel search.

8. The video encoding method according to any one of claims 1 to 7, characterized in that, Performing a pixel-by-pixel search on each of the at least one reference frame includes: In the integer pixel search, a sub-pixel search is performed on each of the at least one reference frame in ascending order of rate-distortion cost.

9. The video encoding method according to claim 8, characterized in that, The step of performing a pixel-by-pixel search on each of the at least one reference frame further includes: For each of the at least one reference frame, if the rate-distortion cost of the reference frame is greater than the rate-distortion cost threshold, the pixel-by-pixel search of the reference frame and subsequent reference frames is skipped.

10. A video encoding device, characterized in that, The video encoding device includes: The integer pixel search unit is configured to perform an integer pixel search for at least one reference frame in the current block to obtain the rate-distortion cost of each of the at least one reference frame. The pixel-by-pixel search unit is configured to perform a pixel-by-pixel search on each of the at least one reference frame, wherein, for each of the at least one reference frame, the pixel-by-pixel search for that reference frame is skipped if the rate-distortion cost of that reference frame is greater than the rate-distortion cost threshold.

11. An electronic device, characterized in that, The electronic device includes: Processor; and Memory used to store processor-executable instructions. Wherein, the processor-executable instructions, when executed by the processor, cause the processor to perform the video encoding method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the video encoding method according to any one of claims 1 to 9.

13. A computer program product having instructions for storing a bit stream, wherein, The bit stream includes: Video data generated by the video encoding method as described in any one of claims 1 to 9.

14. A method for storing a bit stream, characterized in that, The method includes: A bitstream is generated according to the video encoding method as described in any one of claims 1 to 9; and Store the bit stream.

15. A method for transmitting a bit stream, characterized in that, The method includes: A bitstream is generated according to the video encoding method as described in any one of claims 1 to 9; and Send the bit stream.

Images