Motion estimation module for an encoder and encoder

Abstract

The application relates to a motion estimation module for an encoder and the encoder. The motion estimation module comprises: a first intra prediction unit for performing coarse intra prediction on a CTU; a second intra prediction unit for performing accurate intra prediction; a first inter prediction unit for performing coarse inter prediction on the CTU; an integer pixel prediction unit for performing integer pixel prediction based on a coarse inter CU partition result to obtain an integer pixel prediction result; a fractional pixel prediction unit for performing fractional pixel prediction based on the integer pixel prediction result to obtain a fractional pixel prediction result; a second inter prediction unit for performing accurate inter prediction; and a mode selection unit for obtaining a target prediction mode based on an accurate intra CU partition result and an accurate inter CU partition result. At least two of the units are executed in parallel. The method can improve the encoding speed without changing the encoding quality.

Description

Technical Field

[0001] This application relates to the field of image coding technology, and in particular to a motion estimation module for an encoder and an encoder. Background Technology

[0002] With the continuous development of video technology, video resolution is evolving towards 4K, 8K, and even 16K, while frame rates are also evolving towards 120fps. The ever-increasing resolution and frame rate place increasingly higher demands on the processing speed of hardware encoders, making encoding speed one of the most significant bottlenecks affecting the performance of hardware video encoders.

[0003] In traditional technology, many modules of hardware video encoders are executed serially, especially motion estimation. Because motion estimation has many data dependencies, the entire motion estimation process is almost entirely executed serially. Furthermore, due to the need to select prediction modes, a large number of logical operations are involved, which makes motion estimation the most complex and time-consuming part of the video encoder. Motion estimation usually accounts for 60% to 90% of the time when encoding a frame of an image.

[0004] The aforementioned issues result in slow encoding speeds for existing hardware video encoders. Improving encoding speed requires disabling some encoding tools, which, while increasing speed, leads to poor final image quality, making it difficult to balance encoding quality and speed simultaneously. Summary of the Invention

[0005] Therefore, it is necessary to provide a motion estimation module and encoder for the encoder that can improve the encoding speed without changing the encoded image quality, in order to address the above-mentioned technical problems.

[0006] In a first aspect, this application provides a motion estimation module for an encoder, the motion estimation module comprising:

[0007] The first intra-frame prediction unit is used to perform intra-frame prediction of CTU to obtain a coarse intra-frame CU partitioning result;

[0008] The second intra-frame prediction unit is used to obtain the precise intra-frame CU partitioning result based on the coarse intra-frame CU partitioning result.

[0009] The first inter-frame prediction unit is used to perform inter-frame prediction on the CTU to obtain a coarse inter-frame CU partitioning result;

[0010] An integer pixel prediction unit is used to perform integer pixel prediction based on the coarse inter-frame CU division result to obtain an integer pixel prediction result.

[0011] A fractional pixel prediction unit is used to perform fractional pixel prediction based on the integer pixel prediction result to obtain a fractional pixel prediction result.

[0012] The second inter-frame prediction unit is used to perform pattern prediction based on the fractional pixel prediction results to obtain accurate inter-frame CU partitioning results.

[0013] The mode selection unit is used to obtain the target prediction mode based on the precise intra-frame CU partitioning result and the precise inter-frame CU partitioning result.

[0014] At least two of the first intra-frame prediction unit, the second intra-frame prediction unit, the first inter-frame prediction unit, the integer pixel prediction unit, the fractional pixel prediction unit, and the second inter-frame prediction unit are executed in parallel.

[0015] In one optional embodiment, the first intra-frame prediction unit and the first inter-frame prediction unit process CTU[i] in parallel, where i is an integer greater than or equal to 0.

[0016] In one optional embodiment, the integer pixel prediction unit processes CTU[i] in parallel with the first inter-frame prediction unit processes CTU[i+1], and the first inter-frame prediction unit begins to process CTU[i+2] after the integer pixel prediction unit has completed processing each CU in CTU[i]. i is an integer greater than or equal to 0.

[0017] In one optional embodiment, the integer pixel prediction unit, the fractional pixel prediction unit, and the second inter-frame prediction unit are parallel execution units in units of CU;

[0018] The fractional pixel prediction unit processes CU[j] in parallel with the integer pixel prediction unit processes CU[j+1], and the integer pixel prediction unit begins processing CU[j+2] after the fractional pixel prediction unit has finished processing CU[j].

[0019] The processing of CU[j] by the second inter-frame prediction unit is parallel to the processing of CU[j+1] by the fractional pixel prediction unit. After the second inter-frame prediction unit finishes processing CU[j], the fractional pixel prediction unit starts processing CU[j+2], where j is an integer greater than or equal to 0.

[0020] In one optional embodiment, the second inter-frame prediction unit and the second intra-frame prediction unit process CU[j] in parallel, where j is an integer greater than or equal to 0.

[0021] In one optional embodiment, the motion estimation module further includes at least one of the following storage units:

[0022] The first storage unit is used to store the coarse intra-frame CU partitioning result obtained by the first intra-frame prediction unit, and to allow the second intra-frame prediction unit to read the coarse intra-frame CU partitioning result.

[0023] The second storage unit is used to store the coarse inter-frame CU partitioning result obtained by the first inter-frame prediction unit, and to allow the integer pixel prediction unit to read the coarse inter-frame CU partitioning result.

[0024] In one optional embodiment, the first inter-frame prediction unit is further configured to determine the search center point of CTU[i+2], and with the search center point, determine the search range of CTU[i+2] and the reference pixels within the search range of CTU[i+2], wherein the search center points of CTU[0] and CTU[1] are preset, and the search range of CTU[0] and CTU[1] is obtained by searching based on the preset search center points, where i is an integer greater than or equal to 0; wherein the search center point of CTU[i+2] is obtained based on available adjacent motion vectors, or the search center point of CTU[i+2] is obtained based on the motion vectors of co-position blocks in adjacent frames; or the search center point of CTU[i+2] is obtained based on the globally optimal motion vector determined for each CTU during image preprocessing.

[0025] In one optional embodiment, the first inter-frame prediction unit, the integer pixel prediction unit, the fractional pixel prediction unit, and the second inter-frame prediction unit are all used to determine the motion vector of the current CU based on the motion vectors of neighboring CUs;

[0026] Wherein, the motion vector of the adjacent CU is determined based on the integer pixel prediction result of the adjacent CU; or the motion vector of the adjacent CU is determined based on the fractional pixel prediction result of the adjacent CU; or the motion vector of the adjacent CU is determined based on the motion vector of the co-position block of the adjacent frame; or the motion vector of the adjacent CU is the initial motion vector determined for each CU when preprocessing the image.

[0027] Secondly, this application also provides an encoder, which includes the motion estimation module for the encoder described in any of the above embodiments.

[0028] In one alternative embodiment, the encoder further includes:

[0029] The motion compensation and transformation quantization module is used to determine the reconstructed pixels and coding coefficients based on the reference pixels, the original pixels, and the target prediction mode output by the motion estimation module.

[0030] An entropy coding module is used to generate a coded bitstream based on the target prediction pattern output by the motion estimation module and the coding coefficients, and the coded bitstream is stored in memory;

[0031] A loop filtering module is used to perform loop filtering based on the reconstructed pixels to obtain filtered pixels, and store the filtered pixels as reference pixels in the memory;

[0032] At least two of the motion estimation module, the motion compensation and transformation quantization module, the entropy coding module, and the loop filtering module are executed in parallel.

[0033] In one optional embodiment, the processing of CU[j] by the motion compensation and transform quantization module and the processing of CU[j+1] by the second inter-frame prediction unit in the motion estimation module are performed in parallel; or the processing of CU[j] by the motion compensation and transform quantization module and the processing of CU[j+1] by the first intra-frame prediction unit in the motion estimation module are performed in parallel, where j is an integer greater than or equal to 0.

[0034] In one optional embodiment, the entropy encoding module's processing of CU[j] and the motion compensation and transform quantization module's processing of CU[j+1] are parallel; or the loop filtering module's processing of CU[j] and the motion compensation and transform quantization module's processing of CU[j+1] are parallel, where j is an integer greater than or equal to 0.

[0035] In one optional embodiment, the entropy coding module's processing of CU[j] and the loop filtering module's processing of CU[j] are performed in parallel, where j is an integer greater than or equal to 0.

[0036] In one optional embodiment, the encoder further includes at least one of the following storage modules:

[0037] The first storage module is used to store the raw pixels of the CTU pre-read from memory and to be used by the motion estimation module and the motion compensation and transformation quantization module.

[0038] The second storage module is used to pre-read a number of reference pixels from the memory and provide them for use by the motion estimation module and the motion compensation and transformation quantization module.

[0039] The third storage module is used for data caching between the motion estimation module and the motion compensation and transformation quantization module, and is used to store the target prediction mode output by the motion estimation module.

[0040] The fourth storage module is used for data caching between the motion estimation module, the motion compensation and transformation quantization module, and the entropy coding module, and is used to store the target prediction mode output by the motion estimation module and the coding coefficients output by the motion compensation and transformation quantization module.

[0041] The fifth storage module is used as a data buffer between the motion compensation and transform quantization module and the loop filtering module, and is used to store the reconstructed pixels output by the motion compensation and transform quantization module.

[0042] The motion estimation module and encoder described above are for an encoder. The motion estimation module includes a first intra-frame prediction unit for performing intra-frame prediction on the CTU to obtain a coarse intra-frame CU partitioning result; a second intra-frame prediction unit for obtaining a precise intra-frame CU partitioning result based on the coarse intra-frame CU partitioning result; a first inter-frame prediction unit for performing inter-frame prediction on the CTU to obtain a coarse inter-frame CU partitioning result; an integer pixel prediction unit for performing integer pixel prediction based on the coarse inter-frame CU partitioning result to obtain an integer pixel prediction result; and a fractional pixel prediction unit for performing fractional pixel prediction based on the integer pixel prediction result. The system obtains fractional pixel prediction results; a second inter-frame prediction unit is used to perform mode prediction based on the fractional pixel prediction results to obtain accurate inter-frame CU partitioning results; a mode selection unit is used to obtain a target prediction mode based on the accurate intra-frame CU partitioning results and the accurate inter-frame CU partitioning results; wherein at least two of the first intra-frame prediction unit, the second intra-frame prediction unit, the first inter-frame prediction unit, the integer pixel prediction unit, the fractional pixel prediction unit, the second inter-frame prediction unit, and the mode selection unit are executed in parallel, so that parallel operation of these units can greatly improve the hardware encoding speed. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram of a CTU partitioning method in one embodiment;

[0045] Figure 2This is a schematic diagram of the PU partitioning method in one embodiment;

[0046] Figure 3 This is a schematic diagram of the motion estimation module in one embodiment;

[0047] Figure 4 This is a schematic diagram of the pipeline execution of each unit within the motion estimation module in one embodiment;

[0048] Figure 5 A flowchart of the search range determination steps in one embodiment;

[0049] Figure 6 This is a schematic diagram illustrating the calculation of the unavailable motion vector for the search center point of CTU[i+2] in one embodiment;

[0050] Figure 7 This is a schematic diagram of the MV of the current CU's adjacent blocks in one embodiment;

[0051] Figure 8 This is a schematic diagram of the MV of the current CU's adjacent blocks in another embodiment;

[0052] Figure 9 This is a schematic diagram of an encoder in one embodiment;

[0053] Figure 10 This is a schematic diagram of the pipeline execution of each module and unit within the encoder in one embodiment. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] The motion estimation module of an encoder typically needs to perform the following tasks:

[0056] First, determine how the CTU is divided into CUs. According to the HEVC standard, the maximum size of a CTU is 64x64 pixels, and the size of a CU is 4Nx4N (N=2, 4, 8, 16). Figure 1 As shown, Figure 1 This is a schematic diagram of a CTU partitioning method in one embodiment, in which a 64x64 CTU is divided into several 32x32, 16x16, and 8x8 CUs.

[0057] Second, determine how the CU is divided into PUs. For example... Figure 2 As shown, Figure 2 This is a schematic diagram of the PU partitioning method in one embodiment, which shows all possible PU partitioning methods for a 4Nx4N CU.

[0058] Third, find the optimal prediction mode for each PU. Intra-frame prediction modes include various angle prediction modes, and inter-frame prediction modes include normal mode, merge mode, and skip mode.

[0059] Conventional video encoders sequentially traverse all CU partitions, PU partitions, and various prediction modes, calculating the cost of each partition and mode to obtain the optimal partition and prediction mode. While this approach yields the most accurate partitions and prediction modes, ensuring optimal encoded image quality, it is disadvantageous for hardware implementation and extremely slow in encoding speed.

[0060] In this application, the motion estimation module is divided into reasonable modules, and a certain method is used to remove the data dependency problem between modules in the motion estimation process, so that the internal modules of motion estimation can run in parallel pipeline, which greatly improves the running speed of motion estimation.

[0061] Combination Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of a motion estimation module in one embodiment. The motion estimation module of this application includes a first intra-frame prediction unit, a second intra-frame prediction unit, a first inter-frame prediction unit, an integer pixel prediction unit, a fractional pixel prediction unit, a second inter-frame prediction unit, and a mode selection unit.

[0062] The system comprises the following components: a first intra-frame prediction unit for intra-frame prediction of the CTU to obtain a coarse intra-frame CU partitioning result; a second intra-frame prediction unit for obtaining a precise intra-frame CU partitioning result based on the coarse intra-frame CU partitioning result; a first inter-frame prediction unit for inter-frame prediction of the CTU to obtain a coarse inter-frame CU partitioning result; an integer pixel prediction unit for performing integer pixel prediction based on the coarse inter-frame CU partitioning result to obtain an integer pixel prediction result; a fractional pixel prediction unit for performing fractional pixel prediction based on the integer pixel prediction result to obtain a fractional pixel prediction result; a second inter-frame prediction unit for performing mode prediction based on the fractional pixel prediction result to obtain a precise inter-frame CU partitioning result; and a mode selection unit for obtaining a target prediction mode based on the precise intra-frame CU partitioning result and the precise inter-frame CU partitioning result. At least two of the following components are executed in parallel: the first intra-frame prediction unit, the second intra-frame prediction unit, the first inter-frame prediction unit, the integer pixel prediction unit, the fractional pixel prediction unit, and the second inter-frame prediction unit.

[0063] For easier understanding, please refer to Table 1:

[0064] Table 1. Units of the Motion Estimation Module

[0065]

[0066] The first intra-prediction unit (CTU_INP) performs coarse intra-prediction on the entire CTU and determines how to divide the current CTU into CUs. The divided CUs are all in intra-prediction mode, and the prediction result is imprecise. The second intra-prediction unit (CU_INP) performs further intra-prediction based on this imprecise intra-prediction result.

[0067] The first inter-frame prediction unit (CTU_IME) performs coarse inter-frame prediction for the entire CTU and determines how to divide the current CTU into CUs. The divided CUs are all in inter-frame prediction mode, and the prediction results are imprecise. The integer pixel prediction unit (CU_IME) then performs further inter-frame prediction based on this imprecise inter-frame prediction mode.

[0068] The integer pixel prediction unit CU_IME is used to read a CU prediction result, perform further integer pixel predictions based on the result, and pass the prediction result to the fractional pixel prediction unit CU_FME.

[0069] The fractional pixel prediction unit CU_FME receives the prediction result from the integer pixel prediction unit CU_IME, performs further fractional pixel prediction based on the result, and passes the prediction result to the second inter-frame prediction unit CU_MERGE.

[0070] The second inter-frame prediction unit CU_MERGE receives the prediction result from the fractional pixel prediction unit CU_FME, performs Merge / Skip mode prediction based on the result, and passes the prediction result to the mode selection module.

[0071] The second intra-frame prediction unit CU_INP is used to read a CU prediction result, perform further intra-frame prediction based on the result, and pass the prediction result to the mode selection module.

[0072] Assuming that CTU[0] is divided into n CUs after processing by the CTU_IME module, denoted as CU[0]~CU[n]; and that CTU[1] is divided into m CUs after processing by the CTU_IME module, denoted as CU[0]~CU[m], then the pipeline execution diagram of the above units can be found in [reference]. Figure 4 As shown.

[0073] In one of the alternative embodiments, the first intra-frame prediction unit and the first inter-frame prediction unit process CTU[i] in parallel, where i is an integer greater than or equal to 0.

[0074] Among them, the combination Figure 4The intra-frame prediction unit and the inter-frame prediction unit are independent of each other and are executed in parallel in the same pipeline.

[0075] In one of the optional embodiments, the integer pixel prediction unit processes CTU[i] in parallel with the first inter-frame prediction unit processes CTU[i+1], and the first inter-frame prediction unit starts processing CTU[i+2] after the integer pixel prediction unit has completed processing each CU in CTU[i]. i is an integer greater than or equal to 0.

[0076] Among them, combined Figure 4 The integer pixel prediction unit and the first inter-frame prediction unit are executed in parallel pipelined form on a CTU basis. Specifically:

[0077] The integer pixel prediction unit processes CTU[i] in parallel with the first inter-frame prediction unit processes CTU[i+1].

[0078] Once the integer pixel prediction unit has finished processing each CU in CTU[i], the first inter-frame prediction unit begins processing CTU[i+2].

[0079] In one optional embodiment, the integer pixel prediction unit, the fractional pixel prediction unit, and the second inter-frame prediction unit are parallel execution units based on CUs; ​​the fractional pixel prediction unit processes CU[j] in parallel with the integer pixel prediction unit processes CU[j+1], and the integer pixel prediction unit begins processing CU[j+2] after the fractional pixel prediction unit has finished processing CU[j]; the second inter-frame prediction unit processes CU[j] in parallel with the fractional pixel prediction unit processes CU[j+1], and the fractional pixel prediction unit begins processing CU[j+2] after the second inter-frame prediction unit has finished processing CU[j], where j is an integer greater than or equal to 0.

[0080] Among them, combined Figure 4 As shown, the integer pixel prediction unit, fractional pixel prediction unit, and second inter-frame prediction unit are executed in parallel pipelined form on a unit basis. Specifically:

[0081] CU_FME[j] and CU_IME[j+1] are executed in parallel, that is, the fractional pixel prediction unit's processing of CU[j] and the integer pixel prediction unit's processing of CU[j+1] are performed in parallel.

[0082] After CU_FME[j] is executed, CU_IME[j+2] is executed. That is, after the fractional pixel prediction unit has finished processing CU[j], the integer pixel prediction unit begins to process CU[j+2].

[0083] CU_MERGE[j] and CU_FME[j+1] are executed in parallel, that is, the processing of CU[j] by the second inter-frame prediction unit is parallel to the processing of CU[j+1] by the fractional pixel prediction unit.

[0084] After CU_MERGE[j] is executed, CU_FME[j+2] is executed. That is, after the second inter-frame prediction unit has finished processing CU[j], the fractional pixel prediction unit begins to process CU[j+2].

[0085] In one of the alternative embodiments, the second inter-frame prediction unit and the second intra-frame prediction unit process CU[j] in parallel, where j is an integer greater than or equal to 0.

[0086] The second intra-frame prediction unit CU_INP module requires less time and is executed in parallel with the second inter-frame prediction unit CU_MERGE module. After the second inter-frame prediction unit CU_MERGE[k] is completed, the accurate inter-frame prediction result of CU[k] and the accurate intra-frame prediction result of CU[k] can be obtained simultaneously.

[0087] In one optional embodiment, the motion estimation module further includes at least one of the following storage units: a first storage unit for storing the coarse intra-frame CU partitioning result obtained by the first intra-frame prediction unit and for the second intra-frame prediction unit to read the coarse intra-frame CU partitioning result; and a second storage unit for storing the coarse inter-frame CU partitioning result obtained by the first inter-frame prediction unit and for the integer pixel prediction unit to read the coarse inter-frame CU partitioning result.

[0088] Among them, combined Figure 3 As shown, the first storage unit Sram5 is used to store the coarse intra-CU partitioning result obtained by the first intra-frame prediction unit, so that the subsequent second intra-frame prediction unit can read the coarse intra-frame CU partitioning result from it.

[0089] The second storage unit Sram6 is used to store the coarse inter-frame CU partitioning result obtained by the first inter-frame prediction unit, so that the subsequent integer pixel prediction unit can read the coarse inter-frame CU partitioning result from the second storage unit Sram6.

[0090] Introducing the first storage unit SRAM5 and the second storage unit SRAM6 can prevent the execution progress from being blocked by each other, and buffer the data. This can avoid the problem of reduced encoding speed due to the different processing speeds between modules.

[0091] In one optional embodiment, the first inter-frame prediction unit is further configured to determine the search center point of CTU[i+2], and to determine the search range of CTU[i+2] and the reference pixels within the search range of CTU[i+2] based on the search center point, wherein the search center points of CTU[0] and CTU[1] are preset, and the search range of CTU[0] and CTU[1] is obtained by searching based on the preset search center points, where i is an integer greater than or equal to 0; wherein the search center point of CTU[i+2] is obtained based on the available adjacent motion vectors, or the search center point of CTU[i+2] is obtained based on the motion vectors of the co-position blocks of adjacent frames; or the search center point of CTU[i+2] is obtained based on the globally optimal motion vector determined for each CTU during image preprocessing.

[0092] Within a single CTU, the motion vectors (MVs) of each PU are different, corresponding to different reference pixels. If data is read from memory only when reference pixels are needed, the encoding speed will be significantly reduced due to memory latency. In this embodiment, the search range is determined in advance for each CTU, and the reference pixels within the search range are read from memory in advance.

[0093] Among them, combined Figure 5 As shown, Figure 5 A flowchart of the search range determination step in one embodiment.

[0094] Determining the search range requires two steps. The first step is to determine a search center point for each CTU. The second step is to expand a rectangular area outward from the search center point. The size of the rectangular area can be determined by the user or software configuration. However, in order to achieve better search accuracy and encoded image quality, the search center point must be determined by the algorithm. Generally, it is determined by referring to the MV of the adjacent blocks around the current CTU.

[0095] Combination Figure 5 As shown, first set k=0 and initialize it. For example, determine the search center point of CTU[0] and CTU[1] as (0,0), and the search range of CTU[0] and CTU[1] is obtained by searching based on the preset search center point (0,0). Then read the reference pixels in the search range. Then the first inter-frame prediction unit processes CTU[k] and determines the search center point of CTU[k+2]. Based on the search center point, determine the search range to read the reference pixels in the search range. Then add one to the value of k and the first inter-frame prediction unit continues to process CTU[k] until the processing is completed.

[0096] The first inter-frame prediction unit calculates the search range two CTUs in advance. That is, after processing CTU[i], the first inter-frame prediction unit will determine the search range for CTU[i+2] and read the reference pixels within the search range from the inside and store them in the corresponding storage module, specifically the second storage module mentioned below.

[0097] It should be noted that, due to parallelism and the pre-calculation of the search range, the MV of neighboring blocks around the CTU is not always available. For example... Figure 6 As shown, when calculating the search center point of CTU[i+2], its left MV (shaded area in the figure) is unusable because CTU[i+1] has not yet started encoding, and there is no encoding information provided to CTU[i+2] for reference. Therefore, in this embodiment, the reference pixel of CTU[i+1] can be determined by at least one of the following methods:

[0098] Discarding unavailable neighboring motion vectors (MVs) and only referencing available neighboring MVs, such as the adjacent MV above or the temporal adjacent MV, is the simplest method, but it has a certain impact on the accuracy of the search range.

[0099] A video preprocessing module is added to analyze video images in advance and find a globally optimal MV for each CTU. This method is more complex but has higher accuracy.

[0100] In one optional embodiment, the first inter-frame prediction unit, the integer pixel prediction unit, the fractional pixel prediction unit, and the second inter-frame prediction unit are all used to determine the motion vector of the current CU based on the motion vectors of neighboring CUs; ​​wherein, the motion vector of the neighboring CU is determined based on the integer pixel prediction result of the neighboring CU; or the motion vector of the neighboring CU is determined based on the fractional pixel prediction result of the neighboring CU; or the motion vector of the neighboring CU is determined based on the motion vector of the co-position block of the adjacent frame; or the motion vector of the neighboring CU is the initial motion vector determined for each CU when preprocessing the image.

[0101] Inter-frame prediction requires constructing a reference MV list, or MVP list, for each PU within the CU. Whether it's the first inter-frame prediction module CTU_IME, the integer pixel prediction module CU_IME, the fractional pixel prediction module CU_FME, or the second inter-frame prediction module CU_MERGE mentioned in this application, all require constructing an MVP list, i.e., a list of motion vectors.

[0102] When constructing the MVP list, it is necessary to refer to the MV of the current CU's adjacent blocks. Specifically, this can be combined with... Figure 7 As shown, Figure 7This is a schematic diagram of the MV of the current CU's neighboring blocks in one embodiment. The accurate MV of the CU can only be obtained after the second inter-frame prediction module CU_MERGE finishes execution. However, due to parallelism, the MV of the CU's neighboring blocks is not always available, or although available, it is not the final accurate MV. For details, please refer to [link to documentation]. Figure 8 As shown, Figure 8 This is a schematic diagram of the MV of the current CU's adjacent blocks in another embodiment. In this embodiment, when the first inter-frame prediction module processes CU[k+2], the fractional pixel prediction module processes CU[k+1] in parallel, and the second inter-frame prediction module processes CU[k+2] in parallel. At this time, the A1 and B1 positions around CU[k+2] cannot obtain accurate MV. To solve the above problem, the following methods are included, but are not limited to:

[0103] Use available inaccurate prediction values ​​(MVs) to replace unavailable accurate prediction values ​​(MVs). For example, the MV at position A1 of CU[k+2] is replaced by the prediction result of the fractional pixel prediction unit CU_IME at position A1; the MV at position B1 is replaced by the prediction result of the integer pixel prediction unit CU_IME or the fractional pixel prediction unit CU_FME at position B1.

[0104] Use the temporal MV to replace the unavailable precise MV.

[0105] Add a video preprocessing module to analyze video images in advance and find a preliminary MV for each CU.

[0106] In the above embodiments, a reasonable approach is used to remove data dependencies between upper and lower level modules, so that the encoding quality is not reduced due to the increase in encoding speed.

[0107] Combination Figure 9 As shown, Figure 9 This is a schematic diagram of an encoder in one embodiment, which includes a motion estimation module. The motion estimation module further includes several units, including a first intra-frame prediction unit, a second intra-frame prediction unit, a first inter-frame prediction unit, an integer pixel prediction unit, a fractional pixel prediction unit, a second inter-frame prediction unit, and a mode selection unit, wherein at least two of the first intra-frame prediction unit, the second intra-frame prediction unit, the first inter-frame prediction unit, the integer pixel prediction unit, the fractional pixel prediction unit, the second inter-frame prediction unit, and the mode selection unit are executed in parallel.

[0108] In one optional embodiment, the encoder further includes: a motion compensation and transform quantization module, used to determine reconstructed pixels and coding coefficients based on reference pixels, original pixels, and a target prediction mode output by the motion estimation module; an entropy coding module, used to generate a coded bitstream based on the target prediction mode output by the motion estimation module and the coding coefficients, the coded bitstream being stored in memory; and a loop filtering module, used to perform loop filtering on the reconstructed pixels to obtain filtered pixels, and storing the filtered pixels as reference pixels in memory; wherein at least two of the motion estimation module, motion compensation and transform quantization module, entropy coding module, and loop filtering module are executed in parallel.

[0109] The motion estimation module, denoted as ME (Motion Estimation), is primarily used for intra-frame and inter-frame prediction to determine the optimal CU partitioning, PU partitioning, MV prediction, and other information. The motion estimation module is the most complex and time-consuming module in the entire encoding process; its specific limitations can be found above.

[0110] The motion compensation and transformation quantization module, denoted as MC (Motion Compensation) module, mainly receives the optimal prediction information from the motion estimation module and obtains information such as predicted pixels, residuals, reconstructed pixels, and entropy coding coefficients based on the optimal prediction information.

[0111] The loop filtering module, denoted as the ILF (In-Loop Filter) module, mainly receives the reconstructed pixels sent by the motion compensation and transform quantization module, filters the reconstructed pixels to obtain filtered pixels, and then writes the filtered pixels into memory as reference pixels for subsequent frames.

[0112] The entropy coding module, denoted as the EC (Entropy Coding) module, mainly receives the coding coefficients sent by the motion compensation and transform quantization module, as well as the optimal prediction information sent by the motion estimation module. Then, it performs entropy coding on the optimal prediction information and coding coefficients to obtain the final coded bitstream and writes it into memory.

[0113] In order to improve processing efficiency, combined with Figure 10 As shown, at least two of the motion estimation module, motion compensation and transform quantization module, entropy coding module, and loop filtering module are executed in parallel.

[0114] In one optional embodiment, the processing of CU[j] by the motion compensation and transform quantization module and the processing of CU[j+1] by the second inter-frame prediction unit in the motion estimation module are performed in parallel; or the processing of CU[j] by the motion compensation and transform quantization module and the processing of CU[j+1] by the first intra-frame prediction unit in the motion estimation module are performed in parallel, where j is an integer greater than or equal to 0.

[0115] Combination Figure 10 The processing of CU[j] by the motion compensation and transform quantization module can be performed in parallel with the processing of CU[j+1] by the second inter-frame prediction unit in the motion estimation module, or the processing of CU[j+1] by the first intra-frame prediction unit in the motion estimation module.

[0116] In one optional embodiment, the entropy coding module processes CU[j] and the motion compensation and transform quantization module processes CU[j+1] in parallel; or the loop filtering module processes CU[j] and the motion compensation and transform quantization module processes CU[j+1] in parallel, where j is an integer greater than or equal to 0.

[0117] Combination Figure 10 The processing of CU[j] by the entropy coding module or the processing of CU[j] by the loop filtering module is parallel to the processing of CU[j+1] by the motion compensation and transformation quantization module.

[0118] In one alternative embodiment, the entropy coding module processes CU[j] and the loop filtering module processes CU[j] in parallel, where j is an integer greater than or equal to 0.

[0119] Combination Figure 10 The entropy coding module processes CU[j] and the loop filtering module processes CU[j] in parallel.

[0120] In the above embodiments, after adding the pipeline structure mentioned in this application to the HEVC hardware encoder, the encoding time is reduced by 60% or the encoding speed is increased by 150% when all encoding tools are enabled, while ensuring both encoded image quality and encoding speed. It should also be noted that this application only uses the HEVC encoder as an example to illustrate the processing method; other encoding standards can also adopt similar pipeline architectures.

[0121] In one optional embodiment, the encoder further includes at least one of the following storage modules:

[0122] The first storage module is used to store the raw pixels pre-read from the CTU in memory and for use by the motion estimation module and the motion compensation and transform quantization module.

[0123] The second storage module is used to pre-read several reference pixels from memory and provide them for use by the motion estimation module and the motion compensation and transformation quantization module.

[0124] The third storage module is used as a data cache between the motion estimation module and the motion compensation and transformation quantization module, and is used to store the target prediction pattern output by the motion estimation module.

[0125] The fourth storage module is used for data caching between the motion estimation module, the motion compensation and transform quantization module, and the entropy coding module. It stores the target prediction pattern output by the motion estimation module and the coding coefficients output by the motion compensation and transform quantization module.

[0126] The fifth storage module is used for data buffering between the motion compensation and transform quantization module and the loop filtering module, and is used to store the reconstructed pixels output by the motion compensation and transform quantization module.

[0127] In this application, the following continues to be combined Figure 10 Since the motion estimation module, motion compensation and transformation quantization module, entropy coding module and loop filtering module have certain differences in processing speed, in order to avoid these four modules blocking each other's execution progress, a storage module SRAM is added at the connection between any two of these four modules to buffer the data.

[0128] Specifically, five SRAMs need to be added, denoted as SRAM0 (first storage module), SRAM1 (second storage module), SRAM2 (third storage module), SRAM3 (fourth storage module), and SRAM4 (fifth storage module). The functions of these five SRAMs are as follows:

[0129] The first storage module Sram0 is used to store the raw pixels read back from memory. Several raw pixels of CTU can be read in advance and stored in the first storage module Sram0. This is done to avoid the encoding speed being affected by memory latency due to real-time reading of raw pixels.

[0130] The second storage module Sram1 is used to store reference pixels retrieved from memory. The motion estimation module pre-calculates the search range of several CTUs, thereby pre-reading the reference pixels within the search range from memory and storing them in the second storage module Sram1. This is also to avoid the encoding speed being affected by memory latency due to real-time reading of reference pixels.

[0131] The third storage module, SRAM2, is used as a data cache between the motion estimation module and the motion compensation and transformation quantization module. This prevents the progress of the motion compensation and transformation quantization module from being affected by the slow progress of the motion estimation module, or vice versa.

[0132] The fourth storage module, SRAM3, is used for data caching between the motion estimation module, the motion compensation and transform quantization module, and the entropy coding module. This prevents the progress of the entropy coding module from being affected by the slow progress of the motion estimation module, the motion compensation and transform quantization module, or vice versa.

[0133] The fifth storage module, SRAM4, is used for data caching between the motion compensation and transformation quantization module and the loop filter module. This prevents the progress of the loop filter module from being affected by the slow progress of the motion compensation and transformation quantization module, or vice versa.

[0134] The specific specifications of these five SRAMs need to be determined through prior experiments, and no specific restrictions are imposed here.

[0135] This application introduces a pipeline architecture for video encoding, which greatly improves the encoding speed without changing the encoded image quality.

[0136] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0137] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0138] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0139] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A motion estimation module for an encoder, characterized in that, The motion estimation module includes: The first intra-frame prediction unit is used to perform intra-frame prediction of CTU to obtain a coarse intra-frame CU partitioning result; The second intra-frame prediction unit is used to obtain the precise intra-frame CU partitioning result based on the coarse intra-frame CU partitioning result. The first inter-frame prediction unit is used to perform inter-frame prediction on the CTU to obtain a coarse inter-frame CU partitioning result; An integer pixel prediction unit is used to perform integer pixel prediction based on the coarse inter-frame CU division result to obtain an integer pixel prediction result. A fractional pixel prediction unit is used to perform fractional pixel prediction based on the integer pixel prediction result to obtain a fractional pixel prediction result. The second inter-frame prediction unit is used to perform pattern prediction based on the fractional pixel prediction results to obtain accurate inter-frame CU partitioning results. The mode selection unit is used to obtain the target prediction mode based on the precise intra-frame CU partitioning result and the precise inter-frame CU partitioning result. At least two of the first intra-frame prediction unit, the second intra-frame prediction unit, the first inter-frame prediction unit, the integer pixel prediction unit, the fractional pixel prediction unit, and the second inter-frame prediction unit are executed in parallel. The first intra-frame prediction unit and the first inter-frame prediction unit are in the same pipeline and process CTU[i] in parallel, where i is an integer greater than or equal to 0.

2. The motion estimation module for an encoder according to claim 1, characterized in that, The integer pixel prediction unit processes CTU[i] in parallel with the first inter-frame prediction unit processes CTU[i+1]. When the integer pixel prediction unit has completed processing each CU in CTU[i], the first inter-frame prediction unit starts processing CTU[i+2], where i is an integer greater than or equal to 0.

3. The motion estimation module for an encoder according to claim 1, characterized in that, The integer pixel prediction unit, the fractional pixel prediction unit, and the second inter-frame prediction unit are parallel execution units in units of CU; The fractional pixel prediction unit processes CU[j] in parallel with the integer pixel prediction unit processes CU[j+1], and the integer pixel prediction unit begins processing CU[j+2] after the fractional pixel prediction unit has finished processing CU[j]. The processing of CU[j] by the second inter-frame prediction unit is parallel to the processing of CU[j+1] by the fractional pixel prediction unit. After the second inter-frame prediction unit finishes processing CU[j], the fractional pixel prediction unit starts processing CU[j+2], where j is an integer greater than or equal to 0.

4. The motion estimation module for an encoder according to claim 1, characterized in that, The second inter-frame prediction unit and the second intra-frame prediction unit process CU[j] in parallel, where j is an integer greater than or equal to 0.

5. The motion estimation module for an encoder according to any one of claims 1 to 4, characterized in that, The motion estimation module also includes at least one of the following storage units: The first storage unit is used to store the coarse intra-frame CU partitioning result obtained by the first intra-frame prediction unit, and to allow the second intra-frame prediction unit to read the coarse intra-frame CU partitioning result. The second storage unit is used to store the coarse inter-frame CU partitioning result obtained by the first inter-frame prediction unit, and to allow the integer pixel prediction unit to read the coarse inter-frame CU partitioning result.

6. The motion estimation module for an encoder according to claim 1, characterized in that, The first inter-frame prediction unit is further configured to determine the search center point of CTU[i+2], and with the search center point, determine the search range of CTU[i+2] and the reference pixels within the search range of CTU[i+2]. The search center points of CTU[0] and CTU[1] are preset, and the search ranges of CTU[0] and CTU[1] are obtained by searching based on the preset search center points, where i is an integer greater than or equal to 0. The search center point of CTU[i+2] is obtained based on the available adjacent motion vectors, or the search center point of CTU[i+2] is obtained based on the motion vectors of the co-position blocks of adjacent frames; or the search center point of CTU[i+2] is obtained based on the globally optimal motion vector determined for each CTU during image preprocessing.

7. The motion estimation module for an encoder according to claim 1, characterized in that, The first inter-frame prediction unit, the integer pixel prediction unit, the fractional pixel prediction unit, and the second inter-frame prediction unit are all used to determine the motion vector of the current CU based on the motion vectors of adjacent CUs; Wherein, the motion vector of the adjacent CU is determined based on the integer pixel prediction result of the adjacent CU; or the motion vector of the adjacent CU is determined based on the fractional pixel prediction result of the adjacent CU; or the motion vector of the adjacent CU is determined based on the motion vector of the co-position block of the adjacent frame; or the motion vector of the adjacent CU is the initial motion vector determined for each CU when preprocessing the image.

8. An encoder, characterized in that, The encoder includes a motion estimation module for the encoder as described in any one of claims 1 to 7.

9. The encoder according to claim 8, characterized in that, The encoder also includes: The motion compensation and transformation quantization module is used to determine the reconstructed pixels and coding coefficients based on the reference pixels, the original pixels, and the target prediction mode output by the motion estimation module. An entropy coding module is used to generate a coded bitstream based on the target prediction pattern output by the motion estimation module and the coding coefficients, and the coded bitstream is stored in memory; A loop filtering module is used to perform loop filtering based on the reconstructed pixels to obtain filtered pixels, and store the filtered pixels as reference pixels in the memory; At least two of the motion estimation module, the motion compensation and transformation quantization module, the entropy coding module, and the loop filtering module are executed in parallel.

10. The encoder according to claim 9, characterized in that, The processing of CU[j] by the motion compensation and transform quantization module and the processing of CU[j+1] by the second inter-frame prediction unit in the motion estimation module are performed in parallel; or the processing of CU[j] by the motion compensation and transform quantization module and the processing of CU[j+1] by the first intra-frame prediction unit in the motion estimation module are performed in parallel, where j is an integer greater than or equal to 0.

11. The encoder according to claim 9, characterized in that, The entropy encoding module processes CU[j] and the motion compensation and transformation quantization module processes CU[j+1] in parallel; or the loop filtering module processes CU[j] and the motion compensation and transformation quantization module processes CU[j+1] in parallel, where j is an integer greater than or equal to 0.

12. The encoder according to claim 9, characterized in that, The entropy encoding module processes CU[j] and the loop filtering module processes CU[j] in parallel, where j is an integer greater than or equal to 0.

13. The encoder according to any one of claims 9 to 12, characterized in that, The encoder also includes at least one of the following storage modules: The first storage module is used to store the raw pixels of the CTU pre-read from memory and to be used by the motion estimation module and the motion compensation and transformation quantization module. The second storage module is used to pre-read a number of reference pixels from the memory and provide them for use by the motion estimation module and the motion compensation and transformation quantization module. The third storage module is used for data caching between the motion estimation module and the motion compensation and transformation quantization module, and is used to store the target prediction mode output by the motion estimation module. The fourth storage module is used for data caching between the motion estimation module, the motion compensation and transformation quantization module, and the entropy coding module, and is used to store the target prediction mode output by the motion estimation module and the coding coefficients output by the motion compensation and transformation quantization module. The fifth storage module is used as a data buffer between the motion compensation and transform quantization module and the loop filtering module, and is used to store the reconstructed pixels output by the motion compensation and transform quantization module.

Images