Motion estimation module for encoder and encoder

Abstract

The invention relates to a motion estimation module for an encoder and the encoder. The motion estimation module comprises a first intra-frame prediction unit and a second intra-frame prediction unit, wherein the first intra-frame prediction unit is used for performing rough intra-frame prediction on the CTU; the second intra-frame prediction unit is used for performing accurate intra-frame prediction; the first inter-frame prediction unit is used for performing rough inter-frame prediction on the CTU; the integer pixel prediction unit is used for performing integer pixel prediction based on the rough inter-frame CU division result to obtain an integer pixel prediction result; the fractional pixel prediction unit is used for performing fractional pixel prediction based on the integer pixel prediction result to obtain a fractional pixel prediction result; the second inter-frame prediction unit is used for performing accurate inter-frame prediction; the mode selection unit is used for obtaining a target prediction mode based on the accurate intra-frame CU division result and the accurate inter-frame CU division result; at least two of the units are executed in parallel. By adopting the method, the coding speed can be improved on the premise of not changing the coding image quality.

Description

Technical Field

[0001] The present application relates to the technical field of image coding, and in particular to a motion estimation module and an encoder for an encoder. Background Art

[0002] With the continuous development of video technology, video resolution is evolving towards 4K, 8K and even 16K, while the frame rate is also evolving towards 120fps. With the continuous improvement of resolution and frame rate, the requirements for the processing speed of hardware encoders are also increasing. The encoding speed has become one of the important bottlenecks affecting the performance of hardware video encoders.

[0003] In traditional technology, many modules of hardware video encoders are executed serially, especially motion estimation. Since there are many data dependencies in the motion estimation process, the entire motion estimation process is almost executed serially. In addition, since prediction mode selection involves a large number of logical operations, motion estimation becomes the most complex and time-consuming part of the video encoder. Motion estimation usually takes 60% to 90% of the time to encode a frame of image.

[0004] The above problems lead to the low encoding speed of existing hardware video encoders. To increase the encoding speed, some encoding tools can only be turned off. Although this can increase the encoding speed, the final encoding quality is poor due to the turning off of some encoding tools, making it difficult to balance the encoding quality and encoding speed at the same time. Summary of the invention

[0005] Based on this, it is necessary to provide a motion estimation module and encoder for an encoder that can improve the encoding speed without changing the encoding image quality in order to solve the above technical problems.

[0006] In a first aspect, the present application provides a motion estimation module for an encoder, the motion estimation module comprising:

[0007] A first intra prediction unit, used for performing intra prediction on the CTU to obtain a rough intra CU division result;

[0008] A second intra prediction unit, configured to obtain an accurate intra CU division result based on the coarse intra CU division result;

[0009] A first inter-frame prediction unit, configured to perform inter-frame prediction on the CTU to obtain a rough inter-frame CU division result;

[0010] An integer pixel prediction unit, configured to perform integer pixel prediction based on the coarse inter-frame CU division result to obtain an integer pixel prediction result;

[0011] A fractional pixel prediction unit, configured to perform fractional pixel prediction based on the integer pixel prediction result to obtain a fractional pixel prediction result;

[0012] A second inter-frame prediction unit, configured to perform mode prediction based on the fractional pixel prediction result to obtain an accurate inter-frame CU division result;

[0013] A mode selection unit, configured to obtain a target prediction mode based on the precise intra-frame CU division result and the precise inter-frame CU division result;

[0014] At least two of the first intra-frame prediction unit, the second intra-frame prediction unit, the first inter-frame prediction unit, the integer pixel prediction unit, the fractional pixel prediction unit and the second inter-frame prediction unit are executed in parallel.

[0015] In one of the optional embodiments, the first intra-frame prediction unit and the first inter-frame prediction unit process CTU[i] in parallel, where i is an integer greater than or equal to 0.

[0016] In one of the optional embodiments, the integer pixel prediction unit processes CTU[i] in parallel with the first inter-frame prediction unit processes CTU[i+1], and when the integer pixel prediction unit completes processing of each CU in CTU[i], the first inter-frame prediction unit starts processing CTU[i+2], where i is an integer greater than or equal to 0.

[0017] In one of the optional embodiments, the integer pixel prediction unit, the fractional pixel prediction unit and the second inter-frame prediction unit are parallel execution units in units of CU;

[0018] The fractional pixel prediction unit processes CU[j] in parallel with the integer pixel prediction unit processes CU[j+1], and when the fractional pixel prediction unit completes the processing of CU[j], the integer pixel prediction unit starts processing CU[j+2];

[0019] The processing of CU[j] by the second inter-frame prediction unit is in parallel with the processing of CU[j+1] by the fractional pixel prediction unit, and when the processing of CU[j] by the second inter-frame prediction unit is completed, the fractional pixel prediction unit starts to process CU[j+2], where j is an integer greater than or equal to 0.

[0020] In one of the optional embodiments, the second inter-frame prediction unit and the second intra-frame prediction unit process CU[j] in parallel, where j is an integer greater than or equal to 0.

[0021] In one of the optional embodiments, the motion estimation module further includes at least one of the following storage units:

[0022] A first storage unit, configured to store the coarse intra CU division result obtained by the first intra prediction unit, and to allow the second intra prediction unit to read the coarse intra CU division result;

[0023] The second storage unit is used to store the coarse inter-frame CU division result obtained by the first inter-frame prediction unit, and to allow the integer pixel prediction unit to read the coarse inter-frame CU division result.

[0024] In one of the optional embodiments, the first inter-frame prediction unit is further used to determine a search center point of CTU[i+2], and to determine a search range of CTU[i+2] and reference pixels within the search range of CTU[i+2] based on the search center point, wherein the search center points of CTU[0] and CTU[1] are pre-set, and the search ranges of CTU[0] and CTU[1] are obtained by searching based on the pre-set search center point, wherein i is an integer greater than or equal to 0; wherein the search center point of CTU[i+2] is obtained based on available adjacent motion vectors, or the search center point of CTU[i+2] is obtained based on motion vectors of co-located blocks of adjacent frames; or the search center point of CTU[i+2] is obtained based on a global optimal motion vector determined for each CTU when pre-processing the image.

[0025] In one of the optional embodiments, the first inter-frame prediction unit, the integer pixel prediction unit, the fractional pixel prediction unit and the second inter-frame prediction unit are all used to determine the motion vector of the current CU based on the motion vector of the neighboring CU;

[0026] The motion vector of the adjacent CU is determined based on the integer pixel prediction result of the adjacent CU; or the motion vector of the adjacent CU is determined based on the fractional pixel prediction result of the adjacent CU; or the motion vector of the adjacent CU is determined based on the motion vector of the co-located block of the adjacent frame; or the motion vector of the adjacent CU is the initial motion vector determined for each CU when preprocessing the image.

[0027] In a second aspect, the present application further provides an encoder, which includes the motion estimation module for the encoder described in any one of the above embodiments.

[0028] In one of the optional embodiments, the encoder further includes:

[0029] A motion compensation and transform quantization module, used to determine the reconstructed pixels and coding coefficients based on the reference pixels, the original pixels and the target prediction mode output by the motion estimation module;

[0030] An entropy coding module, used for generating a coding stream based on the target prediction mode output by the motion estimation module and the coding coefficients, wherein the coding stream is stored in a memory;

[0031] A loop filtering module, configured to perform loop filtering based on the reconstructed pixels to obtain filtered pixels, and store the filtered pixels as the reference pixels in the memory;

[0032] Wherein, at least two of the motion estimation module, the motion compensation and transform quantization module, the entropy coding module and the loop filtering module are executed in parallel.

[0033] In one of the optional embodiments, the processing of CU[j] by the motion compensation and transform quantization module and the processing of CU[j+1] by the second inter-frame prediction unit in the motion estimation module are parallel; or the processing of CU[j] by the motion compensation and transform quantization module and the processing of CU[j +1] by the first intra-frame prediction unit in the motion estimation module are parallel, and j is an integer greater than or equal to 0.

[0034] In one of the optional embodiments, the entropy coding module processes CU[j] and the motion compensation and transform quantization module processes CU[j +1] in parallel; or the loop filtering module processes CU[j] and the motion compensation and transform quantization module processes CU[j +1] in parallel, and j is an integer greater than or equal to 0.

[0035] In one of the optional embodiments, the processing of CU[j] by the entropy coding module and the processing of CU[j] by the loop filtering module are parallel, and j is an integer greater than or equal to 0.

[0036] In one of the optional embodiments, the encoder further includes at least one of the following storage modules:

[0037] A first storage module, used for storing original pixels of the CTU pre-read from the memory, and for use by the motion estimation module and the motion compensation and transform quantization module;

[0038] A second storage module, used for pre-reading a number of reference pixels from the memory and providing the reference pixels for use by the motion estimation module and the motion compensation and transform quantization module;

[0039] A third storage module, used for data buffering between the motion estimation module and the motion compensation and transform quantization module, and used for storing the target prediction mode output by the motion estimation module;

[0040] a fourth storage module, used for data caching between the motion estimation module, the motion compensation and transform quantization module and the entropy coding module, and used for storing the target prediction mode output by the motion estimation module and the coding coefficient output by the motion compensation and transform quantization module;

[0041] The fifth storage module is used for data caching between the motion compensation and transform quantization module and the loop filtering module, and is used for storing reconstructed pixels output by the motion compensation and transform quantization module.

[0042] The above-mentioned motion estimation module and encoder for encoder, wherein the motion estimation module includes a first intra-frame prediction unit, which is used to perform intra-frame prediction on CTU to obtain a rough intra-frame CU division result; a second intra-frame prediction unit, which is used to obtain an accurate intra-frame CU division result based on the rough intra-frame CU division result; a first inter-frame prediction unit, which is used to perform inter-frame prediction on the CTU to obtain a rough inter-frame CU division result; an integer pixel prediction unit, which is used to perform integer pixel prediction based on the rough inter-frame CU division result to obtain an integer pixel prediction result; a fractional pixel prediction unit, which is used to perform fractional pixel prediction based on the integer pixel prediction result, A fractional pixel prediction result is obtained; a second inter-frame prediction unit is used to perform mode prediction based on the fractional pixel prediction result to obtain an accurate inter-frame CU division result; a mode selection unit is used to obtain a target prediction mode based on the accurate intra-frame CU division result and the accurate inter-frame CU division result; wherein at least two of the first intra-frame prediction unit, the second intra-frame prediction unit, the first inter-frame prediction unit, the integer pixel prediction unit, the fractional pixel prediction unit, the second inter-frame prediction unit and the mode selection unit are executed in parallel, so that the units running in parallel can greatly improve the encoding speed of the hardware. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 is a schematic diagram of a CTU division method in an embodiment;

[0045] Figure 2is a schematic diagram of a PU division method in one embodiment;

[0046] Figure 3 is a schematic diagram of the structure of a motion estimation module in one embodiment;

[0047] Figure 4 A schematic diagram of pipeline execution of each unit in a motion estimation module in one embodiment;

[0048] Figure 5 A flowchart of a search range determination step in one embodiment;

[0049] Figure 6 A schematic diagram of calculating an unavailable motion vector of a search center point of CTU[i+2] in one embodiment;

[0050] Figure 7 is a schematic diagram of MVs of neighboring blocks of a current CU in one embodiment;

[0051] Figure 8 is a schematic diagram of MVs of neighboring blocks of the current CU in another embodiment;

[0052] Fig. 9 is a schematic diagram of an encoder in one embodiment;

[0053] Fig.10 The figure is a schematic diagram of pipeline execution of each module and each unit in an encoder in one embodiment. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] For the motion estimation module of the encoder, the following tasks usually need to be completed:

[0056] First, decide how to divide the CTU into CUs. According to the HEVC standard, the maximum size of a CTU is 64x64 pixels, and the size of a CU is 4Nx4N (N=2, 4, 8, 16). Figure 1 As shown, Figure 1 FIG. 4 is a schematic diagram of a CTU division method in an embodiment, where a 64x64 CTU is divided into a plurality of 32x32, 16x16, and 8x8 CUs.

[0057] Second, decide how to divide the CU into PUs. Figure 2 As shown, Figure 2 FIG. 4 is a schematic diagram of a PU partitioning method in an embodiment, which provides all possible PU partitioning methods of a 4Nx4N CU.

[0058] Third, find the optimal prediction mode for each PU. Intra-frame prediction mode includes various angle prediction modes, and inter-frame prediction mode includes normal mode, merge mode, and skip mode.

[0059] Conventional video encoders will serially traverse all CU divisions, PU divisions and various prediction modes, and obtain the optimal division method and prediction mode by calculating the costs of various divisions and modes. Although this approach can obtain the most accurate division method and prediction mode, thereby ensuring the best encoding quality, it is not conducive to hardware implementation and the encoding speed is extremely slow.

[0060] In this application, each step of the motion estimation module is reasonably divided into modules, and a certain method is adopted to eliminate the data dependency problem between the modules in the motion estimation process, so that the internal modules of the motion estimation can run in parallel in a pipeline, greatly improving the running speed of the motion estimation.

[0061] Combination Figure 3 As shown, Figure 3 This is a structural diagram of a motion estimation module in an embodiment. The motion estimation module of the present application includes a first intra-frame prediction unit, a second intra-frame prediction unit, a first inter-frame prediction unit, an integer pixel prediction unit, a fractional pixel prediction unit, a second inter-frame prediction unit and a mode selection unit.

[0062] Among them, the first intra-frame prediction unit is used to perform intra-frame prediction on the CTU to obtain a rough intra-frame CU division result; the second intra-frame prediction unit is used to obtain an accurate intra-frame CU division result based on the rough intra-frame CU division result; the first inter-frame prediction unit is used to perform inter-frame prediction on the CTU to obtain a rough inter-frame CU division result; the integer pixel prediction unit is used to perform integer pixel prediction based on the rough inter-frame CU division result to obtain an integer pixel prediction result; the fractional pixel prediction unit is used to perform fractional pixel prediction based on the integer pixel prediction result to obtain a fractional pixel prediction result; the second inter-frame prediction unit is used to perform mode prediction based on the fractional pixel prediction result to obtain an accurate inter-frame CU division result; the mode selection unit is used to obtain a target prediction mode based on the accurate intra-frame CU division result and the accurate inter-frame CU division result; wherein at least two of the first intra-frame prediction unit, the second intra-frame prediction unit, the first inter-frame prediction unit, the integer pixel prediction unit, the fractional pixel prediction unit and the second inter-frame prediction unit are executed in parallel.

[0063] For ease of understanding, see Table 1:

[0064] Table 1 Units of the motion estimation module

[0065]

[0066] The first intra prediction unit CTU_INP is used to perform rough intra prediction on the entire CTU and determine how to divide the current CTU into CUs. The divided CUs are all in intra prediction mode, and the prediction results are inaccurate. The second intra prediction unit CU_INP will perform further intra prediction based on this inaccurate intra prediction result.

[0067] The first inter-frame prediction unit CTU_IME is used to perform rough inter-frame prediction on the entire CTU and determine how to divide the current CTU into CUs. The divided CUs are all in inter-frame prediction mode, and the prediction results are inaccurate. The integer pixel prediction unit CU_IME will perform further inter-frame prediction based on this inaccurate inter-frame prediction mode.

[0068] The integer pixel prediction unit CU_IME is used to read a CU prediction result, perform further integer pixel prediction based on the result, and pass the prediction result to the fractional pixel prediction unit CU_FME.

[0069] The fractional pixel prediction unit CU_FME is used to receive the prediction result of the integer pixel prediction unit CU_IME, perform further fractional pixel prediction based on the result, and pass the prediction result to the second inter-frame prediction unit CU_MERGE.

[0070] The second inter-frame prediction unit CU_MERGE is used to receive the prediction result of the fractional pixel prediction unit CU_FME, perform Merge / Skip mode prediction based on the result, and pass the prediction result to the mode selection module.

[0071] The second intra-frame prediction unit CU_INP is used to read a CU prediction result, perform further intra-frame prediction based on the result, and pass the prediction result to the mode selection module.

[0072] Assume that after CTU[0] is calculated by the CTU_IME module, it is divided into n CUs, denoted as CU[0]~CU[n]; after CTU[1] is calculated by the CTU_IME module, it is divided into m CUs, denoted as CU[0]~CU[m]. The pipeline execution diagram of the above units can be seen in Figure 4 shown.

[0073] In one of the optional embodiments, the first intra prediction unit and the first inter prediction unit process CTU[i] in parallel, where i is an integer greater than or equal to 0.

[0074] Among them combined Figure 4The first intra-frame prediction unit and the first inter-frame prediction unit are independent of each other, and the two are executed in parallel in the same level of pipeline.

[0075] In one of the optional embodiments, the integer pixel prediction unit processes CTU[i] in parallel with the first inter-frame prediction unit processes CTU[i+1], and when the integer pixel prediction unit completes processing of each CU in CTU[i], the first inter-frame prediction unit starts processing CTU[i+2], where i is an integer greater than or equal to 0.

[0076] Among them, combined Figure 4 , the integer pixel prediction unit and the first inter-frame prediction unit are pipelined and executed in parallel in units of CTU, specifically:

[0077] The processing of CTU[i] by the integer pixel prediction unit is performed in parallel with the processing of CTU[i+1] by the first inter prediction unit.

[0078] When the integer pixel prediction unit completes processing of each CU in CTU[i], the first inter-frame prediction unit starts processing CTU[i+2].

[0079] In one of the optional embodiments, the integer pixel prediction unit, the fractional pixel prediction unit and the second inter-frame prediction unit are parallel execution units based on CU; the processing of CU[j] by the fractional pixel prediction unit is parallel to the processing of CU[j+1] by the integer pixel prediction unit, and when the fractional pixel prediction unit completes the processing of CU[j], the integer pixel prediction unit starts to process CU[j+2]; the processing of CU[j] by the second inter-frame prediction unit is parallel to the processing of CU[j+1] by the fractional pixel prediction unit, and when the processing of CU[j] by the second inter-frame prediction unit is complete, the fractional pixel prediction unit starts to process CU[j+2], where j is an integer greater than or equal to 0.

[0080] Among them, combined Figure 4 As shown, the integer pixel prediction unit, the fractional pixel prediction unit and the second inter-frame prediction unit are pipelined and executed in parallel in units of CU. Specifically:

[0081] CU_FME[j] is executed in parallel with CU_IME[j+1], that is, the processing of CU[j] by the fractional pixel prediction unit is parallel to the processing of CU[j+1] by the integer pixel prediction unit.

[0082] After CU_FME[j] is executed, CU_IME[j+2] starts to be executed, that is, when the fractional pixel prediction unit completes the processing of CU[j], the integer pixel prediction unit starts to process CU[j+2].

[0083] CU_MERGE[j] is executed in parallel with CU_FME[j+1], that is, the processing of CU[j] by the second inter prediction unit is parallel to the processing of CU[j+1] by the fractional pixel prediction unit.

[0084] After CU_MERGE[j] is executed, CU_FME[j+2] is executed, that is, when the second inter-frame prediction unit completes processing of CU[j], the fractional pixel prediction unit starts processing CU[j+2].

[0085] In one of the optional embodiments, the second inter-frame prediction unit and the second intra-frame prediction unit process CU[j] in parallel, where j is an integer greater than or equal to 0.

[0086] The time required for the second intra-frame prediction unit CU_INP module is shorter and it is executed in parallel with the second inter-frame prediction unit CU_MERGE module. After the second inter-frame prediction unit CU_MERGE[k] is completed, the accurate inter-frame prediction result of CU[k] and the accurate intra-frame prediction result of CU[k] can be obtained at the same time.

[0087] In one of the optional embodiments, the motion estimation module also includes at least one of the following storage units: a first storage unit, used to store the coarse intra-frame CU division result obtained by the first intra-frame prediction unit, and for the second intra-frame prediction unit to read the coarse intra-frame CU division result; a second storage unit, used to store the coarse inter-frame CU division result obtained by the first inter-frame prediction unit, and for the integer pixel prediction unit to read the coarse inter-frame CU division result.

[0088] Among them, combined Figure 3 As shown, the first storage unit Sram5 is used to store the coarse intra-frame CU division result obtained by the first intra-frame prediction unit, so that the subsequent second intra-frame prediction unit can read the coarse intra-frame CU division result therefrom.

[0089] The second storage unit Sram6 is used to store the coarse inter-frame CU division result obtained by the first inter-frame prediction unit, so that the subsequent integer pixel prediction unit can read the coarse inter-frame CU division result from the second storage unit Sram6.

[0090] The introduction of the first storage unit Sram5 and the second storage unit Sram6 can prevent the units from blocking each other's execution progress so as to buffer the data, thus avoiding the problem of reduced encoding speed due to different processing speeds between modules.

[0091] In one of the optional embodiments, the first inter-frame prediction unit is further used to determine a search center point of CTU[i+2], and determine a search range of CTU[i+2] and reference pixels within the search range of CTU[i+2] based on the search center point, wherein the search center points of CTU[0] and CTU[1] are pre-set, and the search ranges of CTU[0] and CTU[1] are obtained by searching based on the pre-set search center point, wherein i is an integer greater than or equal to 0; wherein the search center point of CTU[i+2] is obtained based on available adjacent motion vectors, or the search center point of CTU[i+2] is obtained based on motion vectors of co-located blocks of adjacent frames; or the search center point of CTU[i+2] is obtained based on a global optimal motion vector determined for each CTU when pre-processing the image.

[0092] In a CTU, the MVs (Motion Vectors) of various PUs are different, which correspond to different reference pixels. If the data is read from the memory when the reference pixel is needed, the encoding speed will be greatly reduced due to the memory delay. In this embodiment, the search range is determined in advance for each CTU, and the reference pixels within the search range are read from the memory in advance.

[0093] Among them, combined Figure 5 As shown, Figure 5 The present invention is a flowchart of the search scope determination step in one embodiment.

[0094] The determination of the search range requires two specific steps. The first step is to determine a search center point for each CTU. The second step is to diffuse a rectangular area around the search center point. The size of the rectangular area can be determined by the user or software configuration. However, in order to have better search accuracy and encoding quality, the search center point must be determined by the algorithm. Generally, a search center point is determined for the current CTU by referring to the MV of the adjacent blocks around the current CTU.

[0095] Combination Figure 5 As shown, k is first set to 0 and initialized. For example, the search center point of CTU[0] and CTU[1] is determined to be (0,0), and the search range of CTU[0] and CTU[1] is obtained by searching based on the preset search center point (0,0), and the reference pixels within the search range are read. Then, the first inter-frame prediction unit processes CTU[k], determines the search center point of CTU[k+2], and determines the search range based on the search center point to read the reference pixels within the search range. Then, after adding one to the value of k, the first inter-frame prediction unit continues to process CTU[k] until the processing is completed.

[0096] Among them, the first inter-frame prediction unit will calculate the search range 2 CTUs in advance, that is, after the first inter-frame prediction unit processes CTU[i], it will determine the search range for CTU[i+2], and read the reference pixels within the search range from the inside and store them in the corresponding storage module, specifically the second storage module mentioned below.

[0097] One thing to note is that due to parallelism and pre-calculation of the search range, the MVs of neighboring blocks around the CTU are not always available. Figure 6 As shown in the figure, when calculating the search center point of CTU[i+2], the MV on its left (the shaded part in the figure) is not available, because CTU[i+1] has not started encoding yet, and no encoding information is provided to CTU[i+2] for reference. Therefore, in this embodiment, the reference pixel of CTU[i+1] can be determined by at least one of the following methods:

[0098] Abandon referencing unavailable adjacent MVs (motion vectors) and only referencing available adjacent MVs, such as upper adjacent MVs or time-domain adjacent MVs. This method is the simplest, but it has a certain impact on the accuracy of the search range.

[0099] Add a video pre-processing module to analyze the video image in advance and find a global optimal MV for each CTU. This method is more complex but has higher accuracy.

[0100] In one of the optional embodiments, the first inter-frame prediction unit, the integer pixel prediction unit, the fractional pixel prediction unit and the second inter-frame prediction unit are all used to determine the motion vector of the current CU based on the motion vector of the adjacent CU; wherein the motion vector of the adjacent CU is determined based on the integer pixel prediction result of the adjacent CU; or the motion vector of the adjacent CU is determined based on the fractional pixel prediction result of the adjacent CU; or the motion vector of the adjacent CU is determined based on the motion vector of the co-located block of the adjacent frame; or the motion vector of the adjacent CU is the initial motion vector determined for each CU when pre-processing the image.

[0101] Inter-frame prediction requires building a reference MV list, i.e., an MVP list, for each PU in the CU. Whether it is the first inter-frame prediction module CTU_IME, the integer pixel prediction module CU_IME, the fractional pixel prediction module CU_FME, or the second inter-frame prediction module CU_MERGE mentioned in this application, it is necessary to build an MVP list, i.e., a motion vector list.

[0102] When constructing the MVP list, you need to refer to the MV of the neighboring blocks of the current CU. Figure 7 As shown, Figure 7FIG. 1 is a schematic diagram of the MV of the neighboring blocks of the current CU in an embodiment. The accurate MV of the CU can be obtained only after the second inter-frame prediction module CU_MERGE is executed. However, due to parallelism, the MV of the neighboring blocks around the CU is not always available, or although it is available, it is not the final accurate MV. For details, see Figure 8 As shown, Figure 8 : is a schematic diagram of the MV of the neighboring blocks of the current CU in another embodiment. In this embodiment, when the first inter-frame prediction module processes CU[k+2], the fractional pixel prediction module processes CU[k+1] in parallel, and the second inter-frame prediction module processes CU[k+2] in parallel. At this time, the A1 position and the B1 position around CU[k+2] cannot obtain accurate MVs. In order to solve the above problem, the following methods are included but not limited to:

[0103] The unavailable precise MV is replaced with the available inaccurate MV. For example, the MV at position A1 of CU[k+2] is replaced with the prediction result of the fractional pixel prediction unit CU_IME at position A1; the MV at position B1 is replaced with the prediction result of the integer pixel prediction unit CU_IME at position B1 or the prediction result of the fractional pixel prediction unit CU_FME.

[0104] Time domain MV is used to replace the unavailable precise MV.

[0105] Add a video pre-processing module to analyze the video image in advance and find a preliminary MV for each CU.

[0106] In the above embodiment, a reasonable method is adopted to remove the data dependency between the upper and lower level modules, so that the encoding quality is not reduced due to the increase in encoding speed.

[0107] Combination Fig. 9 As shown, Fig. 9 The present invention is a schematic diagram of an encoder in an embodiment, wherein the encoder includes a motion estimation module. The motion estimation module also includes a plurality of units, wherein the plurality of units include a first intra-frame prediction unit, a second intra-frame prediction unit, a first inter-frame prediction unit, an integer pixel prediction unit, a fractional pixel prediction unit, a second inter-frame prediction unit, and a mode selection unit, wherein at least two of the first intra-frame prediction unit, the second intra-frame prediction unit, the first inter-frame prediction unit, the integer pixel prediction unit, the fractional pixel prediction unit, the second inter-frame prediction unit, and the mode selection unit are executed in parallel.

[0108] In one of the optional embodiments, the encoder also includes: a motion compensation and transform quantization module, which is used to determine the reconstructed pixels and coding coefficients based on the reference pixels, the original pixels and the target prediction mode output by the motion estimation module; an entropy coding module, which is used to generate a coding stream based on the target prediction mode and the coding coefficients output by the motion estimation module, and the coding stream is stored in the memory; a loop filtering module, which is used to perform loop filtering based on the reconstructed pixels to obtain filtered pixels, and store the filtered pixels as reference pixels in the memory; wherein at least two of the motion estimation module, the motion compensation and transform quantization module, the entropy coding module and the loop filtering module are executed in parallel.

[0109] Among them, the motion estimation module is recorded as the ME (Motion Estimation) module: it is mainly used for intra-frame and inter-frame prediction to determine the optimal CU division method, PU division method, MV and other prediction information. The motion estimation module is the most complex and time-consuming module in the entire encoding process. Its specific limitations can be found above.

[0110] Motion compensation and transform quantization module, denoted as MC (Motion Compensation) module: This module mainly receives the optimal prediction information sent by the motion estimation module, and obtains information such as predicted pixels, residuals, reconstructed pixels, and entropy coding coefficients based on the optimal prediction information.

[0111] In-Loop Filter module, denoted as ILF (In-Loop Filter) module: This module mainly receives the reconstructed pixels sent by the motion compensation and transform quantization module, filters the reconstructed pixels to obtain filtered pixels, and then writes the filtered pixels into the memory as reference pixels for subsequent frames.

[0112] Entropy coding module, denoted as EC (Entropy Coding) module: This module mainly receives the coding coefficients sent by the motion compensation and transform quantization module, and the optimal prediction information sent by the motion estimation module, and then performs entropy coding on the optimal prediction information and coding coefficients to obtain the final encoded bit stream and write it into the memory.

[0113] Among them, in order to improve the processing efficiency, combined with Fig.10 As shown, at least two of the motion estimation module, the motion compensation and transform quantization module, the entropy coding module and the loop filtering module are executed in parallel.

[0114] In one of the optional embodiments, the processing of CU[j] by the motion compensation and transform quantization module and the processing of CU[j+1] by the second inter-frame prediction unit in the motion estimation module are parallel; or the processing of CU[j] by the motion compensation and transform quantization module and the processing of CU[j+1] by the first intra-frame prediction unit in the motion estimation module are parallel, and j is an integer greater than or equal to 0.

[0115] Combination Fig.10 The processing of CU[j] by the motion compensation and transform quantization module can be performed in parallel with the processing of CU[j+1] by the second inter-frame prediction unit in the motion estimation module, or the processing of CU[j+1] by the first intra-frame prediction unit in the motion estimation module.

[0116] In one of the optional embodiments, the entropy coding module processes CU[j] and the motion compensation and transform quantization module processes CU[j +1] in parallel; or the loop filtering module processes CU[j] and the motion compensation and transform quantization module processes CU[j +1] in parallel, where j is an integer greater than or equal to 0.

[0117] Combination Fig.10 , the processing of CU[j] by the entropy coding module or the processing of CU[j] by the loop filtering module is parallel to the processing of CU[j +1] by the motion compensation and transform quantization module.

[0118] In one of the optional embodiments, the processing of CU[j] by the entropy coding module and the processing of CU[j] by the loop filtering module are parallel, and j is an integer greater than or equal to 0.

[0119] Combination Fig.10 , the processing of CU[j] by the entropy coding module and the processing of CU[j] by the loop filtering module are parallel.

[0120] In the above embodiment, after adding the pipeline structure mentioned in this application to the HEVC hardware encoder, when all encoding tools are turned on, the encoding time is reduced by 60%, or the encoding speed is increased by 150%, while the encoding quality and encoding speed are guaranteed. In addition, it should be noted that this application only uses the HEVC encoder as an example to illustrate the processing method, and other encoding standards can also adopt a similar pipeline architecture.

[0121] In one of the optional embodiments, the encoder further includes at least one of the following storage modules:

[0122] The first storage module is used to store the original pixels of the CTU pre-read from the memory and is used by the motion estimation module and the motion compensation and transform quantization module.

[0123] The second storage module is used to pre-read a number of reference pixels from the memory and provide them for use by the motion estimation module and the motion compensation and transform quantization module.

[0124] The third storage module is used for data buffering between the motion estimation module and the motion compensation and transform quantization module, and is used for storing the target prediction mode output by the motion estimation module.

[0125] The fourth storage module is used for data caching between the motion estimation module, the motion compensation and transform quantization module and the entropy coding module, and is used for storing the target prediction mode output by the motion estimation module and the coding coefficients output by the motion compensation and transform quantization module.

[0126] The fifth storage module is used for data buffering between the motion compensation and transform quantization module and the loop filtering module, and is used for storing reconstructed pixels output by the motion compensation and transform quantization module.

[0127] Among them, in this application, continue to combine Fig.10 Since the motion estimation module, motion compensation and transform quantization module, entropy coding module and loop filter module have certain differences in processing speed, in order to avoid blocking the execution progress of these four modules, a storage module SRAM is added at the connection between any two modules of these four modules to buffer the data.

[0128] Specifically, five SRAMs need to be added, which are respectively recorded as the first storage module Sram0, the second storage module Sram1, the third storage module Sram2, the fourth storage module Sram3, and the fifth storage module Sram4. The functions of these five SRAMs are as follows:

[0129] The first storage module Sram0 is used to store the original pixels read back from the memory. The original pixels of several CTUs can be read in advance and saved in the first storage module Sram0. This is done to avoid the encoding speed being affected by the memory delay due to real-time reading of the original pixels.

[0130] The second storage module Sram1 is used to store the reference pixels returned from the memory. The motion estimation module will calculate the search range of several CTUs in advance, so as to read the reference pixels in the search range from the memory in advance and save them in the second storage module Sram1. This is also done to avoid the encoding speed being affected by memory delay due to real-time reading of reference pixels.

[0131] The third storage module Sram2 is used for data caching between the motion estimation module and the motion compensation and transform quantization module to avoid the motion estimation module running too slowly affecting the progress of the motion compensation and transform quantization module, or the motion compensation and transform quantization module running too slowly affecting the progress of the motion estimation module.

[0132] The fourth storage module Sram3 is used for data caching between the motion estimation module, the motion compensation and transform quantization module and the entropy coding module to avoid the progress of the entropy coding module being affected by the motion estimation module, the motion compensation and transform quantization module being too slow, or the progress of the motion estimation module, the motion compensation and transform quantization module being affected by the entropy coding module being too slow.

[0133] The fifth storage module Sram4 is used for data caching between the motion compensation and transform quantization module and the loop filtering module to avoid the motion compensation and transform quantization module running too slowly and affecting the progress of the loop filtering module, or the loop filtering module running too slowly and affecting the progress of the motion compensation and transform quantization module.

[0134] The specific specifications of these five SRAMs need to be determined through preliminary experiments, etc., and no specific restrictions are made here.

[0135] This application introduces a pipeline architecture for video encoding, which greatly improves the encoding speed without changing the encoded image quality.

[0136] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0137] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0138] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0139] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A motion estimation module for an encoder, characterized in that The motion estimation module comprises: A first intra prediction unit, used for performing intra prediction on the CTU to obtain a rough intra CU division result; A second intra prediction unit, configured to obtain an accurate intra CU division result based on the coarse intra CU division result; A first inter-frame prediction unit, configured to perform inter-frame prediction on the CTU to obtain a rough inter-frame CU division result; An integer pixel prediction unit, configured to perform integer pixel prediction based on the coarse inter-frame CU division result to obtain an integer pixel prediction result; A fractional pixel prediction unit, configured to perform fractional pixel prediction based on the integer pixel prediction result to obtain a fractional pixel prediction result; A second inter-frame prediction unit, configured to perform mode prediction based on the fractional pixel prediction result to obtain an accurate inter-frame CU division result; A mode selection unit, configured to obtain a target prediction mode based on the precise intra-frame CU division result and the precise inter-frame CU division result; At least two of the first intra-frame prediction unit, the second intra-frame prediction unit, the first inter-frame prediction unit, the integer pixel prediction unit, the fractional pixel prediction unit and the second inter-frame prediction unit are executed in parallel.

2. The motion estimation module for an encoder according to claim 1, characterized in that The first intra-frame prediction unit and the first inter-frame prediction unit process CTU[i] in parallel, where i is an integer greater than or equal to 0.

3. The motion estimation module for an encoder according to claim 1, characterized in that The integer pixel prediction unit processes CTU[i] in parallel with the first inter-frame prediction unit processes CTU[i+1], and when the integer pixel prediction unit completes processing of each CU in CTU[i], the first inter-frame prediction unit starts processing CTU[i+2], where i is an integer greater than or equal to 0.

4. The motion estimation module for an encoder according to claim 1, characterized in that The integer pixel prediction unit, the fractional pixel prediction unit and the second inter-frame prediction unit are parallel execution units in units of CU; The fractional pixel prediction unit processes CU[j] in parallel with the integer pixel prediction unit processes CU[j+1], and when the fractional pixel prediction unit completes the processing of CU[j], the integer pixel prediction unit starts processing CU[j+2]; The processing of CU[j] by the second inter-frame prediction unit is in parallel with the processing of CU[j+1] by the fractional pixel prediction unit, and when the processing of CU[j] by the second inter-frame prediction unit is completed, the fractional pixel prediction unit starts to process CU[j+2], where j is an integer greater than or equal to 0.

5. The motion estimation module for an encoder according to claim 1, characterized in that The second inter-frame prediction unit and the second intra-frame prediction unit process CU[j] in parallel, where j is an integer greater than or equal to 0.

6. The motion estimation module for an encoder according to any one of claims 1 to 5, characterized in that: The motion estimation module further comprises at least one of the following storage units: A first storage unit, configured to store the coarse intra CU division result obtained by the first intra prediction unit, and to allow the second intra prediction unit to read the coarse intra CU division result; The second storage unit is used to store the coarse inter-frame CU division result obtained by the first inter-frame prediction unit, and to allow the integer pixel prediction unit to read the coarse inter-frame CU division result.

7. The motion estimation module for an encoder according to claim 1, characterized in that The first inter-frame prediction unit is further used to determine a search center point of CTU[i+2], and determine a search range of CTU[i+2] and reference pixels within the search range of CTU[i+2] based on the search center point, wherein the search center points of CTU[0] and CTU[1] are pre-set, and the search ranges of CTU[0] and CTU[1] are obtained by searching based on the pre-set search center point, wherein i is an integer greater than or equal to 0; wherein the search center point of CTU[i+2] is obtained based on available adjacent motion vectors, or the search center point of CTU[i+2] is obtained based on motion vectors of co-located blocks of adjacent frames; or the search center point of CTU[i+2] is obtained based on a global optimal motion vector determined for each CTU when pre-processing the image.

8. The motion estimation module for an encoder according to claim 1, characterized in that The first inter-frame prediction unit, the integer pixel prediction unit, the fractional pixel prediction unit, and the second inter-frame prediction unit are all used to determine the motion vector of the current CU based on the motion vector of the neighboring CU; The motion vector of the adjacent CU is determined based on the integer pixel prediction result of the adjacent CU; or the motion vector of the adjacent CU is determined based on the fractional pixel prediction result of the adjacent CU; or the motion vector of the adjacent CU is determined based on the motion vector of the co-located block of the adjacent frame; or the motion vector of the adjacent CU is the initial motion vector determined for each CU when preprocessing the image.

9. An encoder, characterized in that: The encoder comprises a motion estimation module for an encoder as claimed in any one of claims 1 to 8.

10. The encoder according to claim 9, characterized in that The encoder further comprises: A motion compensation and transform quantization module, used to determine the reconstructed pixels and coding coefficients based on the reference pixels, the original pixels and the target prediction mode output by the motion estimation module; An entropy coding module, used for generating a coding stream based on the target prediction mode output by the motion estimation module and the coding coefficients, wherein the coding stream is stored in a memory; A loop filtering module, configured to perform loop filtering based on the reconstructed pixels to obtain filtered pixels, and store the filtered pixels as the reference pixels in the memory; Wherein, at least two of the motion estimation module, the motion compensation and transform quantization module, the entropy coding module and the loop filtering module are executed in parallel.

11. The encoder according to claim 10, characterized in that The processing of CU[j] by the motion compensation and transform quantization module and the processing of CU[j+1] by the second inter-frame prediction unit in the motion estimation module are parallel; or the processing of CU[j] by the motion compensation and transform quantization module and the processing of CU[j+1] by the first intra-frame prediction unit in the motion estimation module are parallel, and j is an integer greater than or equal to 0.

12. The encoder according to claim 10, characterized in that The entropy coding module processes CU[j] and the motion compensation and transform quantization module processes CU[j +1] in parallel; or the loop filtering module processes CU[j] and the motion compensation and transform quantization module processes CU[j +1] in parallel, where j is an integer greater than or equal to 0.

13. The encoder according to claim 10, characterized in that The processing of CU[j] by the entropy coding module and the processing of CU[j] by the loop filtering module are parallel, and j is an integer greater than or equal to 0.

14. The encoder according to any one of claims 9 to 13, characterized in that: The encoder further comprises at least one of the following storage modules: A first storage module, used for storing original pixels of the CTU pre-read from the memory, and for use by the motion estimation module and the motion compensation and transform quantization module; A second storage module, used for pre-reading a number of reference pixels from the memory and providing the reference pixels for use by the motion estimation module and the motion compensation and transform quantization module; A third storage module, used for data caching between the motion estimation module and the motion compensation and transform quantization module, and used for storing the target prediction mode output by the motion estimation module; a fourth storage module, used for data caching between the motion estimation module, the motion compensation and transform quantization module and the entropy coding module, and used for storing the target prediction mode output by the motion estimation module and the coding coefficient output by the motion compensation and transform quantization module; The fifth storage module is used for data caching between the motion compensation and transform quantization module and the loop filtering module, and is used for storing reconstructed pixels output by the motion compensation and transform quantization module.

Images