Video encoding method and device, parallel computing unit and electronic equipment

Abstract

The embodiment of the present application provides a kind of video encoding method and device, parallel computing unit and electronic equipment.Therein, video encoding method includes: the data loading of all to-be-encoded macroblocks in the to-be-encoded area in to-be-encoded video frame is carried out;The prediction of all to-be-encoded macroblocks after loading is carried out, and corresponding prediction result is obtained;According to corresponding prediction result, the transform and quantization of residual data of all to-be-encoded macroblocks are carried out, and corresponding quantization result is obtained;The entropy coding and code stream writing of corresponding quantization result of all to-be-encoded macroblocks are carried out, and corresponding encoding code stream is obtained;Wherein, the operation of at least one step in the above four steps is executed in parallel for all to-be-encoded macroblocks.The method can significantly improve the video encoding efficiency, so that the method can be applied to some scenes with higher real-time requirements.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video coding, in particular to a video coding method and device, a parallel computing unit and an electronic device. BACKGROUND

[0002] To realize the transmission and storage of video content, video coding protocols (for example, H.264) are widely used. Video coding can be understood as a process of converting original video data into a certain format of code stream through specific data compression technology. However, the encoder implemented based on the video coding protocol has a high operation complexity while obtaining a good coding effect, which makes the video coding process time-consuming and difficult to apply in some occasions with high real-time coding requirements. SUMMARY

[0003] Embodiments of the present application aim to provide a video coding method and device, a parallel computing unit and an electronic device to improve the above technical problems.

[0004] To achieve the above purpose, the present application provides the following technical solutions:

[0005] In a first aspect, the embodiments of the present application provide a video coding method, comprising: loading data of all to-be-coded macroblocks in a to-be-coded region in a to-be-coded video frame; predicting all the loaded to-be-coded macroblocks to obtain corresponding prediction results; transforming and quantizing residual data of all the to-be-coded macroblocks according to the corresponding prediction results to obtain corresponding quantization results; entropy coding and code stream writing of the corresponding quantization results of all the to-be-coded macroblocks to obtain corresponding coded code streams; wherein the operation of at least one of the above four steps is performed in parallel for all the to-be-coded macroblocks.

[0006] In the above method, the coding process is divided into four steps, each step is performed for all the to-be-coded macroblocks in the to-be-coded region in the to-be-coded video frame, so that the operation to be performed for each to-be-coded macroblock within each step becomes relatively simple, the dependency between the operations to be performed by each to-be-coded macroblock is reduced, and thus the operation to be performed by each to-be-coded macroblock can be easily implemented in parallel, thereby the video coding efficiency can be significantly improved, and the method can be applied to some scenarios with high real-time coding requirements. It is noted that which step or steps of the above four steps are implemented in parallel is not limited.

[0007] In an implementation form of the first aspect, the predicting the loaded all to-be-encoded macroblocks to obtain corresponding prediction results comprises: obtaining a type of the to-be-encoded video frame; if the type of the to-be-encoded video frame is an intra-coded frame, performing intra prediction on the loaded all to-be-encoded macroblocks to obtain corresponding intra prediction results; if the type of the to-be-encoded video frame is a forward-predicted coded frame or a bi-predicted interpolated coded frame, performing at least parallel inter prediction on the loaded all to-be-encoded macroblocks to obtain corresponding inter prediction results.

[0008] The inventors have found that in a video encoding process, inter prediction is the longest time-consuming part, and the time consumption ratio can reach 60% to 70%, so for inter prediction, parallel processing can be preferentially adopted to improve the efficiency bottleneck problem of video encoding. As for intra prediction, parallel processing or serial processing can be adopted, and because in intra prediction, the prediction operations of each to-be-encoded macroblock are dependent, intra prediction cannot be completely parallel in the granularity of a single to-be-encoded macroblock after loading.

[0009] In the implementation form described above, the type of the to-be-encoded video frame is obtained first, and then the loaded all to-be-encoded macroblocks are predicted in a manner corresponding to the frame type, so as to distinguish the intra prediction and the inter prediction, thereby facilitating different execution manners (for example, intra prediction in series and inter prediction in parallel) for the two manners, and because the judgment logic and the operation logic are separated, the operation of the prediction part can also be executed on a parallel computing unit (the parallel computing unit is good at large-scale parallel operation, but not good at logical judgment).

[0010] In an implementation form of the first aspect, the to-be-encoded macroblock comprises at least one to-be-encoded sub-block, and when the to-be-encoded macroblock is inter predicted, a full search strategy is adopted to calculate a motion vector of the to-be-encoded sub-block, the full search strategy being that, in all sub-blocks of the same size as the to-be-encoded sub-block in a reference frame corresponding to the to-be-encoded video frame, a most matching sub-block of the to-be-encoded sub-block is searched.

[0011] The full search strategy in the implementation form described above can achieve good prediction effect because it traverses all possible positions of the to-be-encoded sub-block in the reference frame. In addition, when the full search strategy is used to calculate the motion vector, it does not depend on other to-be-encoded macroblocks except the current to-be-encoded macroblock, so that the inter prediction for each to-be-encoded macroblock can be performed in parallel, thereby the inter prediction can still be efficiently performed although the full search strategy itself has a large amount of calculation.

[0012] In an implementation form of the first aspect, the macroblock to be encoded comprises at least one subblock to be encoded, and the motion vector of the subblock to be encoded is calculated by using a partial search strategy when performing inter-frame prediction on the macroblock to be encoded, wherein the partial search strategy refers to searching for a most matching subblock of the subblock to be encoded from all subblocks in a reference frame corresponding to the video frame to be encoded, which have the same size as the subblock to be encoded and meet a search condition.

[0013] In the implementation form described above, the prediction efficiency can be further improved compared with the case where a full search strategy is used, because the partial search strategy is used.

[0014] In an implementation form of the first aspect, if the video frame to be encoded comprises a plurality of reference frames, the operation of searching for the most matching subblock of the subblock to be encoded in the reference frames is performed in parallel for each reference frame.

[0015] In the implementation form described above, the most matching subblock is searched in parallel on the plurality of reference frames, which is advantageous to improving the inter-frame prediction efficiency.

[0016] In an implementation form of the first aspect, the operations performed in parallel for all macroblocks to be encoded are executed by a control unit calling a parallel computing unit, and the remaining operations are executed by the control unit itself, wherein the control unit and the parallel computing unit are hardware units.

[0017] In the implementation form described above, the operations performed in parallel for all macroblocks to be encoded are executed by a parallel computing unit (for example, a GPU), and the remaining operations are executed by a control unit (for example, a CPU), which is advantageous to taking advantage of the characteristics of different hardware units and improving the video encoding efficiency.

[0018] In an implementation form of the first aspect, the step of determining the type of the video frame to be encoded as an intra-frame encoding frame, a forward prediction encoding frame or a bidirectional prediction interpolation encoding frame is executed by the control unit, and if the control unit determines that the type of the video frame to be encoded is a forward prediction encoding frame or a bidirectional prediction interpolation encoding frame, the parallel computing unit is called to perform inter-frame prediction in parallel on all macroblocks to be encoded.

[0019] In the implementation form described above, the step of determining the type of the frame as an intra-frame encoding frame, a forward prediction encoding frame or a bidirectional prediction interpolation encoding frame (which is essentially a logical judgment) is executed by the control unit, and the step of inter-frame prediction is executed by the parallel computing unit, which is advantageous to improving the video encoding efficiency by taking advantage of the characteristics that the control unit is good at logical judgment but not good at large-scale parallel operation, and the parallel computing unit is good at large-scale parallel operation but not good at logical judgment.

[0020] In a second aspect, an embodiment of the present application provides a video encoding apparatus, comprising: a loading module configured to load all to-be-encoded macroblocks in a to-be-encoded region in a to-be-encoded video frame; a prediction module configured to predict all the loaded to-be-encoded macroblocks to obtain corresponding prediction results; a quantization module configured to transform and quantize residual data of all the to-be-encoded macroblocks according to the corresponding prediction results to obtain corresponding quantization results; and an entropy encoding module configured to entropy encode the corresponding quantization results of all the to-be-encoded macroblocks and write a code stream to obtain a corresponding encoding code stream; wherein operations of at least one of the four modules are performed in parallel for all the to-be-encoded macroblocks.

[0021] In a third aspect, an embodiment of the present application provides a parallel computing unit, which is a hardware unit, and is configured to perform operations performed in parallel for all the to-be-encoded macroblocks in the method provided in the first aspect or any possible implementation manner of the first aspect.

[0022] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a control unit and a parallel computing unit, both of which are hardware units, and the control unit is configured to invoke the parallel computing unit to perform operations performed in parallel for all the to-be-encoded macroblocks in the method provided in the first aspect or any possible implementation manner of the first aspect, and the control unit itself is further configured to perform remaining operations in the method provided in the first aspect or any possible implementation manner of the first aspect.

[0023] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer program instructions, which, when read and run by a processor, perform the method provided in the first aspect or any possible implementation manner of the first aspect.

[0024] In a sixth aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer program instructions, and the computer program instructions, when read and run by a processor, perform the method provided in the first aspect or any possible implementation manner of the first aspect.

[0025] In a seventh aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, and the memory stores computer program instructions, and the computer program instructions, when read and run by the processor, perform the method provided in the first aspect or any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0026] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 The present application illustrates the steps that may be included in the video encoding method provided in its embodiments.

[0028] Figure 2 This application illustrates the functional modules that the video encoding apparatus provided in the embodiments of this application may include;

[0029] Figure 3 This application illustrates the possible structure of an electronic device according to an embodiment of the present application;

[0030] Figure 4 This illustration shows another possible structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0031] Currently, encoders based on video codec protocols (e.g., H.264) achieve good encoding results, but their computational complexity is significantly increased, making the video encoding process time-consuming. For example, the main encoding principle of an existing encoder can be represented by the following pseudocode:

[0032] for(frame){

[0033] for(slice){

[0034] for(macroblock){

[0035] macroblock_cache_load(); / / Loading steps

[0036] macroblock_analyze(); / / Prediction steps

[0037] macroblock_encode(); / / Quantization steps

[0038] macroblock_write_cabac(); / / Entropy encoding steps

[0039] }

[0040] }

[0041] }

[0042] Here, frame represents a video frame to be encoded, slice represents a slice to be encoded, and macroblock represents a macroblock to be encoded. The relationship between the three is as follows: a video frame to be encoded can contain one or more slices to be encoded, a slice to be encoded can contain multiple macroblocks to be encoded, and a macroblock to be encoded is the basic encoding unit.

[0043] The overall meaning of the above code is: to traverse all video frames to be encoded (outermost for loop), all segments to be encoded (middle for loop), and all macroblocks to be encoded (innermost for loop) in the video to be encoded, and complete the video encoding task.

[0044] Within the innermost for loop, for each macroblock to be encoded, four steps are executed sequentially: loading, prediction, quantization, and entropy coding. These are represented by the functions macroblock_cache_load, macroblock_analyze, macroblock_encode, and macroblock_write_cabac, respectively. The loading step primarily involves loading the data of the macroblock to be encoded; the prediction step involves performing inter-frame and / or intra-frame predictions on the loaded macroblock; the quantization step involves transforming and quantizing the residual data of the macroblock to be encoded based on the prediction results; and the entropy coding step involves entropy coding the quantization results and writing the bitstream.

[0045] After analyzing the working principle of existing encoders, the inventors found that the main reason for the low encoding efficiency of existing encoders is that the encoding of macroblocks is performed completely serially. That is, after the above four steps are performed for the current macroblock to be encoded, the above four steps are performed for the next macroblock to be encoded that is adjacent to it (for example, the next macroblock to be encoded may be located to the right of the current macroblock to be encoded in the video frame to be encoded), and so on.

[0046] Furthermore, some of these four steps cannot be executed completely independently for each macroblock to be encoded: for example, in the inter-frame prediction operation in the prediction step, when the current macroblock to be encoded performs inter-frame prediction, it may use the inter-frame prediction results of several adjacent preceding macroblocks (for example, the preceding macroblocks may be located to the left and / or above the current macroblock to be encoded in the video frame to be encoded). This means that these preceding macroblocks must be encoded or at least have completed inter-frame prediction before the current macroblock to be encoded.

[0047] Therefore, directly expanding the innermost for loop to parallelize the encoding process of each macroblock and improve coding efficiency is difficult to achieve. "For loop unrolling" treats each iteration of the for loop as a basic unit and executes these units in parallel. For example, if the innermost for loop originally needed to encode 100 macroblocks sequentially, 100 threads could be started to replace the innermost for loop (i.e., "unrolling"), and these threads could be executed in parallel to encode these 100 macroblocks to improve coding efficiency. However, based on the above analysis, due to the dependencies between macroblocks during encoding, directly unrolling the inner for loop is difficult to implement.

[0048] To address the problem of low encoding efficiency in existing encoders, the video encoding method, video encoding apparatus, computer-readable storage medium, electronic device, and computer program product proposed in this application significantly improve the parallelism of the encoding process by reorganizing the relationship between the macroblock to be encoded and the above four encoding steps, thereby improving the efficiency of video encoding.

[0049] It should be noted that, apart from the newly proposed solutions in the embodiments of this application, the reasons for the low encoding efficiency of existing encoders and the possible countermeasures (directly expanding the inner for loop) are all conclusions obtained by the inventors during the research and experimentation process. Therefore, they are also the inventors' contributions to the present invention, rather than part of the prior art.

[0050] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. It should be noted that similar reference numerals and letters in the following drawings indicate similar items; therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0051] The terms “comprising,” “including,” or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0052] Figure 1 The steps that may be included in the video encoding method provided in the embodiments of this application are illustrated. (Refer to...) Figure 1 Video encoding methods include:

[0053] Step S110: Load data for all macroblocks to be encoded within the region to be encoded in the video frame to be encoded.

[0054] Step S120: Predict all macroblocks to be encoded after loading to obtain the corresponding prediction results.

[0055] Step S130: Transform and quantize the residual data of all macroblocks to be encoded according to the corresponding prediction results to obtain the corresponding quantization results.

[0056] Step S140: Perform entropy encoding and write the bitstream to the quantization results corresponding to all macroblocks to be encoded, and obtain the corresponding encoded bitstream.

[0057] Referring to existing encoders, for ease of description, steps S110 to S140 can be referred to as loading, prediction, quantization and encoding steps, respectively. However, it should be noted that these steps are only similar to the steps with the same names in existing encoders in terms of functionality, but their implementation methods are likely to be different from existing encoders (as will be known later, at least one step should perform parallel operations, which is different from existing encoders).

[0058] In step S110, the video frame to be encoded is any frame in the video to be encoded. A video frame to be encoded includes at least one region to be encoded, and each region includes multiple macroblocks to be encoded. A macroblock is a basic coding unit. The shape of the region to be encoded is not limited; for example, it can be a rectangular region in the video to be encoded, or it can be an irregular region formed by several consecutive macroblocks (which can span lines) in the video frame to be encoded, etc. Macroblocks to be encoded can have different sizes, such as 16×16, 8×8, etc.

[0059] In the H.264 protocol, a region to be encoded can refer to a slice, but because... Figure 1 The method described here does not necessarily have to be bound to the H.264 protocol. Therefore, the region to be encoded is not referred to as a slice in step S110. However, in the pseudocode below, the case of a slice is still used as an example.

[0060] For different video frames to be encoded, if their encoding processes are independent, they can be encoded in parallel (but not necessarily); for different regions to be encoded within the same video frame, if their encoding processes are independent, they can be encoded in parallel (but not necessarily). Steps S110 to S140 can also be understood as the encoding process for any region to be encoded composed of macroblocks in any frame of the video to be encoded.

[0061] Before performing specific encoding operations on the macroblocks to be encoded, the macroblock data needs to be loaded from its original location into a specific storage space. This "loading" can be understood from different perspectives: from a hardware perspective, the original location could be a computer disk, and the storage space could be a region in computer memory or a region in the memory of a graphics processing unit (GPU), etc. From a software perspective, the original location could be the video file to be encoded, and the storage space could be a cache used to store the macroblocks to be encoded. The specific hardware on which the original location and storage space are located is not limited. For example, if the macroblock data can be stored in an `mb_t` structure, a pointer variable of type `mb_t` can be defined first, and storage space can be allocated based on this pointer variable. The macroblock data of all macroblocks to be encoded in the region to be encoded can then be loaded into this storage space for easy access in subsequent steps.

[0062] Step S120 applies to all macroblocks to be encoded after being loaded in step S110 (here, "all" refers to all within the region to be encoded, and the same interpretation applies when "all" appears in the following text). Prediction is performed in two ways: intra-frame prediction and inter-frame prediction. Intra-frame prediction is suitable for video frames to be encoded that are intra-coded frames (I-frames), while inter-frame prediction is suitable for video frames to be encoded that are forward predictive coded frames (P-frames) or bidirectional predictive interpolation coded frames (B-frames). Optionally, for video frames to be encoded that are P-frames or B-frames, the macroblocks within them can also undergo intra-frame prediction, and the better prediction result (with smaller corresponding residual data) is selected from the inter-frame prediction results and the intra-frame prediction results as the final prediction result.

[0063] The principles of intra-frame prediction and inter-frame prediction are briefly introduced below. For simplicity, the macroblock being processed is referred to as the current macroblock, and the video frame containing the current macroblock is referred to as the current frame:

[0064] First, let's introduce macroblock-level intra-frame prediction: The main process of intra-frame prediction involves using data from several reconstructed macroblocks adjacent to the current macroblock (e.g., macroblocks to the left, upper left, above, and upper right of the current macroblock) and employing different prediction modes (e.g., copying pixel values ​​from the left macroblock, interpolation calculations, etc.) to predict the data of the current macroblock. A residual exists between the prediction result and the actual data of the current macroblock. The smaller this residual, the more accurate (better) the prediction result, resulting in a higher data compression rate during subsequent quantization and entropy coding. Here, a reconstructed macroblock refers to the macroblock recovered from the macroblock to be encoded at the corresponding position after prediction, transformation, quantization, inverse quantization, and inverse transformation.

[0065] To improve intra-frame prediction accuracy, sub-block level intra-frame prediction can also be performed. A macroblock to be coded can be divided into one or more sub-blocks to be coded. For example, for a 16×16 macroblock, considering sub-blocks of 16×16 and 4×4 sizes, two sub-block partitioning methods can be formed for the macroblock: (1) one 16×16 sub-block; (2) 16 4×4 sub-blocks. Intra-frame prediction is performed on each sub-block generated by the partitioning (the prediction method is similar to the prediction method for the entire macroblock introduced above), and then the sub-block partitioning method that minimizes the residual data corresponding to the entire macroblock is selected as part of the prediction result.

[0066] For example, the input to intra-frame prediction can include the data of the current macroblock, the index of the current macroblock in the current frame, and the data of several reconstructed macroblocks adjacent to the current macroblock in the current frame. The output of intra-frame prediction (i.e., the intra-frame prediction result) can include the sub-block partitioning method of the current macroblock, the prediction mode of each sub-block of the current macroblock, and the residual data corresponding to the current macroblock. While the residual data will always be calculated, it can be calculated in the quantization step, not necessarily in the prediction step. Optionally, if only macroblock-level intra-frame prediction is performed, the sub-block content in the intra-frame prediction output needs to be adjusted accordingly.

[0067] First, let's introduce macroblock-level inter-frame prediction: The main process of inter-frame prediction is to search for the macroblock with the highest similarity to the current macroblock in the reference frame corresponding to the current frame based on a certain search strategy. Let's call it the best-matching macroblock. Then, we calculate the motion vector between the current macroblock and the best-matching macroblock. Finally, based on this motion vector, we move the current macroblock to the position of the best-matching macroblock. This allows us to calculate the residual between the two. The smaller the residual data, the more accurate (better) the prediction result is, and the higher the data compression rate will be when performing quantization and entropy coding in the future.

[0068] In this context, if the current frame is a P-frame, the reference frame can be one or more frames preceding the current frame. If the current frame is a B-frame, the reference frame can be one or more frames preceding the current frame, as well as one or more frames following the current frame. All reference frames are reconstructed frames. For the concept of reconstruction, please refer to intra-frame prediction; it will not be repeated here.

[0069] The similarity between two macroblocks can be measured using several similarity metrics. For example, the metric could be the sum of absolute differences (SAD) between the two macroblocks, or the sum of absolute differences (SATD) after transformation (here referring to the Hadamard transform), and so on.

[0070] If the current frame is a P-frame and there are multiple reference frames, a preliminary best-matching macroblock can be found in each reference frame. Then, a final best-matching macroblock is selected from these preliminary best-matching macroblocks to calculate the motion vector. If the current frame is a B-frame, and the processing method for P-frames is referenced for reference frames before the current frame, a best-matching macroblock can be obtained. Similarly, the processing method for P-frames can be referenced for reference frames after the current frame. The final motion vector can be obtained by weighting the motion vectors corresponding to these two best-matching macroblocks.

[0071] There are many existing strategies for searching the best-matching macroblock. For example, a vector is first calculated based on the motion vectors of several reconstructed macroblocks adjacent to the current macroblock in the current frame (e.g., macroblocks to the left, upper left, and above the current macroblock), and then a search is performed in the reference frame near the position indicated by this vector. However, in current search strategies, the inter-frame prediction of the current macroblock depends on the inter-frame prediction results of other macroblocks (e.g., on the motion vectors of these macroblocks). This application will propose some search strategies that do not depend on the inter-frame prediction results of other macroblocks later to improve the parallelizability of inter-frame prediction.

[0072] Similar to intra-frame prediction, inter-frame prediction at the sub-block level can also be performed to improve the accuracy of inter-frame prediction. A macroblock to be encoded can be divided into one or more sub-blocks to be encoded. For example, for a 16×16 macroblock, considering the seven sizes of sub-blocks of 16×16, 16×8, 8×16, 8×8, 8×4, 4×8, and 4×4, multiple sub-block division methods can be formed for the macroblock: for example, (1) one 16×16 sub-block; (2) two 16×8 sub-blocks; (3) eight 4×8 sub-blocks; (4) one 16×8 and two 8×8 sub-blocks (which can be understood as further dividing one of the 16×8 sub-blocks in (2) into two 8×8 sub-blocks), etc. Inter-frame prediction is performed on each sub-block generated by the partitioning (the prediction method is similar to the prediction method for the entire macroblock described above, including steps such as searching for the best matching sub-block and calculating motion vectors). Then, the sub-block partitioning method that minimizes the residual data corresponding to the entire macroblock to be encoded is selected as part of the prediction result.

[0073] For example, the input to inter-frame prediction can include the data of the current macroblock, the index of the current macroblock in the current frame, the data of the reference frame, and the motion vectors of several reconstructed macroblocks adjacent to the current macroblock in the current frame (optional, some search strategies do not use this). The output of inter-frame prediction (i.e., the inter-frame prediction result) can include the sub-block partitioning method of the current macroblock, the index of the reference frame containing the best-matching sub-block, the motion vectors corresponding to the sub-blocks in the current macroblock, and the residual data corresponding to the current macroblock. While the residual data will always be calculated, it can be calculated in the quantization step, not necessarily in the prediction step. Optionally, if only macroblock-level inter-frame prediction is performed, the sub-block content in the output of inter-frame prediction needs to be adjusted accordingly.

[0074] Step S130 is based on the prediction results of all macroblocks to be encoded obtained in step S120. As mentioned earlier, the residual data in step S130 can be part of the prediction results, or the residual data can be calculated based on the prediction results when executing step S130. The transformation and quantization in step S130 are performed sequentially. First, the residual data of the macroblock to be encoded is transformed based on the prediction results corresponding to each macroblock. Then, the coefficients obtained after the transformation are quantized to obtain the quantization result corresponding to each macroblock to be encoded. The transformation in step S130 can be, but is not limited to, the Discrete Cosine Transform (DCT). The coefficients obtained after the transformation can be called DCT coefficients. The transformation first concentrates the energy of the residual data to low frequencies, and then the quantization maps the coefficients generated after the transformation from continuous values ​​to discrete values, which can reduce the dynamic range of the data, thus facilitating the subsequent entropy coding to compress the data. For example, for a macroblock to be encoded, the quantization result of its residual data can be a numerical string, in which most of the values ​​are 0.

[0075] Step S140 is based on the quantization results of all macroblocks to be encoded obtained in step S130. Entropy encoding and bitstream writing in step S140 are performed sequentially: first, entropy encoding is performed on the quantization results corresponding to all macroblocks to be encoded, and then the entropy encoding results are written into the corresponding encoded bitstream. Entropy encoding is primarily based on the principle of information entropy for data compression, and the final encoded bitstream to be written is the encoding result. For example, this encoded bitstream can refer to an encoded video file corresponding to the video to be encoded.

[0076] It should be noted that in addition to the four core steps mentioned above, video coding may involve some additional steps: for example, some encoding initialization steps can be performed when loading the macroblocks to be encoded; another example is that after the prediction step is completed, a filtering step can be performed to improve the block artifacts produced by the prediction, and so on. For the sake of simplicity, these additional steps will be omitted in the following explanation.

[0077] Steps S110~S130 can be represented by the following pseudocode:

[0078] for(frame){

[0079] for(slice){

[0080] / / Loading steps

[0081] for(macroblock){

[0082] macroblock_cache_load();

[0083] }

[0084] / / Prediction Steps

[0085] for(macroblock){

[0086] macroblock_analyze();

[0087] }

[0088] / / Quantification steps

[0089] for(macroblock){

[0090] macroblock_encode();

[0091] }

[0092] / / Entropy coding steps

[0093] for(macroblock){

[0094] macroblock_write_cabac();

[0095] }

[0096] }

[0097] }

[0098] The meanings of frame, slice, and macroblock are the same as in existing encoders and will not be explained again. The functions macroblock_cache_load, macroblock_analyze, macroblock_encode, and macroblock_write_cabac implement the same functionality as their corresponding functions in existing encoders, but their internal logic may not be the same.

[0099] Furthermore, at least one of the steps S110 to S130 is performed in parallel for all macroblocks to be encoded, that is, the operations in the steps are performed in parallel for each macroblock to be encoded.

[0100] For example, all macroblocks to be encoded in the loading step can be loaded in parallel, while the operations on each macroblock in the remaining steps are executed serially. Since the loading processes of the various macroblocks to be encoded do not depend on each other, parallelism can be achieved directly by unrolling a for loop. For instance, if the original for loop in the loading step needs to load 100 macroblocks sequentially, 100 threads can be started to replace the for loop (i.e., "unrolling"), and these threads can be executed in parallel to load these 100 macroblocks, thereby improving data loading efficiency.

[0101] For example, all macroblocks to be encoded in the loading step can be loaded in parallel, and the inter-frame prediction of all macroblocks to be encoded in the prediction step can be performed in parallel (intra-frame prediction cannot simply be performed in parallel on a macroblock-by-macroblock basis). The operations for each macroblock to be encoded in the remaining steps are performed serially. However, to achieve parallel inter-frame prediction, the search strategy in the existing technology needs to be changed so that the inter-frame prediction processes of each macroblock to be encoded do not become interdependent, as detailed later.

[0102] For example, the residual data of all macroblocks to be encoded in the quantization step can be transformed and quantized in parallel, while the operations for each macroblock to be encoded in the remaining steps can be performed serially, and so on.

[0103] The specific steps to be parallelized can be determined based on the ease of parallelization and / or the impact of parallelization on encoding time. For example, in some implementations, parallelization can be implemented according to the priority of prediction, loading, quantization, and entropy coding. This means that if only one step can be parallelized, the prediction step can be parallelized; if two steps can be parallelized, the loading and prediction steps can be parallelized, and so on. Furthermore, for the prediction step, inter-frame prediction has a higher priority than intra-frame prediction because the inventors found that inter-frame prediction is the most time-consuming part in video coding, accounting for 60% to 70% of the total time. Prioritizing its parallel processing can solve the bottleneck problem in video coding. For the loading step, since the logic of data loading is relatively simple, it is easy to implement in parallel, so its priority is also high.

[0104] Note that, firstly, for ease of understanding the code logic, the pseudocode above does not show the for loop unrolling. However, in the actual code, the for loop for the parallel parts is already unrolled, or rather, the part of the actual code that needs to be parallelized does not have a for loop; it has been replaced by the unrolled multi-threaded structure. The for loop in the code above is only for ease of understanding the coding process. Secondly, a step "executes in parallel for all macroblocks to be encoded" does not mean that all macroblocks to be encoded must execute the operations in this step simultaneously. The operations in this step can also be executed in batches in parallel because parallel computing is limited by device resources. For example, if the original for loop in the loading step is to load 10,000 macroblocks sequentially, then in parallel, theoretically 10,000 threads need to be started to replace the for loop. However, the actual device can only run 1,000 threads at a time, so it can be processed in batches. First, use these threads to load 1,000 macroblocks, and then load another 1,000 macroblocks after that, until all 10,000 macroblocks have been loaded.

[0105] As can be seen from the pseudocode of the existing encoder, in the video encoding method proposed in this application embodiment, the innermost for loop of the existing encoder is split into four for loops for four steps. This split makes the operation to be performed for each macroblock to be encoded in each step relatively simple, reduces the dependency between the operations to be performed by each macroblock to be encoded, and makes it easy to implement the operation to be performed by each macroblock to be encoded in parallel.

[0106] Furthermore, the four for loops are executed sequentially. Each step works on all macroblocks within the region to be encoded, and the next step only executes after all macroblocks have been processed. For example, the loading step doesn't load a macroblock first and then immediately start prediction for that block; it waits until all macroblocks have been loaded before beginning the prediction step. This ensures a degree of synchronization in the encoding progress of each macroblock, which also facilitates parallel processing of the macroblocks.

[0107] In summary, the video method described above significantly improves the parallelism of the encoding process by reorganizing the relationship between the macroblocks to be encoded and the four encoding steps. Furthermore, by combining the operation of at least one of the four encoding steps with the behavior of parallel execution of all macroblocks to be encoded, the efficiency of video encoding is significantly improved, making this method applicable to some scenarios with high real-time encoding requirements.

[0108] Optionally, in the above video encoding method, the operations that are performed in parallel for all macroblocks to be encoded are executed by the control unit calling the parallel computing unit, while the remaining operations are executed by the control unit itself.

[0109] The control unit is primarily used to control the overall video encoding process, while the parallel computing unit is mainly used for parallel computing. Both the control unit and the parallel computing unit are hardware units. For example, the control unit can be a central processing unit (CPU), and the parallel computing unit can be a GPU. Of course, it is not limited whether the control unit and the parallel computing unit need to be located on the same device. However, if the control unit and the parallel computing unit are located on the same device, it can save data transmission time between them and improve video encoding efficiency.

[0110] For example, for operations that require parallel execution during the coding process, storage space (referring to the internal storage space of the GPU) can be allocated on the CPU for the GPU program first. Then, the data to be processed by the GPU is stored in this space. The GPU program contains the GPU's kernel functions, which include parallel processing logic (such as parallel inter-frame prediction logic). The CPU can then process the data pre-stored in the GPU in parallel by calling the GPU's kernel functions. For operations that do not require parallel execution, the execution method is no different from that of ordinary programs on the CPU, and existing technologies can be referenced.

[0111] In the above-mentioned alternative scheme, the operations that are to be executed in parallel for all macroblocks to be encoded are placed on the parallel computing unit, and the remaining operations are placed on the control unit. This is beneficial to give full play to the characteristics of different hardware units and improve video coding efficiency. Examples will be given later in conjunction with the inter-frame coding process.

[0112] It's important to note that the execution of the coding steps doesn't necessarily have to follow the methods listed above. For example, all steps can be executed in parallel computing units, or all steps can be executed in control units, and so on. For instance, while CPUs aren't particularly adept at parallel computing, they aren't incapable of it; their efficiency in parallel computing is simply lower than that of GPUs.

[0113] The following section continues to describe the possible parallel methods used in the prediction step. In one implementation, step S120 may further include: obtaining the type of the video frame to be encoded; if the type of the video frame to be encoded is an I-frame, then performing intra-frame prediction on all the loaded macroblocks to be encoded obtained in step S110 to obtain the corresponding intra-frame prediction results; if the type of the video frame to be encoded is a P-frame or a B-frame, then performing parallel inter-frame prediction on at least all the loaded macroblocks to be encoded obtained in step S110 to obtain the corresponding inter-frame prediction results.

[0114] Note that, firstly, the type of the video frame to be encoded can be determined before the prediction step, and the determination result can be saved and directly retrieved when needed. Of course, the type of the video frame to be encoded can also be determined when the prediction step is executed.

[0115] Secondly, if the type of video frame to be encoded is a P-frame or a B-frame, intra-frame prediction can also be performed, and the better prediction result is selected from the inter-frame prediction result and the intra-frame prediction result as the final prediction result. Therefore, the above means that "at least" all macroblocks to be encoded will be subjected to parallel inter-frame prediction.

[0116] Third, the above implementation only mentions that inter-frame prediction should be processed in parallel. However, intra-frame prediction can be processed in parallel or serially by each macroblock to be encoded. The reason is that inter-frame prediction is the most time-consuming part of video coding. Therefore, prioritizing parallel processing for inter-frame prediction can improve the efficiency bottleneck of video coding.

[0117] Furthermore, according to the principles of intra-frame prediction, the prediction operations of each macroblock to be encoded are dependent on each other. Therefore, intra-frame prediction cannot be parallelized entirely at the granularity of a single loaded macroblock to be encoded (for example, if there are 100 macroblocks in the region to be encoded that need to be intra-frame predicted, it is usually not possible to start 100 threads to perform intra-frame prediction simultaneously). Of course, this does not mean that intra-frame prediction cannot be processed in parallel at all. For example, the macroblocks to be encoded can be intra-frame predicted in batches. The prediction operations between batches are independent of each other, and if the prediction operations within each batch are dependent, they can be processed serially.

[0118] The above implementation can be represented by the following pseudocode:

[0119] for(frame){

[0120] for(slice){

[0121] / / Loading steps

[0122] for(macroblock){

[0123] macroblock_cache_load();

[0124] }

[0125] / / Prediction Steps

[0126] if(i_type==SLICE_TYPE_I){ / / Intra-frame prediction

[0127] for(macroblock){

[0128] mb_analyze_intra();

[0129] }

[0130] }

[0131] if(i_type==SLICE_TYPE_P){ / / Inter-frame prediction

[0132] for(macroblock){

[0133] mb_analyze_inter();

[0134] }

[0135] }

[0136] / / Quantification steps

[0137] for(macroblock){

[0138] macroblock_encode();

[0139] }

[0140] / / Entropy coding steps

[0141] for(macroblock){

[0142] macroblock_write_cabac();

[0143] }

[0144] }

[0145] }

[0146] The above pseudocode and Figure 1 Compared to the pseudocode of the Chinese method, the main change is in the prediction step. Here, `i_type` represents the type of video frame to be encoded, and its value can be determined before the prediction step. `SLICE_TYPE_I` represents an I-frame, `SLICE_TYPE_P` represents a P-frame (for simplicity, B-frames are ignored), `mb_analyze_intra` represents intra-frame prediction operation, and `mb_analyze_inter` represents inter-frame prediction operation (for simplicity, it is assumed that P-frames only undergo inter-frame prediction and not intra-frame prediction, i.e., inter-frame prediction results are assumed to be better).

[0147] To perform inter-frame prediction operations in parallel on all macroblocks to be encoded, the corresponding for loop for inter-frame prediction can be expanded. However, to perform intra-frame prediction operations in parallel on all macroblocks to be encoded, as explained earlier, it is not simply a matter of expanding the corresponding for loop for intra-frame prediction. A parallel approach must be designed based on the characteristics of intra-frame prediction.

[0148] Based on the pseudocode above, it can be seen that the above implementation first obtains the type of the video frame to be encoded and then performs prediction on all the macroblocks to be encoded in a manner corresponding to the frame type. This makes it easy to distinguish between intra-frame prediction and inter-frame prediction, thus making it easy to adopt different execution methods for the two (e.g., intra-frame prediction is serial, and inter-frame prediction is parallel).

[0149] Furthermore, since the above implementation separates the judgment logic and the operation logic, it is also convenient to place the prediction operation on the parallel computing unit mentioned earlier. For example, the operation of determining whether the type of the video frame to be encoded is an I-frame, P-frame, or B-frame (corresponding to an if statement, which is a logical judgment operation) can be executed by the control unit mentioned earlier. If the control unit determines that the type of the video frame to be encoded is a P-frame or a B-frame, it calls the parallel computing unit to perform parallel inter-frame prediction on all the macroblocks to be encoded after loading (corresponding to the for loop expansion inside the if statement, which is a parallel operation). This example makes full use of the characteristics of the control unit, which is good at logical judgment but not good at large-scale parallel operation (relatively speaking), while the parallel computing unit is good at large-scale parallel operation but not good at logical judgment (relatively speaking), flexibly allocating the execution subject of the encoding steps, thereby helping to improve the efficiency of video encoding.

[0150] In an alternative implementation, step S120 can also be implemented as follows:

[0151] for(frame){

[0152] for(slice){

[0153] / / Loading steps

[0154] for(macroblock){

[0155] macroblock_cache_load();

[0156] }

[0157] / / Prediction Steps

[0158] for(macroblock){

[0159] if(i_type==SLICE_TYPE_I){ / / Intra-frame prediction

[0160] mb_analyze_intra();

[0161] }

[0162] if(i_type==SLICE_TYPE_P){ / / Inter-frame prediction

[0163] mb_analyze_inter();

[0164] }

[0165] }

[0166] / / Quantification steps

[0167] for(macroblock){

[0168] macroblock_encode();

[0169] }

[0170] / / Entropy coding steps

[0171] for(macroblock){

[0172] macroblock_write_cabac();

[0173] }

[0174] }

[0175] }

[0176] In this implementation, both inter-frame prediction and intra-frame prediction are located inside the for(macroblock) loop, resulting in these two prediction methods not being well isolated. If the for loop is expanded, for a macroblock in a P-frame, although mb_analyze_intra is not executed, the if(i_type==SLICE_TYPE_I) check still has to be performed, which reduces its execution efficiency. Furthermore, if the expanded for loop is placed on a parallel computing unit for execution, since the parallel computing unit is not good at performing logical judgments, it will also lead to a decrease in coding efficiency.

[0177] Furthermore, considering the case where the macroblock to be encoded includes at least one subblock to be encoded (the case without subblocks can be considered a special case where the subblock and macroblock are the same size), to support parallel inter-frame prediction for all macroblocks to be encoded, a new search strategy needs to be set to search for the best-matching subblock corresponding to the subblock to be encoded, and then calculate the motion vector corresponding to the subblock to be encoded, so that the inter-frame prediction process of each macroblock to be encoded can be performed completely independently. Two strategies are listed below:

[0178] (A) Full search strategy: Search for the best matching sub-block of the sub-block to be encoded among all sub-blocks of the same size as the sub-block to be encoded in the reference frame corresponding to the video frame to be encoded.

[0179] The full search strategy achieves better prediction results because it traverses all possible positions the sub-block to be encoded can move to in the reference frame, thus improving the quality of the final encoding result. Furthermore, when calculating motion vectors using the full search strategy, it does not rely on other macroblocks to be encoded besides the current one, allowing inter-frame prediction for each macroblock to be encoded to be performed in parallel. Therefore, although the full search strategy itself is computationally intensive, inter-frame prediction can still be performed efficiently.

[0180] In contrast, existing encoders are not suitable for computationally intensive full search strategies due to the low degree of parallelization in the encoding process.

[0181] (B) Partial search strategy: Among all sub-blocks of the same size as the sub-block to be encoded and that meet the search conditions in the reference frame corresponding to the video frame to be encoded, search for the best matching sub-block to be encoded.

[0182] In some search strategies, search criteria can be freely specified, but it must be ensured that the inter-frame prediction process of each macroblock to be encoded does not become dependent after specification. For example, the search criteria could be: all sub-blocks in the reference frame that are near the position of the sub-block to be encoded (referring to its corresponding position in the reference frame). Since the time interval between the reference frame and the current video frame to be encoded is usually very small, the displacement of objects in the frame is usually not very large. Therefore, searching from "nearby" sub-blocks is only slightly less effective than performing a full search in the reference frame, but the search efficiency improvement is quite significant. Here, "nearby" can be defined in various ways, such as the distance from the position of the sub-block to be encoded being less than a certain distance, etc. Another example is: sub-blocks in the reference frame that are located at certain specific positions (not all positions) near the position of the sub-block to be encoded.

[0183] In summary, partial search strategies improve predictive efficiency compared to full search strategies by narrowing the search scope.

[0184] The inter-frame prediction schemes described above are all parallelized at the macroblock level. That is, the inter-frame prediction process for different macroblocks to be encoded is parallel, but the search for the best-matching sub-block within each sub-block of a macroblock remains serial. An alternative scheme can further improve inter-frame prediction efficiency by parallelizing at the sub-block level. For example, for each sub-block generated in each sub-block partitioning method of the macroblock to be encoded, its corresponding best-matching sub-block is searched in parallel within the reference frame, and the corresponding motion vector is calculated. However, when the number of macroblocks to be encoded is large, parallelization at the sub-block level may generate a large number of threads; therefore, the parallel computing capabilities of the device need to be considered during implementation.

[0185] Furthermore, if the video frame to be encoded includes multiple reference frames, the operation of searching for the best-matching sub-block in each reference frame can be performed in parallel for each reference frame to further improve inter-frame prediction efficiency. For example, if a P-frame has 100 macroblocks to be encoded that require inter-frame prediction, and there are 3 reference frames, without considering parallelism at the sub-block level, 300 threads can be started to search for the best-matching sub-block among the 300 macroblocks. Each macroblock is allocated 3 threads, each searching on one reference frame. Finally, the search results from the 3 threads are aggregated to obtain the best-matching sub-block for that macroblock. When the number of reference frames is small, this implementation does not significantly increase the number of threads.

[0186] Figure 2 The illustration shows the functional modules that may be included in the video encoding apparatus 200 provided in an embodiment of this application. (Refer to...) Figure 2 The video encoding device 200 includes:

[0187] Loading module 210 is used to load data for all macroblocks to be encoded within the region to be encoded in the video frame to be encoded;

[0188] Prediction module 220 is used to predict all macroblocks to be encoded after loading and obtain the corresponding prediction results;

[0189] The quantization module 230 is used to transform and quantize the residual data of all macroblocks to be encoded according to the corresponding prediction results, so as to obtain the corresponding quantization results.

[0190] Entropy coding module 240 is used to entropy code and write the code stream to the quantization results of all macroblocks to be encoded, so as to obtain the corresponding encoded code stream.

[0191] Among them, the operation of at least one of the above four modules is executed in parallel for all macroblocks to be encoded.

[0192] In one implementation of the video encoding apparatus 200, the prediction module 220 predicts all the macroblocks to be encoded after loading, and obtains the corresponding prediction results, including: obtaining the type of the video frame to be encoded; if the type of the video frame to be encoded is an intra-coded frame, then performing intra-frame prediction on all the macroblocks to be encoded after loading, and obtaining the corresponding intra-frame prediction results; if the type of the video frame to be encoded is a forward predictive coded frame or a bidirectional predictive interpolation coded frame, then performing parallel inter-frame prediction on at least all the macroblocks to be encoded after loading, and obtaining the corresponding inter-frame prediction results.

[0193] In one implementation of the video encoding device 200, the macroblock to be encoded includes at least one subblock to be encoded. When the prediction module 220 performs inter-frame prediction on the macroblock to be encoded, it calculates the motion vector of the subblock to be encoded using a full search strategy. The full search strategy means that among all subblocks of the same size as the subblock to be encoded in the reference frame corresponding to the video frame to be encoded, the best matching subblock to be encoded is searched.

[0194] In one implementation of the video encoding device 200, the macroblock to be encoded includes at least one subblock to be encoded. When the prediction module 220 performs inter-frame prediction on the macroblock to be encoded, it calculates the motion vector of the subblock to be encoded using a partial search strategy. The partial search strategy refers to searching for the best matching subblock of the subblock to be encoded among all subblocks of the same size as the subblock to be encoded and that meet the search conditions in the reference frame corresponding to the video frame to be encoded.

[0195] In one implementation of the video encoding apparatus 200, if the video frame to be encoded includes multiple reference frames, the prediction module 220 searches for the best matching sub-block of the sub-block to be encoded in the reference frames in parallel for each reference frame.

[0196] In one implementation of the video encoding device 200, among the operations performed by each module, the operation for parallel execution of all macroblocks to be encoded is executed by the control unit calling the parallel computing unit, and the remaining operations are executed by the control unit itself. Both the control unit and the parallel computing unit are hardware units.

[0197] In one implementation of the video encoding device 200, the operation of the prediction module 220 in determining whether the type of the video frame to be encoded is an intra-frame coded frame, a forward predictive coded frame, or a bidirectional predictive interpolation coded frame is performed by the control unit. If the control unit determines that the type of the video frame to be encoded is a forward predictive coded frame or a bidirectional predictive interpolation coded frame, then the parallel computing unit is invoked to perform parallel inter-frame prediction on all the macroblocks to be encoded after loading.

[0198] The video encoding device 200 provided in this application embodiment can be used to execute the video encoding method provided in this application embodiment. The implementation principle and technical effects of the device have been described in the foregoing method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the method embodiment.

[0199] This application also provides a parallel computing unit, which is a hardware unit used for parallel operations performed on all macroblocks to be encoded in the video encoding method provided in this application. For example, the parallel computing unit may be a GPU.

[0200] Figure 3 The structure of the electronic device 300 provided in an embodiment of this application is shown. (Refer to...) Figure 3 The electronic device 300 includes a control unit 310 and a parallel computing unit 320, both of which are hardware units 320.

[0201] The control unit 310 is used to call the parallel computing unit 320 to execute the operations performed in parallel for all macroblocks to be encoded in the video encoding method provided in the embodiments of this application. The control unit 310 itself is also used to execute other operations in the video encoding method provided in the embodiments of this application (referring to operations other than those performed by the parallel computing unit). For details, please refer to the relevant content in the above method embodiments.

[0202] exist Figure 3 In this configuration, the control unit 310 and the parallel computing unit 320 can be connected via a bus for data exchange. It should also be noted that the control unit 310 and the parallel computing unit 320 are not necessarily connected via a bus. Figure 3 The one-to-one correspondence in the data can be used, for example, one control unit 310 can correspond to multiple parallel computing units 320, and so on.

[0203] For example, the control unit 310 can be a CPU and the parallel computing unit 320 can be a GPU. In this case, the electronic device 300 can be a server, PC, tablet computer, laptop computer, or other devices.

[0204] Understandable. Figure 3 The structure shown is for illustrative purposes only; the electronic device 300 may also include components that are more advanced than those shown. Figure 3 More components shown, or those with the same Figure 3 Different configurations are shown. For example, electronic device 300 may also include one or more components such as a storage unit and a communication unit. Figure 4 The structure of the electronic device 400 provided in an embodiment of this application is shown. (Refer to...) Figure 4 The electronic device 400 includes a processor 410, a memory 420, and a communication interface 430. These components are interconnected and communicate with each other via a communication bus 440 and / or other forms of connection mechanism (not shown).

[0205] The processor 410 includes one or more (only one is shown in the figure), which can be an integrated circuit chip with signal processing capabilities. The processor 410 can be a general-purpose processor, including a Central Processing Unit (CPU), a Microcontroller Unit (MCU), a Network Processor (NP), or other conventional processors; it can also be a special-purpose processor, including a Graphics Processing Unit (GPU), a Neural-network Processing Unit (NPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Furthermore, when there are multiple processors 410, some can be general-purpose processors and others can be special-purpose processors.

[0206] The memory 420 includes one or more (only one is shown in the figure), which may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0207] Processor 410 and other possible components may access memory 420, reading and / or writing data therein. In particular, memory 420 may store one or more computer program instructions, which processor 410 may read and execute to implement the video encoding method provided in the embodiments of this application.

[0208] For example, processor 410 may include the aforementioned control unit (e.g., CPU) and parallel computing unit (e.g., GPU). Memory 420 may be memory used by the control unit (e.g., ordinary computer memory) and memory used by the parallel computing unit (e.g., GPU memory). The computer program instructions can be stored in these two types of memory respectively. The control unit reads instructions from the corresponding memory to implement non-parallelized steps in the encoding process, and the parallel computing unit reads instructions from the corresponding memory to implement parallelized steps in the encoding process. Data interaction can also occur between the control unit and the parallel computing unit; for example, the control unit can control the parallel computing process of the parallel computing unit.

[0209] Communication interface 430 includes one or more (only one is shown in the figure) that can be used to communicate directly or indirectly with other devices for data exchange. Communication interface 430 may include interfaces for wired and / or wireless communication.

[0210] Understandable. Figure 4 The structure shown is for illustrative purposes only; the electronic device 400 may also include more than [other components]. Figure 4 The more or fewer components shown, or having the same Figure 4 Different configurations are shown. For example, if electronic device 400 does not communicate with other devices, it may not include communication interface 430.

[0211] Figure 4 The components shown can be implemented using hardware, software, or a combination thereof. Electronic device 400 may be a physical device, such as a server, PC, mobile phone, tablet, laptop, wearable device, robot, etc., or a virtual device, such as a virtual machine, container, etc. Furthermore, electronic device 400 is not limited to a single device; it can be a combination of multiple devices or a cluster of a large number of devices. Electronic device 300 can also be implemented using the same structure as electronic device 400.

[0212] This application also provides a computer-readable storage medium storing computer program instructions. These computer program instructions are read and executed by a processor to perform the video encoding method provided in this application. For example, the computer-readable storage medium can be implemented as follows: Figure 4 The memory 420 in the electronic device 400.

[0213] This application also provides a computer program product, which includes computer program instructions. These computer program instructions are read and executed by a processor to perform the video encoding method provided in this application.

[0214] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A video encoding method, characterized in that, include: Loading steps: Load data for all macroblocks to be encoded within the region to be encoded in the video frame to be encoded; Prediction steps: Predict all macroblocks to be encoded after loading to obtain the corresponding prediction results; Quantization step: Based on the corresponding prediction results, transform and quantize the residual data of all macroblocks to be encoded to obtain the corresponding quantization results; Entropy coding steps: Entropy coding and bitstream writing are performed on the quantization results corresponding to all macroblocks to be encoded to obtain the corresponding encoded bitstream; In this process, at least one of the above four steps is executed in parallel for all macroblocks to be encoded, and the priority of the above four steps in parallel execution, from high to low, is as follows: the prediction step, the loading step, the quantization step, and the entropy coding step. For the prediction step, the parallel priority of inter-frame prediction is higher than that of intra-frame prediction.

2. The video encoding method according to claim 1, characterized in that, The step of predicting all macroblocks to be encoded after loading to obtain the corresponding prediction results includes: Obtain the type of the video frame to be encoded; If the type of the video frame to be encoded is an intra-coded frame, then intra-prediction is performed on all the macroblocks to be encoded after loading to obtain the corresponding intra-prediction results. If the type of the video frame to be encoded is a forward predictive coding frame or a bidirectional predictive interpolation coding frame, then at least all the macroblocks to be encoded after loading are subjected to parallel inter-frame prediction to obtain the corresponding inter-frame prediction results.

3. The video encoding method according to claim 2, characterized in that, The macroblock to be encoded includes at least one subblock to be encoded. When performing inter-frame prediction on the macroblock to be encoded, the motion vector of the subblock to be encoded is calculated using a full search strategy. The full search strategy means that among all subblocks of the same size as the subblock to be encoded in the reference frame corresponding to the video frame to be encoded, the best matching subblock to be encoded is searched.

4. The video encoding method according to claim 2, characterized in that, The macroblock to be encoded includes at least one subblock to be encoded. When performing inter-frame prediction on the macroblock to be encoded, a partial search strategy is adopted to calculate the motion vector of the subblock to be encoded. The partial search strategy means that among all subblocks of the same size as the subblock to be encoded and satisfying the search conditions in the reference frame corresponding to the video frame to be encoded, the best matching subblock of the subblock to be encoded is searched.

5. The video encoding method according to claim 3 or 4, characterized in that, If the video frame to be encoded includes multiple reference frames, the operation of searching for the best matching sub-block of the sub-block to be encoded in the reference frames is performed in parallel for each reference frame.

6. The video encoding method according to any one of claims 1-4, characterized in that, The operation of parallel execution for all macroblocks to be encoded is executed by the control unit calling the parallel computing unit, while the remaining operations are executed by the control unit itself. Both the control unit and the parallel computing unit are hardware units.

7. The video encoding method according to claim 6, characterized in that, The operation of determining whether the type of the video frame to be encoded is an intra-coded frame, a forward predictive coded frame, or a bidirectional predictive interpolation coded frame is performed by the control unit. If the control unit determines that the type of the video frame to be encoded is a forward predictive coded frame or a bidirectional predictive interpolation coded frame, it calls the parallel computing unit to perform parallel inter-frame prediction on all the macroblocks to be encoded after loading.

8. A video encoding device, characterized in that, include: The loading module is used to load data from all macroblocks to be encoded within the region to be encoded in the video frame to be encoded. The prediction module is used to predict all macroblocks to be encoded after loading and obtain the corresponding prediction results; The quantization module is used to transform and quantize the residual data of all macroblocks to be encoded based on the corresponding prediction results, so as to obtain the corresponding quantization results. The entropy coding module is used to entropy code and write the code stream to the quantization results of all macroblocks to be encoded, so as to obtain the corresponding encoded code stream. Among them, the operation of at least one of the above four modules is executed in parallel for all macroblocks to be encoded, and the priority of the above four modules in parallel implementation from high to low is as follows: the prediction module, the loading module, the quantization module, and the entropy coding module. For the prediction module, the parallel priority of inter-frame prediction is higher than that of intra-frame prediction.

9. A parallel computing unit, characterized in that, The parallel computing unit is a hardware unit, which is used to perform the operations performed in parallel for all macroblocks to be encoded in the video coding method as described in any one of claims 1-7.

10. An electronic device, characterized in that, include: The control unit and the parallel computing unit are both hardware units. The control unit is used to call the parallel computing unit to perform the operation of parallel execution for all macroblocks to be encoded in the video encoding method as described in any one of claims 1-7. The control unit itself is also used to perform the remaining operations in the video encoding method as described in any one of claims 1-7.

Images