Video encoding method and electronic device

Abstract

Provided is a method for encoding a video. The method comprises the steps of: partitioning an image frame into coding tree units (CTUs); applying first low complexity rate-distortion optimization (LC-RDO) to determine prediction mode candidates of the coding tree units, wherein the first LC-RDO includes applying a first correction value for correcting at least one of rate, distortion, or rate-distortion cost; applying a second LC-RDO to determine a prediction mode among the prediction mode candidates, wherein the second LC-RDO includes applying a second correction value for correcting at least one of rate, distortion, or rate-distortion cost; and generating a bitstream on the basis of the determined prediction mode.

Description

Video encoding method and electronic device

[0001] The present disclosure relates to a video encoding method considering rate-distortion cost and an electronic device for encoding video.

[0002] Video codecs divide video frames into smaller units through block segmentation, processing and compressing them. A key technology used in these compression processes is Rate-Distortion Optimization (RDO). RDO considers the trade-off between the bit rate and distortion of the compressed data and selects the optimal compression method among various encoding modes.

[0003] Meanwhile, high-performance video codecs require significant computational complexity to achieve high compression efficiency. However, in resource-constrained environments such as mobile devices, low computational complexity and power consumption become important considerations. Accordingly, Low Complexity Rate-Distortion Optimization (LC-RDO) is being utilized to efficiently compress video data even with low computational complexity.

[0004] According to one aspect of the present disclosure, a method for encoding a video may be provided. The method may include a step of dividing an image frame into coding tree units (CTUs). The method may include a step of determining prediction mode candidates of the coding tree units by applying a first low complexity rate-distortion optimization (LC-RDO). The first LC-RDO may include applying a first correction value that corrects at least one of a rate, a distortion, and a rate-distortion cost. The method may include a step of determining a prediction mode from among the prediction mode candidates by applying a second LC-RDO. The second LC-RDO may include applying a second correction value that corrects at least one of a rate, a distortion, and a rate-distortion cost. The method may include a step of generating a bitstream based on the determined prediction mode.

[0005] According to one aspect of the present disclosure, an electronic device for encoding a video may be provided. The electronic device may include a memory storing one or more instructions; and one or more processors executing the one or more instructions stored in the memory. The one or more processors may, by executing the one or more instructions, divide an image frame into coding tree units (CTUs). The one or more processors may, by executing the one or more instructions, apply a first low complexity rate-distortion optimization (LC-RDO) to determine prediction mode candidates of the coding tree units. The first LC-RDO may include applying a first correction value that corrects at least one of a rate, a distortion, and a rate-distortion cost. The one or more processors may, by executing the one or more instructions, apply a second LC-RDO to determine a prediction mode from among the prediction mode candidates. The second LC-RDO may include applying a second correction value that corrects at least one of a rate, a distortion, and a rate-distortion cost. The one or more processors may generate a bitstream based on the determined prediction mode by executing the one or more instructions.

[0006] According to one aspect of the present disclosure, a computer-readable recording medium having recorded thereon a program for causing an electronic device to execute any one of the aforementioned and hereinafter described methods for encoding a video may be provided.

[0007] FIG. 1 is a diagram illustrating an encoding process of an electronic device according to one embodiment of the present disclosure.

[0008] FIG. 2A is a diagram for explaining the concept of an encoding unit according to one embodiment of the present disclosure.

[0009] FIG. 2b is a diagram for explaining the concept of an encoding unit according to one embodiment of the present disclosure.

[0010] FIG. 2c is a diagram for explaining a recursive search operation for determining a partition structure of a block of an electronic device according to one embodiment of the present disclosure.

[0011] FIG. 3 is a flowchart illustrating an operation of an electronic device encoding a video according to one embodiment of the present disclosure.

[0012] Figure 4a is a block diagram illustrating the operation of calculating RD cost in a general RDO.

[0013] Figure 4b is a block diagram illustrating the operation of calculating the RD cost in a general LC-RDO.

[0014] FIG. 5A is a block diagram illustrating an RD cost calculation operation in an LC-RDO used by an electronic device according to one embodiment of the present disclosure.

[0015] FIG. 5b is a flowchart for explaining an RD cost calculation operation in an LC-RDO used by an electronic device according to one embodiment of the present disclosure.

[0016] FIG. 6A is a block diagram illustrating an RD cost calculation operation in an LC-RDO used by an electronic device according to one embodiment of the present disclosure.

[0017] FIG. 6b is a flowchart for explaining an RD cost calculation operation in an LC-RDO used by an electronic device according to one embodiment of the present disclosure.

[0018] FIG. 7 is a diagram for explaining an operation of an electronic device according to one embodiment of the present disclosure to pre-calculate correction values ​​of an LC-RDO.

[0019] FIG. 8 is a block diagram illustrating a configuration of an electronic device according to one embodiment of the present disclosure.

[0020] Hereinafter, terms used in this specification will be briefly described, and the present disclosure will be described in detail. In this disclosure, the expression “at least one of a, b, or c” can refer to “a,” “b,” “c,” “a and b,” “a and c,” “b and c,” “all of a, b, and c,” or variations thereof.

[0021] The terms used in this disclosure are selected from widely used, common terms, taking into account the functions of the disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, in which case their meanings will be described in detail in the relevant description. Therefore, the terms used in this disclosure should not be defined simply as names, but rather based on the meanings of the terms and the overall content of the disclosure.

[0022] Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art described herein. Furthermore, terms containing ordinal numbers, such as "first" or "second," used herein may be used to describe various components, but such components should not be limited by such terms. Such terms are used solely to distinguish one component from another.

[0023] When a part of the specification is said to "include" a component, unless otherwise specifically stated, this does not exclude other components but rather implies the inclusion of other components. Furthermore, terms such as "part" and "module" used in the specification refer to a unit that processes at least one function or operation, which may be implemented in hardware, software, or a combination of hardware and software.

[0024] Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art can easily practice the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, portions irrelevant to the description have been omitted for clarity of explanation, and similar reference numerals have been used throughout the specification to designate similar parts.

[0025] The present disclosure will be described in detail with reference to the attached drawings below.

[0026] FIG. 1 is a diagram illustrating an encoding process of an electronic device according to one embodiment of the present disclosure.

[0027] In one embodiment, the electronic device may be any type of device that includes a video encoder. For example, the electronic device may include a smartphone, laptop, desktop, digital camera, smart TV, game console, etc. The electronic device may be a device capable of real-time video processing, providing functions such as shooting, recording, and mirroring.

[0028] Meanwhile, electronic devices are distinct from servers capable of processing large amounts of data. Servers can incorporate relatively high-performance hardware compared to electronic devices, enabling them to perform optimal encoding using complex and sophisticated compression algorithms. Conversely, electronic devices may have limitations in computing resource performance (e.g., storage space, processing speed, power consumption, etc.) compared to servers. Consequently, electronic devices can perform encoding using relatively lightweight and fast compression algorithms.

[0029] Referring to FIG. 1, the encoding process of the electronic device may include at least a screen splitting (100), recursive search (110), CABAC (Context-Adaptive Binary Arithmetic Coding) (120), and a post-processing filter (130) application process.

[0030] The picture segmentation (100) process refers to the process of dividing an image into multiple block units for encoding. For example, an image may be divided into multiple blocks, called coding tree units (CTUs). One CTU may in turn be divided into multiple coding units (CUs), which are sub-blocks of various types. The types of coding units may include prediction units (PUs) and transform units (TUs).

[0031] The recursive search (110) process refers to the process of finding the optimal prediction mode of a block. The prediction mode indicates how to make a prediction within a block. Specifically, if a block of the current frame is encoded by predicting it from a block of the previous frame, the prediction mode may be "Inter Mode." Alternatively, if prediction is made based on other blocks within the current frame without comparison with the previous frame, the prediction mode may be "Intra Mode." The recursive search (110) process repeats the process of finding the optimal encoding mode of a block and finding the optimal encoding mode of a sub-block into which the block is divided, thereby determining the type and encoding mode of the block.

[0032] A mode estimator (112) determines optimal prediction mode candidates in a recursive search process (110). In this case, the mode estimator (112) may include an intra estimator and an inter estimator for performing intra prediction and inter prediction on an input block and determining prediction mode candidates. In the operation of the mode estimator (112), low complexity rate-distortion optimization (LC-RDO) may be applied to determine prediction mode candidates. In the present disclosure, it is characterized in that an electronic device uses a correction value when applying LC-RDO. The correction value used when applying LC-RDO will be described later.

[0033] In the mode decision block (114), the final prediction mode is determined. The mode decision block (114) can determine the final prediction mode from among multiple prediction mode candidates determined from the mode estimator (112). In this case, LC-RDO can be applied to determine the final prediction mode.

[0034] The restorer (116) performs encoding-decoding using the prediction mode finally determined for the block.

[0035] The restorer (116) may include an intra compensator for predicting a current block within a video frame and an inter compensator for predicting a block of the current frame using information from a previous frame. In addition, the operation of the restorer (116) may include a transform-quantization (TQ) process for encoding and an inverse transform-dequantization (IT-DQ) process for decoding.

[0036] The reconstructor (116) can generate pixel data of a syntax element and a reconstructed block for bitstream generation through an encoding-decoding process. The syntax element can be passed to the CABAC (120) module for binarization and used to generate a bitstream. In addition, the reconstructed block can be stored in a buffer so that it can be used as a reference frame for the next frame after passing through a post-processing filter (130).

[0037] Meanwhile, Rate-Distortion Optimization (RDO) is an optimization technique used in the encoding process. It achieves optimal compression efficiency by adjusting the trade-off between video bitrate and distortion. In other words, RDO determines the rate and distortion that minimize error within a given bitrate tolerance. This can be expressed mathematically as Equation 1 below.

[0038] [Mathematical Formula 1]

[0039]

[0040]

[0041] Here, is a set of coding parameters, D is the distortion after encoding, which is the difference between the original image and the restored image due to transformation and quantization (Distortion), R is the rate after encoding (the length of the bitstream), R t represents the target rate. That is, RDO is Rt It is equal to or smaller than, and minimizes distortion.

[0042] In this case, the optimization coding parameters is calculated by the mathematical formula 2 below.

[0043] [Equation 2]

[0044]

[0045] Here, J represents the rate-distortion cost (RD Cost) considering the weights of rate R and distortion D, and λ is a Lagrangian multiplier, a coefficient that depends on the quantization parameter.

[0046] In one embodiment, the electronic device uses LC-RDO, which is relatively simple and fast to operate, instead of RDO. LC-RDO, as described above, can be used to determine candidates for prediction modes, determine the prediction mode, and determine the type mode of a post-processing filter. Furthermore, the electronic device can use a correction value that corrects at least one of rate, distortion, and rate-distortion cost to improve compression performance of the LC-RDO. Specific operations and encoding processes by which the electronic device uses the correction value to improve the performance of the LC-RDO will be described in more detail with reference to the drawings described below.

[0047] FIG. 2A is a diagram for explaining the concept of an encoding unit according to one embodiment of the present disclosure.

[0048] In one embodiment, the electronic device may divide an image frame (210) into multiple CTUs (220), which are unit blocks of encoding. The size of the coding tree unit may be predefined. For example, when the image frame (210) has a size of 352x288, the image frame (210) may be divided into 11x9 CTUs (220). In this case, the size of one CTU (220) may be 32x32. However, the size of the CTU (220) is not limited thereto. For example, the size of the CTU (220) may be 64x64, 128x128, etc.

[0049] In one embodiment, a single CTU (220) may be further divided into multiple sub-blocks, or CUs. Each CTU (220) may be converted into CUs, which are sub-blocks of various types, through the recursive search process described above in FIG. 1.

[0050] CU can be divided into "Prediction Unit (PU)" and "Transform Unit (TU)". PU may be a unit used to predict motion in inter mode. It includes prediction mode and motion vector information, and based on this, it uses information from the previous frame to make predictions, and calculates the residual between the predicted block and the actual block to generate compressed data. TU may be a unit used to compress data by performing a transformation in intra mode. TU may be used to transform data into the frequency domain by applying a transformation matrix. In this case, Discrete Cosine Transform (DCT) may be performed, but is not limited thereto.

[0051] FIG. 2b is a diagram for explaining the concept of an encoding unit according to one embodiment of the present disclosure.

[0052] Referring to FIG. 2b, a CTU (220) can be divided into a plurality of sub-blocks, CUs (230). The CTU (220) can be divided in various ways. Block division methods can include Binary-Tree (BT), Ternary Tree (TT), Quad-Tree (QT), etc.

[0053] BT partitioning refers to a method of dividing a CU (230) into two sub-rectangles. BT partitioning can be further categorized as BT-H and BT-V depending on whether it is vertically or horizontally divided. TT partitioning refers to a method of dividing a CU (230) into three sub-rectangles. TT partitioning can be further categorized as TT-H and TT-V depending on whether it is vertically or horizontally divided. QT partitioning refers to a method of dividing a CU (230) into four sub-rectangles.

[0054] When a CTU (220) having a size of 32x32 is divided into QT, a CU (230) having a size of 16x16 is obtained. In one embodiment, each sub-block into which the CU (230) is divided may be further divided until the minimum block size is reached. For example, a sub-block (240) having a size of 8x8 and divided into QT may be further divided into the form of QT, BT-H, BT-V, TT-H, and TT-V. For example, a sub-block (250) having a size of 8x16 and divided into BT may be further divided into the form of BT-V, BT-H, TT-V, and TT-H. For example, a sub-block (260) having a size of 4x16 and divided into TT may be further divided into the form of BT-V, BT-H, TT-V, and TT-H.

[0055] In one embodiment, the electronic device can apply LC-RDO using a correction value when determining the partitioning structure of a block.

[0056] FIG. 2c is a diagram for explaining a recursive search operation for determining a partition structure of a block of an electronic device according to one embodiment of the present disclosure.

[0057] In one embodiment, the electronic device may determine whether to split a CTU (220) into a plurality of CUs (230). If the prediction mode is determined based on the application of LC-RDO (or RDO) to the CTU (220) block, the electronic device may store the RD cost of the CTU (220) block before splitting and split the CTU (220) into CUs (230). If the prediction mode is determined based on the application of LC-RDO (or RDO) to the CU (230), the electronic device may store the RD cost of the CU (230) block after splitting. The electronic device may compare the RD cost of the CTU (220) block before splitting with the RD cost of the CU (230) block after splitting and determine whether to split the block. For example, if the RD cost of the CTU (220) block before division is higher than the RD cost of the CU (230) block after division, the electronic device can divide the CTU (220) block into CU (230) blocks.

[0058] Meanwhile, such block division decisions are recursively searched and repeated. Specifically, for example, it may be determined whether to further divide a divided CU (230) block into sub-blocks. The electronic device may repeatedly perform block division by comparing the RD costs before and after division of blocks (or sub-blocks). In addition, in this case, while performing block division, the type of the block may be determined, and the optimal prediction mode corresponding to the block type may be determined. The above-described recursive search process may involve the electronic device performing LC-RDO using a correction value. Hereinafter, LC-RDO using a correction value, performed by the electronic device of the present disclosure, will be described in more detail.

[0059] FIG. 3 is a flowchart illustrating an operation of an electronic device encoding a video according to one embodiment of the present disclosure.

[0060] In operation S310, the electronic device divides the image frame into CTUs. The electronic device can perform the following operations to divide the image frame into multiple CTUs and determine a prediction mode for each CTU. Fig. 3 illustrates, as an example, determining a prediction mode for a single CTU. However, the following description can also be applied to blocks into which the CTU is divided.

[0061] In operation S320, the electronic device determines prediction mode candidates of the coding tree unit by applying a first LC-RDO using a first compensation value.

[0062] In one embodiment, the electronic device can determine optimal prediction mode candidates for input CTUs through intra-prediction and inter-prediction. In this case, the intra-prediction mode and the inter-prediction mode may each include multiple modes.

[0063] The electronic device may apply a first LC-RDO to determine prediction mode candidates among a plurality of modes. The first LC-RDO may correspond to the LC-RDO included in the operation of the mode estimator illustrated in FIG. 1.

[0064] The electronic device can perform a first LC-RDO that calculates an RD cost for each of the available prediction modes and determines prediction mode candidates that can minimize the RD cost.

[0065] In this case, the electronic device may apply a first correction value before comparing the RD cost of each of the available prediction modes. The first correction value represents a correction value applied in the first LC-RDO. The first correction value may be a value for correcting at least one of the rate R, the distortion D, and the rate-distortion cost (RD Cost). For example, the first correction value may be values ​​for correcting R and D. Alternatively, the first correction value may be values ​​for correcting the RD Cost.

[0066] The electronic device can determine one or more prediction modes capable of minimizing the RD Cost as prediction mode candidates based on the first LC-RDO result to which the first correction value is applied.

[0067] In operation S330, the electronic device determines a prediction mode among the prediction mode candidates by applying a second LC-RDO using a second correction value. The second LC-RDO may correspond to the LC-RDO used in the mode decision block illustrated in FIG. 1.

[0068] The electronic device can perform a second LC-RDO to calculate an RD cost for each of the prediction mode candidates and determine a final prediction mode that can minimize the RD cost.

[0069] In this case, the electronic device may apply a second correction value before comparing the RD cost of each of the prediction mode candidates. The second correction value represents a correction value applied in the second LC-RDO. The second correction value may be a value for correcting at least one of R, D, and RD COST. For example, the second correction value may be values ​​for correcting R and D. Alternatively, the second correction value may be values ​​for correcting RD Cost.

[0070] The electronic device can determine a final prediction mode that minimizes the RD Cost among the prediction mode candidates based on the second LC-RDO result to which the second correction value is applied.

[0071] In operation S340, the electronic device generates a bitstream based on the determined prediction mode.

[0072] In one embodiment, the electronic device may perform encoding-decoding using the finally determined prediction mode. The encoding may include a transform-quantization (TQ) process, and the decoding may include an inverse transform-dequantization (IT-DQ) process. The electronic device may generate pixel data of a syntax element and a reconstructed block for generating a bitstream through the encoding-decoding process. The syntax element may be passed to a CABAC module and binarized. The CABAC module may perform context modeling, which analyzes previous bits and encoded bit patterns to calculate probabilities for possible values ​​of the current bit, context adaptive binarization, which sets the current bit to 0 or 1 based on the context modeling result, and binary arithmetic coding, which encodes the bits to generate a bitstream. In addition, the reconstructed block may be stored in a buffer so that it can be used as a reference frame for the next frame after passing through a post-processing filter.

[0073] Figure 4a is a block diagram illustrating the operation of calculating RD cost in a general RDO.

[0074] Referring to Fig. 4a, blocks representing modules for calculating the RD cost (403) in a typical RDO are illustrated. In a typical RDO process, the RD cost (403) is calculated based on the prediction mode (401) and residual data (RES) (402). The residual data (RES) (402) refers to the original data (ORG) minus the prediction data (PRED) (RES=ORG-PRED).

[0075] The transform-quantization block (TQ) (410) transforms and quantizes residual data (402). The transformation can transform pixel data from the spatial domain to the frequency domain, and a known transformation technique (e.g., discrete cosine transform (DCT)) can be used. The quantization can divide the transformed frequency domain data into several ranges and approximate the values ​​in each range to a specific value.

[0076] The entropy coder block (411) compresses data generated from the transform-quantization block (410). In this case, information on the prediction mode (401) may be included. The entropy coder block (411) may encode data based on the occurrence frequency, probability, etc. of the data, and a known encoding technique (e.g., Huffman coding, Arithmetic coding, etc.) may be used. A bitstream may be output from the entropy coder block (411).

[0077] The rate calculator block (412) calculates the bit rate R output from the entropy coder block (411). R may include mode (or header) information bits and transform coefficient information bits.

[0078] The inverse transform-dequantization block (IT-DQ) (413) inversely transforms and inversely quantizes the data generated from the transform-quantization block (410). Through this, the inverse transform-dequantization block (413) can output restored residual data (RES`).

[0079] The distortion calculator block (414) can calculate the distortion between the input residual data (RES) (402) and the reconstructed residual data (RES`). The distortion calculator block (414) can calculate the distortion D, for example, through the sum of squared differences (SSD).

[0080] The RD cost calculator block (415) can calculate the final RD cost (403) based on R and D and predefined Lagrange multipliers.

[0081] The electronic device can perform the above-described operations for each of the arbitrary input modes and determine the optimal parameters so as to minimize the RD cost.

[0082] Figure 4b is a block diagram illustrating the operation of calculating the RD cost in a general LC-RDO.

[0083] Referring to Fig. 4b, blocks representing modules for calculating the RD cost (404) in a typical LC-RDO are illustrated. In the typical LC-RDO process, the RD cost (404) is calculated based on the prediction mode (401) and residual data (402). The typical LC-RDO has a relatively small amount of computation and a short processing time compared to the typical RDO illustrated in Fig. 4a. Therefore, the compression efficiency of the LC-RDO may be relatively lower than that of the RDO, and the RD cost (404) of the LC-RDO may have a different value from the RD cost (403) of the RDO.

[0084] The transform-quantization and inverse transform-dequantization used in RDO are computationally intensive, and entropy coding incurs long processing times due to its sequential processing. In LC-RDO, Golomb coding is used to replace entropy coding, and the Hadamard transform is used to replace transform-quantization and inverse transform-dequantization.

[0085] Golomb coding block (420) Compresses data and generates a bitstream using Golomb coding, a lossless data compression method. Golomb coding assumes that the probability of input data occurrence geometrically decreases as the absolute value increases, centered around 0, and outputs a bitstream by compressing the probability interval in which data exists using a prefix code and compressing the value in that interval using a suffix code.

[0086] The rate calculator block (421) calculates the bitrate R output from the Golomb coder block (420).

[0087] The Hadamard transform block (422) transforms input data using a Hadamard kernel (matrix). The general formula for the Hadamard kernel is well known and is therefore omitted in this disclosure. The Hadamard kernel may have various sizes. For example, the Hadamard kernel may have sizes such as 1x1, 2x2, 4x4, and 8x8, but is not limited thereto.

[0088] The distortion calculator block (423) can calculate the distortion between the input residual data (RES) (402) and the Hadamard transformed residual data. Meanwhile, in LC-RDO, the Hadamard transformation may be omitted. In this case, the distortion calculator block (423) can also calculate the distortion D through the sum of absolute differences (SAD) of the residual data.

[0089] The RD cost calculator block (424) can calculate the final RD cost (404) based on R and D and a predefined Lagrange multiplier λ.

[0090] FIG. 5A is a block diagram illustrating an RD cost calculation operation in an LC-RDO used by an electronic device according to one embodiment of the present disclosure.

[0091] The electronic device can calculate the RD cost (504) for each prediction mode to determine the optimal prediction mode for the block. In this case, the electronic device can apply a correction value to improve the performance of the LC-RDO.

[0092] Referring to FIG. 5a, blocks representing modules for calculating the RD cost (504) are illustrated.

[0093] In one embodiment, when performing LC-RDO, the electronic device may utilize a rate correction value for correcting the rate and a distortion correction value for correcting distortion. To this end, the electronic device may use an optimized coding parameter. It can perform computational operations to find the optimized coding parameters. is calculated by the mathematical formula 3 below.

[0094] [Equation 3]

[0095]

[0096] Here, J represents the rate-distortion cost (RD Cost) considering the weights of rate R and distortion D, and λ is a Lagrangian multiplier, a coefficient that depends on the quantization parameter. D` represents a distortion correction value, and R` represents a rate correction value.

[0097] In one embodiment, in addition to using data regarding the prediction mode (502) and residual data (503), the electronic device may further use data regarding the prediction type (501) of the block to calculate the correction value. Specifically, the rate adjustment calculator block (510) calculates a rate correction value R` corresponding to the prediction type (501) and the prediction mode (502) of the block. In addition, the distortion adjustment calculator block (511) calculates a distortion correction value D` corresponding to the prediction type (501) and the prediction mode (502) of the block.

[0098] The Golomb coding block (512) compresses data, and the rate calculator block (513) calculates the bit rate R. The Hadamard transform block (514) transforms the input data using a Hadamard kernel (matrix), and the distortion calculator block (515) calculates the distortion D. Here, the operations of the Golomb coding block (512), the rate calculator block (513), the Hadamard transform block (514), and the distortion calculator block (515) can correspond to the operations of a general LC-RDO.

[0099] The RD cost calculator block (516) can calculate the final RD cost (504) based on the bit rate R, rate correction value R`, distortion D, distortion correction value D`, and Lagrange multiplier λ.

[0100] By using LC-RDO that utilizes rate correction value R` and distortion correction value D`, electronic devices can achieve improved compression performance compared to a method that uses general LC-RDO.

[0101] FIG. 5b is a flowchart for explaining an RD cost calculation operation in an LC-RDO used by an electronic device according to one embodiment of the present disclosure.

[0102] Fig. 5b is intended to supplementally explain LC-RDO using the rate correction value R` and distortion correction value D` of Fig. 5a.

[0103] The operation of the bit rate estimator (520) corresponds to the Golomb coding block (512) and the rate calculator block (513) of Fig. 5a. The bit rate estimator (520) can apply Golomb coding to the input prediction mode and calculate the bit rate R from the generated Golomb code.

[0104] The operation of the distortion estimator (530) corresponds to the Hadamard transform block (514) and the distortion calculator block (515) of Fig. 5a. The distortion estimator (530) can apply a Hadamard transform to an input residual data block and calculate a distortion D based on SSD for the transform block.

[0105] The operation of the RD cost adjustment value derivation unit (RD Cost Adjustment Estimator) (540) corresponds to the rate adjustment calculator block (510) and the distortion adjustment calculator block (511) of Fig. 5a. The RD cost adjustment value derivation unit (540) can calculate a bit rate correction value R` based on the input prediction type / mode, and can calculate a distortion correction value D` based on the input prediction type / mode.

[0106] The operation of the RD cost estimator (550) corresponds to the RD cost calculator block (516) of Fig. 5a. The RD cost estimator (550) can calculate a Lagrangian multiplier using a quantization parameter and calculate the RD cost J (J = D + D` + λ(R + R`)).

[0107] By using LC-RDO that utilizes rate correction value R` and distortion correction value D`, electronic devices can achieve improved compression performance compared to a method that uses general LC-RDO.

[0108] FIG. 6A is a block diagram illustrating an RD cost calculation operation in an LC-RDO used by an electronic device according to one embodiment of the present disclosure.

[0109] Referring to FIG. 6a, blocks representing modules for calculating the RD cost (604) are illustrated.

[0110] In one embodiment, the electronic device may utilize a compensation value to compensate for the rate-distortion cost (RD cost) when performing LC-RDO. To this end, the electronic device may use an optimized coding parameter It can perform computational operations to find the optimized coding parameters. is calculated by the mathematical formula 4 below.

[0111] [Equation 4]

[0112]

[0113] Here, J represents the rate-distortion cost (RD Cost) considering the weights of rate R and distortion D, and λ is a Lagrangian multiplier, a coefficient that depends on the quantization parameter. J` represents the RD cost correction value.

[0114] In one embodiment, the electronic device may further utilize data regarding the prediction type (601) of the block, in addition to utilizing data regarding the prediction mode (602) and residual data (603), to calculate the correction value. Specifically, the RD cost adjustment calculator block (610) calculates an RD cost correction value J` corresponding to the prediction type (501) and prediction mode (502) of the block.

[0115] The Golomb coding block (611) compresses data, and the rate calculator block (612) calculates the bit rate R. The Hadamard transform block (613) transforms the input data using a Hadamard kernel (matrix), and the distortion calculator block (614) calculates the distortion D. Here, the operations of the Golomb coding block (611), the rate calculator block (612), the Hadamard transform block (613), and the distortion calculator block (614) can correspond to the operations of a general LC-RDO.

[0116] The RD cost calculator block (615) can calculate the final RD cost (604) based on the bitrate R, distortion D, RD cost correction value J`, and Lagrange multiplier λ.

[0117] By utilizing LC-RDO using rate-distortion correction value J`, electronic devices can achieve improved compression performance compared to the conventional LC-RDO method.

[0118] FIG. 6b is a flowchart for explaining an RD cost calculation operation in an LC-RDO used by an electronic device according to one embodiment of the present disclosure.

[0119] Fig. 6b is for supplementary explanation of LC-RDO using the rate correction value R` and distortion correction value D` of Fig. 6a.

[0120] The operation of the bit rate estimator (620) corresponds to the Golomb coding block (611) and the rate calculator block (612) of Fig. 6a. The bit rate estimator (620) can apply Golomb coding to the input prediction mode and calculate the bit rate R from the generated Golomb code.

[0121] The operation of the distortion estimator (630) corresponds to the Hadamard transform block (613) and the distortion calculator block (614) of Fig. 6a. The distortion estimator (630) can apply a Hadamard transform to an input residual data block and calculate a distortion D based on SSD for the transform block.

[0122] The operation of the RD Cost Adjustment Estimator (640) corresponds to the RD Cost Adjustment Calculator block (610) of Fig. 6a. The RD Cost Adjustment Estimator (640) can calculate the RD cost adjustment value J` based on the input prediction type / mode.

[0123] The operation of the RD cost estimator (660) corresponds to the RD cost calculator block (615) of Fig. 6a. The RD cost estimator (660) can calculate a Lagrangian multiplier using a quantization parameter and calculate the RD cost J (J=D+λR+J`).

[0124] By utilizing LC-RDO that utilizes rate-distortion correction value J`, electronic devices can achieve improved compression performance compared to the conventional LC-RDO method.

[0125] FIG. 7 is a diagram for explaining an operation of an electronic device according to one embodiment of the present disclosure to pre-calculate correction values ​​of an LC-RDO.

[0126] In one embodiment, the correction values ​​used by the electronic device in the LC-RDO may be pre-calculated. For example, the electronic device may pre-calculate the rate correction value R`, the distortion correction value D`, and the RD cost correction value J`.

[0127] In one embodiment, the electronic device can calculate the RD cost (710) of the RDO for each mode / type. The operation of calculating the RD cost (710) of the RDO when the electronic device performs the RDO has been described above in the description of FIG. 4A, and therefore, a repeated description thereof will be omitted.

[0128] In one embodiment, the electronic device can calculate the RD cost (720) of the LC-RDO for each mode / type. The operation of calculating the RD cost (720) of the LC-RDO when the electronic device performs the LC-RDO has been described above in the description of FIG. 4b, and therefore, a repeated description thereof will be omitted.

[0129] Meanwhile, calculating the RD cost (710) of RDO requires relatively more computational effort than calculating the RD cost (720) of LC-RDO. Therefore, the electronic device may receive data generated during the calculation process of the RD cost (710) of RDO from an external device (e.g., a server).

[0130] In one embodiment, the electronic device may generate a lookup table (730) that pre-calculates and stores correction values ​​based on values ​​obtained in the process of calculating the RD cost (710) of the RDO and the RD cost (720) of the LC-RDO.

[0131] For example, the electronic device can calculate the rate correction value R` in advance based on the difference between the rate R1 calculated in the RDO method and the rate R2 calculated in the LC-RDO method. The electronic device can store the calculated rate correction values ​​R` and generate a lookup table (730) corresponding to the rate correction values ​​R`.

[0132] For example, the electronic device can calculate a distortion correction value D` in advance based on the difference between the distortion D1 calculated in the RDO method and the distortion D2 calculated in the LC-RDO method. The electronic device can store the calculated distortion correction values ​​D` and generate a lookup table (730) corresponding to the distortion correction values ​​D`.

[0133] For example, the electronic device can calculate the RD correction value D` in advance based on the difference between the RD cost J1 calculated in the RDO method and the RD cost J2 calculated in the LC-RDO method. The electronic device can store the calculated distortion correction values ​​D` and generate a lookup table (730) corresponding to the distortion correction values ​​D`.

[0134] In one embodiment, when the electronic device applies LC-RDO, the electronic device can apply a correction value that corrects at least one of rate, distortion, and rate-distortion cost based on a pre-computed lookup table.

[0135] For example, the electronic device may apply a rate correction value R` for rate correction and a distortion correction value D` for distortion correction based on a pre-calculated lookup table.

[0136] For example, the electronic device can apply an RD cost correction value J` that corrects the RD cost based on a precomputed lookup table.

[0137] In one embodiment, the lookup table (730) that pre-calculates the rate correction value R`, the distortion correction value D`, and the RD cost correction value J` can be generated in various ways. For example, the lookup table (730) can be generated using a reinforcement learning method. In this case, a reinforcement learning agent can be designed to perform actions that determine the correction values ​​(R`, D`, J`), and can learn a policy to find optimal correction values ​​based on rewards designed to improve the encoding results while observing the encoding results (e.g., the determined prediction mode, compression efficiency, etc.) while encoding blocks.

[0138] FIG. 8 is a block diagram illustrating a configuration of an electronic device according to one embodiment of the present disclosure.

[0139] In one embodiment, the electronic device (1000) may include a memory (1100) and a processor (1200).

[0140] The memory (1100) may store instructions, data structures, and program codes that can be read by the processor (1200). There may be one or more memories (1100). In the disclosed embodiments, operations performed by the processor (1200) may be implemented by executing instructions or codes of a program stored in the memory (1100).

[0141] The memory (1100) may include non-volatile memory such as ROM (Read-only memory) (e.g., PROM (Programmable read-only memory), EPROM (Erasable programmable read-only memory), EEPROM (Electrically erasable programmable read-only memory)), flash memory (e.g., memory card, SSD (Solid-state drive)), and analog recording type (e.g., HDD (Hard disk drive), magnetic tape, optical disk), and volatile memory such as RAM (Random-access memory) (e.g., DRAM (Dynamic random-access memory), SRAM (Static random-access memory)).

[0142] The memory (1100) can store data, instructions, and programs that cause the electronic device (1000) to perform video encoding using LC-RDO using a correction value.

[0143] The processor (1200) can control the overall operations of the electronic device (1000). The processor (1200) can cause the operations of the electronic device (1000) described in the previous drawings to be performed. For example, the processor (1200) can control the overall operations for the electronic device (1000) to perform video encoding using LC-RDO that utilizes a correction value by executing one or more instructions of a program stored in the memory (1100). There can be one or more processors (1200).

[0144] In one embodiment, the one or more processors (1200) may include at least one of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Accelerated Processing Unit (APU), a Many Integrated Core (MIC), a Digital Signal Processor (DSP), and a Neural Processing Unit (NPU). The one or more processors (1200) may be implemented in the form of an integrated system on a chip (SoC) including one or more electronic components. Each of the one or more processors (1200) may also be implemented as separate hardware (H / W).

[0145] When a method according to an embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one processor (1200) or by a plurality of processors (1200). For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by a first processor, or the first operation and the second operation may be performed by a first processor (e.g., a general-purpose processor) and the third operation may be performed by a second processor (e.g., an AI-only processor). Here, an example of the second processor may be an AI-only processor, and the AI-only processor may perform operations for training / inference of an AI model. However, the embodiments of the present disclosure are not limited thereto.

[0146] One or more processors (1200) according to the present disclosure may be implemented as a single-core processor or as a multi-core processor.

[0147] When a method according to one embodiment of the present disclosure includes multiple operations, the multiple operations may be performed by one core or may be performed by multiple cores included in one or more processors (1200).

[0148] Meanwhile, although not illustrated in FIG. 8, the electronic device (1000) may further include additional components to perform the operations described in the above-described embodiment. For example, the electronic device (1000) may further include a display, a camera, etc. However, the additional components of the electronic device (1000) are not limited thereto. The electronic device (1000) may further include components to perform operations related to video processing, including video encoding, such as video acquisition, generation, and playback.

[0149] The present disclosure proposes a video encoding method and an electronic device for performing video encoding that improve the performance of LC-RDO using various types of correction values. Specifically, a method is proposed that can improve compression performance without increasing computational complexity by applying correction values ​​corresponding to the type / mode of a coding block in a low-complexity rate-distortion optimized LC-RDO video encoding process. The technical problems to be achieved in the present disclosure are not limited to those mentioned above, and other technical problems not mentioned above will be clearly understood by those skilled in the art to which the present invention pertains from the description of the present specification.

[0150] According to one aspect of the present disclosure, a method of encoding a video may be provided.

[0151] The method may include a step of dividing an image frame into coding tree units (CTUs).

[0152] The method may include a step of determining prediction mode candidates of the coding tree unit by applying a first low complexity rate-distortion optimization (LC-RDO).

[0153] The first LC-RDO may include applying a first correction value that corrects at least one of rate, distortion, and rate-distortion cost.

[0154] The method may include a step of determining a prediction mode from among the prediction mode candidates by applying a second LC-RDO.

[0155] The second LC-RDO may include applying a second correction value that corrects at least one of rate, distortion, and rate-distortion cost.

[0156] The method may include a step of generating a bitstream based on the determined prediction mode.

[0157] The first correction value and the second correction value may be applied based on a pre-calculated lookup table.

[0158] The above pre-calculated lookup table may be generated based on calculating the rate-distortion cost of RDO by applying Rate-Distortion Optimization (RDO) to the encoding process, calculating the rate-distortion cost of LC-RDO by applying Low Complexity Rate-Distortion Optimization (LC-RDO) to the encoding process, and the difference between the rate-distortion cost of RDO and the rate-distortion cost of LC-RDO.

[0159] The above first LC-RDO may apply a first rate correction value for correcting the rate and a first distortion correction value for correcting distortion.

[0160] The above second LC-RDO may apply a second rate correction value for correcting the rate and a second distortion correction value for correcting distortion.

[0161] The above first LC-RDO may apply a first rate-distortion cost correction value that corrects the rate-distortion cost.

[0162] The above second LC-RDO may apply a second rate-distortion cost correction value that corrects the rate-distortion cost.

[0163] The method may include a step of determining whether to split the coding tree unit into a plurality of coding units.

[0164] The decision on whether to split may be performed recursively for each of the coding units until the optimal prediction mode for each of the coding units is determined.

[0165] The step of determining whether to split the above coding tree unit into a plurality of coding units may be determining whether to split the coding tree unit by applying a third LC-RDO.

[0166] The third LC-RDO may apply a third correction value that corrects at least one of rate, distortion, and rate-distortion cost.

[0167] The step of generating a bitstream based on the above-determined prediction mode may include a step of performing encoding-decoding using the above-determined prediction mode to generate pixel data of a syntax element and a reconstructed block for generating the bitstream.

[0168] The method may include a step of applying a post-processing filter to the restoration block.

[0169] The above method may include a step of storing pixel data of a restoration block to which the post-processing filter is applied in a buffer.

[0170] The step of applying a post-processing filter to the above restoration block may be to determine the type and mode of the post-processing filter by applying the fourth LC-RDO.

[0171] The fourth LC-RDO may include applying a fourth correction value that corrects at least one of rate, distortion, and rate-distortion cost.

[0172] According to one aspect of the present disclosure, an electronic device for encoding a video may be provided.

[0173] The electronic device may include a memory storing one or more instructions; and one or more processors executing the one or more instructions stored in the memory.

[0174] The one or more processors can divide an image frame into coding tree units (CTUs) by executing the one or more instructions.

[0175] The one or more processors can determine prediction mode candidates of the coding tree unit by applying a first low complexity rate-distortion optimization (LC-RDO) by executing the one or more instructions.

[0176] The first LC-RDO may include applying a first correction value that corrects at least one of rate, distortion, and rate-distortion cost.

[0177] The one or more processors can determine a prediction mode among the prediction mode candidates by applying the second LC-RDO by executing the one or more instructions.

[0178] The second LC-RDO may include applying a second correction value that corrects at least one of rate, distortion, and rate-distortion cost.

[0179] The one or more processors can generate a bitstream based on the determined prediction mode by executing the one or more instructions.

[0180] The first correction value and the second correction value may be applied based on a pre-calculated lookup table.

[0181] The above pre-calculated lookup table may be generated based on calculating the rate-distortion cost of RDO by applying Rate-Distortion Optimization (RDO) to the encoding process, calculating the rate-distortion cost of LC-RDO by applying Low Complexity Rate-Distortion Optimization (LC-RDO) to the encoding process, and the difference between the rate-distortion cost of RDO and the rate-distortion cost of LC-RDO.

[0182] The above first LC-RDO may apply a first rate correction value for correcting the rate and a first distortion correction value for correcting distortion.

[0183] The above second LC-RDO may apply a second rate correction value for correcting the rate and a second distortion correction value for correcting distortion.

[0184] The above first LC-RDO may apply a first rate-distortion cost correction value that corrects the rate-distortion cost.

[0185] The above second LC-RDO may apply a second rate-distortion cost correction value that corrects the rate-distortion cost.

[0186] The one or more processors may determine whether to split the coding tree unit into a plurality of coding units by executing the one or more instructions.

[0187] The decision on whether to split may be performed recursively for each of the coding units until the optimal prediction mode for each of the coding units is determined.

[0188] The one or more processors may determine whether to split the coding tree unit by applying the third LC-RDO by executing the one or more instructions.

[0189] The third LC-RDO may include applying a third correction value that corrects at least one of rate, distortion, and rate-distortion cost.

[0190] The one or more processors can perform encoding-decoding using the determined prediction mode by executing the one or more instructions to generate pixel data of a syntax element and a reconstructed block for generating the bitstream.

[0191] The one or more processors may apply a post-processing filter to the restoration block by executing the one or more instructions.

[0192] The one or more processors can store pixel data of a restoration block to which the post-processing filter is applied in a buffer by executing the one or more instructions.

[0193] The one or more processors can determine the type and mode of the post-processing filter by applying the fourth LC-RDO by executing the one or more instructions.

[0194] The fourth LC-RDO may include applying a fourth correction value that corrects at least one of rate, distortion, and rate-distortion cost.

[0195] Meanwhile, embodiments of the present disclosure may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules, executed by a computer. Computer-readable media may be any available media that can be accessed by a computer, and include both volatile and nonvolatile media, removable and non-removable media. Furthermore, computer-readable media may include computer storage media and communication media. Computer storage media include both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Communication media may typically include computer-readable instructions, data structures, or other data in a modulated data signal, such as program modules.

[0196] Additionally, a computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0197] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0198] The above description of the present disclosure is provided for illustrative purposes only, and those skilled in the art will readily appreciate that modifications to other specific forms can be made without altering the technical spirit or essential features of the present disclosure. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, components described as being single may be implemented in a distributed manner, and similarly, components described as being distributed may be implemented in a combined manner.

[0199] The scope of the present disclosure is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present disclosure.

Claims

1. In the video encoding method, A step of dividing an image frame into coding tree units (CTUs); A step of determining prediction mode candidates of the coding tree unit by applying a first low complexity rate-distortion optimization (LC-RDO), wherein the first LC-RDO includes applying a first correction value that corrects at least one of rate, distortion and rate-distortion cost; A step of applying a second LC-RDO to determine a prediction mode from among the prediction mode candidates, wherein the second LC-RDO includes applying a second correction value that corrects at least one of a rate, a distortion, and a rate-distortion cost; and A method comprising the step of generating a bitstream based on the above-determined prediction mode.

2. In paragraph 1, A method wherein the first correction value and the second correction value are applied based on a pre-calculated lookup table.

3. In paragraph 2, The above precomputed lookup table is, A method wherein the rate-distortion cost of RDO is calculated by applying rate-distortion optimization (RDO) to an encoding process, the rate-distortion cost of LC-RDO is calculated by applying low complexity rate-distortion optimization (LC-RDO) to the encoding process, and the rate-distortion cost of LC-RDO is generated based on the difference between the rate-distortion cost of RDO and the rate-distortion cost of LC-RDO.

4. In any one of paragraphs 1 to 3, The above first LC-RDO applies a first rate correction value for correcting the rate and a first distortion correction value for correcting distortion, A method wherein the second LC-RDO applies a second rate correction value for correcting the rate and a second distortion correction value for correcting distortion.

5. In any one of paragraphs 1 to 3, The above first LC-RDO applies a first rate-distortion cost correction value that corrects the rate-distortion cost, A method wherein the second LC-RDO applies a second rate-distortion cost correction value that corrects the rate-distortion cost.

6. In any one of paragraphs 1 to 5, The above method, Further comprising a step of determining whether to split the above coding tree unit into a plurality of coding units, A method wherein the determination of whether to split each of the coding units is performed recursively until the optimal prediction mode of each of the coding units is determined.

7. In paragraph 6, The step of determining whether to split the above coding tree unit into multiple coding units is: A method for determining whether to split the coding tree unit by applying a third LC-RDO, wherein the third LC-RDO includes applying a third correction value that corrects at least one of a rate, a distortion, and a rate-distortion cost.

8. In an electronic device for encoding video, Memory that stores one or more instructions; and comprising one or more processors executing one or more instructions stored in said memory; The one or more processors, by executing the one or more instructions, Divide the image frame into coding tree units (CTUs), Determining prediction mode candidates of the coding tree unit by applying a first low complexity rate-distortion optimization (LC-RDO), wherein the first LC-RDO includes applying a first correction value that corrects at least one of rate, distortion and rate-distortion cost, Applying a second LC-RDO to determine a prediction mode from among the prediction mode candidates, wherein the second LC-RDO includes applying a second correction value that corrects at least one of a rate, a distortion, and a rate-distortion cost, An electronic device that generates a bitstream based on the above-determined prediction mode.

9. In paragraph 8, An electronic device, wherein the first correction value and the second correction value are applied based on a pre-calculated lookup table.

10. In paragraph 9, The above precomputed lookup table is, An electronic device generated based on a difference between the rate-distortion cost of the RDO and the LC-RDO by applying rate-distortion optimization (RDO) to an encoding process and calculating the rate-distortion cost of the RDO, and applying low complexity rate-distortion optimization (LC-RDO) to the encoding process and calculating the rate-distortion cost of the LC-RDO.

11. In any one of paragraphs 8 to 10, The above first LC-RDO applies a first rate correction value for correcting the rate and a first distortion correction value for correcting distortion, An electronic device wherein the second LC-RDO applies a second rate correction value for correcting a rate and a second distortion correction value for correcting distortion.

12. In any one of paragraphs 8 to 10, The above first LC-RDO applies a first rate-distortion cost correction value that corrects the rate-distortion cost, An electronic device, wherein the second LC-RDO applies a second rate-distortion cost compensation value that compensates for the rate-distortion cost.

13. In any one of paragraphs 8 to 12, The one or more processors, by executing the one or more instructions, Decide whether to split the above coding tree unit into multiple coding units, An electronic device, wherein the determination of whether to split each of the coding units is performed recursively until the optimal prediction mode of each of the coding units is determined.

14. In paragraph 13, The one or more processors, by executing the one or more instructions, An electronic device for determining whether to split the coding tree unit by applying a third LC-RDO, wherein the third LC-RDO includes applying a third correction value that corrects at least one of a rate, a distortion, and a rate-distortion cost.

15. A computer-readable recording medium having recorded thereon a program for executing the method of any one of clauses 1 to 7 on a computer.

Images