A method and apparatus for P-frame and / or B-frame image block level rate control

Abstract

This invention discloses a bitrate control method at the image block level for P-frames and / or B-frames. Step S1: Obtain the inter-frame coding mode candidates and their prediction costs for each image block to be encoded. Step S2: Select the minimum prediction cost to characterize the inter-frame coding complexity of the image block to be encoded; use the sum of the inter-frame coding complexities of all image blocks within a video frame as the inter-frame coding complexity of that video frame. Step S3: Calculate the target number of coding bits for each image block to be encoded within the P-frame or B-frame to be encoded. Step S4: Calculate the Lagrange multipliers for the image block to be encoded within the P-frame or B-frame to be encoded. Step S5: Perform video encoding on the image block to be encoded, and then adjust the target number of coding bits for the next image block to be encoded within the P-frame or B-frame to be encoded. This invention has low hardware overhead and low implementation cost; it does not introduce a large number of complex floating-point operations, reducing the difficulty and cost of hardware implementation.

Description

Technical Field

[0001] This invention relates to a digital video coding technique, and more particularly to a block-level bitrate control method for P-frames and / or B-frames suitable for hardware implementation. Background Technology

[0002] Video coding is a technique that compresses redundant components in video images and represents video information using as little data as possible. Common video coding standards include HEVC (High Efficiency Video Coding, also known as H.265) and AVC (Advanced Video Coding, also known as H.264). Video coding technology uses image blocks as the most basic coding unit. For example, in HEVC, the basic coding unit is the CU (Coding Unit). A CU can be an image block of 64 pixels × 64 pixels, 32 pixels × 32 pixels, 16 pixels × 16 pixels, or 8 pixels × 8 pixels. The 64-pixel × 64-pixel image block is also called the LCU (Largest Coding Unit).

[0003] To remove information redundancy in the spatial and temporal domains, input video frames are encoded using intra-frame coding and inter-frame coding techniques. Encoded video frames are generally divided into I-frames (using only intra-frame coding) and P-frames and B-frames (using a hybrid approach). I-frames use only intra-frame coding, employing only the information of the current frame and not referencing information from other encoded frames. All image blocks within an I-frame are encoded as intra-coded blocks. P-frames and B-frames use a hybrid approach, employing both intra-frame and inter-frame coding. They not only use the information of the current frame but also reference information from one or more other encoded frames. Image blocks within P-frames and B-frames can be encoded as either intra-coded or inter-coded blocks. For each image block within a P-frame or B-frame, if intra-frame coding results in a lower rate-distortion cost, the image block will be encoded as an intra-coded block; if inter-frame coding results in a lower rate-distortion cost, the image block will be encoded as an inter-coded block. For each video frame, the type of encoded frame is determined by the image group parameters set by the user. Figure 1 It demonstrates a common image group structure, which includes I-frames, P-frames, or B-frames.

[0004] To maximize video compression, common video coding algorithms are often lossy compression algorithms, such as AVC and HEVC. This results in a difference between the reconstructed video and the original video; that is, the reconstructed video will contain distortion. For lossy compression algorithms, their coding performance needs to be measured by both the coding bitrate and the distortion introduced by the coding. In the video coding process, the coding bitrate and coding distortion are mutually restrictive and contradictory. For example, reducing the coding bitrate often increases coding distortion, while reducing coding distortion often increases the coding bitrate. Rate Distortion Optimization (RDO) aims to minimize coding distortion at a given coding bitrate, or minimize the coding bitrate at a given coding distortion. Currently, most common video coding algorithms use rate distortion optimization methods based on the Lagrange multiplier λ. Assuming D is the coding distortion, R is the coding bitrate, and J is the coding rate distortion cost (RD Cost), J = D + λ × R. Rate-distortion optimization based on the Lagrange multiplier λ can be expressed as min(J), where min() represents minimizing the value. In video coding, each coding mode has its corresponding coding distortion D and coding bit rate R, allowing the calculation of the coding rate-distortion cost J for each mode. During encoding, selecting the coding mode with the minimum rate-distortion cost J yields optimal coding performance. The process by which the encoder selects the optimal coding mode for each image block to be encoded using rate-distortion optimization is simply called mode decision for the image block to be encoded.

[0005] In real-world scenarios, the channel bandwidth (channel capacity) used to transmit compressed video is limited. If the encoding bit rate of the compressed video is too high, exceeding the channel bandwidth capacity, it will cause video transmission congestion or even packet loss. If the encoding bit rate of the compressed video is too low, the channel bandwidth will not be fully utilized, and higher video quality cannot be obtained. Therefore, it is necessary to use bitrate control technology to control the output bitrate of the video encoder to match it with the channel bandwidth capacity.

[0006] Rate control technology adjusts the encoding parameters of the video encoder to make the output bitrate of the video encoder equal to a pre-set target bitrate, while minimizing encoding distortion to improve video encoding quality. Common rate control algorithms generally achieve the task through two stages: target bit allocation and target bit control.

[0007] In common rate control algorithms, target bit allocation typically occurs at three levels: GOP (Group of Pictures, a set of time-series consecutive video frames) level, video frame level, and image block level. After target bit allocation at the GOP and video frame levels, the target number of coded bits for the current video frame to be encoded is determined. The next step is to perform target bit allocation at the image block level, allocating the target number of coded bits to each image block based on its bit allocation weight within the video frame. Then, in the target bit control stage, the corresponding Lagrange multiplier λ is calculated based on the target number of coded bits for each image block, and used in the image block encoding process.

[0008] Because video encoding algorithms involve a large amount of computation, using application-specific integrated circuits (ASICs) to accelerate the video encoding process in hardware has become a common industry practice in order to improve video encoding speed and achieve real-time encoding and transmission. These ASICs that accelerate the video encoding process in hardware are generally referred to as hardware video encoders.

[0009] In common rate control algorithms, the image block-level rate control algorithms for P-frames and B-frames generally involve a large number of complex floating-point operations, which are not conducive to hardware implementation. For example, in the HEVC rate control algorithm proposed in JCTVC-K0103, the image block-level rate control algorithms for P-frames and B-frames not only include a large number of floating-point multiplication and division operations, but also more complex floating-point exponentiation and floating-point logarithmic operations, which are very difficult and costly to implement in hardware. Summary of the Invention

[0010] The technical problem to be solved by this invention is to design a bitrate control method at the image block level for P-frames and / or B-frames suitable for hardware video encoders. Without introducing a large number of floating-point operations, this method can more accurately complete the target bit allocation of image blocks within P-frames and / or B-frames, and calculate the target number of encoded bits and the corresponding Lagrange multipliers for image blocks within P-frames and / or B-frames. This makes the actual number of encoded bits for image blocks within P-frames and / or B-frames closer to the target number of encoded bits, and makes the bitrate control process of P-frames and / or B-frames more precise.

[0011] To address the aforementioned technical problems, this invention proposes a bitrate control method at the image block level for P-frames and / or B-frames, comprising the following steps: Step S1: Predict the inter-frame coding mode for each image block to be encoded, obtaining candidate inter-frame coding modes and their prediction costs for that image block. Step S2: Filter the prediction costs corresponding to each candidate inter-frame coding mode for the image block to be encoded, selecting the minimum prediction cost to characterize the inter-frame coding complexity of that image block; then, the sum of the inter-frame coding complexities of all image blocks within the video frame is taken as the inter-frame coding complexity of that video frame. Step S3: Calculate the target coding bit count for each image block within the P-frame or B-frame to be encoded, based on the ratio of the inter-frame coding complexity of the image blocks to be encoded within the P-frame or B-frame to the overall inter-frame coding complexity of the P-frame or B-frame, and in conjunction with the target coding bit count of the P-frame or B-frame. Step S4: Based on the target number of encoded bits and inter-frame coding complexity of the image block to be encoded within the P-frame or B-frame to be encoded, and combined with the actual number of encoded bits, Lagrange multipliers, and inter-frame coding complexity of the corresponding image block within the previous encoded P-frame or B-frame, calculate the Lagrange multiplier of the image block to be encoded within the P-frame or B-frame to be encoded. Step S5: Perform video coding on the image block to be encoded based on the Lagrange multiplier of the image block to be encoded within the P-frame or B-frame to be encoded, and obtain its actual number of encoded bits; then, adjust the target number of encoded bits for the next image block to be encoded within the P-frame or B-frame to be encoded based on the actual number of encoded bits of the encoded image block. Specifically, the inter-frame coding complexity of each image block within the P-frame or B-frame to be encoded reuses the inter-frame coding complexity of the image block at the same position within the previous encoded P-frame or B-frame, and the inter-frame coding complexity of the P-frame or B-frame to be encoded reuses the inter-frame coding complexity of the previous encoded P-frame or B-frame.

[0012] Further, in step S1, the prediction cost corresponding to each inter-frame coding mode is calculated for each image block to be encoded, and one or more inter-frame coding modes with the lowest prediction cost are selected as inter-frame coding mode candidates; the prediction cost refers to the coding rate distortion cost.

[0013] Furthermore, step S2 calculates at least the inter-frame coding complexity at the image block level and video frame level for each P-frame and B-frame.

[0014] Furthermore, step S3 is defined by the formula: Find; where T blk_curr ω represents the target number of encoded bits for the current image block to be encoded within the current P-frame or B-frame to be encoded. blk_curr ω represents the inter-frame coding complexity of the current image block to be encoded within the current P-frame or B-frame. picT represents the inter-frame coding complexity of the current P-frame or B-frame to be encoded. Pic The target number of encoding bits for the current P-frame or B-frame to be encoded. H The number of bits required for the header information of the current P-frame or B-frame to be encoded, as estimated in advance.

[0015] Furthermore, step S4 is derived from formula five: Find λ; where λ new f represents the Lagrange multiplier of the image block to be encoded within the P-frame or B-frame to be encoded; -1 It is the inverse function of the first function f; I(n) new This represents the actual number of encoded bits for the image block to be encoded within the P-frame or B-frame to be encoded, replaced by the target number of encoded bits for the same image block within the P-frame or B-frame to be encoded; I(m) old λ represents the actual number of encoded bits for the same image block within the preceding encoded P-frame or B-frame to be encoded; g is the second function, g(1) is a constant; λ old The Lagrange multiplier represents the image block at the same position within the previous encoded P-frame or B-frame.

[0016] Furthermore, the first function f is given by Formula 1: Formula 1 states that when different image block-level Lagrange multipliers are used to perform inter-frame coding on the same image block, the ratio of the actual number of coded bits obtained after coding is related to the ratio of the image block-level Lagrange multipliers. This relationship is represented by the first function f.

[0017] Furthermore, the second function g is derived from Formula 2: Get; ω n ω represents the inter-frame coding complexity of the image block to be encoded within the P-frame or B-frame to be encoded. m Formula 2 represents the inter-frame coding complexity of the same image block within the previous encoded P-frame or B-frame to be encoded; Formula 2 means that when using the same image block-level Lagrange multipliers to perform inter-frame coding on image blocks at the same position within different video frames, the ratio of the actual number of encoded bits obtained after coding is related to the ratio of the inter-frame coding complexity of the image block. This relationship is represented by the second function g.

[0018] Furthermore, based on Formula 1 and Formula 2, Formula 3 is derived: Further derivation yields Formula 4: Where, I(n) new The number of target coded bits for the image block to be encoded within the P-frame or B-frame to be encoded is used as the replacement; assuming the preceding encoded P-frame or B-frame of the n-th frame is the m-th frame, ωn Use ω m By making substitutions, we obtain Formula 5.

[0019] Further, step S5 is defined by the formula: Find; where T blk_next T represents the target number of encoded bits for the next image block to be encoded within the current P-frame or B-frame to be encoded. Pic The target number of encoding bits for the current P-frame or B-frame to be encoded. H Coded is the pre-estimated number of bits required for the header information of the current P-frame or B-frame to be encoded. Pic ∑ represents the actual number of encoded bits in the current P-frame or B-frame to be encoded. {all not coded blks} ω blk ω is the sum of the inter-frame coding complexities of all uncoded image blocks within the current P-frame or B-frame to be encoded. blk_next This represents the inter-frame coding complexity of the next image block to be encoded within the current P-frame or B-frame.

[0020] This application also proposes a bitrate control device at the image block level for P-frames and / or B-frames, including an inter-frame coding mode prediction module, an inter-frame coding complexity calculation module, an image block bit allocation module, an image block bit control module, and an image block coding module. The inter-frame coding mode prediction module predicts the inter-frame coding mode for each image block to be encoded, obtaining candidate inter-frame coding modes and their prediction costs for that image block. The inter-frame coding complexity calculation module filters the prediction costs corresponding to each candidate inter-frame coding mode for the image block to be encoded, selecting the minimum prediction cost to characterize the inter-frame coding complexity of that image block; then, the sum of the inter-frame coding complexities of all image blocks within the video frame is taken as the inter-frame coding complexity of the video frame. The image block bit allocation module is used to calculate the target number of encoded bits for each image block within the P-frame or B-frame to be encoded before encoding, based on the ratio of the inter-frame coding complexity of the image blocks to be encoded within the P-frame or B-frame to the inter-frame coding complexity of the P-frame or B-frame itself, and in conjunction with the target number of encoded bits for the P-frame or B-frame. The image block bit allocation module is also used to adjust the target number of encoded bits for the next image block within the P-frame or B-frame to be encoded during the encoding process, based on the actual number of encoded bits of the already encoded image blocks. The image block bit control module is used to calculate the Lagrange multiplier for the image block to be encoded within the P-frame or B-frame to be encoded, based on the target number of encoded bits and the inter-frame coding complexity of the image blocks within the P-frame or B-frame to be encoded, combined with the actual number of encoded bits, Lagrange multipliers, and inter-frame coding complexity of the corresponding image blocks within the previous encoded P-frame or B-frame. The image block encoding module is used to perform video encoding on the image block to be encoded based on the Lagrange multipliers of the image blocks to be encoded within the P-frame or B-frame to be encoded, and to obtain its actual number of encoded bits. Specifically, the inter-frame encoding complexity of each image block within the P-frame or B-frame to be encoded reuses the inter-frame encoding complexity of the image block at the same position within the previous encoded P-frame or B-frame, and the inter-frame encoding complexity of the P-frame or B-frame to be encoded reuses the inter-frame encoding complexity of the previous encoded P-frame or B-frame.

[0021] The technical advantages achieved by this invention are: no additional image preprocessing stage is required, resulting in low hardware overhead and low implementation cost; the logical relationship is simple, avoiding the introduction of a large number of complex floating-point operations, thus reducing the difficulty and cost of hardware implementation. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of a common image group structure.

[0023] Figure 2 This is a flowchart illustrating the image block-level bitrate control method for P-frames and / or B-frames proposed in this application.

[0024] Figure 3 This is a schematic diagram of the same image block within the m-th and n-th frames.

[0025] Figure 4 This is a schematic diagram of the image block-level bitrate control device for P-frames and / or B-frames proposed in this application.

[0026] The following are the labels in the figure: 1 is the inter-frame coding mode prediction module, 2 is the inter-frame coding complexity calculation module, 3 is the image block bit allocation module, 4 is the image block bit control module, and 5 is the image block coding module. Detailed Implementation

[0027] Please see Figure 2 The image block-level bitrate control method for P-frames and / or B-frames suitable for hardware video encoders proposed in this application includes the following steps.

[0028] Step S1: Predict the inter-frame coding mode for each image block to be encoded, obtaining candidate inter-frame coding modes and their prediction costs. Specifically, calculate the prediction cost for each inter-frame coding mode for each image block to be encoded, and select one or more inter-frame coding modes with the lowest prediction costs as candidate inter-frame coding modes. The prediction cost refers to the coding rate distortion cost J, which has relatively low computational cost and low accuracy, J = D + λ × R. Where D is the coding distortion, generally using the SAD (Sum of Absolute Difference) algorithm or the SATD (Sum of Absolute Transformed Difference) algorithm; R is the coding bit rate, generally using a simplified bit rate estimation algorithm, such as the Exponential-Golomb coding algorithm; and generally no transformation or quantization operations are introduced during the calculation process. This step is a step in the existing video coding process, where inter-frame coding mode prediction is performed for all image blocks of P-frames and B-frames.

[0029] Both coding mode prediction and coding mode rate-distortion optimization employ rate-distortion optimization methods to filter coding modes for image blocks. The main difference lies in the precision of the calculation methods used for the coding rate-distortion cost. In coding mode rate-distortion optimization, a more computationally intensive and precise coding rate-distortion cost is generally used. Specifically, the SSD (Sum of Squared Differences) algorithm is typically used to calculate coding distortion, while the CABAC (Context Adaptive Binary Arithmetic Coding) or CAVLC (Context Adaptive Variable Length Coding) algorithm is generally used to calculate the coding bit rate. Furthermore, transformation and quantization operations are typically introduced during the calculation process.

[0030] Step S2: The prediction cost corresponding to each inter-frame coding mode candidate for the image block to be encoded is filtered, and the minimum prediction cost is selected to characterize the inter-frame coding complexity of the image block to be encoded; then, the sum of the inter-frame coding complexities of all image blocks within the video frame is taken as the inter-frame coding complexity of the video frame. Preferably, this step calculates at least the inter-frame coding complexity at the image block level and the video frame level for each P-frame and B-frame, thus serving subsequent P-frames and / or B-frames. This step reuses the calculation results of inter-frame coding mode prediction (step S1) and does not require the introduction of an additional image preprocessing stage, which is an innovation of this application.

[0031] Step S3: Based on the ratio of the inter-frame coding complexity of the image blocks to be encoded within the P-frame or B-frame to the inter-frame coding complexity of the P-frame or B-frame to be encoded, and in conjunction with the target coding bit count of the P-frame or B-frame to be encoded, calculate the target coding bit count for each image block within the P-frame or B-frame to be encoded. For the P-frame or B-frame to be encoded, the inter-frame coding complexity of each image block within it reuses the inter-frame coding complexity of the image block at the same position within the previous encoded P-frame or B-frame, and the inter-frame coding complexity of its video frames reuses the inter-frame coding complexity of the previous encoded P-frame or B-frame. If the current P-frame or B-frame to be encoded is the first P-frame or B-frame after a certain I-frame, then its corresponding previous encoded P-frame or B-frame is the last P-frame or B-frame before that I-frame. Assuming the P-frame or B-frame to be encoded is the nth frame, since encoding has not yet begun, inter-frame coding mode prediction has not started, and therefore the inter-frame coding complexity at the image block level and video frame level cannot be obtained. To solve this problem, this application utilizes the temporal correlation of consecutive video frames, i.e., consecutive video frames are similar in content. It is assumed that the content of the already encoded n-2th and n-1th frames is similar to that of the nth frame. Therefore, the inter-frame coding complexity at the image block level of the nth frame reuses the inter-frame coding complexity of image blocks at the same position in the n-1th or n-2th frame (when the n-1th frame is an I-frame). The inter-frame coding complexity at the video frame level of the nth frame reuses the inter-frame coding complexity at the video frame level of the n-1th or n-2th frame (when the n-1th frame is an I-frame). Since the n-1th and n-2th frames (when the n-1th frame is an I-frame) have already been encoded, the inter-frame coding complexity of their internal image blocks and the inter-frame coding complexity at the video frame level are known. Through this multiplexing, the target number of coding bits for each image block to be encoded can be calculated without special preprocessing before encoding of the P-frame or B-frame. Moreover, the calculation accuracy can meet the requirements of the bit rate control algorithm, which is an innovation of this application.

[0032] As an example, step S3 is defined by the formula: Find T. blk_curr ω represents the target number of encoded bits for the current image block to be encoded. blk_curr Let ω be the inter-frame coding complexity of the current image patch to be encoded. pic T represents the inter-frame coding complexity of the current P-frame or B-frame to be encoded. Pic The target number of encoding bits for the current P-frame or B-frame to be encoded. H The number of bits required for the header information of the current P-frame or B-frame to be encoded, as estimated in advance.

[0033] Step S4: Based on the target number of coding bits and inter-frame coding complexity of the image block to be encoded within the P-frame or B-frame to be encoded, and combined with the actual number of coding bits, Lagrange multipliers, and inter-frame coding complexity of the corresponding image block within the previous encoded P-frame or B-frame, calculate the Lagrange multiplier of the image block to be encoded within the P-frame or B-frame to be encoded. Similar to step S3, the inter-frame coding complexity of each image block within the P-frame or B-frame to be encoded is obtained by multiplexing the inter-frame coding complexity of the corresponding image block within the previous encoded P-frame or B-frame. Assuming the P-frame or B-frame to be encoded is the nth frame, the inter-frame coding complexity of each image block level in the nth frame is multiplexed from the inter-frame coding complexity of the image blocks at the same position within the (n-1)th and (n-2)th frames (if the (n-1)th frame is an I-frame).

[0034] As an example, step S4 is derived from formula five: Find λ. new f represents the Lagrange multiplier of the image block to be encoded within the P-frame or B-frame to be encoded. -1 It is the inverse function of the first function f. I(n) new This indicates that the image block to be encoded within the P-frame or B-frame uses λ. new The actual number of encoded bits after encoding. At this point, the image block to be encoded within the P-frame or B-frame has not yet been encoded, so its actual number of encoded bits cannot be obtained. Here, the target number of encoded bits for the image block within the P-frame or B-frame to be encoded is used as an approximation. I(m) old This represents the actual number of encoded bits for the same image block within the preceding encoded P-frame or B-frame to be encoded. g is the second function, and g(1) is a constant. λ old The Lagrange multiplier represents the image block at the same position within the previous encoded P-frame or B-frame.

[0035] Please see Figure 1 Frame m and frame n are two consecutive P-frames or B-frames, and there may be zero, one, or more video frames between them. Frame m is an encoded P-frame or B-frame, and frame n is a P-frame or B-frame to be encoded. For a certain image block position within a video frame, let λ old and λ new Let I(m) represent the image block-level Lagrange multipliers at that position within two consecutive P-frames or B-frames (i.e., the m-th and n-th frames). old This indicates that the image patch at this location within the m-th frame uses λ. old Let I(m) be the actual number of encoded bits after encoding. new This indicates that the image patch at this location within the m-th frame uses λ. new Let I(n) be the actual number of encoded bits after encoding. oldThis indicates that the image patch at that location within the nth frame uses λ. old Let I(n) be the actual number of encoded bits after encoding. new This indicates that the image patch at that location within the nth frame uses λ. new The actual number of encoded bits after encoding, let ω m Let ω represent the inter-frame coding complexity of the image patch at that position within the m-th frame. n This represents the inter-frame coding complexity of the image block at that position within the nth frame. Figure 3 This is a schematic diagram of the same image block within the m-th and n-th frames.

[0036] The first function f is given by formula one: The physical meaning of Formula 1 is: when different image block-level Lagrange multipliers are used to perform inter-frame coding on the same image block, the ratio of the actual number of coded bits obtained after coding is related to the ratio of the image block-level Lagrange multipliers. This relationship can be represented by the first function f. The specific expression of the first function f can be obtained by fitting a function to both sides of the equal sign in Formula 1, which will not be elaborated here.

[0037] The second function g is derived from formula two: The physical meaning of Formula 2 is as follows: When using the same image block-level Lagrange multipliers to perform inter-frame coding on image blocks at the same position within different video frames, the ratio of the actual number of encoded bits obtained after coding is related to the ratio of the inter-frame coding complexity of the image blocks. This relationship can be represented by the second function g. The specific expression of the second function g can be obtained by fitting a function to both sides of the equality in Formula 2, which will not be elaborated here.

[0038] In its specific implementation, this invention pre-determines the expressions for the first function f and the second function g through extensive experiments and function fitting, storing these expressions in the video encoder. These expressions represent known functional relationships. The first function f and the second function g contain only a small number of floating-point multiplication and division operations, and do not include complex floating-point function operations (such as floating-point exponentiation and floating-point logarithmic operations). The hardware implementation difficulty and cost are far lower than common bitrate control algorithms.

[0039] Based on the derivation of Formula 1 and Formula 2, Formula 3 can be obtained: Further derivation yields Formula 4: Among them, f -1 It is the inverse function of the first function f. Since the first function f is known, then f -1 This is also known. In Formula 4, the m-th frame is an encoded P-frame or B-frame, whose encoding process has been completed. Therefore, the actual number of encoded bits I(m) corresponding to the same position image block within it is...old The image block-level Lagrange multiplier λ used in encoding old The complexity ω of image block-level inter-frame coding m All of these are known quantities. The nth frame is either a P-frame or a B-frame to be encoded, I(n) new Originally, this represented the actual number of encoded bits for the image blocks within a P-frame or B-frame after encoding. However, since the nth frame has not yet begun encoding, the actual number of encoded bits for its image blocks is unavailable. To derive the Lagrange multipliers required for the encoding process of the image blocks before encoding, this application approximates the actual number of encoded bits by using the target number of encoded bits for the image blocks. Since the "target number of encoded bits" for the image blocks is the expected "actual number of encoded bits" after the actual encoding process, this approximation introduces a very small error. I(n) new The number of target coded bits within the image block to be encoded within the P-frame or B-frame to be encoded is used as the replacement; this is a known quantity. ω n λ represents the inter-frame coding complexity of the image block to be encoded at the same position within the nth frame. Since this image block has not yet been encoded, it is an unknown quantity. new This represents the situation where the actual number of encoded bits for the same image block at the same position within the nth frame is equal to the desired target number of encoded bits, I(n). new At this time, the required image block-level Lagrange multipliers are unknown quantities, and also quantities that need to be calculated. This is to calculate the Lagrange multipliers λ of the image blocks to be encoded within the P-frame or B-frame using Formula 4. new This application utilizes the temporal correlation of consecutive video frames to reuse the inter-frame coding complexity of the image block at the same position in the preceding encoded P-frame or B-frame adjacent to the n-th frame, replacing the inter-frame coding complexity of the image block to be encoded within the n-th frame. Assuming the preceding encoded P-frame or B-frame is the m-th frame, since the sampling times of the m-th and n-th frames are very close, the content of the two video frames is very similar. Therefore, the error in replacing the inter-frame coding complexity of the image block at the same position is small, and it will not affect the Lagrange multiplier λ of the image block to be encoded. new The calculation is greatly affected. After this substitution, ω n ≈ω m Then we can get Formula 5.

[0040] In step S4, the Lagrange multiplier λ of the image block to be encoded within the P-frame or B-frame is derived using information from the encoded image blocks within the encoded P-frame or B-frame. new The process, and the replacement operation that utilizes the temporal correlation of consecutive video frames to perform inter-frame coding complexity of the image blocks to be coded within the P-frame or B-frame to be coded, are innovations of this invention.

[0041] Step S5: Based on the Lagrange multiplier λ of the image block to be encoded within the P-frame or B-frame to be encoded. new The image block to be encoded is video encoded to obtain its actual number of encoded bits. Then, based on the actual number of encoded bits of the encoded image block, the target number of encoded bits for the next image block to be encoded within the P-frame or B-frame to be encoded is adjusted.

[0042] As an example, the latter part of step S5 is derived from the formula: Find T. blk_next T represents the target number of encoded bits for the next image block to be encoded within the current P-frame or B-frame to be encoded. Pic The target number of encoding bits for the current P-frame or B-frame to be encoded. H Coded is the pre-estimated number of bits required for the header information of the current P-frame or B-frame to be encoded. Pic ω represents the actual number of encoded bits in the current P-frame or B-frame to be encoded (including the actual number of encoded bits in the previous image block that was just encoded). blk ∑ represents the inter-frame coding complexity of a specific uncoded image block within the current P-frame or B-frame to be encoded. {all not coded blks} ω blk ω is the sum of the inter-frame coding complexities of all uncoded image blocks within the current P-frame or B-frame to be encoded. blk_next This is the inter-frame coding complexity of the next image block to be encoded within the current P-frame or B-frame to be encoded. Similar to step S3, the inter-frame coding complexity of each image block within the P-frame or B-frame to be encoded is obtained by multiplexing the inter-frame coding complexity of the image block at the same position within the previous encoded P-frame or B-frame.

[0043] For a single image block within a P-frame or B-frame to be encoded, the bit allocation process actually involves two steps. The first step is before encoding of the P-frame or B-frame, i.e., before encoding begins for any of the image blocks within that P-frame or B-frame. This is called "pre-allocation," and this process is described in step S3. The second step is during the encoding process of the P-frame or B-frame, i.e., when some image blocks within that P-frame or B-frame have been encoded while others have not. This process involves reallocating bits for the unencoded image blocks within that P-frame or B-frame (called "reallocation"), and this process is described in step S5. The reason for this bit reallocation is that the actual number of bits after encoding a block may differ from the number of bits allocated during the "pre-allocation" process. Therefore, for the remaining unencoded image blocks, the target number of bits needs to be reallocated based on the actual number of bits encoded in the already encoded image blocks.

[0044] Please see Figure 4 The image block-level bitrate control device for P-frames and / or B-frames proposed in this application, suitable for hardware video encoders, includes an inter-frame coding mode prediction module 1, an inter-frame coding complexity calculation module 2, an image block bit allocation module 3, an image block bit control module 4, and an image block coding module 5. Figure 4 The device shown corresponds to Figure 2 The method shown.

[0045] The inter-frame coding mode prediction module 1 is used to predict the inter-frame coding mode of each image block to be encoded, and obtain the inter-frame coding mode candidate and its prediction cost for the image block to be encoded.

[0046] The inter-frame coding complexity calculation module 2 is used to filter the prediction cost corresponding to each inter-frame coding mode candidate of the image block to be encoded, and select the minimum prediction cost to characterize the inter-frame coding complexity of the image block to be encoded; then the sum of the inter-frame coding complexities of all image blocks in the video frame is used as the inter-frame coding complexity of the video frame.

[0047] The image block bit allocation module 3 is used to calculate the target number of encoded bits for each image block within the P-frame or B-frame before encoding, based on the ratio of the inter-frame coding complexity of the image blocks to be encoded within the P-frame or B-frame to the inter-frame coding complexity of the P-frame or B-frame itself, and in conjunction with the target number of encoded bits for the P-frame or B-frame. The image block bit allocation module 3 is also used to adjust the target number of encoded bits for the next image block within the P-frame or B-frame during the encoding process, based on the actual number of encoded bits for the already encoded image blocks.

[0048] The image block bit control module 4 is used to calculate the Lagrange multiplier of the image block to be encoded within the P-frame or B-frame to be encoded, based on the target number of encoded bits and the inter-frame coding complexity of the image block to be encoded within the P-frame or B-frame to be encoded, combined with the actual number of encoded bits, Lagrange multipliers and inter-frame coding complexity of the co-located image block within the previous encoded P-frame or B-frame.

[0049] The image block encoding module 5 is used to determine the Lagrange multiplier λ of the image block to be encoded within the P-frame or B-frame to be encoded. new Perform video encoding on the image block to be encoded and obtain its actual number of encoded bits.

[0050] When the above modules perform calculations, the inter-frame coding complexity of each image block within the P-frame or B-frame to be encoded reuses the inter-frame coding complexity of the image block at the same position within the previous encoded P-frame or B-frame, and the inter-frame coding complexity of the P-frame or B-frame to be encoded reuses the inter-frame coding complexity of the previous encoded P-frame or B-frame.

[0051] Chinese invention patent application CN115550656A, published on December 30, 2022, entitled "A Method and Apparatus for I-Frame Rate Control Suitable for Hardware Implementation," discloses a video frame-level rate control method suitable for hardware implementation of I-frames. Compared with CN115550656A, the main technical innovations of this application are as follows.

[0052] First, the objective of CN115550656A is to calculate the value of the video frame-level Lagrange multiplier for I-frames. For an image frame, there is only one value for the frame-level Lagrange multiplier. The video frame-level Lagrange multiplier can be used for frame-level bitrate control of I-frames, as well as for image block-level bitrate control. When used for image block-level bitrate control, the video frame-level Lagrange multiplier is generally used as a benchmark for threshold conversion to constrain the value of the image block-level Lagrange multiplier. This application is applicable to image block-level bitrate control of P-frames and / or B-frames, and its objective is to calculate the value of the image block-level Lagrange multiplier for each image block within a P-frame and / or B-frame, for use in image block-level bitrate control of P-frames or B-frames.

[0053] Second, both CN115550656A and this application contain formulas one through five, and some formulas are written in the same way. However, the meanings of the variable values ​​in the formulas of the two documents are completely different. The variables in the formulas of CN115550656A are all frame-level variables, while the variables in the formulas of this application are all image block-level variables. For example, in CN115550656A, λ old and λ new All are frame-level Lagrange multipliers, λ in this applicationold and λ new They are all image block-level Lagrange multipliers.

[0054] Third, in this application, to perform image block-level bit allocation, the ratio of the inter-frame coding complexity of the image block within a P-frame or B-frame to the inter-frame coding complexity of the P-frame or B-frame to be encoded is used as the basis for image block bit allocation. CN115550656A, because it uses frame-level bitrate control, does not have an image block-level bit allocation process. Furthermore, in the image block-level bit allocation process of this application, the inter-frame coding complexity of the image block reuses the inter-frame coding complexity of the image block at the same position within the previously encoded P-frame or B-frame, and the inter-frame coding complexity of the video frame reuses the inter-frame coding complexity of the previously encoded P-frame or B-frame. Moreover, the calculation of the inter-frame coding complexity reuses the inter-frame coding mode prediction value, eliminating the need for an additional image preprocessing stage.

[0055] Fourth, in this application, after a certain image block within a P-frame or B-frame to be encoded is encoded, the target number of encoded bits for the next image block to be encoded in the P-frame or B-frame to be encoded needs to be adjusted based on the actual number of encoded bits for that image block. CN115550656A, because it uses frame-level bitrate control, does not have this process.

[0056] The bitrate control method at the P-frame and / or B-frame image block level proposed in this application utilizes the inter-frame coding mode prediction cost generated during the encoding process of image blocks within P-frames or B-frames to characterize the inter-frame coding complexity of image blocks within a video frame, and accordingly completes the target bit allocation for image blocks within P-frames or B-frames. This eliminates the need for additional image preprocessing stages, resulting in low hardware overhead and low implementation cost. Furthermore, this application utilizes information from already encoded image blocks within encoded P-frames or B-frames to calculate the Lagrange multipliers of the image blocks to be encoded within the P-frames or B-frames to be encoded. The logic is simple, avoiding the introduction of numerous complex floating-point operations, effectively reducing the difficulty and cost of hardware implementation.

[0057] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A bitrate control method at the image block level for P-frames and / or B-frames, characterized in that, Includes the following steps; Step S1: Predict the inter-frame coding mode for each image block to be encoded to obtain the inter-frame coding mode candidate and its prediction cost for the image block to be encoded. Step S2: Filter the prediction costs corresponding to each inter-frame coding mode candidate of the image block to be encoded, and select the minimum prediction cost to characterize the inter-frame coding complexity of the image block to be encoded; then take the sum of the inter-frame coding complexities of all image blocks in the video frame as the inter-frame coding complexity of the video frame. Step S3: Calculate the target coding number of each image block within the P-frame or B-frame to be encoded based on the ratio of the inter-frame coding complexity of the image block to be encoded to the inter-frame coding complexity of the P-frame or B-frame to be encoded, and in conjunction with the target coding number of the P-frame or B-frame to be encoded. Step S4: Based on the target number of coding bits and inter-frame coding complexity of the image block to be encoded within the P-frame or B-frame to be encoded, and combined with the actual number of coding bits, Lagrange multipliers, and inter-frame coding complexity of the co-located image block within the previous encoded P-frame or B-frame, calculate the Lagrange multipliers of the image block to be encoded within the P-frame or B-frame to be encoded. Step S5: Perform video encoding on the image block to be encoded according to the Lagrange multipliers of the image block to be encoded within the P-frame or B-frame to be encoded, and obtain its actual number of encoded bits; Then, the target number of encoded bits for the next image block to be encoded within the P-frame or B-frame to be encoded is adjusted based on the actual number of encoded bits for the encoded image block. Specifically, when the inter-frame coding complexity of the image blocks within the P-frame or B-frame to be encoded cannot be obtained, the inter-frame coding complexity of each image block within the P-frame or B-frame to be encoded reuses the inter-frame coding complexity of the image block at the same position within the previous encoded P-frame or B-frame; when the inter-frame coding complexity of the P-frame or B-frame to be encoded cannot be obtained, the inter-frame coding complexity of the P-frame or B-frame to be encoded reuses the inter-frame coding complexity of the previous encoded P-frame or B-frame.

2. The image block-level bitrate control method for P-frames and / or B-frames according to claim 1, characterized in that, In step S1, the prediction cost corresponding to each inter-frame coding mode is calculated for each image block to be encoded, and one or more inter-frame coding modes with the lowest prediction cost are selected as inter-frame coding mode candidates; the prediction cost refers to the coding rate distortion cost.

3. The image block-level bitrate control method for P-frames and / or B-frames according to claim 1, characterized in that, Step S2 calculates at least the inter-frame coding complexity at the image block level and video frame level for each P-frame and B-frame.

4. The image block-level bitrate control method for P-frames and / or B-frames according to claim 1, characterized in that, Step S3 is defined by the formula: Find; where T blk_curr ω represents the target number of encoded bits for the current image block to be encoded within the current P-frame or B-frame to be encoded. blk_curr ω represents the inter-frame coding complexity of the current image block to be encoded within the current P-frame or B-frame. pic T represents the inter-frame coding complexity of the current P-frame or B-frame to be encoded. Pic The target number of encoding bits for the current P-frame or B-frame to be encoded. H The number of bits required for the header information of the current P-frame or B-frame to be encoded, as estimated in advance.

5. The image block-level bitrate control method for P-frames and / or B-frames according to claim 1, characterized in that, Step S4 is derived from Formula 5: Find λ; where λ new f represents the Lagrange multiplier of the image block to be encoded within the P-frame or B-frame to be encoded; -1 It is the inverse function of the first function f; I(n) new This represents the actual number of encoded bits for the image block to be encoded within the P-frame or B-frame to be encoded, replaced by the target number of encoded bits for the same image block within the P-frame or B-frame to be encoded; I(m) old λ represents the actual number of encoded bits for the same image block within the preceding encoded P-frame or B-frame to be encoded; g is the second function, g(1) is a constant; λ old The Lagrange multiplier represents the image block at the same position within the previous encoded P-frame or B-frame.

6. The image block-level bitrate control method for P-frames and / or B-frames according to claim 5, characterized in that, The first function f is given by formula one: Formula 1 states that when different image block-level Lagrange multipliers are used to perform inter-frame coding on the same image block, the ratio of the actual number of coded bits obtained after coding is related to the ratio of the image block-level Lagrange multipliers. This relationship is represented by the first function f.

7. The image block-level bitrate control method for P-frames and / or B-frames according to claim 6, characterized in that, The second function g is derived from formula two: Get; ω n ω represents the inter-frame coding complexity of the image block to be encoded within the P-frame or B-frame to be encoded. m Formula 2 represents the inter-frame coding complexity of the same image block within the previous encoded P-frame or B-frame to be encoded; Formula 2 means that when using the same image block-level Lagrange multipliers to perform inter-frame coding on image blocks at the same position within different video frames, the ratio of the actual number of encoded bits obtained after coding is related to the ratio of the inter-frame coding complexity of the image block. This relationship is represented by the second function g.

8. The image block-level bitrate control method for P-frames and / or B-frames according to claim 7, characterized in that, Based on the derivation of Formula 1 and Formula 2, Formula 3 is obtained: Further derivation yields Formula 4: Where, I(n) new The number of target coded bits for the image block to be encoded within the P-frame or B-frame to be encoded is used as the replacement; assuming the preceding encoded P-frame or B-frame of the n-th frame is the m-th frame, ω n Use ω m By making substitutions, we obtain Formula 5.

9. The image block-level bitrate control method for P-frames and / or B-frames according to claim 1, characterized in that, Step S5 is defined by the formula: Find; where T blk_next T represents the target number of encoded bits for the next image block to be encoded within the current P-frame or B-frame to be encoded. Pic The target number of encoding bits for the current P-frame or B-frame to be encoded. H Coded is the pre-estimated number of bits required for the header information of the current P-frame or B-frame to be encoded. Pic ∑ represents the actual number of encoded bits in the current P-frame or B-frame to be encoded. {allnotcodedblks} ω blk ω is the sum of the inter-frame coding complexities of all uncoded image blocks within the current P-frame or B-frame to be encoded. blk_next This represents the inter-frame coding complexity of the next image block to be encoded within the current P-frame or B-frame.

10. A bitrate control device at the image block level for P-frames and / or B-frames, characterized in that, It includes an inter-frame coding mode prediction module, an inter-frame coding complexity calculation module, an image block bit allocation module, an image block bit control module, and an image block coding module; The inter-frame coding mode prediction module is used to predict the inter-frame coding mode of each image block to be encoded, and obtain the inter-frame coding mode candidate and its prediction cost for the image block to be encoded. The inter-frame coding complexity calculation module is used to filter the prediction cost corresponding to each inter-frame coding mode candidate of the image block to be encoded, and select the minimum prediction cost to characterize the inter-frame coding complexity of the image block to be encoded; then the sum of the inter-frame coding complexities of all image blocks in the video frame is used as the inter-frame coding complexity of the video frame. The image block bit allocation module is used to calculate the target number of encoded bits for each image block within the P-frame or B-frame before encoding, based on the ratio of the inter-frame coding complexity of the image blocks to be encoded within the P-frame or B-frame to the inter-frame coding complexity of the P-frame or B-frame itself, and in conjunction with the target number of encoded bits for the P-frame or B-frame. The image block bit allocation module is also used to adjust the target number of encoded bits for the next image block within the P-frame or B-frame during the encoding process, based on the actual number of encoded bits for the already encoded image blocks. The image block bit control module is used to calculate the Lagrange multiplier of the image block to be encoded within the P-frame or B-frame to be encoded, based on the target number of encoded bits and the inter-frame coding complexity of the image block to be encoded within the P-frame or B-frame to be encoded, combined with the actual number of encoded bits, Lagrange multipliers and inter-frame coding complexity of the co-position image block within the previous encoded P-frame or B-frame. The image block encoding module is used to perform video encoding on the image block to be encoded according to the Lagrange multipliers of the image block to be encoded within the P-frame or B-frame to be encoded, and to obtain its actual number of encoded bits. Specifically, when the inter-frame coding complexity of the image blocks within the P-frame or B-frame to be encoded cannot be obtained, the inter-frame coding complexity of each image block within the P-frame or B-frame to be encoded reuses the inter-frame coding complexity of the image block at the same position within the previous encoded P-frame or B-frame; when the inter-frame coding complexity of the P-frame or B-frame to be encoded cannot be obtained, the inter-frame coding complexity of the P-frame or B-frame to be encoded reuses the inter-frame coding complexity of the previous encoded P-frame or B-frame.

Images