Parallel coding rate control method and device based on virtual frame
By dividing video frames into virtual frames and scheduling them, the problem of inconsistent bitrate control in WPP intra-frame parallel coding is solved, the complexity of hardware design is reduced, and the consistency and efficiency of coding results are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-02-02
- Publication Date
- 2026-04-10
AI Technical Summary
Existing WPP intra-frame parallel coding technology cannot guarantee the consistency of bitrate control during multiple encoding operations, and the strict synchronization requirements increase the complexity of hardware design.
A parallel coding method based on virtual frames is adopted, which divides the video frame to be encoded into multiple independent virtual frames and performs scheduling and synchronization based on the virtual frames. Through target bit allocation and control, the consistency of the coding results is ensured.
While meeting the basic synchronization requirements of WPP technology, consistency of multiple encoding results was achieved, reducing the design difficulty of multi-VPU hardware encoding systems.
Smart Images
Figure CN121842391A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to a digital video coding technology, in particular to a rate control method for intra parallel coding. BACKGROUND
[0002] Video coding is a technology to represent video information using as few data as possible by compressing the redundancy components in video images. Common video coding standards include HEVC (High Efficiency Video Coding, also known as H.265), AVC (Advanced Video Coding, also known as H.264), etc.
[0003] Video coding technology takes image blocks as the most basic coding unit. For example, in HEVC, the basic unit of coding is CU (coding unit). The CU can be an image block of 64 pixels x 64 pixels, 32 pixels x 32 pixels, 16 pixels x 16 pixels, or 8 pixels x 8 pixels. Among them, the image block of 64 pixels x 64 pixels is also called LCU (largest coding unit).
[0004] In order to improve the video compression rate as much as possible, common video coding algorithms often use lossy compression algorithms, which cause the reconstructed video after coding to be different from the original video, i.e., the reconstructed video after coding will be distorted. HEVC and AVC belong to lossy compression algorithms. For lossy compression algorithms, the coding performance needs to be measured by the coding bit rate (rate) and the coding distortion (distortion) introduced. In the video coding process, the coding bit rate and the coding distortion are mutually restrictive and contradictory, such as reducing the coding bit rate often increases the coding distortion, and reducing the coding distortion often leads to an increase in the coding bit rate. Therefore, the rate distortion optimization (RDO) technology is generated, which aims to reduce the coding distortion as much as possible under a certain coding bit rate, or to reduce the coding bit rate as much as possible under a certain coding distortion. At present, common video coding algorithms all use the rate distortion optimization method based on Lagrange multiplier λ. Assuming that D is the coding distortion, R is the coding bit rate, and J is the coding rate distortion cost (RD Cost), then J = D + λ x R, and the rate distortion optimization method based on Lagrange multiplier λ is to find the minimum value of J.
[0005] In video encoding, each encoding mode has its corresponding encoding distortion D and encoding bit rate R, allowing the calculation of the rate-distortion cost J for each encoding mode. During encoding, selecting the encoding mode with the lowest rate-distortion cost J yields optimal encoding performance. The process by which the encoder selects the optimal encoding mode for each image block to be encoded using rate-distortion optimization methods is called mode decision for the image block.
[0006] In real-world scenarios, the channel capacity used to transmit compressed video is limited. If the encoding bitrate of the compressed video is too high, exceeding the channel capacity, it will cause video transmission congestion or even packet loss. If the encoding bitrate of the compressed video is too low, the channel capacity will not be fully utilized, and higher video quality cannot be obtained. Therefore, it is necessary to use rate control technology to control the output bitrate of the video encoder to match it with the channel capacity.
[0007] Bitrate control technology adjusts the encoding parameters of the video encoder to make the output bitrate of the video encoder equal to a pre-set target bitrate, while minimizing encoding distortion to improve video encoding quality. Common bitrate control algorithms typically achieve this through two stages: target bit allocation and target bit control. The target bit allocation stage calculates the target number of coded bits for each image unit to be encoded (such as a group of pictures (GOP), video frame, or image block). The target bit control stage, based on the target number of coded bits for each image unit, calculates its corresponding encoding parameters (such as the Lagrange multiplier λ or quantization parameter QP) for use in the encoding process of that image unit.
[0008] Because video encoding algorithms involve a large amount of computation, using application-specific integrated circuits (ASICs) to accelerate the video encoding process in hardware has become a common industry practice in order to improve video encoding speed and achieve real-time encoding and transmission. Such ASICs that accelerate the video encoding process in hardware are generally called hardware video processing units (VPUs).
[0009] With the development of video coding technology, the compression rate of video coding is getting higher and higher, and the amount of computation in video coding is also increasing. Especially for videos with extremely high resolution and extremely high frame rate, real-time coding must be guaranteed through parallel coding technology.
[0010] One parallel coding technique is intra-frame parallel coding, which distributes image blocks at different locations within a video frame to be encoded onto different coding cores for parallel encoding, thereby improving the speed of video encoding. The coding core refers to a video coding unit that can work in parallel; it can be an encoding thread in a software video encoder or a hardware encoding module in a hardware video encoder.
[0011] For example, HEVC video coding technology can use image segmentation or WPP (wavefront parallel processing) technology to achieve intra-frame parallel coding.
[0012] Intra-frame parallel coding with image segmentation divides video frames spatially into multiple slices and / or tiles, distributing each slice or tile to different coding cores for parallel encoding. This coding method often suffers from significant coding efficiency losses and is prone to blocking artifacts at the boundaries of slices or tiles. Video encoded using intra-frame parallel coding with image segmentation is also susceptible to decoding compatibility issues during decoding.
[0013] WPP's intra-frame parallel coding technique distributes different LCU lines to different coding cores for parallel coding, requiring only that the coding of a particular LCU line lags behind the coding of the preceding LCU line by at least two LCUs. WPP's intra-frame parallel coding technique does not suffer from block artifacts or decoding compatibility issues.
[0014] While WPP's intra-frame parallel coding technology offers significant advantages, its rate control process presents a challenge. This is because rate control is a technique with strong spatiotemporal dependencies, and its final control effect is closely related to the processing order of image blocks during actual encoding. When using WPP's intra-frame parallel coding technology, strict synchronization between the various coding cores is essential to ensure that image blocks within a video frame are encoded sequentially in a fixed order. This guarantees consistent rate control when encoding the same video frame. For hardware parallel coding systems containing multiple VPUs, this synchronization requirement significantly increases the complexity of the hardware design.
[0015] Currently, when using WPP technology for intra-frame parallel coding, there are two main types of existing rate control methods.
[0016] The first type of method does not strictly synchronize the encoding cores of intra-frame parallel coding, but only needs to meet the basic requirement of WPP technology that there is a lag of at least two LCUs between adjacent LCU lines. This type of method is relatively simple to implement, but it cannot guarantee consistent bitrate control when encoding the same video frame multiple times, which does not meet the consistency requirement of video encoders for bitrate control.
[0017] The second type of method involves strict synchronization of the encoding cores of intra-frame parallel coding. This not only meets the basic requirement of WPP technology that adjacent LCU rows lag by at least two LCUs, but also ensures that image blocks within a video frame are encoded sequentially in a fixed order. This type of method can guarantee consistent bitrate control when encoding the same video frame multiple times. However, when implementing this in hardware, such synchronization requirements significantly increase the complexity of the hardware design. Summary of the Invention
[0018] The technical problem to be solved by this invention is how to improve WPP's intra-frame parallel coding technology so that it can ensure consistent bit rate control during multiple coding operations without requiring strict synchronization (only basic synchronization requirements need to be met).
[0019] To address the aforementioned technical problems, this invention proposes a rate control method for parallel coding based on virtual frames, comprising the following steps: Step S1: Divide the video frame to be encoded into multiple independent and non-overlapping subframes. Step S2: Based on the number of coding cores participating in intra-frame parallel coding, divide the video frame to be encoded into multiple virtual frames according to the virtual frame division rules. The virtual frame division rules are as follows: according to the number of coding cores participating in intra-frame parallel coding, each image block row within the first subframe of the video frame to be encoded is sequentially and alternately allocated to each coding core. Once the image block rows within one subframe are allocated, the allocation proceeds to the next subframe, until the image block rows within all subframes of the video frame to be encoded are allocated. All the image block rows allocated to each coding core within the video frame to be encoded constitute a virtual frame. Steps S1 and S2 may be performed simultaneously or arbitrarily preceded. Step S3: Based on the virtual frames, schedule the coding cores participating in intra-frame parallel coding, and schedule each virtual frame inside the video frame to be encoded to different coding cores for intra-frame parallel coding.
[0020] Furthermore, in step S1, the video frame to be encoded is divided into multiple slices using HEVC video coding technology, with each slice being a subframe; or, the video frame to be encoded is divided into multiple tiles using HEVC video coding technology, with each tile being a subframe.
[0021] Furthermore, in step S2, the virtual frame is composed of image block rows; in H.265 encoding, the virtual frame is composed of maximum coding unit (LCU) rows, and in H.264 encoding, the virtual frame is composed of macroblock (MB) rows.
[0022] Furthermore, in step S2, the number of virtual frames within a video frame to be encoded is equal to the number of encoding cores participating in intra-frame parallel encoding.
[0023] Furthermore, in step S2, each virtual frame is confined within a video frame, but each virtual frame spans a single subframe; multiple subframes are allowed to be encoded simultaneously.
[0024] Furthermore, in step S3, each encoding core records the encoding status information of each image block within the virtual frame when encoding a virtual frame; when all image blocks within the virtual frame are encoded, the encoding status information of the encoded virtual frames is statistically analyzed; when all virtual frames within the video frame are encoded, synchronization is performed among the encoding cores, and the encoding status information of the encoded video frames is statistically analyzed; the encoding status information includes one or more of the actual number of encoded bits, encoding distortion D, encoding complexity, Lagrange multiplier λ, and quantization parameter QP.
[0025] Furthermore, the target bit allocation has four levels, from largest to smallest: image group, video frame, virtual frame, and image block; intra-frame parallel coding is performed at the virtual frame level; each virtual frame includes multiple image block rows in multiple subframes within a video frame, that is, each virtual frame is composed of multiple separate regions within a video frame.
[0026] Furthermore, in step S3, the encoding complexity of the virtual frame is obtained by summing the encoding complexities of all image blocks within the virtual frame; the actual number of encoded bits of the virtual frame is obtained by summing the actual number of encoded bits of all image blocks within the virtual frame; when all virtual frames within a video frame are encoded, the encoding complexity of the video frame is obtained by summing the encoding complexities of all virtual frames within the video frame, and the actual number of encoded bits of the video frame is obtained by summing the actual number of encoded bits of all virtual frames within the video frame.
[0027] Furthermore, step S3 further includes the following sub-steps. Step S31: Based on the target number of encoded bits R of the video frame to be encoded. frame The encoding complexity ω of the video frame to be encoded frame The encoding complexity ω of the virtual frame to be encoded virtual_frame The target bit allocation is performed on each virtual frame within the video frame to be encoded to obtain the target encoded bit count R for each virtual frame. virtual_frame . (Formula 1); ω frameAn approximation is made using the encoding complexity of previously encoded video frames, ω. virtual_frame Approximate substitution is performed using the encoding complexity of the virtual frame at the same position in the previously encoded video frame. Step S32: Based on the target number of encoded bits R of the virtual frame to be encoded. virtual_frame The encoding complexity ω of the virtual frame to be encoded virtual_frame The encoding complexity ω of the image block to be encoded blk The target bit allocation is performed on the first image block within the virtual frame to be encoded, resulting in the target number of encoded bits R for the first image block. blk ; (Formula 2); ω virtual_frame An approximation is made using the encoding complexity of a virtual frame at the same position in a previously encoded video frame, ω. blk An approximate substitution is performed using the encoding complexity of an image block at the same position in a previously encoded virtual frame. When an image block within a virtual frame is fully encoded, the target number of encoded bits R of the virtual frame to be encoded is used. virtual_frame The sum of the actual encoded bits of all encoded image blocks within the virtual frame to be encoded. virtual_frame The encoding complexity ω of the next image to be encoded blk_next The sum of the encoding complexity Σ of all image blocks to be encoded within the virtual frame to be encoded {all not coded blks} ω blk Compensated target bit allocation is performed on the second and subsequent image blocks within the virtual frame to be encoded, resulting in the target number of encoded bits R for the second and subsequent image blocks to be encoded. blk_next ; (Formula 3); ω blk_next An approximation is made using the encoding complexity of the image patch at the same position in a previously encoded virtual frame, Σ. {all not coded blks} ω blk Approximate substitution is performed using the sum of the encoding complexities of all image blocks at the same position in the previously encoded virtual frame. Step S33: Based on the target number of encoded bits for each image block within the virtual frame to be encoded, and the encoding state information of the image blocks at the same position in the previously encoded virtual frame, target bit control is performed on each image block within the virtual frame to be encoded to obtain the Lagrange multiplier λ and quantization parameter QP for each image block, which are used in the actual encoding process of the image block.
[0028] This invention also proposes a rate control device for parallel coding based on virtual frames, including a subframe partitioning module, a virtual frame partitioning module, and a virtual frame coding task scheduling module. The subframe partitioning module is used to divide the video frame to be encoded into multiple independent and non-overlapping subframes. The virtual frame partitioning module is used to divide the video frame to be encoded into multiple virtual frames according to the number of coding cores participating in intra-frame parallel coding and according to the virtual frame partitioning rules. The virtual frame partitioning rules are as follows: according to the number of coding cores participating in intra-frame parallel coding, each image block row within the first subframe of the video frame to be encoded is sequentially and alternately allocated to each coding core. After the image block rows within one subframe are allocated, the allocation of image block rows within the next subframe begins, until the image block rows within all subframes of the video frame to be encoded are allocated. All the image block rows within the video frame to be encoded allocated to each coding core constitute a virtual frame. The virtual frame coding task scheduling module is used to schedule the coding cores participating in intra-frame parallel coding based on the virtual frames, scheduling each virtual frame within the video frame to be encoded to different coding cores for intra-frame parallel coding.
[0029] The technical effects achieved by this invention are as follows: When using WPP technology for intra-frame parallel coding, by reasonably dividing the video frames to be encoded into virtual frames and scheduling and synchronizing the bitrate control process of the video frames based on the virtual frames, the consistency of the results when encoding video frames multiple times is ensured; at the same time, each coding core only needs to meet the basic synchronization requirements of WPP technology itself when working, without the need to adopt other strict synchronization measures, thus reducing the design difficulty of multi-VPU hardware coding systems. Attached Figure Description
[0030] Figure 1 This is a flowchart illustrating the rate control method based on virtual frames for parallel coding proposed in this invention.
[0031] Figure 2 This is a schematic diagram of a video frame.
[0032] Figure 3 This is a diagram illustrating how a video frame is divided into four subframes.
[0033] Figure 4 This is a diagram illustrating how a video frame is divided into four virtual subframes.
[0034] Figure 5 This is a schematic diagram of the hierarchy of existing target bit allocation.
[0035] Figure 6 This is a hierarchical diagram of the target bit allocation of the present invention.
[0036] Figure 7 yes Figure 1The detailed flowchart of step S3 is shown below.
[0037] Figure 8 This is a schematic diagram of the rate control device for parallel coding based on virtual frames proposed in this invention.
[0038] The attached diagrams are labeled as follows: Subframe partitioning module 1, Virtual frame partitioning module 2, Virtual frame encoding task scheduling module 3. Detailed Implementation
[0039] Video bitrate control technology generally includes two stages: target bit allocation and target bit control. In existing technologies, the target bit allocation stage first calculates the target coded bit count for a group of pictures (GOP), and then calculates the target coded bit count for the video frames within that GOP. The video bitrate control method proposed in this invention starts with the target coded bit count R of the video frames already obtained. frame Then, improvements were made to the subsequent parts.
[0040] Please see Figure 1 The rate control method for parallel coding based on virtual frames proposed in this invention includes the following steps.
[0041] Step S1: Divide the video frame to be encoded into multiple independent and non-overlapping subframes. For example, this step uses HEVC video coding technology to divide the video frame to be encoded into multiple slices, with each slice serving as a subframe. Alternatively, this step uses HEVC video coding technology to divide the video frame to be encoded into multiple tiles, with each tile serving as a subframe. It is important to note that this invention does not use slices or tiles as scheduling units for parallel encoding.
[0042] Please see Figure 2 This is a video frame consisting of 12 image block rows. For example, in H.265 encoding, an image block row refers to an LCU (maximum coding unit) row, while in H.264 encoding, an image block row refers to an MB (micro block) row.
[0043] Please see Figure 3 This is to Figure 2 The video frame shown is divided into four subframes. Specifically, in... Figure 2 The video frame shown is divided into two parts in both width and height, resulting in four subframes of the same size. The width of each subframe is equal to half the width of a row of image blocks, and the height of each subframe is equal to the sum of the heights of the six rows of image blocks.
[0044] Step S2: Based on the number of coding cores participating in intra-frame parallel coding, the video frame to be encoded is divided into multiple virtual frames according to the virtual frame partitioning rules. The virtual frame partitioning rules are as follows: according to the number of coding cores participating in intra-frame parallel coding, each image block row within the first subframe of the video frame to be encoded is sequentially allocated to each coding core in turn. Once the image block rows within one subframe are allocated, the allocation proceeds to the next subframe, until all image block rows within all subframes of the video frame to be encoded are allocated. All the image block rows allocated to each coding core within the video frame to be encoded constitute a virtual frame. A virtual frame is composed of image block rows. A virtual frame is not a video coding unit as defined in video coding standards such as H.264 and H.265, but rather a bitrate control unit artificially partitioned by this invention for bitrate control during intra-frame parallel coding. For example, in H.265 encoding, a virtual frame consists of LCU rows, and in H.264 encoding, a virtual frame consists of MB rows.
[0045] In this invention, both too many and too few virtual frame divisions within a video frame can have adverse effects. Too many virtual frame divisions result in smaller virtual frame sizes, hindering bitrate control of image blocks within each virtual frame. Too few virtual frame divisions reduce the parallelism of intra-frame parallel coding, leading to wasted coding cores. Preferably, virtual frames are divided according to the number of coding cores participating in intra-frame parallel coding; that is, the number of virtual frames within a single video frame to be encoded equals the number of coding cores participating in intra-frame parallel coding.
[0046] Please see Figure 4 Here Figure 3Taking a video frame as an example, each subframe contains 6 image block rows. Assuming the number of coding cores participating in parallel coding within the subframe is 4, it is preferable to divide the video frame into 4 virtual subframes. Virtual subframe 0 consists of image block rows 0 and 4 within subframe 0, image block rows 2 within subframe 1, image block rows 0 and 4 within subframe 2, and image block rows 2 within subframe 3. Virtual subframe 1 consists of image block rows 1, 5, and 3 within subframe 0, image block rows 1 and 5 within subframe 2, and image block rows 3 within subframe 3. Virtual subframe 2 consists of image block rows 2 within subframe 0, image block rows 0 and 4 within subframe 1, image block rows 2 and 2 within subframe 2, and image block rows 0 and 4 within subframe 3. Virtual subframe 3 consists of image block row 3 within subframe 0, image block row 1 and row 5 within subframe 1, image block row 3 within subframe 2, image block row 1 within subframe 3, and image block row 5 within subframe 3. It is important to note that this invention divides video frames into virtual frames. Each virtual frame is confined within a video frame, but can span a single subframe. This invention does not require waiting for all virtual subframes within a subframe to be encoded, nor does it require adjusting the target bit count for the next subframe; instead, it allows multiple subframes to be encoded simultaneously.
[0047] The order of steps S1 and S2 is not strictly limited; they can be performed simultaneously or either can be performed first or last.
[0048] Step S3: Based on virtual frames, schedule the coding cores participating in intra-frame parallel coding, and schedule each virtual frame within the video frame to be encoded to a different coding core for intra-frame parallel coding. Each coding core records the coding state information of each image block within a virtual frame while encoding it. When all image blocks within a virtual frame are encoded, the coding state information of the encoded virtual frames is statistically analyzed. When all virtual frames within a video frame are encoded, synchronization is performed among the coding cores, and the coding state information of the encoded video frames is statistically analyzed. The coding state information to be recorded is determined by the requirements of the specific rate control algorithm, including, for example, the actual number of encoded bits, coding distortion D, coding complexity, Lagrange multiplier λ, quantization parameter QP, etc.
[0049] Please see Figure 5 The prior art closest to this invention has four levels of target bit allocation, from largest to smallest: image group, video frame, strip or tile, and image block. Prior art performs intra-frame parallel coding at the strip or tile level; a strip is a continuous region within a video frame, and a tile is also a continuous region within a video frame.
[0050] Please see Figure 6 The target bit allocation of this invention has four levels, from largest to smallest: image group, video frame, virtual frame, and image patch. Stripes or tiles are no longer considered as a first level in the target bit allocation for rate control; instead, virtual frames are used. This invention does not perform intra-frame parallel coding at the stripe or tile level, but instead performs it at the virtual frame level (at the same level, but with a different partitioning method than stripes or tiles). Each virtual frame can include multiple image patch rows within multiple subframes of a video frame; that is, each virtual frame can consist of multiple separate regions within a video frame.
[0051] In this invention, a video frame is composed of subframes, and a video frame is also composed of virtual frames, which are composed of image blocks. In step S3, the encoding complexity of a virtual frame is obtained by summing the encoding complexities of all image blocks within that virtual frame. The actual number of encoded bits for a virtual frame is obtained by summing the actual number of encoded bits for all image blocks within that virtual frame. When all virtual frames within a video frame are encoded, the encoding complexity of the video frame is obtained by summing the encoding complexities of all virtual frames within that video frame, and the actual number of encoded bits for the video frame is obtained by summing the actual number of encoded bits for all virtual frames within that video frame.
[0052] Please see Figure 7 Step S3 further includes the following sub-steps.
[0053] Step S31: Based on the target number of coding bits R of the video frame to be encoded frame The encoding complexity ω of the video frame to be encoded frame The encoding complexity ω of the virtual frame to be encoded virtual_frame The target bit allocation is performed on each virtual frame within the video frame to be encoded to obtain the target encoded bit count R for each virtual frame. virtual_frame . (Formula 1). To reduce the latency in hardware implementation of this step, ω frame An approximation is made using the encoding complexity of a previously encoded video frame, ω. virtual_frame An approximation is made using the coding complexity of a virtual frame at the same position in a previously encoded video frame. The number of coding cores participating in intra-frame parallel coding usually remains the same, so the virtual frame partitioning method for the current video frame is usually the same as that for previous video frames.
[0054] In hardware video encoders, to reduce cost and power consumption, when using rate-distortion optimization methods for mode selection of the image block to be encoded, the mode selection process is generally divided into a coding mode prediction stage and a coding mode rate-distortion optimization stage. The main difference between these two stages lies in the precision of the calculation method for the coding rate-distortion cost J. In the coding mode prediction stage, a computationally less complex and less precise method for calculating the coding rate-distortion cost J is generally used; the coding distortion D is generally calculated using the SAD (Sum of Absolute Difference) algorithm or the SATD (Sum of Absolute Transformed Difference) algorithm; the coding bit rate R is generally calculated using a simplified bit rate estimation algorithm, such as the exponential Golomb algorithm; and the coding mode prediction stage generally does not introduce transformation and quantization operations. Conventionally, the low-computation, low-precision coding rate-distortion cost J calculated in the coding mode prediction stage is called the coding mode prediction cost. The prediction cost corresponding to each coding mode is calculated for the image block to be encoded, and the minimum prediction cost is taken as the coding complexity of the image block to be encoded. The sum of the encoding complexities of all image blocks within a virtual frame is used as the encoding complexity of that virtual frame. The sum of the encoding complexities of all image blocks within a video frame is used as the encoding complexity of that video frame.
[0055] Step S32: Based on the target number of encoded bits R of the virtual frame to be encoded virtual_frame The encoding complexity ω of the virtual frame to be encoded virtual_frame The encoding complexity ω of the image block to be encoded blk The target bit allocation is performed on the first image block within the virtual frame to be encoded, resulting in the target number of encoded bits R for the first image block. blk . (Formula 2). To reduce the latency in hardware implementation of this step, ω virtual_frame An approximation is made using the encoding complexity of a virtual frame at the same position in a previously encoded video frame (e.g., the previous one), ω. blk An approximate replacement is made using the encoding complexity of the image patch at the same position in a previously (e.g., the previous) encoded virtual frame.
[0056] In real-world scenarios, when an image block within a virtual frame is encoded, there is often an error between the actual number of encoded bits and the target number of encoded bits calculated before encoding. This error needs to be compensated for in subsequent image block encodings within the virtual frame to reduce bitrate fluctuations and make bitrate control more accurate. When an image block within a virtual frame is encoded, the target number of encoded bits R of the virtual frame to be encoded is calculated... virtual_frame The sum of the actual encoded bits of all encoded image blocks within the virtual frame to be encoded. virtual_frameThe encoding complexity ω of the next image to be encoded blk_next The sum of the encoding complexity Σ of all image blocks to be encoded within the virtual frame to be encoded {all not coded blks} ω blk Compensated target bit allocation is performed on the second and subsequent image blocks within the virtual frame to be encoded, resulting in the target number of encoded bits R for the second and subsequent image blocks to be encoded. blk_next . (Formula 3). To reduce the latency in hardware implementation of this step, ω blk_next An approximate substitution is performed using the encoding complexity of the image patch at the same location in a previously (e.g., the previous) encoded virtual frame, Σ. {all not coded blks} ω blk An approximate replacement is made by summing the encoding complexity of all image blocks at the same position in a previously encoded virtual frame (e.g., the previous one).
[0057] Step S33: Based on the target number of coding bits for each image block within the virtual frame to be encoded, and the coding state information of image blocks at the same position in the already encoded virtual frame, target bit control is performed on each image block within the virtual frame to be encoded to obtain the Lagrange multiplier λ and quantization parameter QP for each image block, which are then used in the actual encoding process of the image block. Different rate control algorithms can be used for the specific calculation process. For example, for H.265 encoding, the rate control algorithm based on the relationship model between coding bit rate R and Lagrange multiplier λ (i.e., the R-λ model) in the JCTVC-K0103 proposal can be used to calculate the Lagrange multiplier λ and quantization parameter QP for the image block, which will not be elaborated here.
[0058] In this invention, when encoding video frames, each encoding core encodes its assigned virtual frames in parallel. For each virtual frame, the encoding state information statistics related to bitrate control are limited to that virtual frame only, neither depending on nor affecting other virtual frames encoded in parallel within the video frame to be encoded. Thus, the bitrate control result of each virtual frame depends only on the partitioning method of the virtual frames, and is independent of the actual encoding order of image blocks between virtual frames. As long as the partitioning method of the virtual frames in the video frame to be encoded remains unchanged, the bitrate control result of the video frame to be encoded will not change. Therefore, when using WPP technology for intra-frame parallel encoding, the consistency of the results of multiple encodings of the same video frame can be guaranteed. Therefore, this invention only requires each encoding core to meet the basic synchronization requirements of WPP technology (the encoding of a certain row of LCUs lags behind the encoding of the LCUs in the row above it by at least two LCUs), without the need for additional strict synchronization measures. This can greatly reduce the design difficulty of a multi-VPU hardware encoding system.
[0059] Compared with existing intra-frame parallel coding techniques for image segmentation and block-based processing, this invention solves the problems of coding efficiency loss and decoding compatibility that exist in existing intra-frame parallel coding techniques for image segmentation and block-based processing.
[0060] Compared to existing WPP intra-parallel coding techniques, this invention only needs to meet the basic synchronization requirements of WPP technology, without requiring strict synchronization, thus solving the problem of consistent bitrate control during repeated coding. The basic synchronization requirements of WPP technology are quite loose and relatively easy to meet, resulting in lower hardware implementation costs. In this invention, each coding core, while meeting the basic synchronization requirements of WPP technology, only needs to focus on encoding the image blocks in the virtual frame it is responsible for, without needing to synchronize with image blocks encoded by other coding cores in real time. This not only ensures the consistency of the encoding results but also significantly reduces the difficulty of hardware design.
[0061] Please see Figure 8 The bitrate control device based on virtual frames for parallel coding proposed in this invention includes a subframe partitioning module 1, a virtual frame partitioning module 2, and a virtual frame coding task scheduling module 3. Figure 8 The device shown corresponds to Figure 1 The method shown.
[0062] The subframe division module 1 is used to divide the video frame to be encoded into multiple independent and non-overlapping subframes.
[0063] The virtual frame partitioning module 2 is used to divide the video frame to be encoded into multiple virtual frames according to the number of coding cores participating in intra-frame parallel coding and the virtual frame partitioning rules. The virtual frame partitioning rules are as follows: according to the number of coding cores participating in intra-frame parallel coding, each image block row within the first subframe of the video frame to be encoded is sequentially and alternately allocated to each coding core. Once the image block rows within one subframe are allocated, the allocation proceeds to the next subframe, until the image block rows within all subframes of the video frame to be encoded are allocated. All the image block rows allocated to each coding core within the video frame to be encoded constitute one virtual frame.
[0064] The virtual frame encoding task scheduling module 3 is used to schedule the encoding cores participating in intra-frame parallel encoding based on virtual frames, and to schedule each virtual frame inside the video frame to be encoded to different encoding cores for intra-frame parallel encoding.
[0065] This invention proposes a rate control method for intra-frame parallel coding based on virtual frames. When using WPP technology for intra-frame parallel coding, this invention rationally divides the video frames to be encoded into virtual frames and schedules and synchronizes the rate control process based on these virtual frames, ensuring consistency of results when encoding video frames multiple times. Using this rate control method, each coding core only needs to meet the basic synchronization requirements of WPP technology itself, eliminating the need for other stringent synchronization measures, which greatly reduces the design complexity of multi-VPU hardware coding systems.
[0066] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A rate control method for parallel coding based on virtual frames, characterized in that, Includes the following steps; Step S1: Divide the video frame to be encoded into multiple independent and non-overlapping subframes; Step S2: Based on the number of coding cores participating in intra-frame parallel coding, the video frame to be encoded is divided into multiple virtual frames according to the virtual frame partitioning rules. The virtual frame partitioning rules are as follows: according to the number of coding cores participating in intra-frame parallel coding, each image block row in the first subframe of the video frame to be encoded is sequentially and alternately allocated to each coding core. Once the image block rows in one subframe are allocated, the allocation of image block rows in the next subframe begins, until the image block rows in all subframes of the video frame to be encoded are allocated. All the image block rows allocated to each coding core within the video frame to be encoded constitute a virtual frame. The order of steps S1 and S2 may be either simultaneous or either can be either one before the other. Step S3: Based on the virtual frames, schedule the coding cores participating in intra-frame parallel coding, and schedule each virtual frame inside the video frame to be encoded to different coding cores for intra-frame parallel coding.
2. The rate control method for parallel coding based on virtual frames according to claim 1, characterized in that, In step S1, the video frame to be encoded is divided into multiple slices using HEVC video coding technology, with each slice being a subframe; or, the video frame to be encoded is divided into multiple tiles using HEVC video coding technology, with each tile being a subframe.
3. The rate control method for parallel coding based on virtual frames according to claim 1, characterized in that, In step S2, the virtual frame is composed of image block rows; in H.265 encoding, the virtual frame is composed of maximum coding unit (LCU) rows, and in H.264 encoding, the virtual frame is composed of macroblock (MB) rows.
4. The rate control method for parallel coding based on virtual frames according to claim 1, characterized in that, In step S2, the number of virtual frames within a video frame to be encoded is equal to the number of encoding cores participating in intra-frame parallel encoding.
5. The rate control method for parallel coding based on virtual frames according to claim 1, characterized in that, In step S2, each virtual frame is confined within a video frame, but each virtual frame spans a single subframe; multiple subframes are allowed to be encoded simultaneously.
6. The rate control method for parallel coding based on virtual frames according to claim 1, characterized in that, In step S3, each encoding core records the encoding status information of each image block within the virtual frame when encoding a virtual frame; when all image blocks within the virtual frame are encoded, the encoding status information of the encoded virtual frames is statistically analyzed; when all virtual frames within the video frame are encoded, synchronization is performed among the encoding cores, and the encoding status information of the encoded video frames is statistically analyzed; the encoding status information includes one or more of the actual number of encoded bits, encoding distortion D, encoding complexity, Lagrange multiplier λ, and quantization parameter QP.
7. The rate control method for parallel coding based on virtual frames according to claim 1, characterized in that, The target bit allocation has four levels, from largest to smallest: image group, video frame, virtual frame, and image block; intra-frame parallel coding is performed at the virtual frame level; each virtual frame includes multiple image block rows in multiple subframes within a video frame, that is, each virtual frame is composed of multiple separate regions within a video frame.
8. The rate control method for parallel coding based on virtual frames according to claim 1, characterized in that, In step S3, the encoding complexity of the virtual frame is obtained by summing the encoding complexities of all image blocks within the virtual frame; the actual number of encoded bits of the virtual frame is obtained by summing the actual number of encoded bits of all image blocks within the virtual frame; when all virtual frames within a video frame are encoded, the encoding complexity of the video frame is obtained by summing the encoding complexities of all virtual frames within the video frame, and the actual number of encoded bits of the video frame is obtained by summing the actual number of encoded bits of all virtual frames within the video frame.
9. The rate control method for parallel coding based on virtual frames according to claim 1, characterized in that, Step S3 further includes the following sub-steps; Step S31: Based on the target number of coding bits R of the video frame to be encoded frame The encoding complexity ω of the video frame to be encoded frame The encoding complexity ω of the virtual frame to be encoded virtual_frame The target bit allocation is performed on each virtual frame within the video frame to be encoded to obtain the target encoded bit count R for each virtual frame. virtual_frame . (Formula 1); ω frame An approximation is made using the encoding complexity of previously encoded video frames, ω. virtual_frame An approximate replacement is made using the encoding complexity of a virtual frame at the same position in a previously encoded video frame; Step S32: Based on the target number of encoded bits R of the virtual frame to be encoded virtual_frame The encoding complexity ω of the virtual frame to be encoded virtual_frame The encoding complexity ω of the image block to be encoded blk The target bit allocation is performed on the first image block within the virtual frame to be encoded, resulting in the target number of encoded bits R for the first image block. blk ; (Formula 2); ω virtual_frame An approximation is made using the encoding complexity of a virtual frame at the same position in a previously encoded video frame, ω. blk An approximate replacement is made using the encoding complexity of the image block at the same position in the previously encoded virtual frame; When the encoding of an image block within a virtual frame is complete, the target number of encoded bits R of the virtual frame to be encoded is determined. virtual_frame The sum of the actual encoded bits of all encoded image blocks within the virtual frame to be encoded. virtual_frame The encoding complexity ω of the next image to be encoded blk_next The sum of the encoding complexity Σ of all image blocks to be encoded within the virtual frame to be encoded {all not coded blks} ω blk Compensated target bit allocation is performed on the second and subsequent image blocks within the virtual frame to be encoded, resulting in the target number of encoded bits R for the second and subsequent image blocks to be encoded. blk_next ; (Formula 3); ω blk_next An approximation is made using the encoding complexity of the image patch at the same position in a previously encoded virtual frame, Σ. {all not coded blks} ω blk An approximate replacement is made by summing the encoding complexity of all image blocks at the same position in the previously encoded virtual frame; Step S33: Based on the target number of coding bits for each image block within the virtual frame to be encoded and the coding status information of the image blocks at the same position in the encoded virtual frame, target bit control is performed on each image block within the virtual frame to be encoded to obtain the Lagrange multiplier λ and quantization parameter QP for each image block, which are used in the actual coding process of the image block.
10. A rate control device for parallel coding based on virtual frames, characterized in that, It includes a subframe partitioning module, a virtual frame partitioning module, and a virtual frame encoding task scheduling module; The subframe division module is used to divide the video frame to be encoded into multiple independent and non-overlapping subframes. The virtual frame partitioning module is used to divide the video frame to be encoded into multiple virtual frames according to the number of coding cores participating in intra-frame parallel coding and the virtual frame partitioning rules. The virtual frame partitioning rules are as follows: according to the number of coding cores participating in intra-frame parallel coding, each image block row in the first subframe of the video frame to be encoded is sequentially and alternately allocated to each coding core. After the image block rows in one subframe are allocated, the allocation of image block rows in the next subframe begins, until the image block rows in all subframes of the video frame to be encoded are allocated. All the image block rows in the video frame to be encoded allocated to each coding core constitute a virtual frame. The virtual frame encoding task scheduling module is used to schedule the encoding cores participating in intra-frame parallel encoding based on virtual frames, and to schedule each virtual frame inside the video frame to be encoded to different encoding cores for intra-frame parallel encoding.