Parallel coding rate control method and device based on virtual subframe

Abstract

The invention discloses a code rate control method for parallel coding based on virtual subframes. The method comprises the following steps: S1, dividing a to-be-coded video frame into one or more sub-frames which are independent from one another and are not overlapped with one another; and S2, calculating a target coding bit number of the to-be-coded sub-frame. And S3, dividing the to-be-coded subframe into a plurality of virtual subframes according to the number of the coding cores participating in parallel coding in the subframe and a division rule of the virtual subframes. And S4, scheduling the coding cores participating in parallel coding in the sub-frames based on the virtual sub-frames, and respectively scheduling each virtual sub-frame in the sub-frame to be coded to different coding cores to carry out parallel coding in the sub-frames. According to the invention, the result consistency of multiple times of coding of the sub-frame is ensured; and meanwhile, strict synchronization is not needed.

Description

Technical Field

[0001] This invention relates to a digital video coding technique, and more particularly to a rate control method for intra-frame parallel coding. Background Technology

[0002] Video coding is a technique that compresses redundant components in video images and represents video information using as little data as possible. Common video coding standards include HEVC (High Efficiency Video Coding, also known as H.265) and AVC (Advanced Video Coding, also known as H.264), among others.

[0003] Video coding techniques use image blocks as the most basic coding unit. For example, in HEVC, the basic coding unit is the CU (coding unit). A CU can be a 64-pixel × 64-pixel, 32-pixel × 32-pixel, 16-pixel × 16-pixel, or 8-pixel × 8-pixel image block. The 64-pixel × 64-pixel image block is also called the LCU (largest coding unit).

[0004] To maximize video compression, common video coding algorithms often employ lossy compression, resulting in differences between the reconstructed video and the original video—that is, distortion occurs in the reconstructed video. HEVC and AVC are examples of lossy compression algorithms. For lossy compression algorithms, their coding performance needs to be measured by both the coding bitrate and the distortion introduced during coding. In video coding, the coding bitrate and coding distortion are mutually restrictive and contradictory; for example, reducing the coding bitrate often increases coding distortion, while reducing coding distortion often leads to an increase in the coding bitrate. Therefore, rate distortion optimization (RDO) techniques were developed, aiming to minimize coding distortion at a given coding bitrate, or vice versa. Currently, common video coding algorithms employ rate distortion optimization methods based on Lagrange multipliers (λ). Assuming D is the coding distortion, R is the coding bit rate, and J is the rate distortion cost (RD Cost), then J = D + λ × R. The rate distortion optimization method based on the Lagrange multiplier λ is to find the minimum value of J.

[0005] In video encoding, each encoding mode has its corresponding encoding distortion D and encoding bit rate R, allowing the calculation of the rate-distortion cost J for each encoding mode. During encoding, selecting the encoding mode with the lowest rate-distortion cost J yields optimal encoding performance. The process by which the encoder selects the optimal encoding mode for each image block to be encoded using rate-distortion optimization methods is called mode decision for the image block.

[0006] In real-world scenarios, the channel capacity used to transmit compressed video is limited. If the encoding bitrate of the compressed video is too high, exceeding the channel capacity, it will cause video transmission congestion or even packet loss. If the encoding bitrate of the compressed video is too low, the channel capacity will not be fully utilized, and higher video quality cannot be obtained. Therefore, it is necessary to use rate control technology to control the output bitrate of the video encoder to match it with the channel capacity.

[0007] Bitrate control technology adjusts the encoding parameters of the video encoder to make the output bitrate of the video encoder equal to a pre-set target bitrate, while minimizing encoding distortion to improve video encoding quality. Common bitrate control algorithms typically achieve this through two stages: target bit allocation and target bit control. The target bit allocation stage calculates the target number of coded bits for each image unit to be encoded (such as a group of pictures (GOP), video frame, or image block). The target bit control stage, based on the target number of coded bits for each image unit, calculates its corresponding encoding parameters (such as the Lagrange multiplier λ or quantization parameter QP) for use in the encoding process of that image unit.

[0008] Because video encoding algorithms involve a large amount of computation, using application-specific integrated circuits (ASICs) to accelerate the video encoding process in hardware has become a common industry practice in order to improve video encoding speed and achieve real-time encoding and transmission. Such ASICs that accelerate the video encoding process in hardware are generally called hardware video processing units (VPUs).

[0009] With the development of video coding technology, the compression rate of video coding is getting higher and higher, and the amount of computation in video coding is also increasing. Especially for videos with extremely high resolution and extremely high frame rate, real-time coding must be guaranteed through parallel coding technology.

[0010] One parallel coding technique is intra-frame parallel coding, which distributes image blocks at different locations within a video frame to be encoded onto different coding cores for parallel encoding, thereby improving the speed of video encoding. The coding core refers to a video coding unit that can work in parallel; it can be an encoding thread in a software video encoder or a hardware encoding module in a hardware video encoder.

[0011] For example, HEVC video coding technology can use image segmentation or WPP (wavefront parallel processing) technology to achieve intra-frame parallel coding.

[0012] Intra-frame parallel coding with image segmentation divides video frames spatially into multiple slices and / or tiles, distributing each slice or tile to different coding cores for parallel encoding. This coding method often suffers from significant coding efficiency losses and is prone to blocking artifacts at the boundaries of slices or tiles. Video encoded using intra-frame parallel coding with image segmentation is also susceptible to decoding compatibility issues during decoding.

[0013] WPP's intra-frame parallel coding technique distributes different LCU lines to different coding cores for parallel coding, requiring only that the coding of a particular LCU line lags behind the coding of the preceding LCU line by at least two LCUs. WPP's intra-frame parallel coding technique does not suffer from block artifacts or decoding compatibility issues.

[0014] While WPP's intra-frame parallel coding technology offers significant advantages, its rate control process presents a challenge. This is because rate control is a technique with strong spatiotemporal dependencies, and its final control effect is closely related to the processing order of image blocks during actual encoding. When using WPP's intra-frame parallel coding technology, strict synchronization between the various coding cores is essential to ensure that image blocks within a video frame are encoded sequentially in a fixed order. This guarantees consistent rate control when encoding the same video frame. For hardware parallel coding systems containing multiple VPUs, this synchronization requirement significantly increases the complexity of the hardware design.

[0015] Currently, when using WPP technology for intra-frame parallel coding, there are two main types of existing rate control methods.

[0016] The first type of method does not strictly synchronize the encoding cores of intra-frame parallel coding, but only needs to meet the basic requirement of WPP technology that there is a lag of at least two LCUs between adjacent LCU lines. This type of method is relatively simple to implement, but it cannot guarantee consistent bitrate control when encoding the same video frame multiple times, which does not meet the consistency requirement of video encoders for bitrate control.

[0017] The second type of method involves strict synchronization of the encoding cores of intra-frame parallel coding. This not only meets the basic requirement of WPP technology that adjacent LCU rows lag by at least two LCUs, but also ensures that image blocks within a video frame are encoded sequentially in a fixed order. This type of method can guarantee consistent bitrate control when encoding the same video frame multiple times. However, when implementing this in hardware, such synchronization requirements significantly increase the complexity of the hardware design. Summary of the Invention

[0018] The technical problem to be solved by this invention is: how to improve the intra-frame parallel coding technology of WPP so that the bit rate control effect can be kept consistent during multiple coding sessions, without the need for strict synchronization (only basic synchronization requirements need to be met).

[0019] To address the aforementioned technical problems, this invention proposes a rate control method for parallel coding based on virtual subframes, comprising the following steps: Step S1: Divide the video frame to be encoded into one or more independent and non-overlapping subframes. Step S2: Based on the target number of coding bits R of the video frame to be encoded... frame The encoding complexity ω of the video frame to be encoded frame The encoding complexity ω of the subframe to be encoded subframe The target bit allocation is performed on the first subframe to be encoded within the video frame to be encoded, resulting in the target number of encoded bits R for the first subframe to be encoded. subframe ; (Formula 1); ω frame An approximation is made using the encoding complexity of previously encoded video frames, ω. subframe An approximation is performed using the encoding complexity of a subframe at the same position within a previously encoded video frame. When the encoding of a subframe within the video frame to be encoded is complete, the target number of encoded bits R of the video frame to be encoded is used. frame The sum of the actual encoded bits of all encoded subframes within the video frame to be encoded. frame The encoding complexity ω of the next subframe to be encoded subframe_next The sum of the encoding complexity Σ of all subframes within the video frame to be encoded. {all not coded subframes} ω subframe Compensated target bit allocation is performed on the second and subsequent subframes to be encoded within the video frame to be encoded, and the target coded bit count R of the second and subsequent subframes to be encoded is calculated.subframe_next ; (Formula 2); ω subframe_next An approximate replacement is performed using the encoding complexity of the subframe at the same position in a previously encoded video frame, Σ {all not coded subframes} ω subframe An approximate replacement is performed using the sum of the encoding complexities of all subframes at the same position in the previously encoded video frame. If, in step S1, the video frame to be encoded is divided into only one subframe, then in step S2, R is directly set... subframe =R frame Step S3: Based on the number of coding cores participating in intra-frame parallel coding, the subframe to be encoded is divided into multiple virtual subframes according to the virtual subframe partitioning rules. The virtual subframe partitioning rules are as follows: according to the number of coding cores participating in intra-frame parallel coding, each image block row within the subframe to be encoded is sequentially and alternately allocated to each coding core until all image block rows within the subframe to be encoded are allocated. All image block rows within the subframe to be encoded allocated to each coding core constitute a virtual subframe. Steps S2 and S3 may be performed simultaneously or either one first. Step S4: Based on the virtual subframes, the coding cores participating in intra-frame parallel coding are scheduled, and each virtual subframe within the subframe to be encoded is scheduled to different coding cores for intra-frame parallel coding.

[0020] Furthermore, in step S1, the video frame to be encoded is divided into multiple slices using HEVC video coding technology, with each slice being a subframe; or, the video frame to be encoded is divided into multiple tiles using HEVC video coding technology, with each tile being a subframe.

[0021] Further, in step S2, the video coding mode selection process is divided into a coding mode prediction stage and a coding mode rate-distortion optimization stage; the coding rate-distortion cost calculated in the coding mode prediction stage is called the prediction cost of the coding mode; the prediction cost corresponding to each coding mode is calculated for the image block to be coded, and the minimum prediction cost is taken as the coding complexity of the image block to be coded; the sum of the coding complexities of all image blocks in the virtual subframe is taken as the coding complexity of the virtual subframe; the sum of the coding complexities of all image blocks in the subframe is taken as the coding complexity of the subframe; the sum of the coding complexities of all image blocks in the video frame is taken as the coding complexity of the video frame.

[0022] Furthermore, in step S3, the virtual subframe is composed of image block rows; in H.265 encoding, the virtual subframe is composed of maximum coding unit (LCU) rows, and in H.264 encoding, the virtual subframe is composed of macroblock (MB) rows.

[0023] Furthermore, in step S3, the number of virtual subframes within a subframe to be encoded is equal to the number of encoding cores participating in parallel encoding within the subframe.

[0024] Furthermore, in step S3, each virtual subframe is confined to a subframe; after all virtual subframes within a subframe are encoded, the target encoding bit count for the next subframe is corrected according to Formula 2, and then the encoding of the next subframe begins.

[0025] Further, in step S4, each encoding core records the encoding status information of each image block within the virtual subframe when encoding a virtual subframe; when all image blocks within the virtual subframe are encoded, the encoding status information of the encoded virtual subframes is statistically analyzed; when all virtual subframes within a subframe are encoded, synchronization is performed among the encoding cores, and the encoding status information of the encoded subframes is statistically analyzed; when all subframes within a video frame are encoded, the encoding status information of the encoded video frame is statistically analyzed; the encoding status information includes one or more of the actual number of encoded bits, encoding distortion D, encoding complexity, Lagrange multiplier λ, and quantization parameter QP.

[0026] Furthermore, the target bit allocation has five levels, from largest to smallest: image group, video frame, subframe, virtual subframe, and image block; parallel coding within the subframe is performed at the virtual subframe level; each virtual subframe includes multiple rows of image blocks within a subframe, that is, each virtual subframe is composed of multiple separate regions within the subframe.

[0027] Further, step S4 further includes the following sub-steps. Step S41: Based on the target number of coded bits R of the subframe to be encoded. subframe The encoding complexity ω of the subframe to be encoded subframe The encoding complexity ω of the virtual subframe to be encoded virtual_subframe Target bit allocation is performed on each virtual subframe within the subframe to be coded to obtain the target coded bit count R for each virtual subframe. virtual_subframe ; (Formula 3); ω subframe An approximation is made using the encoding complexity of the subframe at the same position in a previously encoded video frame, ω. virtual_subframe Approximate substitution is performed using the encoding complexity of the virtual subframe at the same position in the previously encoded subframe. Step S42: Based on the target number of encoded bits R of the virtual subframe to be encoded. virtual_subframe The encoding complexity ω of the virtual subframe to be encoded virtual_subframe The encoding complexity ω of the image block to be encoded blk The target bit allocation is performed on the first image block within the virtual subframe to be encoded, resulting in the target number of encoded bits R for the first image block. blk ; (Formula 4); ωvirtual_subframe An approximation is made using the encoding complexity of a virtual subframe at the same position in a previously encoded subframe, ω. blk An approximate substitution is performed using the encoding complexity of the image block at the same position in a previously encoded virtual subframe. When the encoding of an image block within a virtual subframe is complete, the target number of encoded bits R of the virtual subframe to be encoded is used. virtual_subframe The sum of the actual encoded bits of all encoded image blocks within the virtual subframe to be encoded. virtual_subframe The encoding complexity ω of the next image to be encoded blk_next The sum of the encoding complexity Σ of all image blocks to be encoded within the virtual subframe to be encoded. {all not coded blks} ω blk Compensated target bit allocation is performed on the second and subsequent image blocks within the virtual subframe to be encoded, resulting in the target number of encoded bits R for the second and subsequent image blocks to be encoded. blk_next ; (Formula 5); ω blk_next An approximate replacement is performed using the encoding complexity of the image patch at the same position in the previously encoded virtual subframe, Σ {all not coded blks} ω blk Approximate substitution is performed using the sum of the encoding complexities of all image blocks at the same position in the previously encoded virtual subframe. Step S43: Based on the target number of encoded bits for each image block within the virtual subframe to be encoded, and the encoding state information of the image blocks at the same position in the previously encoded virtual subframe, target bit control is performed on each image block within the virtual subframe to be encoded to obtain the Lagrange multiplier λ and quantization parameter QP for each image block, which are used in the actual encoding process of the image block.

[0028] This invention also proposes a rate control device for parallel coding based on virtual subframes, including a subframe partitioning module, a subframe target bit allocation module, a virtual subframe partitioning module, and a virtual subframe coding task scheduling module. The subframe partitioning module is used to divide the video frame to be encoded into one or more independent and non-overlapping subframes. The subframe target bit allocation module is used to allocate the target number of coding bits R of the video frame to be encoded. frame The encoding complexity ω of the video frame to be encoded frame The encoding complexity ω of the subframe to be encoded subframe The target bit allocation is performed on the first subframe to be encoded within the video frame to be encoded, resulting in the target number of encoded bits R for the first subframe to be encoded. subframe ; (Formula 1); ω frame An approximation is made using the encoding complexity of previously encoded video frames, ω. subframeAn approximate substitution is performed using the encoding complexity of a subframe at the same position in a previously encoded video frame. The subframe target bit allocation module is also used to, when the encoding of a subframe within the video frame to be encoded is completed, allocate bits according to the target number R of the video frame to be encoded. frame The sum of the actual encoded bits of all encoded subframes within the video frame to be encoded. frame The encoding complexity ω of the next subframe to be encoded subframe_next The sum of the encoding complexity Σ of all subframes within the video frame to be encoded. {all not coded subframes} ω subframe Compensated target bit allocation is performed on the second and subsequent subframes to be encoded within the video frame to be encoded, and the target coded bit count R of the second and subsequent subframes to be encoded is calculated. subframe_next ; (Formula 2); ω subframe_next An approximate replacement is performed using the encoding complexity of the subframe at the same position in a previously encoded video frame, Σ {all not coded subframes} ω subframe An approximate replacement is performed using the sum of the encoding complexities of all subframes at the same position in the previously encoded video frame. If the subframe partitioning module divides the video frame to be encoded into only one subframe, the subframe target bit allocation module directly sets R... subframe =R frame The virtual subframe partitioning module is used to divide the subframe to be encoded into multiple virtual subframes according to the number of coding cores participating in intra-frame parallel coding and the virtual subframe partitioning rules. The virtual subframe partitioning rules are as follows: each image block row within the subframe to be encoded is sequentially and alternately assigned to each coding core according to the number of coding cores participating in intra-frame parallel coding, until all image block rows within the subframe to be encoded have been assigned. All image block rows assigned to each coding core within the subframe to be encoded constitute a virtual subframe. The virtual subframe coding task scheduling module is used to schedule the coding cores participating in intra-frame parallel coding based on the virtual subframes, scheduling each virtual subframe within the subframe to be encoded to different coding cores for intra-frame parallel coding.

[0029] The technical effects achieved by this invention are as follows: When using WPP technology for intra-frame parallel coding, by reasonably dividing the subframe to be encoded into virtual subframes and scheduling and synchronizing the bitrate control process of the subframes based on the virtual subframes, the consistency of the results when encoding subframes multiple times is ensured; at the same time, each coding core only needs to meet the basic synchronization requirements of WPP technology itself when working, without the need to adopt other strict synchronization measures, thus reducing the design difficulty of multi-VPU hardware coding systems. Attached Figure Description

[0030] Figure 1This is a flowchart illustrating the parallel coding rate control method based on virtual subframes proposed in this invention.

[0031] Figure 2 This is a schematic diagram of a video frame.

[0032] Figure 3 This is a diagram illustrating how a video frame is divided into four subframes.

[0033] Figure 4 This is a diagram illustrating how a subframe is divided into four virtual subframes.

[0034] Figure 5 This is a schematic diagram of the hierarchy of existing target bit allocation.

[0035] Figure 6 This is a hierarchical diagram of the target bit allocation of the present invention.

[0036] Figure 7 yes Figure 1 The detailed flowchart of step S4 is shown below.

[0037] Figure 8 This is a schematic diagram of the rate control device based on parallel coding of virtual subframes proposed in this invention.

[0038] The attached diagrams are labeled as follows: 1. Subframe partitioning module; 2. Subframe target bit allocation module; 3. Virtual subframe partitioning module; 4. Virtual subframe encoding task scheduling module. Detailed Implementation

[0039] Video bitrate control technology generally includes two stages: target bit allocation and target bit control. In existing technologies, the target bit allocation stage first calculates the target coded bit count for a group of pictures (GOP), and then calculates the target coded bit count for the video frames within that GOP. The video bitrate control method proposed in this invention starts with the target coded bit count R of the video frames already obtained. frame Then, improvements were made to the subsequent parts.

[0040] Please see Figure 1 The rate control method based on parallel coding of virtual subframes proposed in this invention includes the following steps.

[0041] Step S1: Divide the video frame to be encoded into one or more independent and non-overlapping subframes. For example, this step uses HEVC video coding technology to divide the video frame to be encoded into multiple slices, with each slice serving as a subframe. Alternatively, this step uses HEVC video coding technology to divide the video frame to be encoded into multiple tiles, with each tile serving as a subframe. It is important to note that this invention does not use slices or tiles as scheduling units for parallel encoding. In this step, if the video frame to be encoded is divided into only one subframe, the size of that subframe is exactly the same as the size of the original video frame.

[0042] Please see Figure 2 This is a video frame consisting of 12 image block rows. For example, in H.265 encoding, an image block row refers to an LCU (maximum coding unit) row, while in H.264 encoding, an image block row refers to an MB (micro block) row.

[0043] Please see Figure 3 This is to Figure 2 The video frame shown is divided into four subframes. Specifically, in... Figure 2 The video frame shown is divided into two parts in both width and height, resulting in four subframes of the same size. The width of each subframe is equal to half the width of a row of image blocks, and the height of each subframe is equal to the sum of the heights of the six rows of image blocks.

[0044] Step S2: Based on the target number of encoded bits R of the video frame to be encoded frame The encoding complexity ω of the video frame to be encoded frame The encoding complexity ω of the subframe to be encoded subframe The target bit allocation is performed on the first subframe to be encoded within the video frame to be encoded, resulting in the target number of encoded bits R for the first subframe to be encoded. subframe . (Formula 1). In practical scenarios, video encoders typically use the same subframe division method for consecutive video frames. To reduce the latency in hardware implementation of this step, ω frame An approximation is made using the encoding complexity of a previously encoded video frame, ω. subframe An approximation is made using the encoding complexity of the subframe at the same position in a previously encoded video frame (e.g., the previous one).

[0045] In real-world scenarios, when a subframe within a video frame is encoded, there is often an error between the actual number of encoded bits and the target number of encoded bits calculated before encoding. This error needs to be compensated for during the encoding of subsequent subframes within the video frame to reduce bitrate fluctuations and make bitrate control more accurate. When a subframe within the video frame to be encoded is encoded, the target number of encoded bits R of the video frame is calculated... frame The sum of the actual encoded bits of all encoded subframes within the video frame to be encoded. frame The encoding complexity ω of the next subframe to be encoded subframe_next The sum of the encoding complexity Σ of all subframes within the video frame to be encoded. {all not coded subframes} ω subframe Compensated target bit allocation is performed on the second and subsequent subframes to be encoded within the video frame to be encoded, and the target coded bit count R of the second and subsequent subframes to be encoded is calculated. subframe_next . (Formula 2). To reduce the latency in hardware implementation of this step, ω subframe_next An approximation is made using the encoding complexity of the subframe at the same position in a previously encoded video frame, Σ. {all not coded subframes} ω subframe An approximate replacement is made by summing the encoding complexity of all subframes at the same position in the previously encoded video frame (e.g., the previous one).

[0046] In hardware video encoders, to reduce cost and power consumption, when using rate-distortion optimization methods for mode selection of the image block to be encoded, the mode selection process is generally divided into a coding mode prediction stage and a coding mode rate-distortion optimization stage. The main difference between these two stages lies in the precision of the calculation method for the coding rate-distortion cost J. In the coding mode prediction stage, a computationally less complex and less precise method for calculating the coding rate-distortion cost J is generally used; the coding distortion D is generally calculated using the SAD (Sum of Absolute Difference) algorithm or the SATD (Sum of Absolute Transformed Difference) algorithm; the coding bit rate R is generally calculated using a simplified bit rate estimation algorithm, such as the exponential Golomb algorithm; and the coding mode prediction stage generally does not introduce transformation and quantization operations. Conventionally, the low-computation, low-precision coding rate-distortion cost J calculated in the coding mode prediction stage is called the coding mode prediction cost. The prediction cost corresponding to each coding mode is calculated for the image block to be encoded, and the minimum prediction cost is taken as the coding complexity of the image block to be encoded. The sum of the encoding complexities of all image blocks within a virtual subframe is used as the encoding complexity of that virtual subframe. The sum of the encoding complexities of all image blocks within a subframe is used as the encoding complexity of that subframe. The sum of the encoding complexities of all image blocks within a video frame is used as the encoding complexity of that video frame.

[0047] If the video frame to be encoded is divided into only one subframe in step S1, then R in step S2... subframe =R frame No calculation is required.

[0048] Step S3: Based on the number of coding cores participating in intra-frame parallel coding, the subframe to be encoded is divided into multiple virtual subframes according to the virtual subframe partitioning rules. The virtual subframe partitioning rules are as follows: each image block row within the subframe to be encoded is sequentially and alternately allocated to each coding core according to the number of coding cores participating in intra-frame parallel coding, until all image block rows within the subframe to be encoded have been allocated. All the image block rows within the subframe to be encoded allocated to each coding core constitute a virtual subframe. Virtual subframes are composed of image block rows. Virtual subframes are not video coding units as defined in video coding standards such as H.264 and H.265, but rather are artificially partitioned bitrate control units used in this invention for bitrate control during intra-frame parallel coding. For example, in H.265 encoding, virtual subframes consist of LCU rows, and in H.264 encoding, virtual subframes consist of MB rows.

[0049] In this invention, both excessive and insufficient virtual subframe divisions within a single subframe can have adverse effects. Too many virtual subframe divisions result in smaller individual subframe sizes, hindering bitrate control of image blocks within each virtual subframe. Too few virtual subframe divisions reduce the parallelism of intra-frame parallel coding, leading to wasted coding cores. Preferably, virtual subframes are divided according to the number of coding cores participating in intra-frame parallel coding; that is, the number of virtual subframes within a single subframe to be encoded equals the number of coding cores participating in intra-frame parallel coding.

[0050] Please see Figure 4 Here Figure 3 Taking subframe 0 as an example, subframe 0 contains 6 image block rows. Assuming the number of coding cores participating in parallel coding within the subframe is 4, it is preferable to divide this subframe into 4 virtual subframes. Virtual subframe 0 consists of image block rows 0 and 4 within subframe 0. Virtual subframe 1 consists of image block rows 1 and 5 within subframe 0. Virtual subframe 2 consists of image block row 2 within subframe 0. Virtual subframe 3 consists of image block row 3 within subframe 0. It is important to note that this invention divides subframes into virtual subframes, and each virtual subframe is confined to within a subframe. After all virtual subframes within a subframe are encoded, the target coding bit count for the next subframe is corrected (Formula 2), and only then can the encoding of the next subframe begin.

[0051] The order of steps S2 and S3 is not strictly limited; they can be performed simultaneously or either can be performed first or last.

[0052] Step S4: Based on virtual subframes, schedule the coding cores participating in intra-frame parallel coding, assigning each virtual subframe within the subframe to be encoded to a different coding core for intra-frame parallel coding. Each coding core records the coding status information of each image block within a virtual subframe while coding it. When all image blocks within a virtual subframe are encoded, the coding status information of the encoded virtual subframes is statistically analyzed. When all virtual subframes within a subframe are encoded, synchronization is performed among the coding cores, and the coding status information of the encoded subframes is statistically analyzed. When all subframes within a video frame are encoded, the coding status information of the encoded video frame is statistically analyzed. The coding status information to be recorded is determined by the requirements of the specific rate control algorithm, including, for example, the actual number of encoded bits, coding distortion D, coding complexity, Lagrange multiplier λ, quantization parameter QP, etc.

[0053] Please see Figure 5The prior art closest to this invention has four levels of target bit allocation, from largest to smallest: image group, video frame, strip or tile, and image block. Prior art performs intra-frame parallel coding at the strip or tile level; a strip is a continuous region within a video frame, and a tile is also a continuous region within a video frame.

[0054] Please see Figure 6 The target bit allocation of this invention has five levels, from largest to smallest: image group, video frame, subframe (equivalent to stripe or tile), virtual subframe, and image patch. The virtual subframe, as a newly added level in target bit allocation, participates in rate control. Instead of performing intra-frame parallel coding at the stripe or tile level, this invention performs intra-frame parallel coding at the virtual subframe level (the level below). Each virtual subframe can include multiple rows of image patches within a subframe; that is, each virtual subframe can be composed of multiple separate regions within a subframe.

[0055] In this invention, a video frame is composed of subframes, subframes are composed of virtual subframes, and virtual subframes are composed of image blocks. In step S4, the encoding complexity of a virtual subframe is obtained by summing the encoding complexities of all image blocks within that virtual subframe. The actual number of encoded bits for a virtual subframe is obtained by summing the actual number of encoded bits for all image blocks within that virtual subframe. When all virtual subframes within a subframe are encoded, the encoding complexity of the subframe is obtained by summing the encoding complexities of all virtual subframes within that subframe, and the actual number of encoded bits for the subframe is obtained by summing the actual number of encoded bits for all virtual subframes within that subframe. When all subframes within a video frame are encoded, the encoding complexity of the video frame is obtained by summing the encoding complexities of all subframes within that video frame, and the actual number of encoded bits for the video frame is obtained by summing the actual number of encoded bits for all subframes within that video frame.

[0056] Please see Figure 7 Step S4 further includes the following sub-steps.

[0057] Step S41: Based on the target number of coding bits R of the subframe to be encoded subframe The encoding complexity ω of the subframe to be encoded subframe The encoding complexity ω of the virtual subframe to be encoded virtual_subframe Target bit allocation is performed on each virtual subframe within the subframe to be coded to obtain the target coded bit count R for each virtual subframe. virtual_subframe . (Formula 3). To reduce the latency in hardware implementation of this step, ω subframe An approximation is made using the encoding complexity of the subframe at the same position in a previously encoded video frame, ω. virtual_subframeAn approximation is made using the coding complexity of the virtual subframe at the same position in a previously coded subframe. The number of coding cores participating in parallel coding within a subframe usually remains constant, so the virtual subframe partitioning method for the current subframe is usually the same as that for the previous subframe.

[0058] Step S42: Based on the target number of encoded bits R of the virtual subframe to be encoded virtual_subframe The encoding complexity ω of the virtual subframe to be encoded virtual_subframe The encoding complexity ω of the image block to be encoded blk The target bit allocation is performed on the first image block within the virtual subframe to be encoded, resulting in the target number of encoded bits R for the first image block. blk . (Formula 4). To reduce the latency in hardware implementation of this step, ω virtual_subframe An approximation is made using the encoding complexity of a virtual subframe at the same position in a previously encoded subframe, ω. blk An approximate replacement is made using the encoding complexity of the image patch at the same position in a previously (e.g., the previous) encoded virtual subframe.

[0059] In real-world scenarios, when an image block within a virtual subframe is encoded, there is often an error between the actual number of encoded bits and the target number of encoded bits calculated before encoding. This error needs to be compensated for during the encoding of subsequent image blocks within the virtual subframe to reduce bitrate fluctuations and make bitrate control more accurate. When an image block within a virtual subframe is encoded, the target number of encoded bits R of the virtual subframe to be encoded is calculated... virtual_subframe The sum of the actual encoded bits of all encoded image blocks within the virtual subframe to be encoded. virtual_subframe The encoding complexity ω of the next image to be encoded blk_next The sum of the encoding complexity Σ of all image blocks to be encoded within the virtual subframe to be encoded. {all not coded blks} ω blk Compensated target bit allocation is performed on the second and subsequent image blocks within the virtual subframe to be encoded, resulting in the target number of encoded bits R for the second and subsequent image blocks to be encoded. blk_next . (Formula 5). To reduce the latency in hardware implementation of this step, ω blk_next An approximate replacement is performed using the encoding complexity of the image patch at the same position in a previously (e.g., the previous) encoded virtual subframe, Σ. {all not coded blks} ω blk An approximate replacement is made by summing the encoding complexity of all image blocks at the same position in the previously encoded virtual subframe (e.g., the previous one).

[0060] Step S43: Based on the target number of coding bits for each image block within the virtual subframe to be encoded, and the coding state information of image blocks at the same position in the already encoded virtual subframes, target bit control is performed on each image block within the virtual subframe to be encoded to obtain the Lagrange multiplier λ and quantization parameter QP for each image block, which are then used in the actual encoding process of the image blocks. Different rate control algorithms can be used for the specific calculation process. For example, for H.265 encoding, the rate control algorithm based on the relationship model between coding bit rate R and Lagrange multiplier λ (i.e., the R-λ model) in the JCTVC-K0103 proposal can be used to calculate the Lagrange multiplier λ and quantization parameter QP for the image blocks, which will not be elaborated here.

[0061] In this invention, when encoding each subframe in a video frame, each encoding core encodes its assigned virtual subframe in parallel. For each virtual subframe, the bitrate control-related encoding state information statistics are limited to that virtual subframe itself, neither depending on nor affecting other virtual subframes encoded in parallel within that subframe. Thus, the bitrate control result for each virtual subframe depends only on the partitioning method of the virtual subframe, and is independent of the actual encoding order of image blocks between virtual subframes. As long as the partitioning method of the virtual subframes in each subframe of the video frame to be encoded remains unchanged, the bitrate control result of the video frame to be encoded will not change. Therefore, when using WPP technology for intra-frame parallel encoding, the consistency of the results of multiple encodings of the same video frame can be guaranteed. Therefore, this invention only requires each encoding core to meet the basic synchronization requirements of WPP technology (the encoding of a certain row of LCUs lags behind the encoding of the LCUs in the row above it by at least two LCUs), without the need for additional strict synchronization measures. This greatly reduces the design difficulty of a multi-VPU hardware encoding system.

[0062] Compared with existing intra-frame parallel coding techniques that use image segmentation and block-based methods, this invention solves the problems of coding efficiency loss and decoding compatibility that exist in existing intra-frame parallel coding techniques. Furthermore, this invention does not require video frames to be divided into subframes, meaning that a video frame can be treated as a single, unique subframe.

[0063] Compared to existing WPP intra-parallel coding techniques, this invention only needs to meet the basic synchronization requirements of WPP technology, without requiring strict synchronization, thus solving the problem of consistent bitrate control during repeated coding. The basic synchronization requirements of WPP technology are quite loose and relatively easy to meet, resulting in lower hardware implementation costs. In this invention, each coding core, while meeting the basic synchronization requirements of WPP technology, only needs to focus on encoding the image blocks in the virtual subframe it is responsible for, without needing to synchronize with image blocks encoded by other coding cores in real time. This not only ensures the consistency of the coding results but also significantly reduces the difficulty of hardware design.

[0064] Please see Figure 8 The rate control device for parallel coding based on virtual subframes proposed in this invention includes a subframe partitioning module 1, a subframe target bit allocation module 2, a virtual subframe partitioning module 3, and a virtual subframe coding task scheduling module 4. Figure 8 The device shown corresponds to Figure 1 The method shown.

[0065] The subframe division module 1 is used to divide the video frame to be encoded into one or more independent and non-overlapping subframes.

[0066] The subframe target bit allocation module 2 is used to allocate the target number of encoding bits R of the video frame to be encoded. frame The encoding complexity ω of the video frame to be encoded frame The encoding complexity ω of the subframe to be encoded subframe The target bit allocation is performed on the first subframe to be encoded within the video frame to be encoded, resulting in the target number of encoded bits R for the first subframe to be encoded. subframe . (Formula 1). ω frame An approximation is made using the encoding complexity of previously encoded video frames, ω. subframe An approximate substitution is performed using the encoding complexity of a subframe at the same position in a previously encoded video frame. The subframe target bit allocation module 2 is further configured to, when the encoding of a subframe within the video frame to be encoded is completed, allocate bits according to the target number R of the video frame to be encoded. frame The sum of the actual encoded bits of all encoded subframes within the video frame to be encoded. frame The encoding complexity ω of the next subframe to be encoded subframe_next The sum of the encoding complexity Σ of all subframes within the video frame to be encoded. {all not coded subframes} ω subframe Compensated target bit allocation is performed on the second and subsequent subframes to be encoded within the video frame to be encoded, and the target coded bit count R of the second and subsequent subframes to be encoded is calculated. subframe_next . (Formula 2). ω subframe_next An approximate replacement is performed using the encoding complexity of the subframe at the same position in a previously encoded video frame, Σ {all not coded subframes} ω subframe An approximate replacement is made by summing the encoding complexity of all subframes at the same position in the previously encoded video frame.

[0067] If the subframe partitioning module 1 divides the video frame to be encoded into only one subframe, the subframe target bit allocation module 2 directly sets R... subframe =R frame No calculation is required.

[0068] The virtual subframe partitioning module 3 is used to divide the subframe to be encoded into multiple virtual subframes according to the number of coding cores participating in parallel coding within the subframe and the virtual subframe partitioning rules. The virtual subframe partitioning rules are as follows: each image block row within the subframe to be encoded is sequentially and alternately assigned to each coding core according to the number of coding cores participating in parallel coding within the subframe, until all image block rows within the subframe to be encoded have been assigned. All the image block rows within the subframe to be encoded assigned to each coding core constitute one virtual subframe.

[0069] The virtual subframe coding task scheduling module 4 is used to schedule the coding cores participating in intra-subframe parallel coding based on virtual subframes, and to schedule each virtual subframe inside the subframe to be encoded to different coding cores for intra-subframe parallel coding.

[0070] This invention proposes a rate control method for intra-frame parallel coding based on virtual subframes. When using WPP technology for intra-frame parallel coding, this invention rationally divides the subframes to be encoded into virtual subframes and schedules and synchronizes the rate control process of these virtual subframes, ensuring the consistency of results when encoding subframes multiple times, and thus ensuring the consistency of results when encoding video frames multiple times. Using this rate control method, each coding core only needs to meet the basic synchronization requirements of WPP technology itself, eliminating the need for other strict synchronization measures, which greatly reduces the design complexity of multi-VPU hardware coding systems.

[0071] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A rate control method based on parallel coding of virtual subframes, characterized in that, Includes the following steps; Step S1: Divide the video frame to be encoded into one or more independent and non-overlapping subframes; Step S2: Based on the target number of encoded bits R of the video frame to be encoded frame The encoding complexity ω of the video frame to be encoded frame The encoding complexity ω of the subframe to be encoded subframe The target bit allocation is performed on the first subframe to be encoded within the video frame to be encoded, resulting in the target number of encoded bits R for the first subframe to be encoded. subframe ; (Formula 1); ω frame An approximation is made using the encoding complexity of previously encoded video frames, ω. subframe An approximate replacement is made using the encoding complexity of the subframe at the same position in a previously encoded video frame; When a subframe within the video frame to be encoded is completed, the target number of encoded bits R of the video frame is used as the encoding method. frame The sum of the actual encoded bits of all encoded subframes within the video frame to be encoded. frame The encoding complexity ω of the next subframe to be encoded subframe_next The sum of the encoding complexity Σ of all subframes within the video frame to be encoded. {all not coded subframes} ω subframe Compensated target bit allocation is performed on the second and subsequent subframes to be encoded within the video frame to be encoded, and the target coded bit count R of the second and subsequent subframes to be encoded is calculated. subframe_next ; (Formula 2); ω subframe_next An approximate replacement is performed using the encoding complexity of the subframe at the same position in a previously encoded video frame, Σ {all not coded subframes} ω subframe An approximate replacement is made by summing the encoding complexity of all subframes at the same position in the previously encoded video frame; If the video frame to be encoded is divided into only one sub-frame in step S1, then in step S2, R is directly set... subframe =R frame ; Step S3: Based on the number of coding cores participating in parallel coding within the subframe, the subframe to be encoded is divided into multiple virtual subframes according to the virtual subframe partitioning rules. The virtual subframe partitioning rules are as follows: each image block row within the subframe to be encoded is sequentially and alternately assigned to each coding core according to the number of coding cores participating in parallel coding within the subframe, until all image block rows within the subframe to be encoded are assigned. All the image block rows within the subframe to be encoded assigned to each coding core constitute a virtual subframe. Step S2 and step S3 may be performed simultaneously or either be performed first. Step S4: Based on the virtual subframes, schedule the coding cores participating in intra-subframe parallel coding, and schedule each virtual subframe within the subframe to be coded to different coding cores for intra-subframe parallel coding.

2. The rate control method based on parallel coding of virtual subframes according to claim 1, characterized in that, In step S1, the video frame to be encoded is divided into multiple slices using HEVC video coding technology, with each slice being a subframe; or, the video frame to be encoded is divided into multiple tiles using HEVC video coding technology, with each tile being a subframe.

3. The rate control method based on parallel coding of virtual subframes according to claim 1, characterized in that, In step S2, the video coding mode selection process is divided into a coding mode prediction stage and a coding mode rate-distortion optimization stage. The coding rate-distortion cost calculated in the coding mode prediction stage is called the prediction cost of the coding mode. The prediction cost corresponding to each coding mode is calculated for the image block to be coded, and the minimum prediction cost is taken as the coding complexity of the image block to be coded. The sum of the coding complexities of all image blocks in the virtual subframe is taken as the coding complexity of the virtual subframe. The sum of the coding complexities of all image blocks in the subframe is taken as the coding complexity of the subframe. The sum of the coding complexities of all image blocks in the video frame is taken as the coding complexity of the video frame.

4. The rate control method for parallel coding based on virtual subframes according to claim 1, characterized in that, In step S3, the virtual subframe is composed of image block rows; in H.265 encoding, the virtual subframe is composed of maximum coding unit (LCU) rows, and in H.264 encoding, the virtual subframe is composed of macroblock (MB) rows.

5. The rate control method for parallel coding based on virtual subframes according to claim 1, characterized in that, In step S3, the number of virtual subframes within a subframe to be encoded is equal to the number of encoding cores participating in parallel encoding within the subframe.

6. The rate control method based on parallel coding of virtual subframes according to claim 1, characterized in that, In step S3, each virtual subframe is confined to a subframe; after all virtual subframes within a subframe are encoded, the target encoding bit count for the next subframe is corrected according to Formula 2, and then the encoding of the next subframe begins.

7. The rate control method for parallel coding based on virtual subframes according to claim 1, characterized in that, In step S4, each encoding core records the encoding status information of each image block within the virtual subframe when encoding a virtual subframe; when all image blocks within the virtual subframe are encoded, the encoding status information of the encoded virtual subframes is statistically analyzed; when all virtual subframes within a subframe are encoded, synchronization is performed among the encoding cores, and the encoding status information of the encoded subframes is statistically analyzed; when all subframes within a video frame are encoded, the encoding status information of the encoded video frame is statistically analyzed; the encoding status information includes one or more of the actual number of encoded bits, encoding distortion D, encoding complexity, Lagrange multiplier λ, and quantization parameter QP.

8. The rate control method for parallel coding based on virtual subframes according to claim 1, characterized in that, The target bit allocation has five levels, from largest to smallest: image group, video frame, subframe, virtual subframe, and image block; parallel coding is performed within the subframe at the virtual subframe level; each virtual subframe includes multiple rows of image blocks within a subframe, that is, each virtual subframe is composed of multiple separate regions within the subframe.

9. The rate control method for parallel coding based on virtual subframes according to claim 1, characterized in that, Step S4 further includes the following sub-steps; Step S41: Based on the target number of coding bits R of the subframe to be encoded subframe The encoding complexity ω of the subframe to be encoded subframe The encoding complexity ω of the virtual subframe to be encoded virtual_subframe Target bit allocation is performed on each virtual subframe within the subframe to be coded to obtain the target coded bit count R for each virtual subframe. virtual_subframe ; (Formula 3); ω subframe An approximation is made using the encoding complexity of the subframe at the same position in a previously encoded video frame, ω. virtual_subframe An approximate replacement is made using the encoding complexity of a virtual subframe at the same position in a previously encoded subframe; Step S42: Based on the target number of encoded bits R of the virtual subframe to be encoded virtual_subframe The encoding complexity ω of the virtual subframe to be encoded virtual_subframe The encoding complexity ω of the image block to be encoded blk The target bit allocation is performed on the first image block within the virtual subframe to be encoded, resulting in the target number of encoded bits R for the first image block. blk ; (Formula 4); ω virtual_subframe An approximation is made using the encoding complexity of a virtual subframe at the same position in a previously encoded subframe, ω. blk An approximate replacement is made using the encoding complexity of the image patch at the same position in the previously encoded virtual subframe; When the encoding of an image block within a virtual subframe is complete, the target number of encoded bits R of the virtual subframe to be encoded is determined. virtual_subframe The sum of the actual encoded bits of all encoded image blocks within the virtual subframe to be encoded. virtual_subframe The encoding complexity ω of the next image to be encoded blk_next The sum of the encoding complexity Σ of all image blocks to be encoded within the virtual subframe to be encoded. {all not coded blks} ω blk Compensated target bit allocation is performed on the second and subsequent image blocks within the virtual subframe to be encoded, resulting in the target number of encoded bits R for the second and subsequent image blocks to be encoded. blk_next ; (Formula 5); ω blk_next An approximate replacement is performed using the encoding complexity of the image patch at the same position in the previously encoded virtual subframe, Σ {all not coded blks} ω blk An approximate replacement is made by summing the encoding complexity of all image blocks at the same position in the previously encoded virtual subframe; Step S43: Based on the target number of coding bits for each image block within the virtual subframe to be encoded and the coding status information of the image blocks at the same position in the encoded virtual subframe, target bit control is performed on each image block within the virtual subframe to be encoded to obtain the Lagrange multiplier λ and quantization parameter QP for each image block, which are used in the actual coding process of the image block.

10. A rate control device for parallel coding based on virtual subframes, characterized in that, It includes a subframe partitioning module, a subframe target bit allocation module, a virtual subframe partitioning module, and a virtual subframe coding task scheduling module; The subframe division module is used to divide the video frame to be encoded into one or more independent and non-overlapping subframes. The subframe target bit allocation module is used to allocate the target number of encoded bits R of the video frame to be encoded. frame The encoding complexity ω of the video frame to be encoded frame The encoding complexity ω of the subframe to be encoded subframe The target bit allocation is performed on the first subframe to be encoded within the video frame to be encoded, resulting in the target number of encoded bits R for the first subframe to be encoded. subframe ; (Formula 1); ω frame An approximation is made using the encoding complexity of previously encoded video frames, ω. subframe An approximate replacement is made using the encoding complexity of the subframe at the same position in a previously encoded video frame; The subframe target bit allocation module is further configured to, when the encoding of a subframe within the video frame to be encoded is completed, allocate bits according to the target number R of the video frame to be encoded. frame The sum of the actual encoded bits of all encoded subframes within the video frame to be encoded. frame The encoding complexity ω of the next subframe to be encoded subframe_next The sum of the encoding complexity Σ of all subframes within the video frame to be encoded. {all not coded subframes} ω subframe Compensated target bit allocation is performed on the second and subsequent subframes to be encoded within the video frame to be encoded, and the target coded bit count R of the second and subsequent subframes to be encoded is calculated. subframe_next ; (Formula 2); ω subframe_next An approximate replacement is performed using the encoding complexity of the subframe at the same position in a previously encoded video frame, Σ {all not coded subframes} ω subframe An approximate replacement is made by summing the encoding complexity of all subframes at the same position in the previously encoded video frame; If the subframe partitioning module divides the video frame to be encoded into only one subframe, the subframe target bit allocation module directly sets R... subframe =R frame ; The virtual subframe partitioning module is used to divide the subframe to be encoded into multiple virtual subframes according to the number of coding cores participating in parallel coding within the subframe and according to the virtual subframe partitioning rules. The virtual subframe partitioning rules are as follows: according to the number of coding cores participating in parallel coding within the subframe, each image block row inside the subframe to be encoded is sequentially and alternately assigned to each coding core until all image block rows inside the subframe to be encoded have been assigned. All the image block rows inside the subframe to be encoded assigned to each coding core constitute a virtual subframe. The virtual subframe coding task scheduling module is used to schedule the coding cores participating in intra-subframe parallel coding based on virtual subframes, and to schedule each virtual subframe within the subframe to be encoded to different coding cores for intra-subframe parallel coding.

Images