A method and device for controlling bit rate of inter-frame parallel coding based on virtual frame synchronization

Abstract

The present invention discloses a rate control method for inter-frame parallel coding based on virtual frame synchronization. A video frame to be encoded is divided into multiple virtual subframes; and then the multiple virtual subframes are combined into a virtual frame. The encoding core is scheduled based on the virtual frame, and each virtual subframe within the virtual frame is scheduled to different encoding cores for parallel encoding. According to the encoding status information of the current virtual frame, target bits are allocated to each virtual subframe within the next virtual frame to be encoded. The present invention performs an inter-frame parallel encoding process and a rate control process based on the virtual frame. The rate control process of the latter virtual frame can use the encoding status information of the adjacent previous virtual frame that has been completed encoded, so that the rate control is delay-free.

Description

Technical Field

[0001] The present invention relates to a digital video coding technology, in particular to a code rate control method for inter-frame parallel coding based on virtual frame synchronization. Background Art

[0002] Video coding is a technology that compresses redundant components in video images and uses the least amount of data possible to represent video information. Common video coding standards include HEVC (High Efficiency Video Coding, also known as H.265) and AVC (Advanced Video Coding, also known as H.264).

[0003] Video coding technology uses image blocks as the most basic coding unit. For example, in HEVC, the basic unit of video coding is the CU (Coding Unit). A CU can be an image block of 64 pixels x 64 pixels, 32 pixels x 32 pixels, 16 pixels x 16 pixels, or 8 pixels x 8 pixels. The 64 pixels x 64 pixels image block is also called the Largest Coding Unit (LCU).

[0004] To maximize video compression rates, common video coding algorithms often use lossy compression algorithms, resulting in differences (distortion) between the reconstructed and original video. Both HEVC and AVC are lossy compression algorithms. The coding performance of a lossy compression algorithm is measured by both the encoding bitrate (Rate) and the distortion (Distortion) introduced by the encoding. During the video encoding process, the encoding bitrate R and the encoding distortion (D) are mutually constrained and contradictory. Reducing the encoding bitrate R often increases the encoding distortion D, while reducing the encoding distortion D often increases the encoding bitrate R. Rate-distortion optimization (RDO) aims to minimize the encoding distortion D at a given encoding bitrate R, or to minimize the encoding bitrate R at a given encoding distortion D. Common video coding algorithms all use rate-distortion optimization methods based on the Lagrange multiplier λ. This method aims to minimize the rate-distortion cost (RD Cost) J, where J = D + λ × R.

[0005] During video encoding, each coding mode has coding distortion D and coding bit rate R. The rate-distortion cost J for each coding mode can be calculated. Selecting the coding mode with the lowest rate-distortion cost J during video encoding achieves optimal coding performance. The process by which the encoder selects the optimal coding mode for each image block to be encoded using rate-distortion optimization is called mode decision for that image block.

[0006] To remove information redundancy in both the spatial and temporal domains, video coding uses both intra-frame and inter-frame coding techniques to encode input video frames. I-frames are encoded using only intra-frame coding. P-frames and B-frames are encoded using a mix of intra-frame and inter-frame coding techniques. The type of encoded frame is determined by the user-set Group of Pictures (GOP) parameter. Figure 1 It shows a common picture group structure of I frame, P frame and B frame.

[0007] Motion estimation is the core of inter-frame coding technology. Its function is to eliminate the temporal information redundancy of the video signal, thereby improving the coding efficiency. Motion estimation is to search for the best matching block for the image block to be coded in the current coded video frame within a limited range (called the search window) in the coded video frame (called the reference frame), so as to minimize the rate distortion cost. Figure 2 The relative offset between the best matching block in the reference frame and the image block to be encoded in the current encoded video frame at the same position in the reference frame is the optimal motion vector (MV) of the image block to be encoded.

[0008] In real life, the channel bandwidth capacity used for video transmission is limited. If the video encoding bit rate is too high, exceeding the channel bandwidth capacity, it will cause video transmission congestion or even packet loss. If the video encoding bit rate is too low, the channel bandwidth will not be fully utilized, and high video quality cannot be achieved. Therefore, it is necessary to use rate control technology to control the output bit rate of the video encoder to match the channel bandwidth capacity.

[0009] The purpose of rate control is to adjust the encoding parameters of the video encoder so that the output bit rate of the video encoder is equal to the preset target bit rate, while minimizing encoding distortion to improve video encoding quality. Common rate control algorithms generally implement rate control tasks through two steps: target bit allocation and target bit control. Among them, the target bit allocation step is used to calculate the target number of coding bits for the image unit to be encoded at the image group, video frame, and image block levels. The target bit control step is used to calculate the corresponding coding parameters - such as the Lagrange multiplier λ and the quantization parameter (QP) - based on the target number of coding bits for the image unit to be encoded at the video frame and image block levels. These parameters are used in the encoding process of the image unit to be encoded.

[0010] The Lagrange multiplier λ and quantization parameter QP are both defined at the video frame level and the image block level. The video frame-level target bit allocation and target bit control (i.e., video frame-level bit rate control) utilize the video frame-level Lagrange multiplier λ and quantization parameter QP. The image block-level target bit allocation and target bit control (i.e., image block-level bit rate control) utilize the image block-level Lagrange multiplier λ and quantization parameter QP. The video frame-level Lagrange multiplier is also referred to as the video frame-level Lagrange multiplier, frame-level Lagrange multiplier, or video frame Lagrange multiplier. The image block-level Lagrange multiplier is also referred to as the image block-level Lagrange multiplier, block-level Lagrange multiplier, or image block Lagrange multiplier. The abbreviations for the quantization parameters are similar.

[0011] Because video encoding algorithms are computationally intensive, the industry typically uses hardware acceleration, such as application-specific integrated circuits (ASICs), to increase encoding speed and achieve real-time encoding and transmission. Hardware that accelerates the video encoding process is generally called a hardware video encoder, also known as a VPU (Video Processing Unit).

[0012] With the development of video coding technology, the compression rate of video coding is getting higher and higher, and the amount of video coding computation is also getting larger and larger. For high-resolution, high-frame-rate videos, real-time encoding must be guaranteed by parallel coding technology. One parallel coding technology is inter-frame parallel coding, which distributes several consecutive video frames to be encoded to different encoding cores for parallel encoding to increase the speed of video encoding. The "encoding core" here refers to a video encoding operation unit that can work in parallel. It can be either an encoding thread in a software video encoder or a hardware encoding module in a hardware video encoder. When performing inter-frame parallel coding, between the previous and next video frames that have an inter-frame reference relationship, it is necessary to ensure that the image block row to be encoded (i.e., an entire row of image blocks) has been encoded and reconstructed in the search window area corresponding to its reference frame.

[0013] See also Figure 3 , this is a schematic diagram of inter-frame parallel coding. Among them, there are four encoding cores participating in inter-frame parallel coding, namely encoding core 1 to encoding core 4, which are scheduled in sequence to perform encoding in parallel on different video frames. There are 6 video frames that need to be encoded, of which the first frame is an I frame (gray fill), and frames 2-6 are P frames or B frames (no fill). The diamond grid area represents the encoded part of the video frame. It can be seen that frames 1 and 2 have been encoded, completed by encoding core 1 and encoding core 2 respectively; frames 3, 4, and 5 are being encoded, and frame 6 is about to start encoding.

[0014] In traditional rate control algorithms, target bit allocation and target bit control for the video frame to be encoded are based on the encoding status of the previously encoded video frame. However, during inter-frame parallel encoding, multiple consecutive video frames are being encoded simultaneously. In other words, the encoding process of the video frame before the frame to be encoded may not have completed, which greatly complicates the design of the rate control algorithm.

[0015] Currently, there are some methods for bit rate control during inter-frame parallel encoding. One of the most common solutions is to use the encoding status information of the closest completed video frame to perform bit rate control for the video frame to be encoded. This bit rate control solution is as follows: Figure 4As shown in the figure. Frame 6 is the video frame to be encoded, and the encoding of frames 3, 4, and 5 is not yet completed. Frame 2 is the closest encoded video frame to frame 6. Therefore, frame 6 needs to use the encoding status information of frame 2 for rate control. Since there are three video frames between frame 2 and frame 6, the delay for rate control of frame 6 is three frames. It can be seen that although this inter-frame parallel encoding rate control scheme is simple to implement, the delay in the rate control process is very large. Moreover, the higher the parallelism of inter-frame parallel encoding, the more video frames are encoded simultaneously, the greater the delay in the rate control process, and the worse the rate control effect. For video encoding with motion scenes, adopting this scheme will lead to a significant reduction in the effect of rate control, greatly affecting the encoding efficiency. Summary of the Invention

[0016] The technical problem to be solved by the present invention is: how to reduce delay in the code rate control process of inter-frame parallel encoding.

[0017] To solve the above technical problems, the present invention proposes a rate control method for inter-frame parallel coding based on virtual frame synchronization, comprising the following steps. Step S1: Divide the video frame to be encoded into multiple virtual subframes; according to the number of coding cores participating in inter-frame parallel coding, divide each video frame to be encoded into multiple subframes with as equal heights as possible in the vertical direction, each subframe is composed of one or more image block rows, the width of each subframe is equal to the width of the video frame to be encoded, and the height of each subframe at least covers the search window of the first image block row within the subframe; each subframe after such division is a virtual subframe; for the same video sequence, the virtual subframe division method of all video frames to be encoded is consistent. Then, multiple virtual subframes are combined into a virtual frame; one or more virtual subframes that are adjacent in position within video frames with adjacent coding times and meet parallel coding conditions are combined into a virtual frame; the number of virtual subframes contained in a virtual frame is at least 1 and at most equal to the number of coding cores participating in inter-frame parallel coding. Step S2: Scheduling encoding cores based on virtual frames. Each virtual subframe within a virtual frame is assigned to a different encoding core for parallel encoding. Upon completion of virtual frame encoding, the encoding cores are synchronized and coding status information for the virtual frame, each virtual subframe within the virtual frame, and each image block within the virtual subframe is collected. Step S3: Target bit allocation is performed on each virtual subframe within the next virtual frame to be encoded based on the coding status information of the current virtual frame, thereby obtaining a target number of coding bits for each virtual subframe to be encoded. Step S4: Target bit control is performed on the virtual subframe to be encoded based on the target number of coding bits for the virtual subframe to be encoded and the coding status information of previously encoded virtual subframes at the same position, thereby obtaining a Lagrange multiplier λ and a quantization parameter QP for the virtual subframe to be encoded. Step S5: Target bit allocation is performed on each image block within the virtual subframe to be encoded based on the target number of coding bits for the virtual subframe to be encoded and the coding status information of image blocks within previously encoded virtual subframes at the same position, thereby obtaining a target number of coding bits for each image block within the virtual subframe to be encoded. Step S4 and step S5 may be performed in any order or simultaneously. Step S6: Based on the target number of coding bits for each image block within the virtual subframe to be encoded, combined with the Lagrange multiplier λ and quantization parameter QP of the virtual subframe to be encoded, and the encoding status information of the image blocks within the virtual subframe already encoded at the same position, target bit control is performed on each image block within the virtual subframe to be encoded to obtain the Lagrange multiplier λ and quantization parameter QP of each image block.

[0018] Preferably, in step S1, when H.265 video encoding is adopted, the virtual subframe is composed of a maximum coding unit LCU row; when H.264 video encoding is adopted, the virtual subframe is composed of a macroblock MB row.

[0019] Furthermore, in step S1, when the number of video frames to be encoded is M and the number of encoding cores participating in parallel encoding is N, the number of combined virtual frames is M+N-1, where M and N are both positive integers.

[0020] Furthermore, in step S2, the order of video encoding is carried out in the order of virtual frames, that is, encoding starts from the first virtual frame and continues to the last virtual frame; after all the virtual subframes within the current virtual frame are encoded, the virtual subframes on the next virtual frame begin to be encoded; at the video frame level, the encoding of the previous and next video frames is carried out in parallel; at the virtual frame level, the encoding of the previous and next virtual frames is carried out serially.

[0021] Preferably, in step S2, the virtual subframes belonging to the same video frame are always scheduled to be encoded on the same encoding core, so that the encoding state information of the same video frame is continued without the need for additional encoding state information transfer.

[0022] Furthermore, in step S2, the coding status information of the virtual subframe and the coding status information of each image block within the virtual subframe include one or more of the following: the actual number of coded bits of the virtual subframe or each image block within it, distortion cost, prediction cost, Lagrange multiplier λ, and quantization parameter QP. Upon completion of virtual frame coding, the coding core is synchronized, the actual number of coded bits of each virtual subframe is summed to obtain the actual number of coded bits of the virtual frame, and the total number of virtual subframes within all currently encoded virtual frames and the actual number of coded bits of all currently encoded virtual frames in the video sequence are counted; this serves as the coding status information of the virtual frame.

[0023] Furthermore, in step S3, when a virtual frame is encoded, there is often an error between the actual number of encoded bits of each virtual subframe within it and the target number of encoded bits calculated before encoding. This error is smoothed in a sliding window composed of multiple subsequent virtual subframes. The average target number of encoded bits of the virtual subframes in the sliding window is first calculated. The target coding bit numbers of different types of virtual subframes are calculated based on the size ratio of the target coding bit numbers between virtual subframes of different types of video frames and the picture group structure of video coding.

[0024] Furthermore, in step S3, the average target number of coding bits of the virtual subframes in the sliding window is Calculated by the following formula: in, is the average target number of coded bits of the virtual subframe within the sliding window, R avg is the average target coding bit number of each virtual subframe calculated from the target coding rate of the video sequence, N coded is the number of virtual subframes currently encoded in the video sequence, N SW is the size of the sliding window, R coded It is the actual number of coded bits of all currently coded virtual frames in the video sequence, and also the actual number of coded bits of all currently coded virtual subframes.

[0025] Furthermore, in step S5, the target number of coding bits for each image block within the virtual subframe to be encoded is calculated as follows: Among them, R blk_cur is the target number of coding bits for an image block at a certain position within the virtual subframe to be coded, ω blk_cur is the predicted cost value of the image block at the same position in the previously encoded virtual subframe, ω pic is the sum of the predicted cost values ​​of all image blocks in the previously coded virtual subframe, R pic is the target number of coded bits for the current virtual subframe to be coded.

[0026] The present invention also proposes a rate control device for inter-frame parallel coding based on virtual frame synchronization, comprising a virtual subframe and virtual frame division module, a virtual frame coding task scheduling module, a virtual subframe target bit allocation module, a virtual subframe target bit control module, an image block target bit allocation module, and an image block target bit control module. The virtual subframe and virtual frame division module are used to divide a video frame to be coded into multiple virtual subframes, and according to the number of coding cores participating in inter-frame parallel coding, each video frame to be coded is divided into multiple subframes with as equal heights as possible in the vertical direction, each subframe is composed of one or more image block rows, the width of each subframe is equal to the width of the video frame to be coded, and the height of each subframe at least covers the search window of the first image block row within the subframe; each subframe after such division is a virtual subframe; for the same video sequence, the virtual subframe division method of all video frames to be coded is consistent; the virtual subframe and virtual frame division module is also used to combine multiple virtual subframes into a virtual frame; and one or more virtual subframes that are adjacent in position within video frames with adjacent coding times and meet parallel coding conditions are combined into one virtual frame. The number of virtual subframes contained in a virtual frame is at least 1 and at most equal to the number of encoding cores participating in inter-frame parallel encoding. The virtual frame encoding task scheduling module is used to schedule the encoding cores based on the virtual frame, schedule each virtual subframe in the virtual frame to different encoding cores for parallel encoding, and synchronize the encoding cores when the virtual frame encoding is completed, and count the encoding status information of the virtual frame, each virtual subframe in the virtual frame, and each image block in the virtual subframe. The virtual subframe target bit allocation module is used to allocate target bits to each virtual subframe in the next virtual frame to be encoded based on the encoding status information of the current virtual frame, and obtain the target number of encoding bits for each virtual subframe to be encoded. The virtual subframe target bit control module is used to control the target bit of the virtual subframe to be encoded based on the target number of encoding bits of the virtual subframe to be encoded, combined with the encoding status information of the virtual subframe already encoded at the same position, and obtain the Lagrange multiplier λ and quantization parameter QP of the virtual subframe to be encoded. The image block target bit allocation module is used to perform target bit allocation for each image block within the virtual subframe to be encoded based on the target number of coding bits of the virtual subframe to be encoded, combined with the coding status information of the image blocks within the virtual subframe already encoded at the same position, to obtain the target number of coding bits for each image block within the virtual subframe to be encoded. The image block target bit control module is used to perform target bit control for each image block within the virtual subframe to be encoded based on the target number of coding bits for each image block within the virtual subframe to be encoded, combined with the Lagrange multiplier λ and quantization parameter QP of the virtual subframe to be encoded, and the coding status information of the image blocks within the virtual subframe already encoded at the same position, to obtain the Lagrange multiplier λ and quantization parameter QP of each image block.

[0027] The technical effect achieved by the present invention is: based on virtual frames, an inter-frame parallel encoding process and a bit rate control process are performed, and the bit rate control process of the latter virtual frame can use the encoding status information of the adjacent previous virtual frame that has been completed encoded, so that the bit rate control is delay-free. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a schematic diagram of the common image group structure of I frame, P frame, and B frame.

[0029] Figure 2 It is a simple diagram of the motion estimation process.

[0030] Figure 3 It is a simple diagram of inter-frame parallel coding.

[0031] Figure 4 The present invention is a simple schematic diagram of an existing method for rate control during inter-frame parallel encoding.

[0032] Figure 5 It is a flow chart of a code rate control method for inter-frame parallel coding based on virtual frame synchronization proposed by the present invention.

[0033] Figure 6 It is a schematic diagram of dividing a video frame into multiple virtual subframes.

[0034] Figure 7 This is a schematic diagram of an example of how a virtual frame is divided.

[0035] Figure 8 It is a structural diagram of a bit rate control device for inter-frame parallel coding based on virtual frame synchronization proposed by the present invention.

[0036] Explanation of the reference numerals in the figure: 1 is the virtual subframe and virtual frame division module, 2 is the virtual frame encoding task scheduling module, 3 is the virtual subframe target bit allocation module, 4 is the virtual subframe target bit control module, 5 is the image block target bit allocation module, and 6 is the image block target bit control module. DETAILED DESCRIPTION

[0037] See also Figure 5 The bit rate control method of inter-frame parallel coding based on virtual frame synchronization proposed by the present invention includes the following steps.

[0038] Step S1: Divide a video frame to be encoded into multiple virtual sub-frames, and then combine the multiple virtual sub-frames into a virtual frame.

[0039] The concepts of virtual subframes and virtual frames are not existing, but are proposed by the present invention. In the present invention, "virtual subframes" are not video coding units specified in video coding standards such as H.264 and H.265, but are artificially divided rate control units for the purpose of rate control for inter-frame parallel coding. In the present invention, virtual subframes participate in rate control as a first-level target bit allocation. Existing target bit allocation is performed at three levels: picture group, video frame, and image block. The target bit allocation of the present invention is performed at four levels: picture group, virtual frame, virtual subframe, and image block.

[0040] In this step, the virtual subframe division rule is to divide each video frame to be encoded into multiple subframes with the same height as much as possible in the vertical direction according to the number of encoding cores participating in inter-frame parallel encoding. Each subframe consists of one or more image block rows (i.e., one or more rows of image blocks). The width of each subframe is equal to the width of the video frame to be encoded, the height of each subframe is as equal as possible, and the height of each subframe must at least cover the search window of the first image block row in the subframe; each subframe divided in this way is a virtual subframe. For example, in H.265 encoding, the virtual subframe consists of LCU rows, and in H.264 encoding, the virtual subframe consists of MB (Micro Block) rows. For the same video sequence, the virtual subframe division method of all video frames to be encoded is consistent. Please refer to Figure 6 This diagram illustrates how a video frame is divided into multiple virtual subframes. During inter-frame parallel coding, the prerequisite for parallel coding between the previous and next video frames is that the search window area corresponding to the image block rows to be encoded in the reference frame has been reconstructed. When dividing the virtual subframes, the height of the virtual subframe must at least cover the search window of the first image block row within the virtual subframe. This ensures that the encoding process of each virtual subframe belonging to the same virtual frame can be synchronized.

[0041] In this step, if the number of virtual subframes of the video frame is too large or too small, it will have adverse effects. If there are too many virtual subframes, the height of each virtual subframe will be small, which is not conducive to the rate control of the image blocks within the virtual subframe, and the cost of synchronization between the virtual subframes will be higher. If there are too few virtual subframes, the parallelism of inter-frame parallel coding will be reduced, resulting in a waste of coding cores. Preferably, the virtual subframes are divided according to the number of coding cores participating in inter-frame parallel coding, that is, the number of virtual subframes divided into a video frame is equal to the number of coding cores participating in inter-frame parallel coding.

[0042] In the present invention, a "virtual frame" is a virtual video frame composed of one or more virtual subframes that are spatially adjacent (i.e., positionally adjacent) within video frames that are adjacent in coding time and that meet the conditions for parallel coding. The purpose of combining multiple virtual subframes into a virtual frame is to facilitate coding synchronization between virtual subframes belonging to the same virtual frame, and to facilitate target bit allocation and coding status information statistics for virtual subframes. The number of virtual subframes contained within a virtual frame is at least 1 and at most equal to the number of coding cores participating in inter-frame parallel coding.

[0043] See also Figure 7 , which is an example of the combination of virtual frames in the present invention. There are four encoding cores participating in inter-frame parallel encoding, namely encoding core 1-encoding core 4, which are scheduled in sequence to perform parallel encoding on different video frames. There are 6 video frames that need to be encoded, of which the first frame is an I frame (gray), and frames 2-6 are P frames or B frames (colorless). It can be seen that since there are 4 encoding cores, each video frame is divided into 4 virtual subframes. For example, the first frame is divided into 4 virtual subframes, namely 1-1, 1-2, 1-3, and 1-4. In the present invention, a virtual frame is composed of one or more virtual subframes that are spatially adjacent within video frames that are adjacent in encoding time and have parallel encoding conditions. In Figure 7 In the example, each virtual frame is represented by a dotted oblique line. Table 1 lists which virtual subframes each virtual frame is composed of.

[0044]

[0045]

[0046] Table 1: Figure 7 The composition of each virtual frame in

[0047] exist Figure 7In the example, frame 1 and frame 2 belong to video frames with adjacent encoding times, frame 2 and frame 3 belong to video frames with adjacent encoding times, and so on. Virtual subframes 1-1 to 6-1 are located in different video frames. Although their sampling times are different, they are of the same size and located in the same spatial position, so they belong to virtual subframes at the same position (for example, called the first position). Similarly, virtual subframes 1-2 to 6-2, virtual subframes 1-3 to 6-3, and virtual subframes 1-4 to 6-4 also belong to virtual subframes at the same position (for example, called the second position, the third position, and the fourth position, respectively). Any virtual subframe at the first position and any virtual subframe at the second position belong to virtual subframes with adjacent positions, and any virtual subframe at the second position and any virtual subframe at the third position belong to virtual subframes with adjacent positions, and so on. "Having the conditions for parallel encoding" means that each virtual subframe (for example, virtual subframe 5-1) has been encoded and reconstructed in the virtual subframe (virtual subframe 4-1) at the same position in the previous video frame. For example, for virtual subframe 5-1, the prerequisite for it to start encoding is that the search window area corresponding to the image block on the first image block row inside it (located in virtual subframe 4-1) must have been encoded and reconstructed, otherwise motion estimation cannot be performed. When dividing the video frame into virtual subframes, it has been limited that the height of all virtual subframes must at least cover the search window of the first image block row inside the virtual subframe. Therefore, the prerequisite for virtual subframe 5-1 to start encoding is that virtual subframe 4-1 has been encoded and reconstructed. "Having parallel encoding conditions" determines Figure 7 The diagonal line in the figure must connect one or more virtual subframes from the lower left corner to the upper right corner.

[0048] When the number of video frames to be encoded is greater, the division of virtual subframes can be deduced from the situation in this example. According to the virtual frame combination rule of the present invention, when the number of video frames to be encoded is M and the number of encoding cores participating in parallel encoding is N, the number of virtual frames combined is M+N-1, where M and N are both positive integers.

[0049] Step S2: Schedule the encoding core based on the virtual frame, schedule each virtual subframe within the virtual frame to different encoding cores for parallel encoding, synchronize the encoding core when the virtual frame encoding is completed, and count the encoding status information of the virtual frame, each virtual subframe within the virtual frame, and each image block within the virtual subframe.

[0050] In the present invention, since the height of the virtual subframe must at least cover the search window of the first image block row within the virtual subframe, the virtual subframes belonging to the same virtual frame meet the conditions for parallel encoding and can be assigned to different encoding cores for parallel encoding. Parallel encoding can start and end at the same time, that is, each encoding core can be synchronized at the virtual frame level. After all the virtual subframes within the current virtual frame are encoded, the virtual subframes on the next virtual frame begin encoding. Figure 7 For example, when virtual frame 4 is finished encoding, virtual frame 5 is about to start encoding.

[0051] In this step, during the scheduling process, it is preferred that virtual subframes with the same sampling time (i.e., virtual subframes belonging to the same video frame) are always scheduled for encoding on the same encoding core; that is, through reasonable scheduling, when encoding different virtual frames, the encoding of virtual subframes belonging to the same video frame is completed on the same encoding core, so that the encoding state information of the same video frame - such as CABAC (Context-Based Adaptive Binary Arithmetic Coding) state information - can be continued without the need for additional encoding state information transfer. Figure 7 For example, when encoding virtual frame 1, virtual subframe 1-1 is scheduled for encoding on encoding core 1. When encoding virtual frame 2, virtual subframe 1-2, which belongs to the same frame as virtual subframe 1-1, is still scheduled for encoding on encoding core 1, and so on, until the encoding of virtual frame 4 is completed, virtual subframes 1-1, 1-2, 1-3, and 1-4, which belong to the same frame as virtual subframe 1, are all scheduled for encoding on encoding core 1. Figure 7 It can be seen from FIG that after adopting this preferred virtual subframe scheduling method, in the process of encoding virtual frame 1 to virtual frame 9, virtual subframes belonging to the same video frame are all encoded on the same encoding core.

[0052] In this step, each virtual subframe within the virtual frame will be scheduled to different encoding cores for parallel encoding. When encoding a virtual subframe, each encoding core needs to record the encoding status information of the virtual subframe and the encoding status information of each image block within the virtual subframe, for use in the rate control process of the virtual subframe at the same position within the subsequent virtual frame and its internal image blocks. The encoding status information of each virtual subframe and each image block within the virtual subframe that needs to be recorded is determined by the requirements of the specific rate control algorithm. It can be the actual number of encoding bits of the virtual subframe and each image block within it, the distortion cost (such as square error and SSD), the prediction cost (such as absolute error and SAD or absolute transform error and SATD), the Lagrange multiplier λ, the quantization parameter QP, etc., which will not be repeated here.

[0053] In this step, the encoding core is synchronized when the virtual frame encoding is completed, and the encoding status information of the virtual frame, each virtual subframe within the virtual frame, and each image block within the virtual subframe is counted. Among them, the encoding status information of the virtual subframe and each image block within it is counted and recorded by each encoding core, and then the actual number of encoding bits of each virtual subframe is summed up to obtain the actual number of encoding bits of the virtual frame. At the same time, it is also necessary to count the total number of virtual subframes within all virtual frames that have been encoded so far (that is, the number of virtual subframes currently encoded in the video sequence) and the actual number of encoding bits of all virtual frames currently encoded in the video sequence (obtained by accumulating the actual number of encoding bits of each encoded virtual frame). These serve as the encoding status information of the virtual frame.

[0054] In this step, video encoding is performed in the order of virtual frames, starting with the first virtual frame and continuing until the last virtual frame. It can be seen that although the encoding of the previous and next video frames is performed in parallel at the video frame level, at the virtual frame level, the encoding of the previous and next virtual frames is performed serially. By performing rate control based on these serially encoded virtual frames, the rate control process of the subsequent virtual frame can utilize the encoding status information of the adjacent, previously encoded virtual frame without affecting the parallel encoding of the previous and next video frames, achieving delay-free rate control.

[0055] Step S3: performing target bit allocation on each virtual subframe within the next virtual frame to be encoded according to the encoding state information of the current virtual frame, and obtaining a target number of encoding bits for each virtual subframe to be encoded.

[0056] In this step, when a virtual frame is encoded, the actual number of coded bits for each virtual subframe within it often differs from the target number of coded bits calculated before encoding. This error can be smoothed in a sliding window (SW) consisting of multiple subsequent virtual subframes to reduce instantaneous bit rate fluctuations and make bit rate control more accurate. The number of virtual subframes contained in the sliding window is called the sliding window size, and the average target number of coded bits for the virtual subframes within it can be calculated using the following formula: in, is the average target number of coded bits of the virtual subframe within the sliding window, R avg The average target coding bit number of each virtual subframe is calculated from the target coding rate of the video sequence, and R is calculated from the target coding rate of the video sequence. avg The method is: N coded is the number of virtual subframes currently encoded in the video sequence, N SWis the size of the sliding window, R coded is the actual number of coded bits of all currently coded virtual frames in the video sequence (also the actual number of coded bits of all currently coded virtual subframes). The value of will change with the encoding status of the video sequence.

[0057] When a virtual frame is encoded, calculate The target bit allocation of each virtual subframe in the next virtual frame is performed accordingly. Since the next virtual frame may contain virtual subframes of different types (i.e., belonging to I frames, P frames, and B frames), when performing target bit allocation of virtual subframes, one method is to perform the target bit allocation based on the size ratio relationship of the target coding bit numbers between virtual subframes of different types of video frames that are preset. For example, the size of the target coding bit number of the virtual subframe belonging to the I frame is generally set to 6-10 times the size of the target coding bit number of the virtual subframe belonging to the P frame, and the size of the target coding bit number of the virtual subframe belonging to the B frame is generally set to 1 / 4-1 / 2 times the size of the target coding bit number of the virtual subframe belonging to the P frame. According to these preset size ratio relationships of the target coding bit numbers of virtual subframes of different types, combined with the picture group structure of video coding and The target number of coding bits for different types of virtual subframes can be calculated based on the value of , which will not be described here.

[0058] Step S4: According to the target number of coding bits of the virtual subframe to be coded and the coding status information of the coded virtual subframe at the same position, the target bit of the virtual subframe to be coded is controlled to obtain the Lagrange multiplier λ and quantization parameter QP of the virtual subframe to be coded.

[0059] This step calculates the λ and QP of the virtual subframe to be encoded, and different rate control algorithms can be used. For example, for H.265 encoding, the rate control algorithm based on the relationship model between the encoding bit rate R and the Lagrange multiplier λ (i.e., the R-λ model) in the JCTVC-K0103 proposal can be used to calculate the λ and QP of the virtual subframe. I will not go into details here. Virtual subframes do not belong to the prior art. The Lagrange multiplier λ and quantization parameter QP at the virtual subframe level that appear here can refer to the calculation method of the Lagrange multiplier λ and quantization parameter QP at the video frame level.

[0060] After the λ and QP of the virtual subframe are calculated, they can be used as constraints on the λ and QP of each image block within the virtual subframe to participate in the target bit control process of the image block, so that the changes in λ and QP of each image block within the virtual subframe will not be too drastic, thereby reducing the fluctuation of video quality between each image block within the virtual subframe.

[0061] Step S5: Based on the target number of coding bits of the virtual subframe to be encoded and the coding status information of the image blocks within the encoded virtual subframe at the same position, target bits are allocated to each image block within the virtual subframe to be encoded to obtain the target number of coding bits for each image block within the virtual subframe to be encoded.

[0062] In this step, the target bit allocation for each image block in the virtual subframe to be encoded is performed using the predicted cost value of the image block in the previously encoded virtual subframe at the same position. The target coding bit number of the image block is calculated as follows: Among them, R blk_cur is the target number of coding bits for an image block at a certain position within the virtual subframe to be coded, ω blk_cur is the predicted cost value of the image block at the same position in the previously encoded virtual subframe, ω pic is the sum of the predicted cost values ​​of all image blocks in the previously coded virtual subframe, R pic is the target number of coded bits for the current virtual subframe to be coded.

[0063] There is no strict restriction on the order of step S4 and step S5, and they can be performed in any order, or simultaneously.

[0064] Step S6: Based on the target number of coding bits for each image block within the virtual subframe to be encoded, combined with the Lagrange multiplier λ and quantization parameter QP of the virtual subframe to be encoded, and the encoding status information of the image blocks within the encoded virtual subframe at the same position, target bit control is performed on each image block within the virtual subframe to be encoded, and the Lagrange multiplier λ and quantization parameter QP of each image block are obtained, which are used in the actual encoding process of the image block.

[0065] This step calculates the λ and QP for each image block within the virtual subframe to be encoded. Various rate control algorithms can be used for this. For example, for H.265 encoding, the rate control algorithm based on the relationship between the encoding bit rate R and the Lagrange multiplier λ (i.e., the R-λ model) proposed in the JCTVC-K0103 proposal can be used to calculate the λ and QP for each image block. This will not be further explained here.

[0066] See also Figure 8 The bit rate control device for inter-frame parallel coding based on virtual frame synchronization proposed in the present invention includes a virtual subframe and virtual frame division module 1, a virtual frame coding task scheduling module 2, a virtual subframe target bit allocation module 3, a virtual subframe target bit control module 4, an image block target bit allocation module 5 and an image block target bit control module 6. Figure 8 The device shown corresponds to Figure 5 The method shown.

[0067] The virtual subframe and virtual frame division module 1 is used to divide the video frame to be encoded into multiple virtual subframes. According to the number of encoding cores participating in inter-frame parallel encoding, each video frame to be encoded is divided vertically into multiple subframes of as equal height as possible. Each subframe is composed of one or more image block rows. The width of each subframe is equal to the width of the video frame to be encoded, and the height of each subframe at least covers the search window of the first image block row within the subframe. Each subframe thus divided is a virtual subframe. For the same video sequence, the virtual subframe division method of all video frames to be encoded remains consistent. Multiple virtual subframes are then combined into a virtual frame. One or more virtual subframes that are adjacent in position within video frames with adjacent encoding times and meet the parallel encoding conditions are combined into a virtual frame. The number of virtual subframes contained in a virtual frame is at least one and at most equal to the number of encoding cores participating in inter-frame parallel encoding.

[0068] The virtual frame encoding task scheduling module 2 is used to schedule the encoding core based on the virtual frame, schedule each virtual subframe within the virtual frame to different encoding cores for parallel encoding, and synchronize the encoding core when the virtual frame encoding is completed, and count the encoding status information of the virtual frame, each virtual subframe within the virtual frame, and each image block within the virtual subframe.

[0069] The virtual subframe target bit allocation module 3 is used to perform target bit allocation on each virtual subframe in the next virtual frame to be encoded according to the encoding state information of the current virtual frame, so as to obtain the target encoding bit number of each virtual subframe to be encoded.

[0070] The virtual subframe target bit control module 4 is used to perform target bit control on the virtual subframe to be encoded based on the target number of encoding bits of the virtual subframe to be encoded and the encoding status information of the virtual subframe encoded at the same position, so as to obtain the Lagrange multiplier λ and quantization parameter QP of the virtual subframe to be encoded.

[0071] The image block target bit allocation module 5 is used to allocate target bits to each image block within the virtual subframe to be encoded based on the target number of encoding bits of the virtual subframe to be encoded and the encoding status information of the image blocks within the encoded virtual subframe at the same position, so as to obtain the target number of encoding bits for each image block within the virtual subframe to be encoded.

[0072] The image block target bit control module 6 is used to perform target bit control on each image block within the virtual subframe to be encoded based on the target coding bit number of each image block within the virtual subframe to be encoded, combined with the Lagrange multiplier λ and quantization parameter QP of the virtual subframe to be encoded, and the encoding status information of the image blocks within the encoded virtual subframe at the same position, to obtain the Lagrange multiplier λ and quantization parameter QP of each image block for the actual encoding process of the image block.

[0073] The present invention proposes a rate control method for inter-frame parallel coding based on virtual frame synchronization. This method rationally divides the video frame to be encoded into virtual subframes, then combines multiple virtual subframes into virtual frames. The parallel encoding process and rate control process of the video frames are scheduled and synchronized based on the virtual frames. While ensuring the parallel encoding of the video frames to be encoded, the rate control process of the subsequent virtual frame can utilize the encoding status information of the adjacent, previously encoded virtual frame. This allows the rate control process based on virtual frames to be performed serially, achieving delay-free rate control and significantly improving the rate control effect during inter-frame parallel coding.

[0074] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A rate control method for inter-frame parallel coding based on virtual frame synchronization, characterized in that: The method includes the following steps: Step S1: Divide the video frame to be encoded into multiple virtual subframes; divide each video frame to be encoded into multiple subframes in the vertical direction according to the number of encoding cores participating in inter-frame parallel encoding, where the number of subframes is equal to the number of encoding cores, each subframe is composed of one or more image block rows, the width of each subframe is equal to the width of the video frame to be encoded, and the height of each subframe at least covers the search window of the first image block row within the subframe; each subframe divided in this way is a virtual subframe; for the same video sequence, the virtual subframe division method of all video frames to be encoded is consistent; One or more virtual subframes that are adjacent in position within video frames with adjacent coding times and meet parallel coding conditions are combined into a virtual frame; the number of virtual subframes contained in a virtual frame is at least one and at most equal to the number of coding cores participating in inter-frame parallel coding; the parallel coding condition is that the search window area corresponding to the image block row to be coded in its reference frame has been coded and reconstructed; Step S2: Scheduling the encoding cores based on the virtual frame, scheduling each virtual subframe within the virtual frame to different encoding cores for parallel encoding, synchronizing the encoding cores when the virtual frame encoding is completed, and collecting statistics on the encoding status of the virtual frame, each virtual subframe within the virtual frame, and each image block within the virtual subframe; Step S3: performing target bit allocation on each virtual subframe within the next virtual frame to be encoded according to the encoding state information of the current virtual frame, and obtaining a target number of encoding bits for each virtual subframe to be encoded; In step S3, the average target coding bit number of the virtual subframes in the sliding window composed of multiple virtual subframes is calculated. The target coding bit numbers of different types of virtual subframes are calculated based on the target coding bit ratio between virtual subframes of different types of video frames and the picture group structure of video coding; Step S4: According to the target number of coding bits of the virtual subframe to be coded and in combination with the coding state information of the coded virtual subframe at the same position, the target bit of the virtual subframe to be coded is controlled to obtain the Lagrange multiplier λ and quantization parameter QP of the virtual subframe to be coded; Step S5: performing target bit allocation on each image block within the virtual subframe to be encoded based on the target number of encoding bits of the virtual subframe to be encoded and combining the encoding status information of the image blocks within the encoded virtual subframe at the same position to obtain the target number of encoding bits for each image block within the virtual subframe to be encoded; The order of step S4 and step S5 may be any one before the other, or performed simultaneously; Step S6: Based on the target number of coding bits for each image block within the virtual subframe to be encoded, combined with the Lagrange multiplier λ and quantization parameter QP of the virtual subframe to be encoded, and the encoding state information of the image blocks within the encoded virtual subframe at the same position, target bit control is performed on each image block within the virtual subframe to be encoded, and the Lagrange multiplier λ and quantization parameter QP of each image block are obtained.

2. The bit rate control method for inter-frame parallel coding based on virtual frame synchronization according to claim 1, characterized in that: In step S1, when H.265 video encoding is adopted, the virtual subframe is composed of a maximum coding unit LCU row, and when H.264 video encoding is adopted, the virtual subframe is composed of a macroblock MB row.

3. The bit rate control method for inter-frame parallel coding based on virtual frame synchronization according to claim 1, characterized in that: In step S1, when the number of video frames to be encoded is M and the number of encoding cores participating in parallel encoding is N, the number of combined virtual frames is M+N-1, where M and N are both positive integers.

4. The bit rate control method for inter-frame parallel coding based on virtual frame synchronization according to claim 1, characterized in that: In step S2, the order of video encoding is carried out in the order of virtual frames, that is, encoding starts from the first virtual frame and continues to the last virtual frame; after all the virtual subframes in the current virtual frame are encoded, the virtual subframes on the next virtual frame begin to be encoded; at the video frame level, the encoding of the previous and next video frames is carried out in parallel; at the virtual frame level, the encoding of the previous and next virtual frames is carried out serially.

5. The bit rate control method for inter-frame parallel coding based on virtual frame synchronization according to claim 1, wherein: In step S2, the virtual subframes belonging to the same video frame are always scheduled to be encoded on the same encoding core, so that the encoding state information of the same video frame is continued without the need for additional encoding state information transfer.

6. The bit rate control method for inter-frame parallel coding based on virtual frame synchronization according to claim 1, characterized in that: In step S2, the coding status information of the virtual subframe and the coding status information of each image block within the virtual subframe include one or more of the following: the actual number of coding bits of the virtual subframe or each image block within it, the distortion cost, the prediction cost, the Lagrange multiplier λ, and the quantization parameter QP; The coding status information of the virtual frame includes: the actual coding bit number of the virtual frame obtained by summing the actual coding bit number of each virtual subframe, the total number of virtual subframes within all virtual frames that have been encoded so far, and the actual coding bit number of all virtual frames currently encoded in the video sequence.

7. The bit rate control method for inter-frame parallel coding based on virtual frame synchronization according to claim 1, characterized in that: In step S5, the target number of coding bits for each image block within the virtual subframe to be encoded is calculated as follows: ;in, is the target number of coding bits for an image block at a certain position within the virtual subframe to be coded, is the predicted cost value of the image block at the same position in the previously encoded virtual subframe, is the sum of the predicted cost values ​​of all image blocks in the previously encoded virtual subframe, is the target number of coded bits for the current virtual subframe to be coded.

8. A rate control device for inter-frame parallel coding based on virtual frame synchronization, characterized in that: It includes a virtual subframe and virtual frame division module, a virtual frame encoding task scheduling module, a virtual subframe target bit allocation module, a virtual subframe target bit control module, an image block target bit allocation module and an image block target bit control module; The virtual subframe and virtual frame division module is used to divide the video frame to be encoded into multiple virtual subframes, and divide each video frame to be encoded into multiple subframes in the vertical direction according to the number of encoding cores participating in inter-frame parallel encoding. The number of subframes is equal to the number of encoding cores, each subframe is composed of one or more image block rows, the width of each subframe is equal to the width of the video frame to be encoded, and the height of each subframe at least covers the search window of the first image block row within the subframe; each subframe after such division is a virtual subframe; for the same video sequence, the virtual subframe division method of all video frames to be encoded is consistent; the virtual subframe and virtual frame division module is also used to combine one or more virtual subframes that are adjacent in position within video frames with adjacent encoding time and have parallel encoding conditions into one virtual frame. The number of virtual subframes contained in a virtual frame is at least one and at most equal to the number of encoding cores participating in inter-frame parallel encoding. The parallel encoding condition is that the search window area corresponding to the image block row to be encoded in its reference frame has been encoded and reconstructed. The virtual frame encoding task scheduling module is used to schedule encoding cores based on virtual frames, schedule each virtual subframe within a virtual frame to different encoding cores for parallel encoding, synchronize the encoding cores when virtual frame encoding is completed, and collect statistics on encoding status information of the virtual frame, each virtual subframe within the virtual frame, and each image block within the virtual subframe; The virtual subframe target bit allocation module is used to allocate target bits to each virtual subframe within the next virtual frame to be encoded according to the encoding state information of the current virtual frame, and obtain the target encoding bit number of each virtual subframe to be encoded; calculate the average target encoding bit number of virtual subframes in the sliding window composed of multiple virtual subframes The target coding bit numbers of different types of virtual subframes are calculated based on the target coding bit ratio between virtual subframes of different types of video frames and the picture group structure of video coding; The virtual subframe target bit control module is used to perform target bit control on the virtual subframe to be encoded based on the target number of coding bits of the virtual subframe to be encoded and the coding state information of the virtual subframe already encoded at the same position, so as to obtain the Lagrange multiplier λ and quantization parameter QP of the virtual subframe to be encoded; The image block target bit allocation module is used to perform target bit allocation on each image block in the virtual subframe to be encoded based on the target number of coding bits of the virtual subframe to be encoded and in combination with the coding state information of the image blocks in the virtual subframe that has been encoded at the same position, so as to obtain the target number of coding bits for each image block in the virtual subframe to be encoded; The image block target bit control module is used to perform target bit control on each image block within the virtual subframe to be encoded based on the target coding bit number of each image block within the virtual subframe to be encoded, combined with the Lagrange multiplier λ and quantization parameter QP of the virtual subframe to be encoded, and the encoding state information of the image block within the virtual subframe encoded at the same position, to obtain the Lagrange multiplier λ and quantization parameter QP of each image block.

Images