Video processing method and device, video coding method and device, electronic equipment and storage medium

Abstract

The invention relates to a video processing method and device, a video coding method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a to-be-processed video, and segmenting a video frame in the to-be-processed video to obtain coding blocks; calculating the complexity and bit rate error of the coding block; calculating by using the complexity and the bit rate error to obtain an increment quantization parameter; determining a quantization parameter of the frame according to a frame-level code rate control algorithm; calculating a target quantization parameter by using the increment quantization parameter and the quantization parameter of the frame; and performing encoding processing on the encoding block based on the target quantization parameter. And summing the calculated increment quantization parameter and the frame-level quantization parameter to obtain a target quantization parameter. Due to the fact that the contents contained in the coding blocks are different, the corresponding increment quantization parameters are different, when the quantization parameters are utilized to participate in coding, differential coding can be achieved, the coding quality can be guaranteed, meanwhile, the bit rate of unnecessary coding blocks is reduced, and the coding efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to video processing, encoding methods, devices, electronic equipment and storage media. Background Art

[0002] Video coding is the process of compressing digital video data to reduce its size while maintaining a certain visual quality. Block-level rate control plays a vital role in video coding. Coding blocks are the basic building blocks in modern video codecs, such as HEVC and AV1. Block-level rate control refers to the process of adjusting the bitrate distribution between blocks to achieve the desired overall bitrate while maintaining consistent visual quality between video frames and avoiding buffer overflow / underflow.

[0003] Block-level rate control is very important because video encoding is a highly computationally intensive task that requires a lot of computing power and storage capacity. In addition, video data is usually transmitted over networks with limited bandwidth, which can cause transmission delays and buffering issues if the video bitrate is not managed properly. Therefore, block-level rate control is essential to optimize the encoding process and ensure that the receiving device can effectively transmit and decode the resulting video data. A solution that can balance encoding efficiency and encoding quality is needed. Summary of the invention

[0004] The purpose of this application is to provide a video processing, encoding method, device, electronic device and storage medium, which can achieve a solution that balances encoding efficiency and encoding quality.

[0005] According to a first aspect of an embodiment of the present application, a video processing method is provided, including:

[0006] Obtaining a video to be processed, and segmenting video frames in the video to be processed to obtain coding blocks;

[0007] Calculating the complexity and bit rate error of the coding block;

[0008] Calculating an incremental quantization parameter using the complexity and the bit rate error;

[0009] Determining a quantization parameter of a frame according to a frame-level rate control algorithm;

[0010] Calculating a target quantization parameter using the incremental quantization parameter and the quantization parameter of the frame;

[0011] The coding block is encoded based on the target quantization parameter.

[0012] Optionally, the bit rate error calculation method includes:

[0013] Performing frame-level encoding on the video frame using the quantization parameter of the frame to determine an actual number of bits corresponding to each encoding block;

[0014] Determining an average number of bits using the frame level rate control algorithm;

[0015] A bit rate error is calculated based on the actual number of bits and the average number of bits.

[0016] Optionally, determining the average number of bits by using the frame-level rate control algorithm includes:

[0017] The number of bits per frame predicted using the frame-level rate control algorithm;

[0018] The average number of bits is calculated using the number of bits per frame and the number of coding blocks included in the video frame.

[0019] Optionally, the complexity calculation method includes:

[0020] Dividing the coding block into a coding unit CU;

[0021] The standard deviation representing the complexity of the block is calculated using the coding unit variance and the number of coding units; wherein the complexity is the complexity of the coding block in at least one dimension of a time dimension, a space dimension, and a texture dimension.

[0022] Optionally, the calculating an incremental quantization parameter by using the complexity and the bit rate error includes:

[0023] Calculating a buffer size and a scaling factor for the coding block;

[0024] The incremental quantization parameter is calculated using the complexity, the bit rate error, the buffer size and the scaling factor.

[0025] Optionally, the calculating the buffer size of the coding block includes:

[0026] The buffer size is calculated using the average number of bits and the total number of coding blocks in the video frame.

[0027] Optionally, after calculating and obtaining the incremental quantization parameter, the method further includes:

[0028] Setting a maximum quantization parameter and a minimum quantization parameter of the coding block;

[0029] In response to a comparison result of the incremental quantization parameter with the maximum quantization parameter and the minimum quantization parameter, if the incremental quantization parameter is smaller than the minimum quantization parameter, clipping the incremental quantization parameter to the minimum quantization parameter;

[0030] If the incremental quantization parameter is greater than the maximum quantization parameter, the incremental quantization parameter is clipped to the maximum quantization parameter.

[0031] According to a second aspect of an embodiment of the present application, a video encoding method is provided, including:

[0032] Obtaining a video to be processed, and segmenting video frames in the video to be processed to obtain various coding blocks;

[0033] Calculating the complexity and bit rate error of each of the coding blocks;

[0034] Obtaining incremental quantization parameters corresponding to each of the coding blocks respectively by using the complexity and the bit rate error calculation;

[0035] Determining a quantization parameter of a frame according to a frame-level rate control algorithm;

[0036] Calculate the target quantization parameters corresponding to each of the coding blocks by using the incremental quantization parameters corresponding to each of the coding blocks and the frame quantization parameters corresponding to the video frame;

[0037] The video frame is encoded according to the target quantization parameters corresponding to each of the encoding blocks.

[0038] According to a third aspect of an embodiment of the present application, a video processing device is provided, including:

[0039] An acquisition module is used to acquire a video to be processed and segment video frames in the video to be processed to obtain coding blocks;

[0040] A first calculation module, used for calculating the complexity and bit rate error of the coding block;

[0041] A second calculation module, configured to calculate an incremental quantization parameter using the complexity and the bit rate error;

[0042] A determination module, used to determine a quantization parameter of a frame according to a frame level bit rate control algorithm;

[0043] A third calculation module, configured to calculate a target quantization parameter using the incremental quantization parameter and the frame quantization parameter;

[0044] The encoding module is used to perform encoding processing on the encoding block based on the target quantization parameter.

[0045] According to a fourth aspect of an embodiment of the present application, an electronic device is provided, comprising a memory and a processor, wherein the memory is used to store a computer program executable by the processor; and the processor is used to execute the computer program in the memory to implement the above method.

[0046] According to a fifth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that when the executable computer program in the storage medium is executed by a processor, the above method can be implemented.

[0047] Compared with the prior art, the beneficial effect of the present application is that: in a video processing system, after the video to be processed is segmented, a plurality of coding blocks are obtained. In order to achieve more accurate coding block-level coding, corresponding quantization parameters will be set for different coding blocks. Specifically, the complexity and bit rate error of the coding block are calculated respectively. Then, the incremental quantization parameter is calculated by comprehensively utilizing the complexity and bit rate error of the coding block. At the same time, the quantization parameter of the frame is calculated using the frame-level bit rate control algorithm. The target quantization parameter is obtained by summing the calculated incremental quantization parameter and the frame-level quantization parameter. Since the contents contained in each coding block are different, it means that the corresponding incremental quantization parameters are different. When the quantization parameters are used in the coding, differentiated coding can be achieved, which can ensure the coding quality while reducing the bit rate of unnecessary coding blocks and improving the coding efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of motion estimation provided in an embodiment of the present application.

[0049] Figure 2 A schematic diagram of sub-pixels provided in an embodiment of the present application.

[0050] Figure 3 A schematic diagram of the coding block division provided in an embodiment of the present application.

[0051] Figure 4 A schematic diagram of the reference relationship provided for the embodiments of the present application.

[0052] Figure 5 A flowchart of a video processing method provided in an embodiment of the present application.

[0053] Figure 6 A flowchart of a video encoding method provided in an embodiment of the present application.

[0054] Figure 7 Schematic diagram of the encoding system used to illustrate this application.

[0055] Figure 8 is a block diagram of a video processing device according to an exemplary embodiment.

[0056] Fig. 9 is a block diagram of a video encoding device according to an exemplary embodiment.

[0057] Fig.10It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0058] Unless otherwise defined, the technical terms or scientific terms used in this specification and claims shall have the usual meanings understood by persons with ordinary skills in the technical field to which the invention belongs. The specific embodiments of the present invention will be described below in conjunction with the accompanying drawings. It should be noted that in the specific description of these embodiments, in order to provide a concise description, it is impossible for this specification to provide a detailed description of all the features of the actual embodiments. Without departing from the spirit and scope of the present invention, those skilled in the art may modify and replace the embodiments of the present invention, and the resulting embodiments are also within the scope of protection of the present invention.

[0059] In modern video compression standards, inter-frame prediction coding is usually used, that is, the correlation between frames at different times is used to achieve the purpose of video compression through predictive coding. ME is the basis of inter-frame prediction. Inter-frame prediction finds the best match by performing complex ME operations in the reference frame, that is, finding an image block in the reference image that is most similar to the current coding block in the current frame, as shown below: Figure 1 A schematic diagram of motion estimation provided in an embodiment of the present application.

[0060] ME performs motion search based on blocks in the reference frame, that is, within a certain search range, it moves to different positions to obtain the corresponding reference blocks, and by comparing the pixel difference between the reference block and the current coding block, finds a position with the smallest difference as the best position. The block corresponding to this position is used as the best reference block (equivalent to the CU or PU mentioned above). The offset corresponding to this position is the motion vector, such as Figure 1 The solid arrow in the figure. Usually the best position is not located at the integer pixel position of the original image, but most of the time it is located somewhere between two integer pixels. Therefore, before motion search, it is necessary to divide the blank area between the integer positions into equally spaced fractional pixel positions, and perform interpolation calculations on the fractional pixel positions so that the fractional pixel positions obtain the corresponding pixel values. Then, based on this, motion search can be performed, which can greatly improve the search accuracy. Figure 2 Schematic diagram of pixel division provided in the embodiment of the present application. Figure 2 The "o" shown here represents a pixel point at an integer pixel position, and the "x" represents a pixel at a fractional pixel position.

[0061] The processing object of the video encoder is a sequence of video images (Sequence). A video sequence is a collection of multiple video images (Frame) in time order. The encoder encodes by Frame, and each Frame will be encoded with various divisions, decisions, and selections, and finally start encoding with each coding unit CU. Figure 3 Schematic diagram of coding block division provided in the embodiment of the present application. For the currently common video encoder, the basic processing flow is as follows Figure 3 As shown,

[0062] (1) The first step is to divide each frame into LCBs (Largest Coding Blocks) of fixed size. Each LCB is a coding entry. There are many LCBs in a reference frame. Generally speaking, the number of LCBs in a reference frame is the same as the number of integer pixels, that is, one LCB is divided into one integer pixel.

[0063] (2) In the second step, each LCB is divided into coding units (CUs) of various sizes using a quadtree. An LCB can have a combination of CUs of various sizes, forming a coding tree unit (CTU).

[0064] (3) In the third step, each CU may be divided into prediction units (PUs) of different shapes.

[0065] exist Figure 3 In the coding block division shown in the figure, except for step (1), the other two steps require RDO to decide how to divide, the size of the CU, and which PU mode to select for each CU. When encoding a CU / PU, the image blocks in the adjacent reference frame are used as references. According to the similarity, inter-frame prediction is used to achieve data compression. There may be multiple reference relationships here. For each reference relationship, the RDO decision is usually made by traversing and comparing various possible divisions, and selecting the one with the lowest cost as the final coding mode. ME and its pixel-by-pixel interpolation are used. Moreover, different reference frames correspond to different reference relationships and different pixel-by-pixel interpolations. The selection of reference frames is also decided by RDO. Figure 4 Schematic diagram of the reference relationship provided in the embodiment of the present application. Figure 4 As shown in the figure (the arrow in the figure indicates a possible reference relationship when CU / PU is encoded). For sub-pixel interpolation in ME, the common practice of encoders at present is that the encoder will perform sub-pixel interpolation every time RDO needs to use sub-pixels as references. Although this is more flexible, it brings a lot of redundant calculations, which affects the speed of the encoder.

[0066] like Figure 5 A schematic diagram of a video processing method provided in an embodiment of the present application. The method comprises the following steps:

[0067] Step 501: Obtain a video to be processed, and segment video frames in the video to be processed to obtain coding blocks.

[0068] Step 502: Calculate the complexity and bit rate error of the coding block.

[0069] Step 503: Calculate an incremental quantization parameter using the complexity and bit rate error.

[0070] Step 504: Determine the quantization parameter of the frame according to the frame level rate control algorithm.

[0071] Step 505: Calculate a target quantization parameter using the incremental quantization parameter and the frame quantization parameter.

[0072] Step 506: Perform encoding processing on the coding block based on the target quantization parameter.

[0073] Specifically, the video file to be processed is raw data, including a series of continuous video frames, which together constitute dynamic video content. The video to be processed can be obtained by uploading a file, receiving a video stream, or other video input methods.

[0074] After obtaining the video file to be processed, the next step is to segment each frame in the video to be processed. Video frames are the basic units of video data. They are arranged at certain time intervals and together constitute the complete video content. Segmentation refers to dividing each frame into smaller, easier to manage and process units. These units are called coding blocks (Coding Block, CB) or (Largest Coding Block, LCB) in video coding. This application will be explained using LCB coding blocks as an example.

[0075] Through the segmentation process in the previous step, each video frame is decomposed into multiple coding blocks (LCBs). These LCBs are the basic operation units in the video encoding process. Subsequent encoding, compression, bit rate control and other operations will be based on these LCBs. The size of LCB can be determined according to the video encoding standard, encoder implementation and specific application requirements. Different sizes of LCBs will affect the complexity and efficiency of encoding and the quality of the final video.

[0076] In the block-level rate control method, the division and selection of LCB is crucial because it directly affects the accuracy and efficiency of subsequent rate allocation. A reasonable division strategy can make the encoding process more efficient while maintaining good video quality.

[0077] The complexity mentioned here refers to the overall complexity of each coding block in the time dimension, space dimension and texture dimension. When calculating, the standard deviation is used to represent the complexity. The specific calculation process will be described in the following embodiments, and will not be repeated here. The complexity of the time dimension can be obtained by processing the standard deviation of different frames through a certain algorithm, for example, subtracting the standard deviation of different frames, or using other algorithm models.

[0078] Bitrate error: This is the difference between the target bitrate and the actual bitrate used to encode each block. This error value is used to adjust the quantization parameters to ensure that the overall bitrate meets the preset target while avoiding buffer overflow or underflow.

[0079] Quantization Parameter (QP): Quantization is a key step in video encoding that compresses data by reducing the differences between pixel values. QP is a parameter used in the quantization process that determines the degree of quantization. A lower QP value means less quantization, preserving more details but generating more encoded data; a higher QP value means more quantization, more loss of details but generating less encoded data.

[0080] Incremental Quantization Parameter (Block QP): A QP adjustment value calculated based on the complexity and bitrate error of the coding block. This value is used to adjust the frame-level QP to achieve more precise bitrate control for each coding block.

[0081] The frame-level rate control algorithm is an algorithm used to control the bit allocation and QP value of each frame during the video encoding process. It determines the QP value of each frame based on the bit rate target of the entire video sequence and the current encoding status (such as buffer status, number of bits of the encoded frame, etc.).

[0082] The frame quantization parameter (QPframe) is the quantization parameter value of each frame calculated by the frame-level bit rate control algorithm.

[0083] At the beginning of video processing, the video stream is split into a series of video frames, which are then further divided into multiple largest coding blocks (LCBs). The size of the LCB depends on the specific video codec standard. For example, in HEVC, the size of the LCB may be 64×64 pixels, while in AV1 it may be 128×128 pixels. Each LCB is then further divided into smaller coding units (CUs), and the size of these CUs can change dynamically depending on the complexity of the video content.

[0084] To calculate the complexity of LCB, we first need to divide each LCB into coding units (CUs). Then, we calculate the variance of each CU and the number of CUs to calculate the standard deviation of the block complexity. The block complexity reflects the details and texture complexity inside the LCB.

[0085] The calculation of the bitrate error involves encoding the video frame using the frame quantization parameter (QPframe) determined by the frame-level rate control algorithm to determine the actual number of bits corresponding to each coded block. The average number of bits is then calculated using the number of bits per frame predicted by the frame-level rate control algorithm. The bitrate error is the difference between the actual number of bits and the average number of bits.

[0086] Next, the buffer size for the LCB is calculated, as well as the scaling factor. The buffer size is calculated based on the average number of bits and the total number of blocks in the coded block in the video frame. The scaling factor is a tuning parameter used to adjust the quantization parameters to adapt to the characteristics of the video content. With this information, the incremental quantization parameter (block QP) can be calculated, which reflects the amount by which the quantization parameters of each LCB need to be adjusted after taking into account the complexity and bitrate errors.

[0087] The frame quantization parameter (QPframe) determined by the frame-level rate control algorithm is used to control the average quantization level of the entire video frame. This parameter is determined based on the overall complexity of the video frame and the desired bit rate target.

[0088] Finally, the target quantization parameter can be calculated by combining the incremental quantization parameter (block QP) and the frame quantization parameter (QPframe). This parameter will be used in the actual video encoding process to adjust the quantization level of each LCB to achieve the best visual quality at a given bit rate.

[0089] Assume that a HEVC video is being processed and the size of the video frame is 1920x1080 pixels. Each video frame is split into multiple LCBs of 64x64 pixels. For a particular LCB, its complexity is calculated and it is found that it contains rich textures and details. Through the frame-level rate control algorithm, the QPframe of the current frame is determined to be 28. Next, the bit rate error of the LCB is calculated and it is found that its actual number of bits is higher than the average number of bits, which means that the LCB requires more bits to encode.

[0090] Based on the complexity and bitrate error, a positive incremental quantization parameter is calculated, indicating that the quantization parameter of the LCB needs to be increased to reduce the bitrate. Assuming the calculated incremental quantization parameter is +3, the target quantization parameter of the LCB will be QPframe plus the incremental quantization parameter, that is, 28+3=31. Based on this target quantization parameter, the LCB can be encoded to meet the given bitrate target while maintaining good visual quality.

[0091] In one or more embodiments of the present application, the bit rate error calculation method includes:

[0092] Performing frame-level encoding on the video frame using the quantization parameter of the frame to determine an actual number of bits corresponding to each encoding block;

[0093] Determining an average number of bits using the frame level rate control algorithm;

[0094] A bit rate error is calculated based on the actual number of bits and the average number of bits.

[0095] In practical applications, the entire video frame is preliminarily encoded according to a frame-level quantization parameter (QPframe) determined by a frame-level rate control algorithm (such as an R-lambda model, a quadratic rate distortion model, etc.).

[0096] During the encoding process, the actual number of bits (blockrealbits) used by each coding block (LCB) after encoding is counted and recorded. This data reflects the actual compression efficiency of each coding block during the encoding process.

[0097] Based on the frame-level rate control algorithm and the target bitrate, the average number of bits that should be allocated to each coding block in each frame (blockaveragebits) is calculated. This usually involves allocating the total target number of bits for the current frame, taking into account the complexity of the coding blocks within the frame and other coding constraints.

[0098] The calculation formula can be: blockaveragebits = (total target bits / total number of blocks in a frame), where the total target bits are given by the frame-level rate control algorithm, and the total number of blocks in a frame depends on the resolution of the video frame and the segmentation strategy of the codec.

[0099] After obtaining the actual number of bits and the average number of bits through the above scheme, the bit rate error can be further calculated according to the preset algorithm. The specific calculation process can be to calculate the difference between the actual number of bits (blockrealbits) and the average number of bits (blockaveragebits) of each coding block to obtain the bit rate error (blockbiterror) of each coding block. The calculation formula (1) of the bit rate error is as follows:

[0100] blockbiterror = blockrealbits – blockaveragebits (1)

[0101] This bit rate error reflects the degree to which the coding block deviates from the average distribution during the encoding process.

[0102] In one or more embodiments of the present application, determining the average number of bits by using the frame-level rate control algorithm includes:

[0103] The number of bits per frame predicted using the frame-level rate control algorithm;

[0104] The average number of bits is calculated using the number of bits per frame and the number of coding blocks included in the video frame.

[0105] The average number of bits blockaveragebits is calculated by the following formula (2):

[0106]

[0107] Where bpf represents the number of bits per frame predicted using the frame-level rate control algorithm and Nblocks is the total number of blocks in the frame. Nblocks depends on the type of codec, as the size of each block may vary depending on the codec.

[0108] There are many ways to calculate the number of bits per frame (bpf). You can use a specified algorithm or model to calculate it. For example, you can use a statistical model or a machine learning model to predict the number of bits per frame. These models can predict based on historical data (such as the encoding results of the previous frame) and the characteristics of the current frame.

[0109] You can also use the R-λ model to calculate: In video coding, the R-λ model is a commonly used rate-distortion model that establishes the relationship between the encoding bit rate (R) and the quantization parameter (λ). By adjusting λ, the encoding bit rate can be controlled.

[0110] Alternatively, linear programming or optimization algorithms may be used for calculation: the rate control problem may be expressed as an optimization problem, such as a linear programming or quadratic programming problem, and the optimal bit allocation solution may be found by solving the problem.

[0111] Therefore, in practical applications, the average number of bits can be calculated through the above-mentioned multiple calculation methods.

[0112] In one or more embodiments of the present application, the complexity calculation method includes: dividing the coding block to obtain a coding unit CU (coding unit); using the coding unit variance and the number of coding units to calculate the standard deviation of the coding block representing the complexity; wherein the complexity is the complexity of the coding block in at least one dimension of the time dimension, the space dimension, and the texture dimension.

[0113] Another important consideration when designing an effective block-level rate control algorithm is to consider the complexity of each block to determine the QP value for the encoded block. The complexity of a block determines the number of bits required to encode it. Blocks with higher complexity generally require more bits to encode, while blocks with lower complexity require fewer bits. Methods that consider block complexity differ from some state-of-the-art methods because they mainly consider the sum of absolute transform differences (SATD) of spatial or temporal correlations between blocks and do not consider standard deviation as a measure of complexity. In doing so, these measures result in higher computational complexity because computing SATD is more time-consuming compared to standard deviation.

[0114] Standard deviation and variance are two common metrics for evaluating block complexity. The coding unit variance is the average of the squared differences from the mean, while the standard deviation is the square root of the variance. Although both metrics provide similar information, standard deviation is considered a measure of block complexity for a number of reasons. Using standard deviation to represent block complexity has the following advantages:

[0115] Standard deviation has the same units as the input data: One limitation of variance is that it is in squared units, which makes it difficult to interpret. On the other hand, standard deviation has the same units as the data, which makes it easier to understand and interpret.

[0116] Standard deviation is more sensitive to outliers: Standard deviation is a better measure of complexity because it is more sensitive to outliers than variance. The square root in the standard deviation formula reduces the impact of outliers, while variance gives more weight to outliers.

[0117] An example of calculating the standard deviation blockσ of each coded block is calculated as follows

[0118]

[0119] Where blockvar is the variance of the coded block. Var(i) is the variance of the i-th 8×8 block, N 8×8 is the total number of 8×8 blocks within each coding block.

[0120] To calculate the standard deviation of each block, first divide it into 8×8 blocks. Then, for each 8×8 block, calculate its variance. Then, sum the variances of each block within the block. The variance of the block is calculated by dividing the sum of the variances of each 8×8 block by the total number of 8×8 blocks within the block. Finally, the standard deviation of the block is equal to the square root of its variance.

[0121] In one or more embodiments of the present application, the step of calculating an incremental quantization parameter (block QP) using the complexity and the bit rate error includes:

[0122] Calculating a buffer size and a scaling factor for the coding block;

[0123] The incremental quantization parameter (block QP) is calculated using the complexity, the bit rate error, the buffer size and the scaling factor.

[0124] The method for calculating the buffer size of the coding block specifically includes: using the average number of bits and the total number of blocks of the coding block in the video frame to calculate the buffer size. After calculating the incremental quantization parameter (block QP), it also includes: setting the maximum quantization parameter and the minimum quantization parameter of the coding block; in response to the comparison result of the size relationship between the incremental quantization parameter and the maximum quantization parameter and the minimum quantization parameter, if the incremental quantization parameter is less than the minimum quantization parameter, then the incremental quantization parameter is clipped to the minimum quantization parameter; if the incremental quantization parameter is greater than the maximum quantization parameter, then the incremental quantization parameter is clipped to the maximum quantization parameter.

[0125] In order to consider the impact of small and large video buffer verifiers (VBV), a hypothetical block buffer size is defined. Doing so makes the proposed block-level rate control more sensitive to the changes in different VBV sizes and makes the proposed algorithm more effective and efficient. The hypothetical block buffer size, blockbuffer size, is calculated as follows:

[0126] blockbuffer size = blockaverage bits × Nblock (4)

[0127] where blockbuffersize is the average number of bits per block calculated in equation (2). Nblock is the total number of blocks in the frame.

[0128] In order to find the quantization parameter QP value for each coding block, the quantization parameter QP of the frame needs to be adjusted. This is done by calculating the incremental quantization parameter delta QP. The incremental quantization parameter delta QP represents the amount by which the quantization parameter QP value of each block needs to be adjusted to meet the required visual quality level while also achieving the target bitrate. By adjusting the quantization parameter QP value for each coding block, the encoder can allocate more bits to complex or visually important areas of the frame while reducing the bitrate of less important areas. In the proposed method, the incremental quantization parameter deltaQP of each block is calculated using the following formula (5):

[0129]

[0130] where blockΔQP and QPframe are the block delta QP and frame QP, respectively. QPmax is the maximum quantization parameter QP value allowed by each encoder. blockbiterror is the bitrate difference tracking measure proposed in equation (1). blockσ is the block complexity measure proposed in equation (3), and blockbuffersize is the hypothetical blockbuffer size calculated according to equation (4).

[0131] In order to take into account the different block sizes of each codec, formula (5) is further modified to take into account different block sizes. This is achieved by introducing a codec-dependent scaling factor s. The final calculation formula (6) for Delta QP is as follows:

[0132]

[0133] Where s is a codec-related constant.

[0134] An important consideration in formula (6) is that other complexity metrics can be considered instead of standard deviation, such as variance, normalized variance, normalized standard deviation, and logarithm of standard deviation or variance. Among them, standard deviation is a preferred option because it shows the highest coding efficiency gain.

[0135] The value of blockΔQP in formula (6) can be positive or negative, and its value represents the quantization parameter increment of the block. It controls the degree of quantization applied to the block during the video encoding process. Therefore, after its value is calculated, it needs to be clipped to limit its fluctuations. One of the main reasons for limiting the variation of the calculated block delta QP is to minimize quality fluctuations and bitrate usage. Doing so can reduce the amount of quantization applied to each block, thereby helping to maintain a consistent level of visual quality throughout the video sequence. This in turn can reduce quality fluctuations and provide viewers with a smoother viewing experience. In addition, limiting the variation of the calculated delta QP can also save bitrate because when a lower level of quantization is applied, less bitrate is required to encode each block. This is especially important for applications with limited bandwidth and storage resources, such as video streaming or mobile devices. In the proposed block-level bitrate control, BlockΔQP is limited according to the following formula (7):

[0136]

[0137] Where ΔQPmax and ΔQPmin are constants, indicating the maximum and minimum quantization parameters that the user can set to allow Delta QP to change.

[0138] The last step of the proposed block-level rate control algorithm is to calculate the QP of the block. After limiting the delta QP variation of the block, the QP of the entire block can be found using formula (8), as follows:

[0139] blockQP = QPframe + blockΔQP (8)

[0140] Where blockQP is the quantization parameter QP of the coding block, QPframe is the quantization parameter QP of the frame calculated using the frame-level rate control algorithm. blockΔQP is the delta QP of the cropped block. After calculating the QP of the block, you can continue the encoding process.

[0141] like Figure 6 A schematic diagram of a video encoding method provided in an embodiment of the present application. Figure 6 As can be seen in , the method specifically includes the following steps:

[0142] Step 601: Obtain a video to be processed, and segment video frames in the video to be processed to obtain coding blocks.

[0143] Step 602: Calculate the complexity and bit rate error of each coding block.

[0144] Step 603: Calculate the incremental quantization parameters corresponding to each of the coding blocks using the complexity and bit rate error.

[0145] Step 604: Determine the quantization parameter of the frame according to the frame level rate control algorithm.

[0146] Step 605: Calculate the target quantization parameters corresponding to each of the coding blocks using the incremental quantization parameters corresponding to each of the coding blocks and the frame quantization parameters corresponding to the video frame.

[0147] Step 606: Encode the video frame according to the target quantization parameters corresponding to the encoding blocks.

[0148] In practical applications, the system obtains the video data to be processed from the video source (such as files, network streams, etc.). Subsequently, the system decodes the video data and divides the video frame into coding blocks LCB, or smaller coding units (CUs) or coding tree units (CTUs), which are usually used as basic processing blocks in video coding. The specific division method depends on the video coding standard adopted (such as HEVC, AV1, etc.).

[0149] The complexity of each coding block is calculated by analyzing the amount of pixel variation (including temporal and spatial activity), texture complexity (such as the level of detail and high-frequency content), and other indicators. These characteristics can be quantified using standard deviation, variance, or other statistical methods.

[0150] Formula example: As described in the document, the standard deviation formula is used to calculate the variance of the 8×8 sub-blocks within the block, and then the standard deviation of the entire block (blockσ) is obtained as a measure of complexity.

[0151] Calculate the bitrate error:

[0152] Definition: The bit rate error (blockbiterror) refers to the difference between the number of bits actually used to encode each block (blockrealbits) and the average number of bits predicted by the frame-level rate control algorithm (blockaveragebits).

[0153] Formula: blockbiterror=blockrealbits-blockaveragebits

[0154] BlockAverageBits calculation method: BlockAverageBits = bpf / Nblocks, where bpf is the number of bits per frame predicted by the frame-level rate control algorithm, and Nblocks is the total number of blocks in the frame.

[0155] Calculate the incremental quantization parameter (blockΔQP): According to the complexity and bit rate error of each coding block, the incremental quantization parameter of each block is calculated by a specific algorithm (as shown in formulas (5) and (6) in the document). The incremental QP represents the amount that needs to be adjusted relative to the frame-level quantization parameter (QPframe) to optimize the bit rate allocation and visual quality of each block.

[0156] In practical applications, considering the block size differences of different codecs, the calculation of incremental QP can be adjusted by introducing a codec-related scaling factor s.

[0157] The frame-level rate control algorithm determines the quantization parameter (QPframe) for each frame based on the overall complexity of the video, the frame rate, the target bitrate, and the buffer status. This usually involves a complex control loop designed to balance the overall bitrate requirement and video quality.

[0158] When calculating the target quantization parameter (blockQP), the frame-level quantization parameter (QPframe) is added to the incremental quantization parameter (blockΔQP) of each block to obtain the target quantization parameter (blockQP) of each coding block. When calculating, it is also necessary to pay attention to clipping the incremental QP to avoid excessive fluctuations.

[0159] Each coding block is encoded using the calculated target quantization parameter (blockQP). Through the above scheme, the content contained in each coding block in the video frame to be encoded is different, and the corresponding target quantization parameter is also different. The larger the target quantization parameter of the coding block, the more content is contained in the coding block and the higher the bit rate is; the smaller the target quantization parameter of the coding block, the less content is contained in the coding block and the lower the bit rate is; therefore, different encoding methods can be used for different coding blocks in the video frame during the encoding process. The target quantization parameter QP of the coding block is determined by considering the complexity and bit rate error of the coding block. The final value of the target quantization parameter QP will be a combination of the complexity and bit rate error of the coding block.

[0160] In addition, the scheme is more efficient in bitrate allocation because it allocates more bits to complex blocks and fewer bits to blocks with lower complexity, thereby maintaining the visual quality and visible details of each block. The scheme provides higher visual quality by providing more bitrate to visually more important coding blocks. And, the scheme is more accurate and robust because it takes into account the bitrate error of each coding block and adjusts the quantization parameter QP value of different coding blocks accordingly.

[0161] The present application scheme can be applied in various video coding systems, and practical application scenarios include but are not limited to video streaming, monitoring, broadcasting, and video conferencing. In addition, the proposed coding scheme can be implemented using software or hardware, and can be easily integrated into existing video coding systems, so that coding operations can be implemented without complex hardware support. The coding process will follow the selected video coding standard (such as HEVC, AV1, etc.), and adjust the coding parameters such as transform coefficients, quantization matrices, etc. based on the QP value to achieve the expected bit rate and visual quality.

[0162] like Figure 7 A schematic diagram of the encoding system used to illustrate this application. Figure 7 As can be seen in the figure, the schematic diagram includes:

[0163] The preset target bit rate and the acquired video to be processed are input into the frame level rate control module (a technology in video coding), which is responsible for setting a suitable frame quantization parameter (QPframe) for the entire video frame to control the bit rate of the frame and maintain the overall visual quality of the video as much as possible. At the same time, the average bit rate of the coding block is also provided.

[0164] The quantization parameters of the frame are input into the encoding module for encoding processing to obtain the actual bit rate of the encoding block. Then, the bit rate error is calculated using the actual bit rate and the average bit rate. Furthermore, in order to consider the impact of small and large video buffer verifiers (VBV), a hypothetical block buffer size is defined. Doing so makes the block-level rate control more sensitive to changes in different VBV sizes and makes the algorithm more effective and efficient. The buffer size range is obtained by performing a buffer calculation based on the input average bit rate.

[0165] Then, the quantization parameter ΔQP is calculated within the buffer size constraint using the previously calculated standard deviation representing the complexity of the coding block, the bit rate error, and the frame quantization parameter. ΔQP is trimmed based on the maximum quantization parameter and the minimum quantization parameter to obtain the required incremental quantization parameter.

[0166] The incremental quantization parameter and the previously calculated frame quantization parameter are then used to calculate the target quantization parameter.

[0167] The encoding of the video frame is performed using the target quantization parameter, which is the quantization parameter of the coding block. Different coding blocks in the video frame have different target quantization parameters. Therefore, when encoding the video frame, it is possible to achieve differential encoding between different coding blocks in the frame, improve coding efficiency, ensure coding quality, and reduce bit rate.

[0168] Figure 7The encoding module is the process of converting video signals or data streams into a format more suitable for storage or transmission.

[0169] Hypothetical CTU buffer size, CTU (Coding Tree Unit) is the basic processing unit in modern video codecs such as HEVC (High Efficiency Video Coding). The hypothetical CTU buffer size is a theoretical value set to simulate the actual buffer behavior, which is used to adjust and control the encoding process of CTU to avoid buffer overflow or underflow.

[0170] CTU standard deviation calculation is a method used to measure the degree of variation of pixel values ​​within a CTU. This calculation helps to evaluate the complexity of a CTU, which in turn affects the setting of its quantization parameter (QP).

[0171] ΔQP calculation refers to the quantization parameter (QP) adjustment calculated for each block. This value is dynamically calculated based on factors such as block complexity and bit rate error, and is used to adjust the QP value of the block to achieve the desired encoding effect and visual quality.

[0172] ΔQP clipping refers to the process of limiting or clipping the ΔQP value after it is calculated. The purpose of this step is to ensure that the ΔQP value is within a reasonable range to avoid video quality fluctuations or improper bitrate usage caused by ΔQP being too large or too small.

[0173] Through the above scheme, in the video processing system, after the processed video is segmented, multiple coding blocks are obtained. In order to achieve more accurate block-level coding, corresponding quantization parameters will be set for different coding blocks. Specifically, the complexity and bit rate error of the coding block are calculated respectively. Then, the incremental quantization parameter is calculated by comprehensively utilizing the complexity and bit rate error of the coding block. At the same time, the quantization parameter of the frame is calculated using the frame-level bit rate control algorithm. The target quantization parameter is obtained by summing the calculated incremental quantization parameter and the frame-level quantization parameter. Since the contents of each coding block are different, it means that the corresponding incremental quantization parameters are different. When the quantization parameters are used in the encoding, differentiated encoding can be achieved, which can ensure the encoding quality while reducing the bit rate of unnecessary coding blocks and improving the encoding efficiency.

[0174] In addition, the technical solution of this application has the following advantages:

[0175] Precise control of QP: By considering the complexity and rate error of each block, the algorithm of this application provides more precise control of QP, which may lead to better compression efficiency and visual quality.

[0176] Consistent visual quality: By adjusting the QP of each block, the method of the present application can help ensure consistent visual quality throughout the video, even in parts with high or low complexity.

[0177] Improved texture detail preservation: By taking into account the complexity when adjusting QP, the method of the present application may be able to better preserve fine texture details in the video, resulting in higher visual quality and less compression artifacts.

[0178] Optimal balance between bitrate and quality: An approach that balances overall encoding bitrate and visual quality can help achieve better overall video compression, resulting in higher quality video at a lower bitrate.

[0179] Potential for improved video streaming: The ability of this application's approach to balance bitrate and visual quality can provide users with a smoother and more consistent streaming experience, especially when network bandwidth fluctuates.

[0180] Advantages for high-resolution video: High-resolution video usually requires higher bitrates for compression, but the algorithm's precise QP control can help reduce the required bitrate while maintaining good visual quality.

[0181] Wide range of industry applications: Effective video compression is critical to many industries, including entertainment, education, healthcare, etc.

[0182] Figure 8 FIG. 1 is a block diagram of a video processing device according to an exemplary embodiment. Figure 8 As shown, in this embodiment, the video processing device includes:

[0183] The acquisition module 81 is used to acquire a video to be processed and segment video frames in the video to be processed to obtain coding blocks.

[0184] The first calculation module 82 is used to calculate the complexity and bit rate error of the coding block.

[0185] The second calculation module 83 is used to calculate the incremental quantization parameter using the complexity and the bit rate error.

[0186] The determination module 84 is used to determine the quantization parameter of the frame according to the frame level rate control algorithm.

[0187] The third calculation module 85 is used to calculate a target quantization parameter using the incremental quantization parameter and the frame quantization parameter.

[0188] The encoding module 86 is used to perform encoding processing on the encoding block based on the target quantization parameter.

[0189] A first calculation module 82, configured to perform frame-level encoding on the video frame using the quantization parameter of the frame, and determine an actual number of bits corresponding to each encoding block;

[0190] Determining an average number of bits using the frame level rate control algorithm;

[0191] A bit rate error is calculated based on the actual number of bits and the average number of bits.

[0192] A first calculation module 82, configured to predict the number of bits per frame using the frame level rate control algorithm;

[0193] The average number of bits is calculated using the number of bits per frame and the number of coding blocks included in the video frame.

[0194] A first calculation module 82, configured to divide the coding block into coding units CU;

[0195] The standard deviation representing the complexity of the block is calculated using the coding unit variance and the number of coding units; wherein the complexity is the complexity of the coding block in at least one dimension of a time dimension, a space dimension, and a texture dimension.

[0196] A first calculation module 82, configured to calculate a buffer size and a scaling factor of the coding block;

[0197] The incremental quantization parameter is calculated using the complexity, the bit rate error, the buffer size and the scaling factor.

[0198] The first calculation module 82 is configured to calculate the buffer size by using the average number of bits and the total number of coding blocks in the video frame.

[0199] A first calculation module 82, configured to set a maximum quantization parameter and a minimum quantization parameter of the coding block;

[0200] In response to a comparison result of the incremental quantization parameter with the maximum quantization parameter and the minimum quantization parameter, if the incremental quantization parameter is smaller than the minimum quantization parameter, clipping the incremental quantization parameter to the minimum quantization parameter;

[0201] If the incremental quantization parameter is greater than the maximum quantization parameter, the incremental quantization parameter is clipped to the maximum quantization parameter.

[0202] Fig. 9 FIG. 1 is a block diagram of a video encoding device according to an exemplary embodiment. Fig. 9As shown, in this embodiment, the video encoding device includes:

[0203] The acquisition module 91 is used to acquire a video to be processed and segment video frames in the video to be processed to obtain various coding blocks.

[0204] The calculation module 92 is used to calculate the complexity and bit rate error of each coding block.

[0205] The calculation module 92 is used to calculate the incremental quantization parameters corresponding to each of the coding blocks by using the complexity and the bit rate error.

[0206] The determination module 93 is used to determine the quantization parameter of the frame according to the frame level rate control algorithm.

[0207] The calculation module 92 is used to calculate the target quantization parameters corresponding to each of the coding blocks by using the incremental quantization parameters corresponding to each of the coding blocks and the frame quantization parameters corresponding to the video frame.

[0208] The encoding module 94 is used to encode the video frame according to the target quantization parameters corresponding to each of the encoding blocks.

[0209] An embodiment of the present application also proposes an electronic device, comprising a processor and a memory; the memory is used to store a computer program executable by the processor; the processor is used to execute the computer program in the memory to implement the video processing and video encoding methods described in any of the above embodiments.

[0210] The embodiments of the present application further provide a computer-readable storage medium. When the executable computer program in the storage medium is executed by a processor, the video processing and video encoding methods described in any of the above embodiments can be implemented.

[0211] Regarding the device in the above embodiment, the specific manner in which the processor performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0212] Fig.10 900 is a block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 900 may be provided as a server. Fig.10 , the device 900 includes a processing component 922, which further includes one or more processors, and a memory resource represented by a memory 932 for storing instructions, such as applications, that can be executed by the processing component 922. The application stored in the memory 932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 922 is configured to execute instructions to perform the above-mentioned video processing method.

[0213] The device 900 may also include a power component 926 configured to perform power management of the device 900, a wired or wireless network interface 950 configured to connect the device 900 to a network, and an input / output (I / O) interface 958. The device 900 may operate based on an operating system stored in the memory 932, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™, or the like.

[0214] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 932 including instructions, and the instructions can be executed by the processing component 922 of the device 900 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0215] In the present invention, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance. The term "plurality" refers to two or more than two, unless otherwise clearly defined.

[0216] The above description of the embodiments is to facilitate those of ordinary skill in the art to understand and apply the present application. It is obvious that those familiar with the art can easily make various modifications to these embodiments and apply the general principles described herein to other embodiments without having to pay creative labor. Therefore, the present application is not limited to the embodiments herein, and improvements and modifications made by those skilled in the art based on the contents disclosed in the present application without departing from the scope and spirit of the present application are within the scope of the present application.

Claims

1. A video processing method, characterized in that: The method comprises: Obtaining a video to be processed, and segmenting video frames in the video to be processed to obtain coding blocks; Calculating the complexity and bit rate error of the coding block; Calculating an incremental quantization parameter using the complexity and the bit rate error; Determining a quantization parameter of a frame according to a frame-level rate control algorithm; Calculating a target quantization parameter using the incremental quantization parameter and the quantization parameter of the frame; The coding block is encoded based on the target quantization parameter.

2. The method according to claim 1, characterized in that The bit rate error calculation method includes: Performing frame-level encoding on the video frame using the quantization parameter of the frame to determine an actual number of bits corresponding to each encoding block; Determining an average number of bits using the frame level rate control algorithm; A bit rate error is calculated based on the actual number of bits and the average number of bits.

3. The method according to claim 2, characterized in that The determining the average number of bits by using the frame level rate control algorithm comprises: The number of bits per frame predicted using the frame-level rate control algorithm; The average number of bits is calculated using the number of bits per frame and the number of coding blocks included in the video frame.

4. The method according to claim 1, characterized in that: The complexity calculation method includes: Dividing the coding block into a coding unit CU; The standard deviation representing the complexity is calculated using the coding unit variance and the number of coding units; wherein the complexity is the complexity of the coding block in at least one dimension of a time dimension, a space dimension, and a texture dimension.

5. The method according to claim 3, characterized in that: The step of calculating the incremental quantization parameter using the complexity and the bit rate error includes: Calculating a buffer size and a scaling factor for the coding block; The incremental quantization parameter is calculated using the complexity, the bit rate error, the buffer size and the scaling factor.

6. The method according to claim 5, characterized in that The calculating the buffer size of the coding block includes: The buffer size is calculated using the average number of bits and the total number of coding blocks in the video frame.

7. The method according to claim 5, characterized in that After the incremental quantization parameter is obtained by calculation, the method further includes: Setting a maximum quantization parameter and a minimum quantization parameter of the coding block; In response to a comparison result of the incremental quantization parameter with the maximum quantization parameter and the minimum quantization parameter, if the incremental quantization parameter is smaller than the minimum quantization parameter, clipping the incremental quantization parameter to the minimum quantization parameter; If the incremental quantization parameter is greater than the maximum quantization parameter, the incremental quantization parameter is clipped to the maximum quantization parameter.

8. A video encoding method, characterized in that: The method comprises: Obtaining a video to be processed, and segmenting video frames in the video to be processed to obtain various coding blocks; Calculating the complexity and bit rate error of each of the coding blocks; Obtaining incremental quantization parameters corresponding to each of the coding blocks respectively by using the complexity and the bit rate error calculation; Determining a quantization parameter of a frame according to a frame-level rate control algorithm; Calculate the target quantization parameters corresponding to each of the coding blocks by using the incremental quantization parameters corresponding to each of the coding blocks and the frame quantization parameters corresponding to the video frame; The video frame is encoded according to the target quantization parameters corresponding to each of the encoding blocks.

9. A video processing device, characterized in that: The device comprises: An acquisition module is used to acquire a video to be processed and segment video frames in the video to be processed to obtain coding blocks; A first calculation module, used for calculating the complexity and bit rate error of the coding block; A second calculation module, configured to calculate an incremental quantization parameter using the complexity and the bit rate error; A determination module, used to determine a quantization parameter of a frame according to a frame level bit rate control algorithm; A third calculation module, configured to calculate a target quantization parameter using the incremental quantization parameter and the frame quantization parameter; The encoding module is used to perform encoding processing on the encoding block based on the target quantization parameter.

10. An electronic device comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 8.

11. A computer-readable medium having stored thereon at least one instruction, at least one program, code set or instruction set, wherein the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the method according to any one of claims 1 to 8.

Images