A code rate control method, device and electronic equipment

Abstract

The embodiment of the application discloses a code rate control method, device and electronic equipment, relates to the technical field of video coding, and can stabilize code rate and improve the accuracy of code rate control. The method comprises the following steps: acquiring two continuous images, wherein the two images comprise a target frame to be encoded and a previous frame of the target frame; calculating a semantic difference degree between the target frame and the previous frame, wherein the semantic difference degree is the distance between feature vectors of original images of the target frame and the previous frame, and the feature vectors are extracted through a convolutional neural network; and based on the semantic difference degree, a code rate control strategy is selected to encode the target frame. The application is suitable for code rate control processing in a video coding scene.

Description

Technical Field

[0001] This invention relates to the field of video coding technology, and in particular to a bitrate control method, apparatus, and electronic device. Background Technology

[0002] Video rate control (RC) is a core supporting technology for low-latency real-time interactive scenarios such as remote desktops, cloud computers, and cloud gaming. By controlling the quantization parameter (QP) of image encoding, it achieves a balance between high image quality and low latency in environments with limited network bandwidth and complex fluctuations, avoiding stuttering or blurry images due to bandwidth limitations.

[0003] Current bitrate control algorithms primarily rely on statistical models fitted from historical coding data to predict the quantization parameters of the next frame, thereby achieving bitrate control. However, this strategy has significant limitations. When there are significant visual changes between adjacent frames (such as rapid object movement, sudden changes in lighting, or scene switching), the statistical model based on historical data often underestimates the actual coding complexity of the predicted frame (P-frame), resulting in a large deviation between the predicted quantization parameters and the actual requirements. Consequently, the number of bits in the generated P-frame far exceeds the expected threshold, causing drastic bitrate fluctuations. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a bitrate control method, apparatus, and electronic device that can stabilize the bitrate and improve the accuracy of bitrate control.

[0005] In a first aspect, embodiments of the present invention provide a bitrate control method, the method comprising: acquiring two consecutive frames of images, the two frames of images including a target frame to be encoded and a frame preceding the target frame; calculating a semantic difference degree between the target frame and the preceding frame, the semantic difference degree being the distance between feature vectors of the original images of the target frame and the preceding frame, the feature vectors being extracted by a convolutional neural network; and selecting a bitrate control strategy based on the semantic difference degree to encode the target frame.

[0006] In one implementation, selecting a bitrate control strategy based on the semantic difference degree includes: if the semantic difference degree is less than a first threshold, selecting a first encoding strategy, wherein the first encoding strategy determines the encoding type of the target frame as a prediction frame; if the semantic difference degree is greater than or equal to the first threshold and less than a second threshold, selecting a second encoding strategy, wherein the second encoding strategy is used to determine the encoding type of the target frame as a prediction frame and to perform encoding compensation on the target frame according to the semantic difference degree, wherein the first threshold is less than the second threshold; if the semantic difference degree is greater than or equal to the second threshold, selecting a third encoding strategy, wherein the third encoding strategy determines the encoding type of the target frame as a keyframe.

[0007] In one implementation, the rate control strategy is selected as a first encoding strategy. The step of selecting the rate control strategy based on the semantic difference and encoding the target frame includes: determining a preliminary target number of bits according to the encoding type of the target frame and the state of the video buffer validator model, wherein the state of the video buffer validator model is determined based on the deviation between the current water level and the target water level of the leaky bucket model in the video buffer validator model; inputting the preliminary target number of bits into an inter-frame rate-quantization model to obtain quantization parameters; and encoding the target frame according to the quantization parameters.

[0008] In one implementation, the rate control strategy is selected as a second encoding strategy. The step of selecting the rate control strategy based on the semantic dissimilarity and encoding the target frame includes: determining a preliminary target bit count based on the encoding type of the target frame and the state of the video buffer validator model, wherein the state of the video buffer validator model is determined based on the deviation between the current water level and the target water level of the leaky bucket model in the video buffer validator model; predicting compensation coefficients using a pre-trained rate compensation model based on the semantic dissimilarity; adjusting the preliminary target bit count based on the compensation coefficients to obtain a corrected target bit count; inputting the corrected target bit count into an inter-frame rate-quantization model to obtain quantization parameters; and encoding the target frame based on the quantization parameters.

[0009] In one implementation, after encoding the target frame according to the quantization parameters, the method further includes: keeping the parameters of the inter-frame bitrate-quantization model unchanged and updating the state of the video buffer validator model.

[0010] In one implementation, the rate control strategy is selected as a third encoding strategy. The step of selecting the rate control strategy based on the semantic difference and encoding the target frame includes: determining a preliminary target number of bits based on the encoding type of the target frame and the state of the video buffer validator model, wherein the state of the video buffer validator model is determined based on the deviation between the current water level and the target water level of the leaky bucket model in the video buffer validator model; obtaining quantization parameters based on the preliminary target number of bits using an intra-frame rate-quantization model; and encoding the target frame based on the quantization parameters.

[0011] In one implementation, after encoding the target frame according to the quantization parameters, the method further includes: updating the parameters of the intra-frame bitrate-quantization model and the state of the video buffer validator model; and resetting the parameters of the inter-frame bitrate-quantization model to their initial parameter states.

[0012] In one implementation, the feature vector represents the high-level semantic content of the corresponding image.

[0013] In one implementation, the convolutional neural network is a ResNet-18 network or a visual geometry group network.

[0014] In one implementation, before predicting compensation coefficients using a pre-trained bitrate compensation model based on the semantic dissimilarity, the method further includes: acquiring multiple video image sequences; encoding each video image sequence in an encoding mode with fixed quantization parameters, and recording the actual number of encoded bits for each frame during the encoding process; calculating the relative bit growth ratio of the current frame relative to the previous frame for each pair of consecutive frames based on the actual number of encoded bits; calculating the semantic dissimilarity for each pair of consecutive frames; selecting samples whose semantic dissimilarity is greater than or equal to a first threshold and less than a second threshold to form a training dataset, wherein each sample in the training dataset includes a semantic dissimilarity and a corresponding relative bit growth ratio; and determining the parameters of the bitrate compensation model based on the training dataset, wherein the parameters minimize the error between the relative bit overhead prediction value output by the bitrate compensation model based on the semantic dissimilarity and the relative bit growth ratio in the samples.

[0015] In one implementation, the rate compensation model is R = Where D is the semantic dissimilarity and R is the relative bit overhead. , These are the parameters of the bitrate compensation model.

[0016] In one embodiment, the convolutional neural network is deployed on a neural network processing unit.

[0017] Secondly, embodiments of the present invention also provide a bitrate control device, the device comprising: an acquisition unit, configured to acquire two consecutive frames of images, the two frames including a target frame to be encoded and a frame preceding the target frame; a calculation unit, configured to calculate a semantic difference degree between the target frame and the preceding frame, the semantic difference degree being the distance between feature vectors of the original images of the target frame and the preceding frame, the feature vectors being extracted by a convolutional neural network; and an encoding unit, configured to select a bitrate control strategy based on the semantic difference degree and encode the target frame.

[0018] In one implementation, the encoding unit is specifically configured to: if the semantic difference is less than a first threshold, select a first encoding strategy, the first encoding strategy being used to determine the encoding type of the target frame as a prediction frame; if the semantic difference is greater than or equal to the first threshold and less than a second threshold, select a second encoding strategy, the second encoding strategy being used to determine the encoding type of the target frame as a prediction frame, and perform encoding compensation on the target frame according to the semantic difference, wherein the first threshold is less than the second threshold; if the semantic difference is greater than or equal to the second threshold, select a third encoding strategy, the third encoding strategy determining the encoding type of the target frame as a keyframe.

[0019] In one implementation, the rate control strategy is selected as a first encoding strategy, and the encoding unit is specifically used to: determine a preliminary target number of bits based on the encoding type of the target frame and the state of the video buffer validator model, wherein the state of the video buffer validator model is determined based on the deviation between the current water level and the target water level of the leaky bucket model in the video buffer validator model; input the preliminary target number of bits into the inter-frame rate-quantization model to obtain quantization parameters; and encode the target frame according to the quantization parameters.

[0020] In one implementation, the rate control strategy is selected as a second encoding strategy, and the encoding unit is specifically used to: determine a preliminary target bit count based on the encoding type of the target frame and the state of the video buffer validator model, wherein the state of the video buffer validator model is determined based on the deviation between the current water level and the target water level of the leaky bucket model in the video buffer validator model; predict compensation coefficients using a pre-trained rate compensation model based on the semantic difference degree; adjust the preliminary target bit count according to the compensation coefficients to obtain a corrected target bit count; input the corrected target bit count into an inter-frame rate-quantization model to obtain quantization parameters; and encode the target frame according to the quantization parameters.

[0021] In one embodiment, the apparatus further includes: a first update unit, configured to, after encoding the target frame according to the quantization parameters, keep the parameters of the inter-frame bitrate-quantization model unchanged, and update the current level of the video buffer validator model according to the actual number of encoded bits of the target frame.

[0022] In one implementation, the rate control strategy is selected as a third encoding strategy, and the encoding unit is specifically used to: determine a preliminary target number of bits based on the encoding type of the target frame and the state of the video buffer validator model, wherein the state of the video buffer validator model is determined based on the deviation between the current water level and the target water level of the leaky bucket model in the video buffer validator model; obtain quantization parameters based on the preliminary target number of bits through an intra-frame rate-quantization model; and encode the target frame based on the quantization parameters.

[0023] In one embodiment, the apparatus further includes: a second update unit, configured to update the parameters of the intra-frame bitrate-quantization model and the state of the video buffer validator model after encoding the target frame according to the quantization parameters; and a reset unit, configured to reset the parameters of the inter-frame bitrate-quantization model to the initial parameter state.

[0024] In one implementation, the feature vector represents the high-level semantic content of the corresponding image.

[0025] In one implementation, the convolutional neural network is a ResNet-18 network or a visual geometry group network.

[0026] In one embodiment, the apparatus further includes a training unit, the training unit being configured to: acquire multiple video image sequences; encode each video image sequence in an encoding mode with fixed quantization parameters, and record the actual number of encoded bits for each frame during the encoding process; calculate the relative bit growth ratio of the current frame relative to the previous frame in each pair of consecutive frames based on the actual number of encoded bits; calculate the semantic difference degree of each pair of consecutive frames; select samples whose semantic difference degree is greater than or equal to a first threshold and less than a second threshold to form a training dataset, wherein each sample in the training dataset includes a semantic difference degree and a corresponding relative bit growth ratio; and determine the parameters of the rate compensation model based on the training dataset, wherein the parameters minimize the error between the relative bit overhead prediction value output by the rate compensation model based on the semantic difference degree and the relative bit growth ratio in the samples.

[0027] In one implementation, the rate compensation model is R = Where D is the semantic dissimilarity and R is the relative bit overhead. , These are the parameters of the bitrate compensation model.

[0028] In one embodiment, the convolutional neural network is deployed on a neural network processing unit.

[0029] Thirdly, embodiments of the present invention also provide an electronic device, the electronic device comprising: a processor and a memory, the processor being electrically connected to the memory; the memory being used to store executable program code; the processor running a program corresponding to the executable program code by reading the executable program code stored in the memory, thereby implementing the bitrate control method provided in any one of the embodiments of the present invention.

[0030] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement any of the bitrate control methods provided in the embodiments of the present invention.

[0031] The bitrate control method, apparatus, electronic device, and storage medium provided in the embodiments of the present invention can acquire two consecutive frames of images, calculate the semantic difference between a target frame and the previous frame, wherein the semantic difference is the distance between the feature vectors of the original images of the target frame and the previous frame, the feature vectors are extracted by a convolutional neural network, and a bitrate control strategy is selected to encode the target frame based on the semantic difference. By introducing a convolutional neural network to extract the semantic features of the image to quantify the content difference between the two frames and selecting different bitrate control strategies accordingly, the problem of large deviations between the predicted quantization parameters and actual requirements is avoided, thus stabilizing the bitrate and improving the accuracy of bitrate control. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 A flowchart of a rate control method provided in an embodiment of the present invention;

[0034] Figure 2 A schematic diagram of a bitrate control method provided in an embodiment of the present invention;

[0035] Figure 3 Another flowchart of the bitrate control method provided for an embodiment of the present invention;

[0036] Figure 4 A flowchart illustrating a training method for a rate compensation model provided in an embodiment of the present invention;

[0037] Figure 5 A detailed flowchart of a rate control method provided for embodiments of the present invention;

[0038] Figure 6 A schematic diagram of a bitrate control device provided in an embodiment of the present invention;

[0039] Figure 7 A schematic diagram of an electronic device provided as an embodiment of the present invention. Detailed Implementation

[0040] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0041] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0042] In a first aspect, embodiments of the present invention provide a bitrate control method that can stabilize the bitrate and improve the accuracy of bitrate control.

[0043] like Figure 1 As shown, an embodiment of the present invention provides a bitrate control method, the method including:

[0044] S11, acquire two consecutive frames of images, the two frames of images including the target frame to be encoded and the frame preceding the target frame.

[0045] In this step, the target frame to be encoded and its preceding frame in time sequence can be obtained. Encoding refers to the process of converting raw image data into a bitstream according to a specific compression standard (such as H.264 / H.265). The preceding frame serves as a reference image, and its encoding information (such as the actual number of bits) is recorded by the system and used to guide the bitrate allocation of the current frame.

[0046] S12, calculate the semantic difference between the target frame and the previous frame, whereby the semantic difference is the distance between the feature vectors of the original images of the target frame and the previous frame, and the feature vectors are extracted by a convolutional neural network.

[0047] In this step, a Convolutional Neural Network (CNN) is used to extract feature vectors from the target frame and the previous frame, respectively. A CNN is a deep learning model that can automatically learn abstract, robust feature representations from the original image through multiple layers of convolution and pooling operations, such as object contours, texture structures, or scene layouts. Subsequently, the distance between these two feature vectors (e.g., Euclidean distance) is calculated, and this distance value is defined as the semantic dissimilarity. It can be understood that a larger semantic dissimilarity indicates a more significant change in content between the two frames (such as rapid object movement, sudden changes in lighting, or scene transitions); conversely, a smaller semantic dissimilarity indicates a more gradual change in content between the two frames.

[0048] S13, Based on the semantic difference, select a bitrate control strategy to encode the target frame.

[0049] In this step, a differentiated bitrate control strategy is selected to encode the target frame based on the calculated semantic dissimilarity. The bitrate control strategy can refer to dynamic bit allocation and quantization parameter adjustment rules designed to adapt to changes in different video content. For example, one bitrate control strategy can be selected when the semantic dissimilarity is greater than a certain threshold, and another bitrate control strategy can be selected when the semantic dissimilarity is less than a certain threshold. In this way, the target frame can be encoded using different bitrate control strategies.

[0050] The bitrate control method provided by embodiments of the present invention can acquire two consecutive image frames, calculate the semantic difference between a target frame and the previous frame, wherein the semantic difference is the distance between the feature vectors of the original images of the target frame and the previous frame, the feature vectors are extracted by a convolutional neural network, and a bitrate control strategy is selected to encode the target frame based on the semantic difference. By introducing a convolutional neural network to extract semantic features of the image to quantify the content difference between the two image frames and selecting different bitrate control strategies accordingly, the problem of large deviations between predicted quantization parameters and actual requirements is avoided, thus stabilizing the bitrate and improving the accuracy of bitrate control.

[0051] Specifically, in this embodiment of the invention, the feature vector of the image can be extracted using different forms of convolutional neural networks. For example, the convolutional neural network can be a ResNet-18 network or a Visual GeometryGroup (VGG) network. The semantic content of a convolutional neural network is a hierarchical feature representation from bottom to top, from concrete to abstract, and from detail to concept; the high-level semantic content is the abstract features learned by the deep layers of the network, corresponding to object categories, overall concepts, and scene understanding. To accurately measure the degree of semantic change between two frames, an appropriate feature vector representation method can be selected. For example, in one implementation, the feature vector represents the high-level semantic content of the corresponding image. For example, in this embodiment of the invention, the convolutional neural network uses the ResNet-18 model. After the input image undergoes standard preprocessing such as uniform scaling and normalization, and the last fully connected layer is removed, a 512-dimensional feature vector is finally output to represent the high-level semantic content of the frame.

[0052] In practice, feature vectors can be extracted from the target frame and the original image of the previous frame, and the Euclidean distance between them can be calculated to obtain the semantic difference between frames. Compared with traditional low-order feature methods based on pixels or histograms, the semantic difference can effectively reflect the essence of content changes, such as object movement, the appearance of new objects, changes in lighting, or scene switching.

[0053] After obtaining the semantic difference, different bitrate control strategies can be selected based on the semantic difference between the target frame and the previous frame. For example, in one implementation, selecting a bitrate control strategy based on the semantic difference may include: if the semantic difference is less than a first threshold, selecting a first encoding strategy, which determines the encoding type of the target frame as a prediction frame; if the semantic difference is greater than or equal to the first threshold and less than a second threshold, selecting a second encoding strategy, which determines the encoding type of the target frame as a prediction frame and performs encoding compensation on the target frame according to the semantic difference, wherein the first threshold is less than the second threshold; if the semantic difference is greater than or equal to the second threshold, selecting a third encoding strategy, which determines the encoding type of the target frame as a keyframe.

[0054] For example, based on the semantic difference degree D and the first threshold Second threshold Based on this relationship, semantic differences can be divided into three levels, and differentiated bitrate control strategies can be implemented for different levels. Details are as follows.

[0055] First level (D< ):

[0056] The change is determined to be a normal content change, such as slow movement or slight lighting adjustment. In this scenario, the first encoding strategy is selected, and the encoding type of the target frame is determined to be a predicted frame. No additional intervention is required, and the RQ model is updated normally.

[0057] Second level ( ≤D< ):

[0058] The scene is characterized by drastic content changes, such as large-scale object movement, the entry of a new object into the frame, or significant lighting changes. While this may not constitute a complete scene transition, it can easily trigger a surge in P-frame bitrate. In this scenario, a second encoding strategy is selected, determining the encoding type of the target frame as a predicted frame while simultaneously performing bit compensation on the encoding parameters based on semantic dissimilarity.

[0059] Third level (D≥) ):

[0060] The scenario is determined to be a scene transition, such as a program change or interface jump. In this scenario, continuing to use P-frame encoding would result in a severe bit count overload, and forced compression would cause a sharp drop in image quality. Therefore, the current frame can be encoded as an I-frame to ensure that the starting frame of the new scene has sufficient quality, while preventing subsequent frames from going out of control due to the continuation of the error state.

[0061] In this embodiment, a first encoding strategy, a second encoding strategy, or a third encoding strategy can be selected based on the relationship between the semantic difference degree and the first threshold and the second threshold, respectively, to achieve a graded response to normal fluctuations, drastic changes, and scene switching. This avoids the waste of I-frames caused by misjudging scene switching and can also actively compensate for the bit rate during drastic changes, thereby improving the overall bit rate stability.

[0062] For example, in one implementation, the rate control strategy is selected as a first encoding strategy. The step of selecting the rate control strategy based on the semantic difference and encoding the target frame may include: determining a preliminary target number of bits according to the encoding type of the target frame and the state of the video buffer validator model, wherein the state of the video buffer validator model is determined based on the deviation between the current water level and the target water level of the leaky bucket model in the video buffer validator model; inputting the preliminary target number of bits into the inter-frame rate-quantization model to obtain quantization parameters; and encoding the target frame according to the quantization parameters.

[0063] Figure 2The diagram illustrates a bitrate control method provided by an embodiment of the present invention. Specifically, in this embodiment, a convolutional neural network is introduced to calculate the semantic difference between the target frame and the previous frame to quantify the content difference between them. Before each frame is formally encoded, the bit allocator combines the structure of the current GOP (Group of Pictures) and the position of the frame within the GOP to determine its encoding type (e.g., I-frame or P-frame) and allocates an initial target bitrate i_bits. Considering that the actual number of bits generated by encoding often deviates from the expected target, potentially causing the overall bitrate to deviate from the set value, a Video Buffering Verifier (VBV) model can be introduced, and a "leaky bucket" mechanism is used to dynamically compensate for the error. Specifically, based on the initial target bitrate i_bits, the deviation between the current water level and the target water level of the leaky bucket model is added to obtain the initial target bitrate v_bits.

[0064] In this embodiment, the rate compensation module uses the initial target bit count v_bits as the input to the inter-RQ model to obtain quantization parameters. Then, based on these quantization parameters, the target frame is encoded. After encoding, the leaky bucket model and the inter-RQ model are synchronously updated using the actual output encoded bit count and the quantization parameter QP value. This embodiment is suitable for scenarios where image content changes smoothly and can reduce encoding overhead.

[0065] It's important to note that the RC algorithm maintains independent RQ (Rate-Quantization Model) parameters for I-frames and P-frames, namely the intra-frame rate-quantization model and the inter-frame rate-quantization model. Both share the same function structure, differing only in their parameters. The RQ model establishes a mathematical mapping between the bit rate and the quantization step size, enabling precise control of coded bit resources. The intra-frame RQ model is primarily used in the bit allocation stage to estimate the bit overhead required for I-frames, thus reserving a reasonable bit budget for subsequent P-frames. The quantization parameter QP for P-frames is calculated through the inter-frame RQ model to achieve precise bit rate control for inter-frame coding.

[0066] For example, such as Figure 3 As shown, in one embodiment, the rate control strategy is selected as a second encoding strategy, and the process of selecting the rate control strategy based on the semantic difference and encoding the target frame may include:

[0067] S301. Determine the initial target number of bits based on the encoding type of the target frame and the state of the video buffer validator model, wherein the state of the video buffer validator model is determined based on the deviation between the current water level of the leaky bucket model in the video buffer validator model and the target water level.

[0068] S302. Based on the semantic difference degree, predict the compensation coefficient using a pre-trained bitrate compensation model;

[0069] S303. Adjust the initial target number of bits according to the compensation coefficient to obtain the corrected target number of bits;

[0070] S304. Input the corrected target bit count into the inter-frame bit rate-quantization model to obtain quantization parameters;

[0071] S305. Encode the target frame according to the quantization parameters.

[0072] The difference between the second encoding strategy and the first encoding strategy provided in this embodiment is that, in this embodiment, the rate compensation module can adjust the initial target bit number v_bits according to the semantic difference degree to obtain the corrected target bit number. Then, the corrected target bit number is input into the inter-frame rate-quantization model (inter RQ model) to obtain the quantization parameters.

[0073] Specifically, in this embodiment of the invention, a rate compensation model can be established for the Rate–Dissimilarity relationship using a mathematical function, i.e., let R = f(D), where R represents the relative bit overhead caused by changes in content between frames, and D represents the semantic dissimilarity calculated based on CNN feature vectors.

[0074] In practice, the rate compensation model can employ various methods to characterize the relationship between relative bit overhead R and semantic dissimilarity D, such as logarithmic functions and polynomials with multiple powers. For example, in one implementation, the rate compensation model is R = Where D is the semantic dissimilarity and R is the relative bit overhead. , Here are the parameters of the rate compensation model. The rate compensation model uses a polynomial function R = Due to its strong fitting ability, it is easy to update the gradient descent method quickly, balancing model performance and ease of engineering implementation.

[0075] Based on this, step S302, which calculates the compensation coefficient using a bitrate compensation model according to the semantic difference degree, may include:

[0076] The initial quantization parameter QP is calculated once using the inter RQ model with a preliminary target number of bits v_bits, and is denoted as QP.k ;

[0077] Based on the current QP range, obtain the corresponding rate compensation model parameter a. k With b k ;

[0078] semantic difference D k The compensation coefficient R is calculated using the bitrate compensation model. k :

[0079] R k = ;

[0080] In step S303, the target number of bits is corrected using a rate compensation model, and finally the corrected target number of bits t_bits is calculated using the inter RQ model:

[0081] ( )

[0082] In step S304, the corrected target number of bits t_bits is input into the inter RQ model to obtain the final QP used for encoding, which is then sent to the encoder to complete the encoding. After encoding, the actual number of encoded bits o_bits is obtained.

[0083] In this embodiment, the initial target bit count is corrected based on the compensation coefficient calculated by semantic difference. Therefore, it is possible to actively suppress the surge in P-frame bit rate and smooth bit rate fluctuations by dynamically compensating the bit budget through semantic difference in the face of drastic content changes.

[0084] In a specific implementation, after encoding the target frame, the state of the video buffer validator model can be updated. For example, in one embodiment, after encoding the target frame according to the quantization parameters, the method may further include: keeping the parameters of the inter-frame bitrate-quantization model unchanged, and updating the current level of the video buffer validator model according to the actual number of encoded bits of the target frame.

[0085] Because drastic content changes can easily lead to inaccurate RQ model predictions, to avoid model contamination, the results of this frame will not be used to update the inter-RQ model, but rather to update the leaky bucket model. Simultaneously, the rate compensation model can also be updated. Specifically, in the H.264 and H.265 coding standards, for every 6-degree difference in the quantization parameter QP, the coding rate (bit usage) approximately doubles or decreases. Therefore, in QP... l Not equal to QP k At that time, the actual number of encoded bits o_bits is approximately converted to the QP before adjustment. k Calculate R based on the number of bits. k :

[0086] R k = o_bits / v_bits

[0087] Bitrate compensation model parameter a k b k The update is as follows:

[0088] a k a k +2 (R k - - ) ;

[0089] b k b k +2 (R k - - );in, Let be the learning rate, a constant.

[0090] For example, in one implementation, the rate control strategy is selected as a third encoding strategy. The step of selecting the rate control strategy based on the semantic difference and encoding the target frame may include: determining a preliminary target number of bits based on the encoding type of the target frame and the state of the video buffer validator model, wherein the state of the video buffer validator model is determined based on the deviation between the current water level and the target water level of the leaky bucket model in the video buffer validator model; obtaining quantization parameters based on the preliminary target number of bits using an intra-frame rate-quantization model; and encoding the target frame based on the quantization parameters.

[0091] Specifically, in the third encoding strategy, the target frame is forcibly encoded as an I-frame. Unlike traditional methods that rely on weighted estimation of the QP of P-frames in the previous GOP, this embodiment does not rely on weighted estimation of the QP of P-frames in the previous GOP. Instead, it directly sets the corrected target bit count t_bits to the initial target bit count v_bits, inputs it into the intraRQ model, calculates the QP value of the I-frame, and sends this QP value to the encoder to complete the I-frame encoding. After encoding, the leaky bucket model and the intraRQ model are updated, and the interRQ model is reset to its initial parameter state to avoid the statistical information of the old scene interfering with the bitrate control of the new scene.

[0092] In this embodiment of the invention, based on the initial target bit count, the quantization parameter QP is obtained through the intra-frame RQ model, and the target frame is encoded as an I-frame. This enables direct encoding of I-frames during real-scene switching, avoiding transmission delays and stuttering caused by ultra-large P-frames, and ensuring the image quality of the starting frame of the new scene.

[0093] In specific implementations, after encoding the target frame, the parameters of the intra-frame bitrate-quantization model and the state of the video buffer validator model can be updated. For example, in one embodiment, after encoding the target frame according to the quantization parameters, the method further includes: updating the parameters of the intra-frame bitrate-quantization model and the state of the video buffer validator model; and resetting the parameters of the inter-frame bitrate-quantization model to their initial parameter states.

[0094] In this embodiment of the invention, after the third encoding strategy is completed, the intra-frame RQ model and VBV model states are updated, and the inter-frame RQ model parameters are reset to their initial states. This can cut off the interference of old scene statistics on the new scene, ensure that the bitrate control converges quickly after the switch, and avoid loss of control.

[0095] Furthermore, embodiments of the present invention also provide a training method for a rate compensation model. For example, in one implementation, before predicting compensation coefficients using a pre-trained rate compensation model based on the semantic dissimilarity, the method may further include: acquiring multiple video image sequences; encoding each video image sequence in an encoding mode with fixed quantization parameters, recording the actual number of encoded bits for each frame during the encoding process; calculating the relative bit growth ratio of the current frame relative to the previous frame in each pair of consecutive frames based on the actual number of encoded bits; calculating the semantic dissimilarity for each pair of consecutive frames; selecting samples whose semantic dissimilarity is greater than or equal to a first threshold and less than a second threshold to constitute a training dataset, wherein each sample in the training dataset includes a semantic dissimilarity and a corresponding relative bit growth ratio; and determining the parameters of the rate compensation model based on the training dataset, wherein the parameters minimize the error between the relative bit overhead prediction value output by the rate compensation model based on the semantic dissimilarity and the relative bit growth ratio in the samples.

[0096] Specifically, such as Figure 4 As shown in this embodiment of the invention, the training process of the bitrate compensation model may include:

[0097] S401. Obtain multiple video image sequences.

[0098] Multiple representative original video image sequences were collected as training and testing sets. Since the encoder's rate-distortion optimization behavior differs significantly under different quantization parameters (QP), a set of rate compensation model parameters was maintained independently for each QP value to improve model generalization ability and adaptation accuracy. Let the range of quantization parameter QP be... If ~QPmax is maintained, then a total of max+1 sets of independent parameters are maintained.

[0099] S402. In the encoding mode with fixed quantization parameters, each segment of the video image sequence is encoded, and the actual number of encoded bits for each frame during the encoding process is recorded.

[0100] The fixed GOP structure is IPPPPP…, and the encoder adopts constant QP mode, encoding each video segment frame by frame and recording the actual output bit count of each frame after encoding.

[0101] S403. Based on the actual number of encoded bits, calculate the relative bit growth ratio of the current frame relative to the previous frame in each pair of consecutive frames.

[0102] Taking the i-th P-frame as an example, the number of encoded bits in the previous frame is The current frame encoding bit count is The relative bit growth ratio .

[0103] S404. Calculate the semantic difference between each pair of consecutive frames.

[0104] The pre-trained CNN module is used to extract 512-dimensional feature vectors from the current frame and the previous frame, denoted as follows: and Semantic difference Euclidean distance is usually used.

[0105] S405. Select samples whose semantic difference is greater than or equal to the first threshold and less than the second threshold to form a training dataset, wherein each sample in the training dataset includes a semantic difference and a corresponding relative bit growth ratio.

[0106] Set the first threshold Second threshold Select the one that satisfies The training dataset consists of samples; each sample includes semantic dissimilarity. and the corresponding relative bit growth ratio .

[0107] S406. Based on the training dataset, determine the parameters of the rate compensation model, wherein the parameters minimize the error between the relative bit overhead prediction value output by the rate compensation model according to the semantic difference degree and the relative bit growth ratio in the sample.

[0108] Choose a polynomial model R = To characterize the relationship between relative bit overhead R and semantic dissimilarity D, a loss function is constructed:

[0109] L(a, b) =

[0110] Gradient descent iterative optimization is used, with a learning rate η of 0.01. The parameter update formula is as follows:

[0111] a new = a old +

[0112] b new = b old +

[0113] After iterative convergence, the current value is obtained. Corresponding model initial parameters .

[0114] The bitrate compensation model training method provided in this invention provides stable and reliable initial parameters for the bitrate compensation model through fixed QP encoding, modeling the correlation between inter-frame semantic differences and bit growth, interval sample selection and polynomial fitting, enabling the model to accurately predict bit overhead caused by differences in video content and improve the accuracy of bitrate control.

[0115] It should be noted that, thanks to the widespread use of NPUs (Neural Processing Units) in modern SoC architectures, CNN feature extraction can be efficiently deployed on dedicated hardware in this embodiment of the invention. For example, in one implementation, the convolutional neural network is deployed on a neural processing unit. Specifically, the step of encoding the target frame can be performed by a VPU (Video Processing Unit). By deploying the computationally intensive task of CNN feature extraction on the NPU, and the bitrate control and encoding tasks on the VPU, the two work together heterogeneously. The semantic difference degree D quickly calculated by the NPU is directly used as the basis for the VPU to perform bitrate control and encoding decisions. In this embodiment of the invention, by running the convolutional neural network on the NPU, utilizing its dedicated instruction set and parallel computing architecture, the feature calculation process such as convolution and pooling can be significantly accelerated, reducing the latency and power consumption of image feature extraction, and avoiding the performance bottleneck of general-purpose processors in intensive neural network operations. At the same time, this approach can effectively reduce CPU load, allowing the encoding system to make fuller use of heterogeneous computing resources, thereby improving overall throughput efficiency.

[0116] The bitrate control method provided by the embodiments of the present invention will be described in detail below through specific examples.

[0117] like Figure 5 As shown, the bitrate control method provided in the embodiments of the present invention may include:

[0118] S501. Acquire two consecutive frames of images, the two frames of images including the target frame to be encoded and the frame preceding the target frame;

[0119] S502. Calculate the semantic difference between the target frame and the previous frame, wherein the semantic difference is the distance between the feature vectors of the original images of the target frame and the previous frame, and the feature vectors are extracted by a convolutional neural network.

[0120] S503. If the semantic difference is greater than or equal to the first threshold and less than the second threshold, a second coding strategy is selected. The second coding strategy determines the coding type of the target frame as a predicted frame and performs coding compensation on the target frame according to the semantic difference.

[0121] S504. Determine the initial target number of bits based on the encoding type of the target frame and the state of the video buffer validator model, wherein the state of the video buffer validator model is determined based on the deviation between the current water level of the leaky bucket model in the video buffer validator model and the target water level.

[0122] S505. Based on the semantic difference degree, predict the compensation coefficient using a pre-trained bitrate compensation model;

[0123] S506. Adjust the initial target number of bits according to the compensation coefficient to obtain the corrected target number of bits;

[0124] S507. Input the corrected target bit count into the inter-frame bit rate-quantization model to obtain quantization parameters;

[0125] S508. Encode the target frame according to the quantization parameters;

[0126] S509. Keeping the parameters of the inter-frame bitrate-quantization model unchanged, update the current level of the video buffer verifier model according to the actual number of encoded bits of the target frame.

[0127] Secondly, embodiments of the present invention provide a bitrate control device that can stabilize the bitrate and improve the accuracy of bitrate control.

[0128] like Figure 6 As shown, embodiments of the present invention also provide a bitrate control device, which may include:

[0129] Acquisition unit 31 is used to acquire two consecutive frames of images, the two frames of images including the target frame to be encoded and the frame preceding the target frame;

[0130] The calculation unit 32 is used to calculate the semantic difference between the target frame and the previous frame, wherein the semantic difference is the distance between the feature vectors of the original images of the target frame and the previous frame, and the feature vectors are extracted by a convolutional neural network.

[0131] Encoding unit 33 is used to select a rate control strategy based on the semantic difference degree and encode the target frame.

[0132] The bitrate control device provided in the embodiments of the present invention can acquire two consecutive frames of images, calculate the semantic difference between a target frame and the previous frame, wherein the semantic difference is the distance between the feature vectors of the original images of the target frame and the previous frame, the feature vectors are extracted by a convolutional neural network, and a bitrate control strategy is selected to encode the target frame based on the semantic difference. By introducing a convolutional neural network to extract the semantic features of the image to quantify the content difference between the two frames and selecting different bitrate control strategies accordingly, the problem of large deviations between the predicted quantization parameters and actual requirements is avoided, thus stabilizing the bitrate and improving the accuracy of bitrate control.

[0133] In one implementation, the encoding unit 33 is specifically configured to: if the semantic difference is less than a first threshold, select a first encoding strategy, wherein the first encoding strategy determines the encoding type of the target frame as a prediction frame; if the semantic difference is greater than or equal to the first threshold and less than a second threshold, select a second encoding strategy, wherein the second encoding strategy determines the encoding type of the target frame as a prediction frame, and performs encoding compensation on the target frame according to the semantic difference, wherein the first threshold is less than the second threshold; if the semantic difference is greater than or equal to the second threshold, select a third encoding strategy, wherein the third encoding strategy determines the encoding type of the target frame as a keyframe.

[0134] In one implementation, the rate control strategy is selected as a first encoding strategy, and the encoding unit 33 is specifically used to: determine the initial target bit count based on the encoding type of the target frame and the state of the video buffer validator model, wherein the state of the video buffer validator model is determined based on the deviation between the current water level and the target water level of the leaky bucket model in the video buffer validator model; input the initial target bit count into the inter-frame rate-quantization model to obtain quantization parameters; and encode the target frame according to the quantization parameters.

[0135] In one implementation, the rate control strategy is selected as a second encoding strategy, and the encoding unit 33 is specifically used to: determine the initial target bit count based on the encoding type of the target frame and the state of the video buffer validator model, wherein the state of the video buffer validator model is determined based on the deviation between the current water level and the target water level of the leaky bucket model in the video buffer validator model; predict compensation coefficients based on the semantic difference degree using a pre-trained rate compensation model; adjust the initial target bit count based on the compensation coefficients to obtain a corrected target bit count; input the corrected target bit count into the inter-frame rate-quantization model to obtain quantization parameters; and encode the target frame based on the quantization parameters.

[0136] In one embodiment, the apparatus further includes: a first update unit, configured to, after encoding the target frame according to the quantization parameters, keep the parameters of the inter-frame bitrate-quantization model unchanged, and update the current level of the video buffer validator model according to the actual number of encoded bits of the target frame.

[0137] In one implementation, the rate control strategy is selected as a third encoding strategy, and the encoding unit 33 is specifically used to: determine the initial target bit count based on the encoding type of the target frame and the state of the video buffer validator model, wherein the state of the video buffer validator model is determined based on the deviation between the current water level and the target water level of the leaky bucket model in the video buffer validator model; obtain quantization parameters based on the initial target bit count through the intra-frame rate-quantization model; and encode the target frame based on the quantization parameters.

[0138] In one embodiment, the apparatus further includes: a second update unit, configured to update the parameters of the intra-frame bitrate-quantization model and the state of the video buffer validator model after encoding the target frame according to the quantization parameters; and a reset unit, configured to reset the parameters of the inter-frame bitrate-quantization model to the initial parameter state.

[0139] In one implementation, the feature vector represents the high-level semantic content of the corresponding image.

[0140] In one implementation, the convolutional neural network is a ResNet-18 network or a visual geometry group network.

[0141] In one embodiment, the apparatus further includes a training unit, the training unit being configured to: acquire multiple video image sequences; encode each video image sequence in an encoding mode with fixed quantization parameters, and record the actual number of encoded bits for each frame during the encoding process; calculate the relative bit growth ratio of the current frame relative to the previous frame in each pair of consecutive frames based on the actual number of encoded bits; calculate the semantic difference degree of each pair of consecutive frames; select samples whose semantic difference degree is greater than or equal to a first threshold and less than a second threshold to form a training dataset, wherein each sample in the training dataset includes a semantic difference degree and a corresponding relative bit growth ratio; and determine the parameters of the rate compensation model based on the training dataset, wherein the parameters minimize the error between the relative bit overhead prediction value output by the rate compensation model based on the semantic difference degree and the relative bit growth ratio in the samples.

[0142] In one implementation, the rate compensation model is R = Where D is the semantic dissimilarity and R is the relative bit overhead. , These are the parameters of the bitrate compensation model.

[0143] In one embodiment, the convolutional neural network is deployed on a neural network processing unit.

[0144] Thirdly, embodiments of the present invention also provide an electronic device capable of stabilizing the bit rate and improving the accuracy of bit rate control.

[0145] like Figure 7 As shown, the electronic device provided in the embodiments of the present invention may include: a processor 71 and a memory 72, wherein the processor 71 and the memory 72 are electrically connected; the memory 72 is used to store executable program code; the processor 71 runs a program corresponding to the executable program code by reading the executable program code stored in the memory 72, so as to implement any of the bitrate control methods provided in the foregoing embodiments.

[0146] The specific execution process of the above steps by the processor 71, as well as the steps further executed by the processor 71 by running executable program code, can be found in the description of the foregoing embodiments, and will not be repeated here.

[0147] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement any of the bitrate control methods provided in the foregoing embodiments, thus achieving the corresponding technical effects. This has been described in detail above and will not be repeated here.

[0148] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0149] The various embodiments in this specification are described in a related manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0150] In particular, the device embodiment is basically similar to the method embodiment, so the description is relatively simple. For relevant details, please refer to the description of the method embodiment.

[0151] For ease of description, the above apparatus is described by dividing it into various functional units / modules. Of course, in implementing this invention, the functions of each unit / module can be implemented in one or more software and / or hardware.

[0152] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0153] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A bitrate control method, characterized in that, include: Acquire two consecutive frames of images, the two frames of images including the target frame to be encoded and the frame preceding the target frame; Calculate the semantic difference between the target frame and the previous frame, where the semantic difference is the distance between the feature vectors of the original images of the target frame and the previous frame, and the feature vectors are extracted by a convolutional neural network; Based on the semantic difference, a bitrate control strategy is selected to encode the target frame; The step of selecting a bitrate control strategy based on the semantic difference includes: If the semantic difference is less than a first threshold, a first encoding strategy is selected, which is used to determine the encoding type of the target frame as a predicted frame. If the semantic difference is greater than or equal to the first threshold and less than the second threshold, a second encoding strategy is selected. The second encoding strategy is used to determine the encoding type of the target frame as a predicted frame and to perform encoding compensation on the target frame according to the semantic difference, wherein the first threshold is less than the second threshold. If the semantic difference is greater than or equal to the second threshold, a third encoding strategy is selected, and the third encoding strategy determines the encoding type of the target frame as a key frame.

2. The bitrate control method according to claim 1, characterized in that, The bitrate control strategy is selected as the first encoding strategy; The step of selecting a rate control strategy based on the semantic difference and encoding the target frame includes: Based on the encoding type of the target frame and the state of the video buffer validator model, the initial target number of bits is determined, wherein the state of the video buffer validator model is determined based on the deviation between the current water level of the leaky bucket model in the video buffer validator model and the target water level. The initial target number of bits is input into the inter-frame bitrate-quantization model to obtain the quantization parameters; The target frame is encoded according to the quantization parameters.

3. The bitrate control method according to claim 1, characterized in that, The bitrate control strategy is selected as the second encoding strategy; The step of selecting a rate control strategy based on the semantic difference and encoding the target frame includes: Based on the encoding type of the target frame and the state of the video buffer validator model, the initial target number of bits is determined, wherein the state of the video buffer validator model is determined based on the deviation between the current water level of the leaky bucket model in the video buffer validator model and the target water level. Based on the semantic difference, the compensation coefficient is predicted using a pre-trained bitrate compensation model; The initial target number of bits is adjusted according to the compensation coefficient to obtain the corrected target number of bits; The corrected target bit count is input into the inter-frame bit rate-quantization model to obtain the quantization parameters; The target frame is encoded according to the quantization parameters.

4. The bitrate control method according to claim 3, characterized in that, After encoding the target frame according to the quantization parameters, the method further includes: Keeping the parameters of the inter-frame bitrate-quantization model unchanged, the current level of the video buffer validator model is updated according to the actual number of encoded bits of the target frame.

5. The bitrate control method according to claim 1, characterized in that, The bitrate control strategy is selected as the third encoding strategy; The step of selecting a rate control strategy based on the semantic difference and encoding the target frame includes: Based on the encoding type of the target frame and the state of the video buffer validator model, the initial target number of bits is determined, wherein the state of the video buffer validator model is determined based on the deviation between the current water level of the leaky bucket model in the video buffer validator model and the target water level. Based on the initial target number of bits, the quantization parameters are obtained through the intra-frame bitrate-quantization model; The target frame is encoded according to the quantization parameters.

6. The bitrate control method according to claim 5, characterized in that, After encoding the target frame according to the quantization parameters, the method further includes: Update the parameters of the intra-frame bitrate-quantization model and the state of the video buffer validator model; Reset the parameters of the inter-frame bitrate-quantization model to their initial state.

7. The bitrate control method according to claim 3, characterized in that, Before predicting the compensation coefficients using a pre-trained bitrate compensation model based on the semantic dissimilarity, the method further includes: Acquire multiple video image sequences; In the encoding mode with fixed quantization parameters, each segment of the video image sequence is encoded, and the actual number of encoded bits for each frame during the encoding process is recorded. Based on the actual number of encoded bits, calculate the relative bit growth ratio of the current frame relative to the previous frame in each pair of consecutive frames; Calculate the semantic difference between each pair of consecutive frames; Samples with semantic dissimilarity greater than or equal to a first threshold and less than a second threshold are selected to form a training dataset, wherein each sample in the training dataset includes semantic dissimilarity and a corresponding relative bit growth ratio; Based on the training dataset, the parameters of the rate compensation model are determined, wherein the parameters minimize the error between the relative bit overhead prediction value output by the rate compensation model according to the semantic dissimilarity and the relative bit growth ratio in the sample.

8. A bit rate control device, characterized in that, include: An acquisition unit is used to acquire two consecutive frames of images, the two frames of images including the target frame to be encoded and the frame preceding the target frame; A computing unit is used to calculate the semantic difference between the target frame and the previous frame, wherein the semantic difference is the distance between the feature vectors of the original images of the target frame and the previous frame, and the feature vectors are extracted by a convolutional neural network. An encoding unit is configured to select a rate control strategy based on the semantic difference and encode the target frame. Specifically, the encoding unit is used for: If the semantic difference is less than a first threshold, a first encoding strategy is selected, which is used to determine the encoding type of the target frame as a predicted frame. If the semantic difference is greater than or equal to the first threshold and less than the second threshold, a second encoding strategy is selected. The second encoding strategy is used to determine the encoding type of the target frame as a predicted frame and to perform encoding compensation on the target frame according to the semantic difference, wherein the first threshold is less than the second threshold. If the semantic difference is greater than or equal to the second threshold, a third encoding strategy is selected, and the third encoding strategy determines the encoding type of the target frame as a key frame.

9. An electronic device, characterized in that, The electronic device includes: a processor and a memory, wherein the processor is electrically connected to the memory; the memory is used to store executable program code; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the bit rate control method according to any one of claims 1 to 7.

Images