DVFS method and device of hardware video encoder
By using the number of ‘to be encoded cache pairs’ in the hardware video encoder to characterize the load and adjust the frequency in real time, the problem of power consumption and performance imbalance in the existing methods is solved, and efficient encoding in complex systems is achieved.
Patent Information
- Application Number
- CN202510430156.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-08
- Publication Date
- 2025-08-15
AI Technical Summary
The existing hardware video encoder DVFS method cannot accurately characterize the mapping relationship between encoding performance and system requirements in complex embedded systems, resulting in unbalanced power consumption and performance, especially when system resources fluctuate frequently.
By counting the number of video frame caches to be encoded and stream caches to be filled in the video encoding system, the workload of the hardware video encoder is characterized as a 'buffered cache pair', and its operating frequency is adjusted according to the real-time load relationship to meet the balance of encoding performance and power consumption.
It realizes that the performance of hardware video encoder in complex systems meets real-time requirements, while minimizing power consumption, avoiding encoding delay and frame drop problems, and has high adaptability and real-time performance.
Smart Images
Figure CN120499394A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to a power consumption control method, and in particular to a DVFS method suitable for a hardware video encoder. Background Art
[0002] DVFS (dynamic voltage and frequency scaling) is an energy management technology widely used to reduce energy consumption in processors and computing devices. DVFS adjusts voltage and frequency in real time based on the processor workload. It reduces voltage and frequency to save energy when the processor workload is low, and increases voltage and frequency to provide higher performance when the processor workload is high. By dynamically adjusting processor voltage and frequency to adapt to varying workloads, DVFS achieves energy savings and thermal management.
[0003] Video coding is a technology that compresses redundant components in video images and uses the least amount of data possible to represent video information. Common video coding standards include HEVC (high-efficiency video coding, also known as H.265) and AVC (advanced video coding, also known as H.264).
[0004] Because video encoding algorithms require a lot of computation, using an application-specific integrated circuit (ASIC) to accelerate the video encoding process has become a common practice in the industry to improve the speed and achieve real-time encoding and transmission. This type of ASIC that accelerates the video encoding process is generally referred to as a hardware video encoder, or VPU (video processing unit). "VPU" will be used to refer to "hardware video encoder" in this article.
[0005] The VPU can operate at different clock frequencies and voltages. Generally speaking, the higher the VPU's operating frequency, the stronger its encoding performance and the higher the required voltage. The lower the VPU's operating frequency, the weaker its encoding performance and the lower the required voltage.
[0006] See also Figure 1 A typical video coding system includes a video acquisition module 11, a video encoding module 12, and a bitstream storage and transmission module 13, which are connected in sequence. The core component of the video acquisition module 11 is generally a camera, which is responsible for acquiring the video frames to be encoded. The core component of the video encoding module 12 is the VPU, which is responsible for encoding the video frames to be encoded and generating the encoded video stream (abbreviated as "bitstream"). The core component of the bitstream storage and transmission module 13 is generally a file system or network transmission system, which is responsible for storing or transmitting the encoded video stream.
[0007] In video coding systems, the VPU must not only meet real-time encoding requirements but also minimize power consumption. If the VPU performance is too low to meet the system's requirements, encoding latency will increase and, in severe cases, frames to be encoded may be lost, significantly reducing the quality of the encoded video stream. If the VPU performance is too high, exceeding the system's requirements, power consumption will increase, increasing the system's operating cost.
[0008] Therefore, it is necessary to design a DVFS method suitable for the VPU based on the characteristics of the video coding system. When the VPU workload is low, its operating frequency is reduced to save energy. When the VPU workload is high, its operating frequency is increased to provide better performance.
[0009] Currently, there are some DVFS methods suitable for VPUs, the most typical of which are the DVFS method based on video encoding configuration parameters and the DVFS method based on VPU duty cycle.
[0010] The DVFS method based on video coding configuration parameters pre-measures the impact of different video coding configuration parameters on VPU performance. Before the VPU starts operating, the VPU's operating frequency is selected based on the user-set video coding configuration parameters to meet the performance requirements of the video coding system. These video coding configuration parameters include, for example, the number of video coding channels, the video coding standard used by each channel, video resolution, encoding frame rate, and encoding frame type. The essence of this DVFS method is to use video coding configuration parameters to characterize the VPU workload. Different video coding configuration parameters represent different VPU workloads, which in turn correspond to different VPU operating frequencies. This DVFS method has the following disadvantages. First, the mapping relationship between different video coding configuration parameters and VPU performance must be measured in advance. The more video coding configuration parameters there are, the greater the measurement workload. Furthermore, the mapping relationship may vary for different VPUs, requiring re-measurement, making adaptive use impossible. Second, it is not effective on complex embedded systems (such as smartphones). This is because the VPU's performance is affected not only by its own frequency but also by available resources such as the CPU and memory. In complex embedded systems, due to competition from other hardware devices, the CPU, memory and other resources available to the VPU are often unstable, resulting in a large difference between the actual performance of the VPU and the measured performance, making it difficult to match the requirements of the encoding system.
[0011] The VPU duty cycle-based DVFS method periodically monitors the VPU's duty cycle—the ratio of active operating time to total operating time—while the VPU is operating, and adjusts the VPU frequency based on the duty cycle. Essentially, this DVFS method uses the VPU's duty cycle to characterize the VPU's workload. A high VPU duty cycle indicates a heavy workload, requiring an increase in the VPU frequency to provide better video encoding performance. A low VPU duty cycle indicates a light workload, requiring a decrease in the VPU frequency to conserve system power. Compared to DVFS methods based on video encoding configuration parameters, this DVFS method hides system complexity by normalizing the relative relationship between VPU workload and VPU encoding performance to the VPU's duty cycle. It does not consider the impact of video encoding configuration parameters and available system resources such as CPU and memory on the VPU's actual encoding performance, enabling adaptive adjustment of the VPU frequency. This DVFS method has the following disadvantages. First, VPU duty cycle information must rely on statistics collected over a period of time, resulting in an inherent lag. Using lagged duty cycle information to predict the current VPU workload inevitably introduces certain errors, especially when the system's instantaneous state undergoes significant changes. Predicting the current VPU workload based on duty cycle information can be highly inaccurate. Secondly, this DVFS method cannot effectively handle situations where the video capture module proactively reduces the encoding frame rate. Because this DVFS method relies on statistics about the VPU's performance over a period of time, if the system state fluctuates dramatically during this statistical period, resulting in a significant drop in video encoding system performance, the video capture module may proactively reduce the frame rate to prevent significant frame drops. Consequently, the VPU's duty cycle may actually drop significantly during this statistical period, causing the DVFS method to mistakenly believe that the VPU workload is low and further reduce the VPU's frequency. This drop in VPU frequency further degrades video encoding performance, prompting the video capture module to even more aggressively reduce the encoding frame rate, creating an irreversible vicious cycle.
[0012] As can be seen, due to the inherent complexity of video encoding systems, the actual encoding performance of the VPU is not only related to its own operating frequency but also affected by available system resources such as the CPU and memory. This makes the DVFS method based on video encoding configuration parameters poorly applicable in complex embedded systems. While the DVFS method based on the VPU duty cycle can hide system complexity, it relies on statistics of VPU operating conditions over a period of time, which leads to an inevitable lag and prevents accurate real-time assessment of the VPU workload. Summary of the Invention
[0013] The technical problem to be solved by this application is to provide a DVFS method suitable for hardware video encoders. The method can hide the complexity of the system in a video encoding system where the system's available resources (such as CPU and memory) fluctuate frequently, and use clear and easily statistically analyzed data indicators to adaptively characterize the mapping relationship between the encoding performance of the hardware video encoder and the requirements of the video encoding system in real time. The method can then adjust the frequency of the hardware video encoder accordingly to meet the video encoding system's requirements for balanced performance and power consumption.
[0014] In order to solve the above technical problems, the present application discloses a DVFS method for a hardware video encoder, comprising the following steps. Step S31: Count the number of video frame buffers to be encoded and code stream buffers to be filled in the video encoding system, and call a "video frame buffer to be encoded" and a "code stream buffer to be filled" a "cache pair to be encoded", and use the number of "cache pairs to be encoded" at a certain moment to represent the workload of the hardware video encoder at that moment. Step S32: According to the real-time workload of the hardware video encoder, determine the relative relationship between the encoding performance and encoding requirements of the hardware video encoder at that moment, and calculate the new operating frequency of the hardware video encoder accordingly. Step S33: Adjust the hardware video encoder to the new operating frequency.
[0015] Furthermore, in step S31, when the number of "video frame buffers to be encoded" and "code stream buffers to be filled" in the video encoding system is not equal at a certain moment, the smaller one is taken as the number of "buffer pairs to be encoded".
[0016] Furthermore, in step S31, the workload of the hardware video encoder at a certain moment is the number of "buffer pairs to be encoded" in the video encoding system at that moment.
[0017] Furthermore, in step S31, the following three situations may occur. Situation 1: The speed at which "buffer pairs to be encoded" are generated in the video encoding system is greater than the speed at which the hardware video encoder processes the "buffer pairs to be encoded". At this time, the encoding performance of the hardware video encoder cannot meet the requirements of the video encoding system, and the number of "buffer pairs to be encoded" gradually increases. Situation 2: The speed at which "buffer pairs to be encoded" are generated in the video encoding system is less than the speed at which the hardware video encoder processes the "buffer pairs to be encoded". At this time, the encoding performance of the hardware video encoder exceeds the requirements of the video encoding system, and the number of "buffer pairs to be encoded" gradually decreases. Situation 3: When the speed at which "buffer pairs to be encoded" are generated in the video encoding system is equal to the speed at which the hardware video encoder processes the "buffer pairs to be encoded", the encoding performance of the hardware video encoder just meets the requirements of the video encoding system, and the number of "buffer pairs to be encoded" remains basically unchanged.
[0018] Furthermore, in step S31, the greater the number of "buffer pairs to be encoded", the greater the workload of the hardware video encoder and the greater the encoding delay; the fewer the number of "buffer pairs to be encoded", the smaller the workload of the hardware video encoder and the smaller the encoding delay.
[0019] Furthermore, in step S32, firstly, it is determined whether the current encoding performance of the hardware video encoder is surplus, insufficient, or suitable; and secondly, a new operating frequency of the hardware video encoder is calculated.
[0020] Furthermore, in step S32, M represents a threshold for determining excess encoding performance of the hardware video encoder, and N represents a threshold for determining insufficient encoding performance of the hardware video encoder, where 0≤M<N. If the number of "buffer pairs to be encoded" in the current video encoding system is ≤M, then the current encoding performance of the hardware video encoder is determined to be excess. If the number of "buffer pairs to be encoded" in the current video encoding system is ≥N, then the current encoding performance of the hardware video encoder is determined to be insufficient. If the number of "buffer pairs to be encoded" in the current video encoding system is >M and <N, then the current encoding performance of the hardware video encoder is determined to be adequate.
[0021] Furthermore, in step S32, the optional operating frequency values of the hardware video encoder are arranged in ascending order, and a unique frequency number is assigned to each operating frequency value starting from 0 and incremented by 1, thereby obtaining a mapping table between frequency numbers and optional operating frequency values of the hardware video encoder; min_freq_idx represents the lowest frequency number, max_freq_idx represents the highest frequency number, cur_freq_idx represents the frequency number corresponding to the current operating frequency value of the hardware video encoder, freq_up_step represents the frequency increase step size, and freq_down_step represents the frequency decrease step size; min_freq_idx ≤ cur_freq_idx ≤ max_freq_idx. When the current encoding performance of the hardware video encoder is excessive, the difference between cur_freq_idx and freq_down_step is compared with the value of min_freq_idx, and the larger value is used as the new frequency number of the hardware video encoder. When the hardware video encoder's current encoding performance is insufficient, the sum of cur_freq_idx + freq_up_step is compared with the value of max_freq_idx, and the smaller value is used as the new frequency number of the hardware video encoder. When the hardware video encoder's current encoding performance is adequate, the current frequency number cur_freq_idx of the hardware video encoder is directly used as the new frequency number of the hardware video encoder. After calculating the new frequency number of the hardware video encoder, a mapping table between the frequency number and all optional operating frequency values of the hardware video encoder is searched to obtain the new operating frequency of the hardware video encoder.
[0022] Furthermore, in step S32, the value of cur_freq_idx is updated using the calculated new frequency number of the hardware video encoder.
[0023] The present application also discloses a DVFS device for a hardware video encoder, including an encoder workload calculation module, an encoder frequency calculation module, and an encoder frequency adjustment module. The encoder workload calculation module is used to count the number of video frame buffers to be encoded and code stream buffers to be filled in the video encoding system, and a "video frame buffer to be encoded" and a "code stream buffer to be filled" are called a "buffer pair to be encoded", and the number of "buffer pairs to be encoded" at a certain moment is used to characterize the workload of the hardware video encoder at that moment. The encoder frequency calculation module is used to determine the relative relationship between the encoding performance and encoding requirements of the hardware video encoder at that moment according to the real-time workload of the hardware video encoder, and calculate the new operating frequency of the hardware video encoder accordingly. The encoder frequency adjustment module is used to adjust the hardware video encoder to the new operating frequency.
[0024] The technical effect achieved by this application is: compared with the DVFS method of the existing hardware video encoder, this application has the characteristics of small computational complexity, good adaptability, and strong real-time performance, so that the performance of the hardware video encoder can not only meet the real-time requirements of encoding, but also minimize the encoding power consumption. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 This is a schematic diagram of the structure of a typical video coding system.
[0026] Figure 2 yes Figure 1 The flowchart of the video encoding method of the video encoding system is shown.
[0027] Figure 3 This is a flowchart of the DVFS method suitable for hardware video encoders proposed in this application.
[0028] Figure 4 Schematic diagram of the structure of the DVFS device suitable for hardware video encoder proposed in this application.
[0029] Explanation of the reference numerals in the figure: video acquisition module 11, video encoding module 12, code stream storage and transmission module 13, encoder workload calculation module 31, encoder frequency calculation module 32, encoder frequency adjustment module 33. DETAILED DESCRIPTION
[0030] See also Figure 2 , Figure 1 The video encoding method of the video encoding system shown specifically includes the following steps.
[0031] Step S21: The video acquisition module 11 acquires the video frames to be encoded and stores them in a "video frame buffer to be encoded" for encoding by the video encoding module 12. The video acquisition module 11 maintains a software-level buffer queue, which contains multiple "video frame buffers to be encoded" of equal size. Each "video frame buffer to be encoded" is used to fill a video frame to be encoded.
[0032] The code stream storage and filling module 13 provides a "code stream buffer to be filled" for the video encoder 12 to fill the encoded video stream. The code stream storage and filling module 13 also maintains a software-level buffer queue, which contains multiple "code stream buffers to be filled" of equal size. Each "code stream buffer to be filled" is used to fill the encoded code stream of an encoded video frame (i.e., the encoded video stream).
[0033] The purpose of providing multiple "buffers for video frames to be encoded" and multiple "buffers for bitstreams to be filled" is to enable the video capture module 11, video encoding module 12, and bitstream storage and transmission module 13 to operate in parallel, thereby improving the performance of the video encoding system. For example, at a given moment, the video encoding module 12 may occupy a portion of the "buffer for video frames to be encoded" for encoding, while the video capture module 11 can still concurrently fill other "buffers for video frames to be encoded" with video frames. Alternatively, at a given moment, the video encoding module 12 may occupy a portion of the "buffer for bitstreams to be filled" for encoding, while the bitstream storage and transmission module 13 can still concurrently store and / or transmit the encoded video streams in other "buffers for bitstreams to be filled".
[0034] When encoding a video, the video encoding module 12 (e.g., the VPU) needs to read a video frame from a "video frame buffer to be encoded" for encoding and simultaneously write the corresponding encoded video stream to a "stream buffer to be filled." In other words, having at least one "video frame buffer to be encoded" and one "stream buffer to be filled" is a prerequisite for the video encoding module 12 to be able to encode video frames. In video encoding systems, multiple "video frame buffers to be encoded" and multiple "stream buffers to be filled" are typically allocated. Their number changes in real time with the state of the video encoding system and is not necessarily equal.
[0035] Step S22: The video encoding module 12 reads the video frames to be encoded stored in the "video frame buffer to be encoded" and encodes them, and stores the resulting encoded video stream in the "stream buffer to be filled." When the video encoding module 12 completes encoding of the video frames to be encoded in the "video frame buffer to be encoded," the original "video frame buffer to be encoded" is renamed the "encoded video frame buffer." When the "stream buffer to be filled" is filled with the encoded video stream, the original "stream buffer to be filled" is renamed the "filled stream buffer."
[0036] Step S23: After encoding the video frame to be encoded, the video acquisition module 11 reclaims the corresponding "encoded video frame buffer" to store the new video frame to be encoded. The stream storage and filling module 13 stores the encoded video stream stored in the "filled stream buffer" in a file or transmits it over the network, then reclaims the corresponding "filled stream buffer" to fill it with the new encoded video stream.
[0037] See also Figure 3 The DVFS method suitable for hardware video encoder proposed in this application includes the following steps.
[0038] Step S31: Count the number of video frame buffers to be encoded and code stream buffers to be filled in the video encoding system, and call a "video frame buffer to be encoded" and a "code stream buffer to be filled" a "cache pair to be encoded". The number of "cache pairs to be encoded" at a certain moment is used to represent the workload of the video encoding module 12 (i.e., the hardware video encoder) at that moment.
[0039] For example, at a certain moment, if there are 5 "video frame buffers to be encoded" and 3 "bitstream buffers to be filled" in the video encoding system, then there are 3 "buffer pairs to be encoded". Alternatively, at a certain moment, if there are 3 "video frame buffers to be encoded" and 0 "bitstream buffers to be filled" in the video encoding system, then there are 0 "buffer pairs to be encoded".
[0040] There are three possible situations in this step.
[0041] Case 1: The speed at which the video acquisition module 11 and the code stream storage and filling module 13 generate "buffer pairs to be encoded" is faster than the speed at which the video encoding module 12 processes "buffer pairs to be encoded". At this time, the encoding performance of the video encoding module 12 cannot meet the requirements of the video encoding system, and the number of "buffer pairs to be encoded" gradually increases.
[0042] Case 2: The speed at which the video acquisition module 11 and the code stream storage and filling module 13 generate "buffer pairs to be encoded" is slower than the speed at which the video encoding module 12 processes "buffer pairs to be encoded". At this time, the encoding performance of the video encoding module 12 exceeds the requirements of the video encoding system, and the number of "buffer pairs to be encoded" gradually decreases.
[0043] Case 3: When the speed at which the video acquisition module 11 and the code stream storage and filling module 13 generate "buffer pairs to be encoded" is exactly equal to the speed at which the video encoding module 12 processes "buffer pairs to be encoded", the encoding performance of the video encoding module 12 just meets the requirements of the video encoding system, and the number of "buffer pairs to be encoded" remains basically unchanged.
[0044] From the above three situations, it can be seen that the number of “buffer pairs to be encoded” in the video encoding system can reflect in real time the mapping relationship between the encoding performance of the current video encoding module 12 and the requirements of the video encoding system.
[0045] This application uses the number of "cache pairs to be encoded" to characterize the current workload of the video encoding module 12. The more "cache pairs to be encoded", the greater the workload of the video encoding module 12, and the greater the encoding delay. The fewer "cache pairs to be encoded", the smaller the workload of the video encoding module 12, and the smaller the encoding delay. At the same time, the change in the number of "cache pairs to be encoded" also reflects the match between the encoding performance of the video encoding module 12 and the requirements of the video encoding system. The number of "cache pairs to be encoded" gradually increases, indicating that the encoding performance of the video encoding module 12 is lower than the requirements of the video encoding system. The number of "cache pairs to be encoded" gradually decreases, indicating that the encoding performance of the video encoding module 12 exceeds the requirements of the video encoding system. The number of "cache pairs to be encoded" remains unchanged, indicating that the encoding performance of the video encoding module 12 just meets the requirements of the video encoding system.
[0046] It can be seen that selecting the number of "buffer pairs to be encoded" in the video encoding system to represent the real-time workload of video encoding module 12 conceals the complexity of the video encoding system and normalizes the relationship between the encoding performance of video encoding module 12 and the requirements of the video encoding system into a simple data indicator. This is highly adaptable, clear, and easy to statistically analyze. Furthermore, the above analysis shows that the current number of "buffer pairs to be encoded" in the video encoding system can reflect the current workload of video encoding module 12 in real time without lag, effectively coping with large fluctuations in the system's instantaneous state.
[0047] As an example, the workload of the video encoding module 12 at a certain moment is the number of “buffer pairs to be encoded” in the video encoding system at that moment.
[0048] Step S32: Determine the relative relationship between the encoding performance and encoding requirements of the video encoding module 12 at that moment according to the real-time workload of the video encoding module 12 , and calculate a new operating frequency of the video encoding module 12 accordingly.
[0049] First, determine whether the current encoding performance of the video encoding module 12 is excessive, insufficient, or appropriate. M represents the threshold for determining the excessive encoding performance of the video encoding module 12, and N represents the threshold for determining the insufficient encoding performance of the video encoding module 12, where 0 ≤ M < N. The specific values of M and N can be adjusted based on the video encoding system's sensitivity to encoding delay and system power consumption. If the video encoding system is more sensitive to encoding delay, M and N can be selected as smaller integer values. If the video encoding system is more sensitive to system power consumption, M and N can be selected as larger integer values.
[0050] If the number of "buffer pairs to be encoded" in the current video encoding system is ≤ M, it is determined that the encoding performance of the current video encoding module 12 is excessive and exceeds the requirements of the current video encoding system, and the operating frequency of the video encoding module 12 needs to be reduced.
[0051] If the number of "buffer pairs to be encoded" in the current video encoding system is ≥ N, it is determined that the encoding performance of the current video encoding module 12 is insufficient and lower than the requirements of the current video encoding system, and the operating frequency of the video encoding module 12 needs to be increased.
[0052] If the number of "buffer pairs to be encoded" in the current video encoding system is greater than M and less than N, it is determined that the encoding performance of the current video encoding module 12 is appropriate, and the operating frequency of the video encoding module 12 is maintained unchanged.
[0053] Next, calculate the new operating frequency of the video encoding module 12. Pre-arrange the optional operating frequency values of the video encoding module 12 in ascending order, and assign a unique frequency number, starting from 0 and incrementing by 1, to each operating frequency value. For example, if the video encoding module 12 has N optional operating frequency values, then arrange these N optional operating frequency values in ascending order and, starting with the lowest operating frequency value, assign frequency numbers 0, 1, 2, ...N-1. The lowest operating frequency value is numbered 0, and the highest operating frequency value is numbered N-1. This creates a mapping table between frequency numbers and the optional operating frequency values of the video encoding module 12. Hereinafter, min_freq_idx will be used to represent the lowest frequency number, max_freq_idx will be used to represent the highest frequency number, cur_freq_idx will be used to represent the frequency number corresponding to the current operating frequency value of the video encoding module 12, freq_up_step will be used to represent the frequency increase step, and freq_down_step will be used to represent the frequency decrease step. min_freq_idx≤cur_freq_idx≤max_freq_idx. freq_up_step indicates the step size by which the frequency number of the video encoding module 12 needs to be increased when the encoding performance of the video encoding module 12 is insufficient, and is a positive integer greater than 0. freq_down_step indicates the step size by which the frequency number of the video encoding module 12 needs to be reduced when the encoding performance of the video encoding module 12 is excessive, and is a positive integer greater than 0. The user can adjust the speed at which the frequency of the video encoding module 12 is increased or decreased by adjusting the values of freq_up_step and freq_down_step. The larger the values of freq_up_step and freq_down_step, the faster the speed at which the frequency of the video encoding module 12 is increased or decreased. The smaller the values of freq_up_step and freq_down_step, the slower the speed at which the frequency of the video encoding module 12 is increased or decreased.
[0054] When the current encoding performance of the video encoding module 12 is in excess, the difference between cur_freq_idx and freq_down_step is compared with the value of min_freq_idx, and the larger one is taken as the new frequency index of the video encoding module 12 .
[0055] When the current encoding performance of the video encoding module 12 is insufficient, the sum of cur_freq_idx+freq_up_step and the value of max_freq_idx are compared, and the smaller one is taken as the new frequency index of the video encoding module 12 .
[0056] When the current encoding performance of the video encoding module 12 is suitable, the current frequency number cur_freq_idx of the video encoding module 12 is directly used as the new frequency number of the video encoding module 12 .
[0057] After calculating the new frequency number for the video encoding module 12, a mapping table is searched for the frequency number and all optional operating frequency values for the video encoding module 12 to obtain the new operating frequency for the video encoding module 12. At the same time, the value of cur_freq_idx needs to be updated for subsequent adjustments to the operating frequency of the video encoding module 12.
[0058] Step S33: adjusting the video encoding module 12 to a new operating frequency. At this time, the performance of the video encoding module 12 can meet the real-time encoding requirements and can also reduce the encoding power consumption to the maximum extent.
[0059] See also Figure 4 The DVFS device suitable for a hardware video encoder proposed in this application includes an encoder workload calculation module 31 , an encoder frequency calculation module 32 , and an encoder frequency adjustment module 33 . Figure 4 The device shown corresponds to Figure 3 The method shown.
[0060] The encoder workload calculation module 31 is used to count the number of video frame buffers to be encoded and code stream buffers to be filled in the video encoding system. A "video frame buffer to be encoded" and a "code stream buffer to be filled" are called a "buffer pair to be encoded", and the number of "buffer pairs to be encoded" at a certain moment is used to represent the workload of the video encoding module (i.e., the hardware video encoder) at that moment.
[0061] The encoder frequency calculation module 32 is used to determine the relative relationship between the encoding performance and encoding requirements of the video encoding module at that moment according to the real-time workload of the video encoding module, and calculate the new operating frequency of the video encoding module accordingly.
[0062] The encoder frequency adjustment module 33 is used to adjust the video encoding module to a new operating frequency.
[0063] This application proposes a DVFS method suitable for hardware video encoders. It selects the number of "buffer pairs to be encoded" of a VPU (hardware video encoder) to characterize the VPU's workload. This method hides the complexity of the video encoding system and normalizes the relationship between the VPU's encoding performance and the system's requirements into a simple data indicator. This method is highly adaptable, clear, and easy to statistically analyze. Furthermore, the number of "buffer pairs to be encoded" of the current VPU can reflect the current VPU's workload in real time without lag, effectively coping with large fluctuations in the system's instantaneous state and avoiding the occurrence of severe frame drops.
[0064] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.
Claims
1. A DVFS method for a hardware video encoder, characterized in that: The method includes the following steps: Step S31: Counting the number of video frame buffers to be encoded and code stream buffers to be filled in the video encoding system. A "video frame buffer to be encoded" and a "code stream buffer to be filled" are referred to as a "buffer pair to be encoded." The number of "buffer pairs to be encoded" at a certain moment represents the workload of the hardware video encoder at that moment. Step S32: determining the relative relationship between the encoding performance of the hardware video encoder and the encoding requirement at that moment according to the real-time workload of the hardware video encoder, and calculating a new operating frequency of the hardware video encoder accordingly; Step S33: Adjust the hardware video encoder to a new operating frequency.
2. The DVFS method of the hardware video encoder according to claim 1, wherein: In step S31, at a certain moment, when the number of "video frame buffers to be encoded" and "stream buffers to be filled" in the video encoding system is not equal, the smaller one is taken as the number of "buffer pairs to be encoded".
3. The DVFS method of the hardware video encoder according to claim 2, wherein: In step S31, the workload of the hardware video encoder at a certain moment is the number of “buffer pairs to be encoded” in the video encoding system at that moment.
4. The DVFS method of the hardware video encoder according to claim 1, wherein: In step S31, the following three situations may occur: Case 1: The video encoding system generates "buffer pairs to be encoded" faster than the hardware video encoder can process them. In this case, the hardware video encoder's encoding performance cannot meet the requirements of the video encoding system, and the number of "buffer pairs to be encoded" gradually increases. Case 2: The rate at which the video encoding system generates "buffer pairs to be encoded" is slower than the rate at which the hardware video encoder processes them. In this case, the encoding performance of the hardware video encoder exceeds the requirements of the video encoding system, and the number of "buffer pairs to be encoded" gradually decreases. Case 3: When the speed at which the video encoding system generates "buffer pairs to be encoded" is equal to the speed at which the hardware video encoder processes them, the encoding performance of the hardware video encoder just meets the requirements of the video encoding system, and the number of "buffer pairs to be encoded" remains essentially unchanged.
5. The DVFS method of the hardware video encoder according to claim 4, wherein: In step S31, the greater the number of "buffer pairs to be encoded", the greater the workload of the hardware video encoder and the greater the encoding delay; the fewer the number of "buffer pairs to be encoded", the smaller the workload of the hardware video encoder and the smaller the encoding delay.
6. The DVFS method of the hardware video encoder according to claim 1, wherein: In step S32, firstly, it is determined whether the current encoding performance of the hardware video encoder is surplus, insufficient, or suitable; and secondly, a new operating frequency of the hardware video encoder is calculated.
7. The DVFS method of the hardware video encoder according to claim 6, wherein: In step S32, M represents a threshold for determining excess encoding performance of the hardware video encoder, and N represents a threshold for determining insufficient encoding performance of the hardware video encoder, where 0≤M<N; If the number of "buffer pairs to be encoded" in the current video encoding system is ≤ M, it is determined that the current encoding performance of the hardware video encoder is excessive; If the number of "buffer pairs to be encoded" in the current video encoding system is ≥ N, it is determined that the current encoding performance of the hardware video encoder is insufficient; If the number of “buffer pairs to be encoded” in the current video encoding system is greater than M and less than N, it is determined that the current encoding performance of the hardware video encoder is appropriate.
8. The DVFS method of the hardware video encoder according to claim 7, wherein: In step S32, the optional operating frequency values of the hardware video encoder are arranged in order from low to high, and a unique frequency number is set for each operating frequency value starting from 0 and incremented by 1, thereby obtaining a mapping table between the frequency number and the optional operating frequency value of the hardware video encoder; min_freq_idx is used to represent the lowest frequency number, max_freq_idx is used to represent the highest frequency number, cur_freq_idx is used to represent the frequency number corresponding to the current operating frequency value of the hardware video encoder, freq_up_step is used to represent the frequency increase step size, and freq_down_step is used to represent the frequency decrease step size; min_freq_idx≤cur_freq_idx≤max_freq_idx; When the current encoding performance of the hardware video encoder is excessive, the difference between cur_freq_idx and freq_down_step is compared with the value of min_freq_idx, and the larger one is used as the new frequency number of the hardware video encoder. When the current encoding performance of the hardware video encoder is insufficient, compare the sum of cur_freq_idx + freq_up_step with the value of max_freq_idx, and take the smaller one as the new frequency number of the hardware video encoder; When the current encoding performance of the hardware video encoder is suitable, the current frequency number cur_freq_idx of the hardware video encoder is directly used as the new frequency number of the hardware video encoder; After the new frequency number of the hardware video encoder is calculated, a mapping table between the frequency number and all optional operating frequency values of the hardware video encoder is searched to obtain the new operating frequency of the hardware video encoder.
9. The DVFS method of the hardware video encoder according to claim 8, wherein: In step S32, the value of cur_freq_idx is updated using the calculated new frequency number of the hardware video encoder.
10. A DVFS device for a hardware video encoder, characterized in that: It includes an encoder workload calculation module, an encoder frequency calculation module, and an encoder frequency adjustment module; The encoder workload calculation module is used to count the number of video frame buffers to be encoded and code stream buffers to be filled in the video encoding system. A "video frame buffer to be encoded" and a "code stream buffer to be filled" are referred to as a "buffer to be encoded pair". The number of "buffer pairs to be encoded" at a certain moment is used to represent the workload of the hardware video encoder at that moment. The encoder frequency calculation module is used to determine the relative relationship between the encoding performance and encoding requirements of the hardware video encoder at a given moment according to the real-time workload of the hardware video encoder, and calculate a new operating frequency of the hardware video encoder accordingly; The encoder frequency adjustment module is used to adjust the hardware video encoder to a new operating frequency.