Dynamic frequency and voltage adjustment method, device, equipment, storage medium and program product

Abstract

The application relates to a dynamic frequency and voltage adjustment method, device, equipment, storage medium and program product. The method comprises the following steps: determining the value of the utilization time length under the upper and lower limits of the utilization time length of a video decoder according to the current decoding information of the video decoder; the upper and lower limit conditions comprise a frame loss limit condition and a statistical error condition of the video decoder; and the dynamic frequency and voltage adjustment strategy parameters of the video decoder are determined according to the value of the utilization time length, and the dynamic frequency and voltage adjustment strategy parameters are used for dynamically adjusting the frequency and voltage of the video decoder. The method can more accurately dynamically adjust the frequency and voltage of the video decoder.

Description

Technical Field

[0001] This application relates to the field of chip technology, and in particular to a dynamic frequency and voltage modulation method, apparatus, device, storage medium, and program product. Background Technology

[0002] Dynamic Voltage and Frequency Scaling (DVFS) is a common power-saving technique used by processors and various computing units. DVFS can dynamically adjust the voltage and frequency of the processor or computing unit based on its current workload (e.g., device utilization).

[0003] In related technologies, the device utilization of a hardware video decoder (VPU) depends on the number of clock cycles reported when decoding each frame. By accumulating the number of clock cycles consumed during decoding over a period of time, and then dividing the accumulated value by the total number of clock cycles within the corresponding statistical period, the device utilization of the VPU during this period can be calculated.

[0004] However, for video playback scenarios, the duration of the statistical utilization rate should not be too short; it needs to cover several frame intervals to avoid excessive fluctuations and calculation errors in device utilization statistics. At the same time, the duration of the statistical utilization rate should not be too long to avoid excessive latency in triggering frequency upsampling. Currently, it is still unclear how to reasonably set the utilization rate statistical duration, which could lead to inaccurate dynamic frequency and voltage modulation of the video decoder. Summary of the Invention

[0005] Therefore, it is necessary to provide a dynamic frequency and voltage modulation method, apparatus, device, storage medium, and program product that can more accurately perform dynamic frequency and voltage modulation on video decoders, addressing the aforementioned technical problems.

[0006] In a first aspect, this application provides a dynamic frequency and voltage modulation method, including:

[0007] Under the upper and lower limits of the utilization statistics duration of the video decoder, the value of the utilization statistics duration is determined based on the current decoding information of the video decoder; the upper and lower limits include the frame loss restriction condition and the statistical error condition of the video decoder.

[0008] Based on the value of the utilization statistics duration, the dynamic frequency and voltage modulation strategy parameters of the video decoder are determined, and the dynamic frequency and voltage modulation strategy parameters are used to dynamically adjust the frequency and voltage of the video decoder.

[0009] In one embodiment, the frame dropping constraint includes a decoding timeout constraint and a buffer overflow constraint;

[0010] The step of determining the value of the utilization statistics duration based on the current decoding information of the video decoder, under the conditions of the upper and lower limits of the utilization statistics duration of the video decoder, includes:

[0011] Under the decoding timeout limit and the buffer overflow limit, the upper limit of the utilization statistics duration is determined based on the decoding information.

[0012] In one embodiment, the decoding information includes frame interval time, latency factor, load data, load rate, and decoding buffer depth;

[0013] The step of determining the upper limit of the utilization statistics duration based on the decoding information, under the conditions of the decoding timeout limit and the buffer overflow limit, includes:

[0014] Under the decoding timeout constraint, a first constraint parameter for the utilization statistics duration is determined based on the frame interval time, the latency factor, the load rate, and the load data.

[0015] Under the buffer overflow constraint, a second constraint parameter for the utilization statistics duration is determined based on the frame interval time, the decoding buffer depth, the load rate, and the load data.

[0016] The upper limit of the utilization rate statistics duration is determined based on the first limiting parameter and the second limiting parameter.

[0017] In one embodiment, determining the value of the utilization statistics duration based on the current decoding information of the video decoder, under the condition of the upper and lower limits of the utilization statistics duration of the video decoder, includes:

[0018] Under the statistical error conditions, the lower limit of the utilization rate statistical duration is determined based on the decoding information.

[0019] In one embodiment, the decoding information includes frame interval time, payload data, and the maximum value of the calculation error;

[0020] Under the statistical error condition, determining the lower limit of the utilization rate statistical duration based on the decoding information includes:

[0021] Under the statistical error conditions, a third limiting parameter for the utilization statistical duration is determined based on the frame interval time, the load data, and the maximum value of the calculation error.

[0022] The lower limit of the utilization rate statistics duration is determined based on the third limiting parameter.

[0023] In one embodiment, the method further includes:

[0024] Determine the current frame decoding duration of the video decoder;

[0025] Determine a first correlation between the decoding latency of the target frame and the utilization statistics duration, wherein the target frame is the last frame under the utilization statistics duration;

[0026] Under the condition that the decoding delay of the target frame is constrained to be less than the upper limit of the frame decoding duration, the decoding timeout restriction condition is determined according to the first association relationship.

[0027] In one embodiment, the decoding information includes the decoding buffer depth; the method further includes:

[0028] Determine a second correlation between the number of decoded frames of the video decoder within the utilization statistics period and the utilization statistics period, and a third correlation between the number of input frames of the video decoder within the utilization statistics period and the utilization statistics period;

[0029] With the difference between the number of decoded frames and the number of input frames constrained to a preset proportion less than the maximum value of the decoding buffer depth, the buffer overflow restriction condition is determined based on the second correlation, the third correlation, and the decoding buffer depth.

[0030] In one embodiment, the decoding information includes frame interval time and payload data; the method further includes:

[0031] Determine the maximum value of the computational error corresponding to the load of the video decoder;

[0032] The load calculation error ratio is determined based on the frame interval time, the utilization statistics duration, and the load data.

[0033] The statistical error condition is constrained to the point that the ratio of the calculated error of the load is less than the maximum value of the calculated error.

[0034] In one embodiment, the decoding information includes frame interval time and load data; the dynamic frequency and voltage modulation strategy parameters include an upper limit value and a lower limit value for utilization configuration; determining the dynamic frequency and voltage modulation strategy parameters of the video decoder based on the value of the utilization statistics duration includes:

[0035] Obtain the down-ratio and down-ratio safety margin of the video decoder;

[0036] Based on the value of the utilization statistics duration, the frame interval time, and the load data, the actual value of the calculation error of the load corresponding to the video decoder is determined;

[0037] The upper limit of the utilization configuration is determined based on the actual value of the calculation error and the actual upper limit of the utilization of the video decoder.

[0038] The lower limit of the utilization rate is determined based on the actual lower limit of the utilization rate, the down-frequency ratio, and the down-frequency safety margin.

[0039] In one embodiment, determining the dynamic frequency modulation and voltage regulation strategy parameters of the video decoder based on the value of the utilization statistics duration includes:

[0040] Obtain the resolution of each video data stream currently being decoded by the video decoder;

[0041] When the resolution of each video data stream is less than the preset resolution, the dynamic frequency and voltage modulation strategy parameters of the video decoder are determined based on the value of the utilization statistics duration.

[0042] In one embodiment, the method further includes:

[0043] When the resolution of the target video data is greater than or equal to the preset resolution, the video decoder is configured in the highest performance mode, and the target video data is any video data currently being decoded.

[0044] Secondly, this application also provides a dynamic frequency and voltage modulation device, comprising:

[0045] The statistical duration determination module is used to determine the value of the utilization statistical duration based on the current decoding information of the video decoder, under the condition of the upper and lower limits of the utilization statistical duration of the video decoder; the upper and lower limits include the frame loss restriction condition and the statistical error condition of the video decoder.

[0046] The parameter determination module is used to determine the dynamic frequency and voltage modulation strategy parameters of the video decoder based on the value of the utilization statistics duration. The dynamic frequency and voltage modulation strategy parameters are used to perform dynamic frequency and voltage modulation on the video decoder.

[0047] In one embodiment, the frame dropping constraint includes a decoding timeout constraint and a buffer overflow constraint;

[0048] The statistical duration determination module is specifically used to determine the upper limit of the utilization statistical duration based on the decoding information, under the decoding timeout limit and the buffer overflow limit.

[0049] In one embodiment, the decoding information includes frame interval time, latency factor, load data, load rate, and decoding buffer depth;

[0050] The statistical duration determination module is specifically used to determine a first limiting parameter for the utilization statistical duration based on the frame interval time, the latency factor, the load rate, and the load data under the decoding timeout limitation condition; to determine a second limiting parameter for the utilization statistical duration based on the frame interval time, the decoding buffer depth, the load rate, and the load data under the buffer overflow limitation condition; and to determine an upper limit value for the utilization statistical duration based on the first limiting parameter and the second limiting parameter.

[0051] In one embodiment, the statistical duration determination module is specifically used to determine the lower limit of the utilization statistical duration based on the decoding information under the statistical error condition.

[0052] In one embodiment, the decoding information includes frame interval time, payload data, and the maximum value of the calculation error;

[0053] The statistical duration determination module is specifically used to determine a third limiting parameter for the utilization statistical duration based on the frame interval time, the load data, and the maximum value of the calculation error under the statistical error conditions; and to determine a lower limit value for the utilization statistical duration based on the third limiting parameter.

[0054] In one embodiment, the statistical duration determination module is further configured to determine the current frame decoding duration of the video decoder; determine a first correlation between the decoding latency of the target frame and the utilization statistical duration, wherein the target frame is the last frame under the utilization statistical duration; and determine the decoding timeout restriction condition based on the first correlation condition, under the condition that the decoding latency of the target frame is constrained to be less than the upper limit of the frame decoding duration.

[0055] In one embodiment, the decoding information includes a decoding buffer depth; the statistical duration determination module is further configured to determine a second correlation between the number of decoded frames of the video decoder and the utilization statistical duration within the utilization statistical duration, and a third correlation between the number of input frames of the video decoder and the utilization statistical duration within the utilization statistical duration; and, under the condition that the difference between the number of decoded frames and the number of input frames is constrained to be less than a preset proportion of the maximum value of the decoding buffer depth, the buffer overflow restriction condition is determined based on the second correlation, the third correlation, and the decoding buffer depth.

[0056] In one embodiment, the decoding information includes frame interval time and load data; the statistical duration determination module is further configured to determine the maximum value of the calculation error corresponding to the load of the video decoder; determine the load calculation error ratio based on the frame interval time, the utilization statistical duration and the load data; and constrain the statistical error condition to ensure that the load calculation error ratio is less than the maximum value of the calculation error.

[0057] In one embodiment, the decoding information includes frame interval time and load data; the dynamic frequency and voltage adjustment strategy parameters include an upper limit and a lower limit for utilization configuration; the statistical duration determination module is further configured to obtain the down-frequency ratio and down-frequency safety margin of the video decoder; determine the actual value of the calculation error of the load corresponding to the video decoder based on the value of the utilization statistical duration, the frame interval time, and the load data; determine the upper limit for utilization configuration based on the actual value of the calculation error and the actual lower limit for utilization of the video decoder; and determine the lower limit for utilization configuration based on the actual lower limit for utilization, the down-frequency ratio, and the down-frequency safety margin.

[0058] In one embodiment, the parameter determination module is specifically used to obtain the resolution of each video data currently being decoded by the video decoder; when the resolution of each video data is less than the preset resolution, the dynamic frequency and voltage modulation strategy parameters of the video decoder are determined according to the value of the utilization statistics duration.

[0059] In one embodiment, the parameter determination module is further configured to use the highest performance mode to configure the video decoder when the resolution of the target video data is greater than or equal to the preset resolution, wherein the target video data is any video data currently being decoded.

[0060] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the dynamic frequency and voltage modulation method of the first aspect described above.

[0061] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the dynamic frequency and voltage modulation method of the first aspect described above.

[0062] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the dynamic frequency and voltage modulation method of the first aspect described above.

[0063] The aforementioned dynamic frequency and voltage modulation method, apparatus, equipment, storage medium, and program product, under the upper and lower limits of the video decoder utilization statistics duration, determine the value of the utilization statistics duration based on the current decoding information of the video decoder. The upper and lower limits include frame loss constraints and statistical error conditions for the video decoder. Based on the value of the utilization statistics duration, the dynamic frequency and voltage modulation strategy parameters of the video decoder are determined, and these parameters are used to dynamically modulate the frequency and voltage of the video decoder. Because the utilization statistics duration is determined based on the current decoding information of the video decoder under the frame loss constraints and statistical error conditions, it ensures that frame loss will not occur within the utilization statistics duration due to decoding timeouts or buffer overflows, and also reduces the impact of statistical errors on the utilization statistics duration. This makes the determined utilization statistics duration more reasonable, and the dynamic frequency and voltage modulation strategy parameters determined through the utilization statistics duration are also more reasonable, thereby improving the accuracy of dynamic frequency and voltage modulation. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 A flowchart illustrating a dynamic frequency and voltage modulation method provided in an embodiment of this application;

[0066] Figure 2 A flowchart illustrating another dynamic frequency and voltage modulation method provided in this application embodiment;

[0067] Figure 3 A flowchart illustrating another dynamic frequency and voltage modulation method provided in an embodiment of this application;

[0068] Figure 4 A structural block diagram of a dynamic frequency and voltage modulation device provided in this application embodiment;

[0069] Figure 5 This is an internal structural diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0071] The relevant technologies will be explained below.

[0072] Dynamic Voltage and Frequency Scaling (DVFS) is a common power-saving technique used in processors and various computing units. The principle of DVFS is to dynamically adjust the voltage and frequency of the processor or computing unit based on its workload (e.g., device utilization). When the workload is high, the voltage and frequency are increased to meet performance requirements; when the workload is low, the voltage and frequency are decreased to save power.

[0073] DVFS can be applied not only to Central Processing Units (CPUs) but also widely to various computing units or hardware accelerators, such as Graphics Processing Units (GPUs), Vision Processing Units (VPUs), and Neural Processing Units (NPUs). Because these processors or computing units consume a significant amount of power during computation, high computational energy efficiency ratios are often pursued for environmental protection and heat dissipation considerations; that is, the lower the energy consumption while performing the same amount of computation, the better. In the field of mobile computing, power supply generally relies on batteries; therefore, DVFS becomes particularly important in mobile computing or edge computing. DVFS plays a role in saving power consumption, reducing heat dissipation, and extending battery life.

[0074] It should be understood that DVFS includes both hardware and software implementations. The hardware implementation of DVFS involves modules such as a voltage regulator and a clock generator. These modules work together to control the voltage and clock frequency of the computing unit. The software modules include power management firmware and device drivers within the operating system. The power management firmware is responsible for directly or indirectly adjusting the device's voltage and frequency. The device driver, through a predefined DVFS power policy, periodically obtains the device's load status and, based on the obtained load status and the DVFS power policy, determines whether to trigger a frequency change operation.

[0075] It should be understood that the parameters of the device-driven DVFS power consumption strategy can be finely tuned to ensure that the device can accurately and quickly change the voltage frequency, thereby maximizing power savings and minimizing the impact on performance. If the parameters of the DVFS power consumption strategy are set flawedly, it will have negative consequences. For example, it may cause excessive power loss, leading to excessive battery heating and affecting battery life; or it may affect the processor's computing performance, thus impacting the user experience.

[0076] In related technologies, DVFS power management strategies include simple on-demand mode and performance mode. Simple on-demand mode provides the lowest voltage and frequency required by the current computing load, dynamically adjusting the voltage and frequency according to the load—using high frequency and high voltage for high loads and low frequency and low voltage for low loads. Performance mode, on the other hand, keeps the voltage and frequency at their highest levels to meet maximum performance requirements.

[0077] For example, when the DVFS power consumption strategy is configured in simple on-demand mode, if the device driver sets the boost threshold to 90%, the down threshold to 50%, and the polling utilization interval to 100ms, then DVFS will automatically trigger a device utilization read operation every 100 milliseconds (MS). If the read utilization is higher than 90%, it will increase to a higher voltage frequency; if the read utilization is lower than 50%, it will decrease to a lower voltage frequency.

[0078] It's important to note that when accurate real-time device utilization is obtained, the parameter settings for frequency and voltage regulation offer considerable flexibility. To minimize the impact on computing performance, the polling interval can be set shorter; to save more power, the upsampling threshold can be set larger. However, for VPUs, due to hardware limitations, obtaining accurate real-time utilization is not feasible. Overemphasizing the accuracy of utilization statistics will inevitably increase the latency of triggering frequency conversion, leading to frame drops. Therefore, a reasonable utilization statistics duration needs to be set, and accordingly, appropriate DVFS parameters should be configured to minimize power consumption while meeting performance requirements.

[0079] In related technologies, the device utilization of a hardware video decoder (VPU) depends on the number of clock cycles reported when decoding each frame. By accumulating the number of clock cycles consumed during decoding over a period of time, and then dividing the accumulated value by the total number of clock cycles within the corresponding statistical period, the device utilization of the VPU during this period can be calculated.

[0080] However, for video playback scenarios, the frequency of decoding and transmitting the bitstream data depends on the video's frame rate. If the video's frame rate is 30 frames per second (FPS), the clock cycles required to decode one frame are reported every 33.3ms. Therefore, the duration of the utilization statistics should not be too short; it needs to cover several frame intervals to avoid excessive fluctuations in the device utilization statistics.

[0081] Furthermore, an excessively short statistical utilization period can also affect the calculation error of device utilization. For example, decoding a frame of 1080P video takes 2-4ms, which accounts for 6-12% of the frame interval time; while decoding a frame of 4K video typically takes 8-16ms, which accounts for 24-48% of the frame interval time. If the clock count of even one frame is not included in the calculation, the resulting error in utilization calculation will be huge. The shorter the utilization statistical duration, the greater the statistical error caused by missing a frame. If the utilization calculation error is too large, the DVFS mechanism will fail to trigger frequency upsampling.

[0082] Furthermore, the duration for utilization statistics should not be too long. If the utilization statistics duration is set too long, it will lead to excessive latency in triggering frequency upsampling when the device is under high load. Playing videos may experience frame drops due to insufficient decoding performance, which will greatly affect the user experience.

[0083] Therefore, it is currently impossible to determine how to reasonably set the utilization rate statistics duration, which leads to inaccurate dynamic frequency and voltage modulation of the video decoder.

[0084] To address the aforementioned issues, this application provides a dynamic frequency and voltage modulation method, apparatus, device, storage medium, and program product. Under the constraints of frame loss and statistical error, the utilization statistical duration is determined based on the current decoding information of the video decoder. This ensures that frame loss does not occur within the utilization statistical duration due to decoding timeouts or buffer overflows, and also reduces the impact of statistical errors on the utilization statistical duration. Consequently, the determined utilization statistical duration is more reasonable, and the dynamic frequency and voltage modulation strategy parameters determined by the utilization statistical duration are also more reasonable, thereby improving the accuracy of dynamic frequency and voltage modulation.

[0085] The dynamic frequency and voltage modulation method provided in this application can be applied to a dynamic frequency and voltage modulation device, which can be a video decoder or any device containing a video decoder. This application does not limit this.

[0086] The aforementioned devices containing video decoders can include, but are not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can include virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses.

[0087] In one exemplary embodiment, such as Figure 1 As shown, a dynamic frequency and voltage modulation method is provided. Taking the application of this method to a video decoder as an example, the method includes the following steps S201 to S202. Wherein:

[0088] S201. Under the upper and lower limits of the utilization statistics duration of the video decoder, determine the value of the utilization statistics duration based on the current decoding information of the video decoder.

[0089] The following section will first explain the current decoding information of the video decoder.

[0090] It should be understood that the embodiments of this application do not limit when the current decoding information of the video decoder is obtained. In some embodiments, the VPU can obtain the current decoding information in real time, while in other embodiments, the VPU can obtain the current decoding information when a change in load is detected.

[0091] In some embodiments, the decoding information includes frame interval time, delay factor, load data, load rate, decoding buffer depth, and the maximum value of the calculation error.

[0092] The frame interval is used to handle multiple stages involved in the pipeline of the player when playing video. That is, the processing of multiple stages involved in the pipeline, such as decoding, post-processing, rendering, display, caching or synchronization margin, needs to be completed within one frame interval.

[0093] For example, taking a 30fps video as an example, the frame interval is 33.3ms. The decoding stage typically takes 8 to 16ms, usually accounting for 24% to 48% of the frame interval; the post-processing stage typically takes 3 to 8ms, usually accounting for 9% to 24% of the frame interval, and may include color space conversion, scaling, deblocking, or deblocking filtering; the rendering stage typically takes 2 to 5ms, usually accounting for 6% to 15% of the frame interval; the display stage typically takes 2ms, usually accounting for 6% of the frame interval; and the buffering or synchronization buffer stage typically takes 5.3 to 12.3ms, usually accounting for 16% to 37% of the frame interval.

[0094] The latency factor can be the maximum multiple by which the frame decoding time is allowed to be delayed relative to the frame interval time. For example, the latency factor can be set to 1.8.

[0095] The load data may include the number of video streams currently supported by the VPU and / or the number of video streams it will support. For example, the load data may include an increase in the number of video streams supported by the VPU from 2 to 3.

[0096] The load rate can be defined as the computational load of the VPU when processing the current video data, and it can be collected based on the operating status of the video decoder. The decoding buffer depth is used to store bitstream data that is not processed immediately, thereby avoiding frequent frame drops.

[0097] It should be understood that the embodiments of this application do not limit the value of the decoding buffer depth. For example, it can be set to 5. When the buffer approaches the overflow of the decoding buffer depth, the newly arrived frame bitstream data will be discarded.

[0098] The maximum value of the calculation error can be a preset maximum calculation error that the device can utilize. For example, the maximum value of the calculation error can be set to 10%.

[0099] The following explains how to determine the value of the utilization rate statistics duration.

[0100] It should be understood that in the embodiments of this application, the upper and lower limits of the utilization rate statistics duration can be determined first, and then the value of the utilization rate statistics duration can be determined based on the upper and lower limits of the utilization rate statistics duration.

[0101] In some embodiments, the frame dropping constraints for the upper limit of the utilization statistics duration include decoding timeout constraints and buffer overflow constraints. Accordingly, the VPU can determine the upper limit of the utilization statistics duration based on the decoding information under the decoding timeout constraints and buffer overflow constraints.

[0102] The decoding timeout constraint is a condition to avoid decoding timeouts.

[0103] In some embodiments, the decoding information used for the upper limit of the utilization statistics duration includes frame interval time, latency factor, load data, load rate, and decoding buffer depth. Accordingly, under decoding timeout constraints, the VPU can determine a first constraint parameter for the utilization statistics duration based on the frame interval time, latency factor, load rate, and load data. Under buffer overflow constraints, the VPU can determine a second constraint parameter for the utilization statistics duration based on the frame interval time, decoding buffer depth, load rate, and load data. Subsequently, the VPU can determine the upper limit of the utilization statistics duration based on the first and second constraint parameters.

[0104] It should be understood that the reasonable calculation time for VPU device utilization and the corresponding dynamic frequency and voltage adjustment strategy parameters are intended to achieve frequency increase under heavy load and frequency decrease under light load while minimizing frame drop. Two factors determine whether frame drop occurs: whether the decoding time of the current frame is too long and whether the current buffer is close to full. That is, when the buffer is close to overflow, newly arriving frame data will be discarded. For example, if the number of frames already buffered is greater than 4, approaching the maximum decoding buffer depth of 5, frame drop is triggered. Alternatively, if the time taken to decode a frame is too long, exceeding a preset multiple of the frame interval time, that frame will also be discarded. For example, if the time taken to decode a frame is greater than 1.8 times the frame interval time, frame drop is triggered.

[0105] In this embodiment of the application, the upper limit of the utilization statistics duration can be limited by the decoding timeout limit and the buffer overflow limit, thereby avoiding the VPU not being able to increase its frequency in time due to the excessively long statistics duration. This ensures that the current decoding performance of the VPU can meet the new decoding requirements and avoids frame dropping caused by increased frame decoding time or increased use of the buffer.

[0106] The decoding timeout limits for video decoders are explained below.

[0107] In some embodiments, under decoding timeout constraints, the VPU can determine a first constraint parameter for the utilization statistics duration based on frame interval time, latency factor, load rate, and load data.

[0108] For example, the VPU can input the frame interval time, latency factor, load rate and load data into the formula (1) corresponding to the decoding timeout limit condition to determine the first limit parameter of the utilization statistics duration.

[0109] Bound0=Ti*d*Lf / (Lf-N / (N+1)) (1)

[0110] Wherein, Bound0 is the first limiting parameter, i.e. the decoding timeout limiting parameter; Ti is the frame interval time; d is the latency factor; Lf is the load rate; N is the number of currently supported video channels, and N+1 is the number of video channels to be increased to. Both N and N+1 are the load data mentioned above.

[0111] It should be understood that the embodiments of this application do not limit how the decoding timeout constraint is determined. In some embodiments, the VPU can determine the current frame decoding duration of the video decoder. Subsequently, the VPU determines a first correlation between the decoding latency of the target frame and the utilization statistics duration. Finally, under the condition that the decoding latency of the target frame is constrained to be less than the upper limit of the frame decoding duration, the VPU determines the decoding timeout constraint based on the first correlation.

[0112] The target frame is the last frame within the utilization statistics duration. The decoding latency of the target frame can be the end-to-end latency, which is the time from when the target frame enters the queue of the utilization statistics duration T to when decoding is completed.

[0113] This application does not limit the frame decoding time in its embodiments. In some embodiments, it can be determined by load rate, load data and the number of frames of video data.

[0114] For example, taking VPU decoding 4K resolution 30fps video data as an example, assuming the VPU utilization statistics duration is T, and the frame interval time is Ti = 1000ms / 30 = 33.3ms, the VPU is close to full load at its current operating frequency, and it can support N video decoding. When the VPU increases its support from N video decoding to supporting N+1 video decoding, a frequency increase is expected after time T. Assuming N is 2, the load rate Lf of the VPU decoding 2 channels of 4K resolution 30fps video data is 0.9, and the current frame decoding time of the VPU is 33.3ms / 2*0.9 = 15ms. The upper limit of the frame decoding time is Ti*d = 33.3ms*1.8 = 60ms, where d is 1.8.

[0115] It should be understood that the larger the number of frames within the utilization statistics period, the longer the waiting time and the greater the decoding latency. After the utilization statistics period is delayed, frequency upsampling is triggered, reducing the decoding time of previously waiting frames and decreasing the decoding latency. Therefore, the decoding latency of the last frame within the utilization statistics period is the largest (i.e., the target frame). Accordingly, the decoding timeout constraint can be determined by constraining the decoding latency of the last frame within the utilization statistics period to be less than the upper limit of the frame decoding time.

[0116] For example, taking VPU decoding of 4K resolution 30fps video data as an example, we can set M as the number of frames decoded for each video in the time period T, M = rounded up [T / 15 / (2+1)] (one of the N+1 video channels can trigger upsampling). The expected value of M is T / 15 / (2+1)+0.5. The Mth frame is the last frame in the time period T, and its decoding delay is the largest. Since the start time of the utilization calculation may be any time before the start of a frame, we take an expected value of 0.5*Ti. At this time, the first correlation between the decoding delay of the Mth frame (the time interval between the decoding start time of the 1st frame and the decoding completion time of the Mth frame in the time period T, i.e., the decoding delay of the target frame mentioned above) and the utilization statistics duration can be shown in formula (2). By constraining the decoding delay of the Mth frame in formula (2) to be less than the upper limit of the frame decoding duration of 60ms, we can obtain formula (3), and calculate T<231ms.

[0117] Delay=T-(M+0.5-1)*33.3ms (2)

[0118] T-((T / 15 / 3)+0.5+0.5-1)*33.3ms<60ms (3)

[0119] Where T is the utilization statistics duration, and M is the last frame within the T time period.

[0120] For example, taking VPU decoding 4K resolution 60fps video data as an example, its frame interval time is Ti=1000ms / 60=16.7ms. If, at the current frequency, when decoding N video streams at near full load, the decoding time for one frame is still 15ms when increasing to N+1 video streams, but N=1. (Decoding one 4K resolution 60fps video stream is equivalent to decoding two 4K resolution 30fps video streams), the upper limit of the decoding latency is 30ms. Correspondingly, the Mth frame in the T time period is T / 15 / (1+1)+0.5. By constraining the decoding latency of the Mth frame to be less than the upper limit of the frame decoding time of 30ms, we can obtain formula (4), and calculate that T<67.5ms.

[0121] T-((T / 15 / 2)+0.5+0.5-1)* 16.7ms <30ms (4)

[0122] Therefore, for 4K video data, due to the long decoding time per frame, the maximum decoding latency becomes the limiting factor in determining whether frame dropping is triggered. For 4K resolution 30fps video data, the upper limit of the utilization statistics duration is 231ms; for 4K resolution 60fps video data, the upper limit of the utilization statistics duration is 67.5ms. When the VPU decodes 4K resolution 60fps video data, the load changes more significantly with each additional channel decoded, requiring faster upsampling response and stricter timeout requirements, resulting in a lower upper limit for the corresponding utilization statistics duration.

[0123] For example, based on the above formulas (3) and (4), formula (5) can be obtained to characterize the constraint that the maximum decoding latency is less than the upper limit of the frame decoding time. By simplifying formula (5), formula (6) is obtained to characterize the decoding latency requirement. Subsequently, formula (1) corresponding to the decoding timeout constraint is derived from formula (6).

[0124] T-(T*N / (Ti*Lf*(N+1))*Ti <Ti*d (5)

[0125] T <Ti*d*Lf / (Lf-N / (N+1)) (6)

[0126] Where T is the utilization statistics duration; Ti is the frame interval time; N is the number of currently supported video channels, and N+1 is the number of video channels to be added to. Both N and N+1 are the load data mentioned above; Lf is the load rate, which is 0.9 when close to full load; d is the latency impact factor, which is 1.8; Lf-N / (N+1)>0, Lf>N / (N+1), that is, N cannot exceed the load factor limit to avoid exceeding the hardware performance limit.

[0127] The following explains the buffer overflow restrictions for video decoders.

[0128] In some embodiments, under buffer overflow constraints, the VPU determines a second constraint parameter for the utilization statistics duration based on the frame interval time, decoding buffer depth, load rate, and load data.

[0129] For example, the VPU can input the frame interval time, decoding buffer depth, load rate and load data into the formula (7) corresponding to the buffer overflow limit condition to determine the second limit parameter of the utilization statistics duration.

[0130] Bound1=0.8*B*Ti*Lf / (Lf-N / (N+1)) (7)

[0131] Wherein, Bound1 is the second limiting parameter, namely the buffer overflow limiting parameter; Ti is the frame interval time; B is the decoding buffer depth; Lf is the load rate; N is the number of currently supported video channels, and N+1 is the number of supported video channels to be increased. Both N and N+1 are the load data mentioned above.

[0132] It should be understood that the embodiments of this application do not limit how the buffer overflow constraint is determined. In some embodiments, the decoding information involved in the buffer overflow constraint includes the decoding buffer depth. The VPU first determines a second correlation between the number of decoded frames of the video decoder and the utilization statistics duration within the utilization statistics duration, and a third correlation between the number of input frames of the video decoder and the utilization statistics duration within the utilization statistics duration. Subsequently, the VPU determines the buffer overflow constraint based on the second correlation, the third correlation, and the decoding buffer depth, while constraining the difference between the number of decoded frames and the number of input frames to a preset proportion less than the maximum value of the decoding buffer depth.

[0133] For example, the maximum value of the decoding buffer depth can be set to 5, and the preset ratio can be set to 80%.

[0134] It should be understood that if the statistical duration of utilization is T, the decoding buffer depth will also increase within time T as the load increases. After time T, a frequency upsampling is triggered, which speeds up the decoding of a frame, and the decoding buffer depth will decrease accordingly. Therefore, at time T, the number of undecoded frames is the maximum depth of the decoding buffer, from which the buffer overflow constraint to avoid frame dropping can be derived.

[0135] For example, taking VPU decoding 4K resolution 30fps video data as an example, the utilization statistics duration is T. The load rate for decoding two channels of 4K resolution 30fps video data is 0.9. Then, at the current frequency, the time to decode one frame is 1000 / 30 / 2*0.9=15ms. Then M = rounded down [T / 15 / (2+1)], where M is the number of decoded frames for each channel within the time period T, taking the channel with the fewest decoded frames as the standard. After taking the desired value, we obtain the second correlation between the number of decoded frames and the utilization statistics duration: M=T / 15 / (2+1)-0.5.

[0136] For example, K is the number of input frames of one video data stream within time T. K = rounded down [T / 33.3]. Taking the desired value of it, we can obtain the third correlation between the number of input frames and the utilization statistics duration: K = T / 33.3 - 0.5.

[0137] For example, since the difference between the number of decoded frames and the number of input frames is constrained to a preset ratio that is less than the maximum value of the decoding buffer depth, formula (8) can be obtained through the second correlation, the third correlation, and the decoding buffer depth, and T < 514ms is calculated. Similarly, taking the VPU decoding of 4K resolution 30fps video data as an example, N is 1, Ti is 16.7ms, then formula (9) is obtained, and T < 514ms is calculated.

[0138] T / 33.3-T / 15 / 3 <5*0.8 (8)

[0139] T / 16.7-T / 15 / 2 < 5*0.8 (9)

[0140] For example, based on the above formulas (8) and (9), formula (10) can be obtained to characterize the preset ratio that constrains the difference between the number of decoded frames and the number of input frames to be less than the maximum value of the decoding buffer depth. By simplifying formula (10), formula (11) is obtained to characterize the buffer upper limit requirement. Subsequently, formula (7) corresponding to the buffer overflow restriction condition is derived from formula (11).

[0141] T / Ti-T*N / (Ti*Lf*(N+1)) <B*0.8 (10)

[0142] T<0.8*B*Ti*Lf / (Lf-N / (N+1)) (11)

[0143] Where T is the utilization statistics duration; Ti is the frame interval time; N is the number of currently supported video channels, and N+1 is the number of video channels to be added to. N and N+1 are the load data mentioned above; Lf is the load rate, which is 0.9 when close to full load; B is the decoding buffer depth, which is 5 at the maximum value; Lf-N / (N+1)>0, Lf>N / (N+1), that is, N cannot exceed the load factor limit to avoid exceeding the hardware performance limit.

[0144] In some embodiments, the VPU can also determine the lower limit of the utilization statistics duration based on the decoding information, under statistical error conditions.

[0145] In some embodiments, the decoding information used for the lower limit of the utilization statistics duration includes the frame interval time, the maximum value of the load data and the calculation error. Accordingly, under the statistical error condition, the VPU can determine a third limiting parameter for the utilization statistics duration based on the frame interval time, the maximum value of the load data and the calculation error. Subsequently, the VPU can determine the lower limit of the utilization statistics duration based on the third limiting parameter.

[0146] The statistical error conditions for the video decoder are explained below.

[0147] In some embodiments, under statistical error conditions, the VPU can determine a third limiting parameter for the utilization statistical duration based on the frame interval time, load data, and the maximum value of the calculation error.

[0148] For example, the VPU can input the frame interval time, load data and the maximum value of the calculation error into the formula (12) corresponding to the statistical error condition to determine the third limiting parameter of the utilization statistical duration.

[0149] Bound2 = Ti / (e max *N) (12)

[0150] Where Bound2 is the third limiting parameter, i.e., the statistical error limiting parameter; Ti is the frame interval time; (e max The maximum value for the calculation error can be set to 10%; N is the number of currently supported video channels, and N and the above load data.

[0151] It should be understood that since video stream data is sent at frame intervals, the calculation time for device utilization must span at least one frame interval cycle. Furthermore, because the statistics for the number of decoding cycles depend on the reporting of statistical information after each frame is decoded, it is highly likely that the VPU will still be in decoding mode and has not yet issued an interrupt when the utilization statistics period ends, resulting in the omission of statistical information for one frame. The longer the decoding time for a frame, the greater the probability of losing the statistical information for the last frame, and the greater the error in calculating device utilization. To minimize the error in calculating device utilization, the utilization statistics period should be as long as possible, or in other words, as much frame decoding information as possible should be collected within a single utilization statistics period. Based on this, statistical error conditions can be generated.

[0152] In some embodiments, the decoding information involved in the statistical error constraint includes frame interval time and load data. The VPU can first determine the maximum value of the computational error corresponding to the load of the video decoder. Subsequently, the VPU determines the computational error ratio of the load based on the frame interval time, utilization statistics duration, and load data. Finally, the VPU constrains the statistical error condition to ensure that the computational error ratio of the load is less than the maximum value of the computational error.

[0153] For example, taking 4K resolution 30fps video data as an example, assuming that the current load satisfies N-channel video decoding, N=2, its load rate is 90%, the total number of frames decoded in time T is T / Ti*N, and the number of frames counted is T / Ti*N-1. Then, the time spent on each frame is Ti*Lf / N, the calculated device utilization rate is (T / Ti*N-1)*Ti*Lf / N / T, and the calculation error ratio between the calculated device utilization rate and the actual load rate Lf is (T*N / Ti-1)*Ti / (T*N)=1-Ti / (T*N). If the maximum value of the obtained calculation error is 10%, then by constraining the calculation error ratio to be less than the maximum value of the calculation error, we get the result as shown in formula (13), and calculate T>167ms. Subsequently, formula (12) corresponding to the buffer overflow constraint condition is derived from formula (13).

[0154] T>Ti / ( e max *N) (13)

[0155] Where T is the utilization statistics duration; Ti is the frame interval time; e max The maximum value for the calculation error can be set to 10%; N is the number of video channels currently supported, and N is the load data mentioned above.

[0156] In some embodiments, after determining the first limiting parameter, the second limiting parameter, and the third limiting parameter, the VPU can determine the upper limit of the utilization statistics duration based on the first limiting parameter and the second limiting parameter, and determine the lower limit of the utilization statistics duration based on the third limiting parameter.

[0157] For example, if the first limiting parameter corresponding to the decoding timeout limit is Bound0 and the second limiting parameter corresponding to the buffer overflow condition is Bound1, then min{Bound0, Bound1} can be used as the upper limit of the range of utilization statistics duration, while the third limiting parameter Bound2 corresponding to the statistical error condition can be directly used as the lower limit of the range of utilization statistics duration, so the range of utilization statistics duration is [Bound2, min{Bound0, Bound1}].

[0158] In some embodiments, it can be seen from the formula (1) for Bound0 and the formula (7) for Bound1 that when d < 0.8*B, Bound0 can be used as the upper limit of the range of values ​​for the utilization rate statistics duration; when d > 0.8*B, Bound1 can be used as the upper limit of the range of values ​​for the utilization rate statistics duration; and when d = 0.8*B, Bound0 and Bound1 are equal and both can be used as the upper limit of the range of values ​​for the utilization rate statistics duration.

[0159] For example, taking 4K resolution 30fps video data as an example, if the current VPU is close to full load, it can decode 2 video data channels, and the load rate is 0.9, the latency factor is 1.8, and the maximum calculation error is 5. The VPU can calculate Bound0 as 231ms using formula (1), Bound1 as 514ms using formula (7), and Bound2 as 167ms using formula (12). The range of utilization statistics duration can be [167, 231]ms. Subsequently, the VPN can select any value from the range of utilization statistics duration [167, 231]ms as the utilization statistics duration value.

[0160] In this embodiment of the application, the range of utilization rate statistics duration is determined by different limiting parameters, and then the value of utilization rate statistics duration is determined from the range of utilization rate statistics duration, so that the value is more reasonable.

[0161] S202. Based on the value of the utilization rate statistics duration, determine the dynamic frequency and voltage modulation strategy parameters of the video decoder.

[0162] Among them, the dynamic frequency and voltage modulation strategy parameters are used to dynamically modulate the frequency and voltage of the video decoder.

[0163] In this step, once the VPU determines the value of the utilization statistics duration, the dynamic frequency modulation and voltage regulation strategy parameters of the video decoder can be determined based on the value of the utilization statistics duration.

[0164] In some embodiments, the decoding information includes frame interval time and load data, and the dynamic frequency and voltage adjustment strategy parameters include an upper limit and a lower limit for utilization configuration. Accordingly, the VPU can obtain the down-rate and down-rate safety margin of the video decoder. The VPU can determine the actual value of the calculation error of the load corresponding to the video decoder based on the utilization statistics duration, frame interval time, and load data. Subsequently, the VPU can determine the upper limit for utilization configuration based on the actual value of the calculation error and the actual upper limit for video decoder utilization, and determine the lower limit for utilization configuration based on the actual upper limit, the down-rate, and the down-rate safety margin.

[0165] The utilization configuration upper limit is the maximum utilization rate of the device that triggers frequency upsampling. The downsampling ratio can be the ratio of the next frequency that the video decoder is expected to reduce to to the current frequency. The downsampling safety margin is an additional safety margin reserved in the utilization configuration to prevent insufficient decoding performance of the video decoder due to downsampling.

[0166] In this embodiment, the conditions for frequency increase are set with a certain margin. Therefore, under ideal circumstances where equipment utilization statistics are accurate, the upper limit of utilization configuration is usually set to 90-95%. However, in actual equipment utilization calculations, calculation errors are introduced. Therefore, the actual value of the calculation error can be derived, and by combining the actual value of the calculation error with the actual upper limit of utilization, the upper limit of utilization configuration can be determined.

[0167] For example, due to the larger calculation error of 4K resolution video data, let's take 4K resolution video data as a benchmark and assume the utilization statistics duration is T. Then the actual value of the calculation error is e = Ti / (T * N). For 4K30 video, Ti = 1000ms / 30, N = 2. Assuming T = 200ms (the same applies to 60FPS and 30FPS), the actual value of the calculation error can be determined by the value of the utilization statistics duration, frame interval time, and load data. For example, the actual value of the calculation error is e = 1000ms / (30 * 200ms * 2) = 0.08. Subsequently, the utilization configuration upper limit is obtained by multiplying the actual utilization limit by (1 - e). If the actual utilization limit is 90%, then the utilization configuration upper limit is calculated to be 83%.

[0168] It should be noted that, in order to adapt to 4K resolution video data, the upper limit of the utilization configuration has been lowered. From another perspective, this affects the upsampling behavior of 1080P and lower resolution video data during the decoding process. For 1080P and lower resolution video data, the calculation error of its utilization is relatively smaller, which will trigger upsampling prematurely, resulting in additional power consumption.

[0169] Among them, the utilization rate configuration lower limit is the lower limit of the device utilization rate that triggers frequency reduction.

[0170] It should be understood that the lower limit of equipment utilization depends on the frequency levels supported by the VPU, i.e., the ratio between adjacent frequencies. If the VPU expects to reduce the next frequency level to 50% of the current frequency, then the lower limit of utilization configuration corresponding to this frequency reduction will also be 50% of the upper limit of utilization configuration. Furthermore, the lower limit of utilization configuration also needs to consider the error value of utilization statistics. To avoid the ping-pong effect caused by repeated frequency increases and decreases, the lower limit of utilization configuration needs to be further reduced by a frequency reduction safety margin (delta), which can be set to 0.03-0.05.

[0171] For example, if the down-frequency ratio between adjacent low and high frequencies is f, the actual value of the calculation error is e, the upper limit of the utilization configuration is Lf_up, and the down-frequency safety margin is delta, then the lower limit of the utilization configuration Lf_lo can be configured as Lf_lo=Lf_up*f*(1-e)-delta.

[0172] With Lf_up set to 90%, f set to 50%, e set to 0.08, and delta set to 0.03, Lf_lo can be calculated to be 38%.

[0173] It should be noted that for video data with resolutions of 1080P and lower, the calculation error for utilization will be smaller, so the expected lower limit for frequency reduction should be slightly higher than the calculated value for 4K video. This will delay the timing of frequency reduction, resulting in additional power consumption.

[0174] In this embodiment, the upper limit of the utilization configuration is calculated by calculating the actual value of the error, and then the lower limit of the utilization configuration is calculated, which makes the determined dynamic frequency and voltage regulation strategy parameters more accurate.

[0175] In other embodiments, the dynamic frequency and voltage modulation strategy parameters also include device status acquisition time. The VPU can determine the device status acquisition time based on the frame interval. Subsequently, the VPU can poll the device utilization rate of the video decoder based on the device status acquisition time.

[0176] The device status acquisition time mentioned above can be the time interval (pollinterval) for acquiring device status (utilization).

[0177] For example, the device status acquisition time characterizes how often the VPU polls the device status under DVFS driving to obtain its utilization. Since video playback inputs frame stream data according to frame intervals, it is necessary to ensure that device utilization is updated within a frame interval and to trigger frequency upsampling or downsampling. Accordingly, the device status acquisition time can be determined based on the frame interval. After determining the device status acquisition time, the VPU polls the video decoder's device utilization according to the device status acquisition time.

[0178] For example, if the frame interval of 60fps video data is 16.7ms, then an integer less than the frame interval and close to the frame interval can be determined as the device status acquisition time. For example, the device status acquisition time can be determined as 15ms.

[0179] In some embodiments, the VPU can obtain the resolution of each video data stream currently being decoded by the video decoder. If the resolution of each video data stream is less than a preset resolution, the dynamic frequency and voltage modulation strategy parameters of the video decoder are determined based on the value of the utilization statistics duration.

[0180] That is, when the resolution of each video data stream is less than the preset resolution, it can be determined that the video is in dynamic frequency and voltage modulation mode. At this time, the VPU can use the value of the statistical duration to determine the dynamic frequency and voltage modulation strategy parameters of the video decoder, and then use the dynamic frequency and voltage modulation strategy parameters to perform dynamic frequency and voltage modulation on the video decoder.

[0181] For example, in dynamic frequency and voltage adjustment mode, the VPU can poll the video decoder's device utilization based on the device status acquisition time. If the acquired device utilization is greater than the configured upper limit, the VPU's frequency and voltage are increased. If the acquired device utilization is less than the configured lower limit, the VPU's frequency and voltage are decreased.

[0182] The VPU's processing modes can include dynamic frequency and voltage adjustment mode and maximum performance mode. The VPU can use different modes to process video data of different resolutions. In dynamic frequency and voltage adjustment mode, the VPU dynamically adjusts the voltage and frequency according to the load, while in maximum performance mode, the VPU keeps the voltage and frequency at their highest levels to meet maximum performance requirements.

[0183] In other embodiments, when the resolution of the target path video data is greater than or equal to a preset resolution, the VPU can configure the video decoder in the highest performance mode.

[0184] The target video data refers to any one of the currently decoded video streams. The preset resolution can be set according to the actual situation, for example, to 8K resolution.

[0185] For example, switching between dynamic frequency modulation and voltage regulation mode and the highest performance mode can be achieved through processing mode indication information. When the resolution of the target video data is greater than or equal to a preset resolution, the VPU can update the processing mode indication information of the video decoder to the first type of indication information. At this time, the VPU configures the video decoder using dynamic frequency modulation and voltage regulation mode. Subsequently, in response to the first type of indication information, the VPU configures the video decoder using the highest performance mode.

[0186] For example, the VPU can obtain the resolution of each video data stream currently being decoded and determine whether it is 8K resolution video data or video data with a resolution greater than 8K. If it is 8K resolution video data or video data with a resolution greater than 8K, the processing mode indication information can be set to the first type indication information (perf mode).

[0187] It should be noted that, since the VPU may decode multiple video streams simultaneously, the highest performance mode can be set to atomic integer. If 8K resolution video data is identified, increment the perf mode by one. After decoding is complete, decrement the perf mode by one.

[0188] For example, when setting the VPU frequency, if the first type of indication information exists, the VPU driver will set the VPU voltage and frequency to the maximum. If the first type of indication information does not exist, the VPU driver will adjust the VPU according to the target voltage and frequency determined by dynamic frequency hopping voltage regulation.

[0189] In this embodiment of the application, when the resolution of video data is greater than or equal to a preset resolution, the highest performance mode of the video decoder is configured by using the first type of indication information, so that the upsampling response processing of high-resolution video data can be freed from dependence on device utilization.

[0190] In this embodiment, the utilization statistics duration is determined by upper and lower limits. This not only ensures accurate feedback of device utilization without frame loss but also keeps the calculation error within a certain range. Based on this, this embodiment achieves real-time and accurate adjustment of the VPU's voltage frequency, thereby dynamically adjusting the voltage frequency according to the VPU load and reducing overall power consumption. Furthermore, by providing a basis for calculating VPU device utilization and configuring dynamic frequency and voltage adjustment strategy parameters, the reliability of parameters is improved, significant parameter tuning time is saved, and timely response to dynamic frequency adjustments, especially frequency upscaling operations, is ensured, minimizing the impact on performance and reducing overall VPU power consumption.

[0191] The dynamic frequency and voltage modulation method provided in this application determines the value of the utilization statistics duration based on the current decoding information of the video decoder, under the upper and lower limits of the utilization statistics duration. The upper and lower limits include frame loss constraints and statistical error conditions of the video decoder. Based on the value of the utilization statistics duration, dynamic frequency and voltage modulation strategy parameters of the video decoder are determined, and these parameters are used to dynamically adjust the frequency and voltage of the video decoder. Because the utilization statistics duration is determined based on the current decoding information of the video decoder under the frame loss constraints and statistical error conditions, it ensures that no frame loss occurs within the utilization statistics duration due to decoding timeouts or buffer overflows. It also reduces the impact of statistical errors on the utilization statistics duration, making the determined utilization statistics duration more reasonable. The dynamic frequency and voltage modulation strategy parameters determined through the utilization statistics duration are also more reasonable, thereby improving the accuracy of dynamic frequency and voltage modulation.

[0192] It should be noted that the dynamic frequency and voltage regulation performed in the embodiments of the present application takes video data with 4K resolution as an example. However, the dynamic frequency and voltage regulation method derived from 4K resolution video data is also applicable to video data with resolutions lower than 4K.

[0193] For example, for a 1080P video, it takes the VPU about 2-4 ms to decode one frame. If the video has a frame rate of 30FPS, the VPU can decode 8-16 channels of video simultaneously (this document only discusses the case of a single decoding core). Adding one more channel of 1080P video to the VPU does not cause a large extra load, so it does not bring a great impact on the utilization statistics. For a 4K video, it takes the VPU about 8-16 ms to decode one frame, which means the VPU can decode 2-4 channels of 4K 30FPS video simultaneously. Therefore, adding one more channel of 4K video will bring large fluctuations to the load of the VPU. This imposes more stringent requirements on the calculation of utilization and the accuracy of DVFS management strategy parameters. Therefore, once an ideal utilization statistics method and DVFS management strategy parameters are obtained for 4K resolution video data, they will definitely also be applicable to 1080P and lower resolution videos.

[0194] For example, different from 4K resolution video data, when the VPU processes 1080P resolution video data, the number of video channels N is different under the same load. Generally, under the same computing power of the VPU, the number of 1080P resolution video channels that can be decoded is four times that of 4K resolution video channels.

[0195] For example, if the load rate is 0.9, the delay factor is 1.8, and the maximum decoding buffer depth is 5, then for 1080P resolution 30fps video data, Ti=1000ms / 30=33.3ms, and the number of video channels N is 8. Under the decoding timeout limitation condition: T<Ti*d*Lf / (Lf-N / (N+1)), it is calculated that the utilization statistics duration T is less than 4855ms. Under the buffer overflow limitation condition: T<0.8*B*Ti*Lf / (Lf-N / (N+1)), it is calculated that the utilization statistics duration T is less than 10898ms. Under the statistical error condition: T>1000ms / (30*8*e), it is calculated that the utilization statistics duration T is greater than 42ms.

[0196] For example, if the load factor is 0.9, the latency factor is 1.8, and the maximum decoding buffer depth is 5, then the time taken for 1080P resolution 60fps video data is Ti = 1000ms / 60 = 16.7ms, and the number of video data streams N is 4. Under the decoding timeout constraint, the calculated utilization statistics duration T is less than 4855ms. Under the buffer overflow constraint, the calculated utilization statistics duration T is less than 601ms. Under the statistical error constraint, the calculated utilization statistics duration T is greater than 42ms.

[0197] The following explains how to generate the upper and lower limits of the video decoder utilization statistics duration. Figure 2 This is a flowchart illustrating another dynamic frequency and voltage modulation method provided in an embodiment of this application. This dynamic frequency and voltage modulation method is applied to a video decoder, such as... Figure 2 The dynamic frequency and voltage regulation method includes S301 to S310:

[0198] S301. Determine the current frame decoding duration of the video decoder.

[0199] S302. Determine the first correlation between the decoding latency of the target frame and the utilization statistics duration, where the target frame is the last frame under the utilization statistics duration.

[0200] S303. Under the condition that the decoding delay of the target frame is constrained to be less than the upper limit of the frame decoding time, the decoding timeout limit condition is determined according to the first association relationship.

[0201] S304. Determine the second correlation between the number of decoded frames and the utilization statistics duration of the video decoder, and the third correlation between the number of input frames and the utilization statistics duration of the video decoder.

[0202] S305. Under the condition that the difference between the number of decoded frames and the number of input frames is constrained to a preset ratio that is less than the maximum value of the decoding buffer depth, the buffer overflow restriction condition is determined according to the second association relationship, the third association relationship and the decoding buffer depth.

[0203] S306. Determine the maximum value of the calculation error corresponding to the load of the video decoder.

[0204] S307. Determine the load calculation error ratio based on the frame interval time, utilization statistics duration, and load data.

[0205] S308. The statistical error condition is constrained to the ratio of the calculated error of the load to the maximum value of the calculated error.

[0206] S309. Under the upper and lower limits of the utilization statistics duration of the video decoder, determine the value of the utilization statistics duration based on the current decoding information of the video decoder.

[0207] The upper and lower limits include the video decoder's decoding timeout limit, buffer overflow limit, and statistical error limit.

[0208] S310. Based on the value of the utilization statistics duration, determine the dynamic frequency and voltage modulation strategy parameters of the video decoder. The dynamic frequency and voltage modulation strategy parameters are used to dynamically adjust the frequency and voltage of the video decoder.

[0209] The following explains how to determine the value of the utilization rate statistics duration. Figure 3 This is a flowchart illustrating another dynamic frequency modulation and voltage modulation method provided in an embodiment of this application. This dynamic frequency modulation and voltage modulation method is applied to a video decoder, such as... Figure 3 The dynamic frequency and voltage regulation method includes S401 to S406:

[0210] S401. Under the decoding timeout constraint, determine the first constraint parameter of the utilization statistics duration based on the frame interval time, delay factor, load rate and load data.

[0211] S402. Under the buffer overflow constraint, determine the second constraint parameter of the utilization statistics duration based on the frame interval time, decoding buffer depth, load rate and load data.

[0212] S403. Determine the upper limit of the utilization rate statistics duration based on the first and second limiting parameters.

[0213] S404. Under statistical error conditions, determine the third limiting parameter for utilization statistical duration based on the frame interval time, load data, and the maximum value of the calculation error.

[0214] S405. Determine the lower limit value of the utilization rate statistical duration based on the third limiting parameter.

[0215] S406. Based on the value of the utilization statistics duration, determine the dynamic frequency and voltage modulation strategy parameters of the video decoder. The dynamic frequency and voltage modulation strategy parameters are used to dynamically adjust the frequency and voltage of the video decoder.

[0216] The dynamic frequency and voltage modulation method provided in this application determines the value of the utilization statistics duration based on the current decoding information of the video decoder, under the upper and lower limits of the utilization statistics duration. The upper and lower limits include frame loss constraints and statistical error conditions of the video decoder. Based on the value of the utilization statistics duration, dynamic frequency and voltage modulation strategy parameters of the video decoder are determined, and these parameters are used to dynamically adjust the frequency and voltage of the video decoder. Because the utilization statistics duration is determined based on the current decoding information of the video decoder under the frame loss constraints and statistical error conditions, it ensures that no frame loss occurs within the utilization statistics duration due to decoding timeouts or buffer overflows. It also reduces the impact of statistical errors on the utilization statistics duration, making the determined utilization statistics duration more reasonable. The dynamic frequency and voltage modulation strategy parameters determined through the utilization statistics duration are also more reasonable, thereby improving the accuracy of dynamic frequency and voltage modulation.

[0217] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0218] Based on the same inventive concept, this application also provides a dynamic frequency and voltage regulation device for implementing the dynamic frequency and voltage regulation method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more embodiments of the dynamic frequency and voltage regulation device provided below can be found in the limitations of the dynamic frequency and voltage regulation method described above, and will not be repeated here.

[0219] In one exemplary embodiment, such as Figure 4 As shown, a dynamic frequency and voltage regulation device 500 is provided, including: a statistical duration determination module 501 and a parameter determination module 502, wherein:

[0220] The statistical duration determination module 501 is used to determine the value of the utilization statistical duration based on the current decoding information of the video decoder, under the upper and lower limits of the utilization statistical duration of the video decoder; the upper and lower limits include the frame loss limit condition and the statistical error condition of the video decoder.

[0221] The parameter determination module 502 is used to determine the dynamic frequency and voltage modulation strategy parameters of the video decoder based on the value of the utilization statistics duration. The dynamic frequency and voltage modulation strategy parameters are used to dynamically adjust the frequency and voltage of the video decoder.

[0222] In one embodiment, the frame dropping constraints include a decoding timeout constraint and a buffer overflow constraint;

[0223] The statistical duration determination module 501 is specifically used to determine the upper limit of the utilization statistical duration based on the decoding information under decoding timeout and buffer overflow constraints.

[0224] In one embodiment, the decoding information includes frame interval time, latency factor, load data, load rate, and decoding buffer depth;

[0225] The statistical duration determination module 501 is specifically used to determine a first limiting parameter of the utilization statistical duration based on the frame interval time, latency factor, load rate and load data under the decoding timeout limit condition; to determine a second limiting parameter of the utilization statistical duration based on the frame interval time, decoding buffer depth, load rate and load data under the buffer overflow limit condition; and to determine an upper limit value of the utilization statistical duration based on the first limiting parameter and the second limiting parameter.

[0226] In one embodiment, the statistical duration determination module 501 is specifically used to determine the lower limit of the utilization statistical duration based on the decoding information under statistical error conditions.

[0227] In one embodiment, the decoding information includes frame interval time, payload data, and the maximum value of the calculation error;

[0228] The statistical duration determination module 501 is specifically used to determine the third limiting parameter of the utilization statistical duration based on the frame interval time, load data and the maximum value of the calculation error under statistical error conditions; and to determine the lower limit of the utilization statistical duration based on the third limiting parameter.

[0229] In one embodiment, the statistical duration determination module 501 is further configured to determine the current frame decoding duration of the video decoder; determine a first correlation between the decoding latency of the target frame and the utilization statistical duration, wherein the target frame is the last frame under the utilization statistical duration; and determine a decoding timeout limit condition based on the first correlation condition under the condition that the decoding latency of the target frame is constrained to be less than the upper limit of the frame decoding duration.

[0230] In one embodiment, the decoding information includes the decoding buffer depth; the statistical duration determination module 501 is further configured to determine a second correlation between the number of decoded frames of the video decoder and the utilization statistical duration within the utilization statistical duration, and a third correlation between the number of input frames of the video decoder and the utilization statistical duration within the utilization statistical duration; and, under the condition that the difference between the number of decoded frames and the number of input frames is constrained to a preset proportion less than the maximum value of the decoding buffer depth, a buffer overflow restriction condition is determined based on the second correlation, the third correlation, and the decoding buffer depth.

[0231] In one embodiment, the decoding information includes frame interval time and load data; the statistical duration determination module 501 is further used to determine the maximum value of the calculation error corresponding to the load of the video decoder; determine the load calculation error ratio based on the frame interval time, utilization statistical duration and load data; and constrain the statistical error condition to the point that the load calculation error ratio is less than the maximum value of the calculation error.

[0232] In one embodiment, the decoding information includes frame interval time and load data; the dynamic frequency and voltage adjustment strategy parameters include an upper limit and a lower limit of utilization configuration; the statistical duration determination module 501 is further used to obtain the down-frequency ratio and down-frequency safety margin of the video decoder; determine the actual value of the calculation error of the load corresponding to the video decoder based on the value of the utilization statistical duration, frame interval time, and load data; determine the upper limit of utilization configuration based on the actual value of the calculation error and the actual upper limit of utilization of the video decoder; and determine the lower limit of utilization configuration based on the actual upper limit of utilization, the down-frequency ratio, and the down-frequency safety margin.

[0233] In one embodiment, the parameter determination module 502 is specifically used to obtain the resolution of each video data currently being decoded by the video decoder; when the resolution of each video data is less than the preset resolution, the dynamic frequency and voltage modulation strategy parameters of the video decoder are determined based on the value of the utilization statistics duration.

[0234] In one embodiment, the parameter determination module 502 is further configured to use the highest performance mode to configure the video decoder when the resolution of the target video data is greater than or equal to a preset resolution, wherein the target video data is any video data currently being decoded.

[0235] Each module in the aforementioned dynamic frequency and voltage regulation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0236] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a dynamic frequency and voltage modulation method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0237] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0238] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described dynamic frequency and voltage modulation method.

[0239] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described dynamic frequency and voltage modulation method.

[0240] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described dynamic frequency and voltage modulation method.

[0241] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0242] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0243] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A dynamic frequency and voltage modulation method, characterized in that, The method includes: Under the upper and lower limits of the utilization statistics duration of the video decoder, the value of the utilization statistics duration is determined based on the current decoding information of the video decoder; the upper and lower limits include the frame loss restriction condition and the statistical error condition of the video decoder. Based on the value of the utilization statistics duration, the dynamic frequency and voltage modulation strategy parameters of the video decoder are determined. The dynamic frequency and voltage modulation strategy parameters are used to dynamically modulate the frequency and voltage of the video decoder. The frame dropping constraints include a decoding timeout constraint and a buffer overflow constraint; the decoding timeout constraint and the buffer overflow constraint are used to limit the upper limit of the utilization statistics duration; the statistical error constraint is used to limit the lower limit of the utilization statistics duration.

2. The method according to claim 1, characterized in that, The step of determining the value of the utilization statistics duration based on the current decoding information of the video decoder, under the conditions of the upper and lower limits of the utilization statistics duration of the video decoder, includes: Under the conditions of the decoding timeout limit and the buffer overflow limit, the upper limit of the utilization statistics duration is determined based on the decoding information.

3. The method according to claim 2, characterized in that, The decoding information includes frame interval time, delay factor, load data, load rate, and decoding buffer depth; The step of determining the upper limit of the utilization statistics duration based on the decoding information, under the conditions of the decoding timeout limit and the buffer overflow limit, includes: Under the decoding timeout constraint, a first constraint parameter for the utilization statistics duration is determined based on the frame interval time, the latency factor, the load rate, and the load data. Under the buffer overflow constraint, a second constraint parameter for the utilization statistics duration is determined based on the frame interval time, the decoding buffer depth, the load rate, and the load data; The upper limit of the utilization rate statistics duration is determined based on the first limiting parameter and the second limiting parameter.

4. The method according to claim 1, characterized in that, The step of determining the value of the utilization statistics duration based on the current decoding information of the video decoder, under the conditions of the upper and lower limits of the utilization statistics duration of the video decoder, includes: Under the statistical error conditions, the lower limit of the utilization rate statistical duration is determined based on the decoding information.

5. The method according to claim 4, characterized in that, The decoding information includes the frame interval time, payload data, and the maximum value of the calculation error; Under the statistical error condition, determining the lower limit of the utilization rate statistical duration based on the decoding information includes: Under the statistical error conditions, a third limiting parameter for the utilization statistical duration is determined based on the frame interval time, the load data, and the maximum value of the calculation error. The lower limit of the utilization rate statistics duration is determined based on the third limiting parameter.

6. The method according to claim 2, characterized in that, The method further includes: Determine the current frame decoding duration of the video decoder; Determine a first correlation between the decoding latency of the target frame and the utilization statistics duration, wherein the target frame is the last frame under the utilization statistics duration; Under the condition that the decoding delay of the target frame is constrained to be less than the upper limit of the frame decoding duration, the decoding timeout restriction condition is determined according to the first association relationship.

7. The method according to claim 2, characterized in that, The decoding information includes the decoding buffer depth; the method further includes: Determine a second correlation between the number of decoded frames of the video decoder within the utilization statistics period and the utilization statistics period, and a third correlation between the number of input frames of the video decoder within the utilization statistics period and the utilization statistics period; With the difference between the number of decoded frames and the number of input frames constrained to a preset proportion less than the maximum value of the decoding buffer depth, the buffer overflow restriction condition is determined based on the second correlation, the third correlation, and the decoding buffer depth.

8. The method according to claim 1, characterized in that, The decoding information includes frame interval time and payload data; the method further includes: Determine the maximum value of the computational error corresponding to the load of the video decoder; The load calculation error ratio is determined based on the frame interval time, the utilization statistics duration, and the load data. The statistical error condition is constrained to the point that the ratio of the calculated error of the load is less than the maximum value of the calculated error.

9. The method according to claim 1, characterized in that, The decoding information includes frame interval time and load data; the dynamic frequency and voltage modulation strategy parameters include an upper limit and a lower limit for utilization configuration; determining the dynamic frequency and voltage modulation strategy parameters of the video decoder based on the value of the utilization statistics duration includes: Obtain the down-ratio and down-ratio safety margin of the video decoder; Based on the value of the utilization statistics duration, the frame interval time, and the load data, the actual value of the calculation error of the load corresponding to the video decoder is determined; The upper limit of the utilization configuration is determined based on the actual value of the calculation error and the actual upper limit of the utilization of the video decoder. The lower limit of the utilization rate is determined based on the actual lower limit of the utilization rate, the down-frequency ratio, and the down-frequency safety margin.

10. The method according to claim 1, characterized in that, The step of determining the dynamic frequency and voltage modulation strategy parameters of the video decoder based on the value of the utilization statistics duration includes: Obtain the resolution of each video data stream currently being decoded by the video decoder; When the resolution of each video data stream is less than the preset resolution, the dynamic frequency and voltage modulation strategy parameters of the video decoder are determined based on the value of the utilization statistics duration.

11. The method according to claim 10, characterized in that, The method further includes: When the resolution of the target video data is greater than or equal to the preset resolution, the video decoder is configured in the highest performance mode, and the target video data is any video data currently being decoded.

12. A dynamic frequency and voltage regulation device, characterized in that, The device includes: The statistical duration determination module is used to determine the value of the utilization statistical duration based on the current decoding information of the video decoder, under the condition of the upper and lower limits of the utilization statistical duration of the video decoder; the upper and lower limits include the frame loss restriction condition and the statistical error condition of the video decoder. The parameter determination module is used to determine the dynamic frequency and voltage modulation strategy parameters of the video decoder based on the value of the utilization statistics duration. The dynamic frequency and voltage modulation strategy parameters are used to perform dynamic frequency and voltage modulation on the video decoder. The frame dropping constraints include a decoding timeout constraint and a buffer overflow constraint; the decoding timeout constraint and the buffer overflow constraint are used to limit the upper limit of the utilization statistics duration; the statistical error constraint is used to limit the lower limit of the utilization statistics duration.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

Images