A frame rendering optimization method and device, electronic equipment and readable storage medium

Abstract

The application discloses a frame rendering optimization method and device, electronic equipment and a readable storage medium, and relates to the technical field of graphic processing. The method calculates the predicted rendering time and the predicted rendering completion time of the next frame through a prediction algorithm; generates a virtual display rhythm sequence based on an actual refresh rate, calibrates an ideal frame period in combination with a clock drift offset, and generates an expected display time according to the calibrated frame period; pairs the expected display time with the predicted rendering completion time, and generates a time matching table; generates an optimal matching scheme of the candidate frame and the expected display time based on a weighted cost parameter in the time matching table; and reconstructs the out-of-order sequence of the rendering queue according to the optimal matching scheme, and performs layered processing on the output frame of the reconstructed rendering sequence in the display period. The method can effectively improve the fluency and stability of picture output, and reduce picture tearing and freezing phenomena through optimization of the frame rendering process.

Description

Technical Field

[0001] This application relates to the field of graphics processing technology, specifically to a frame rendering optimization method and apparatus, electronic device and readable storage medium. Background Technology

[0002] As the performance of display devices continues to improve and application scenarios become more complex, the requirements for timing control precision in ensuring smooth visuals are becoming increasingly stringent. The matching problem between image rendering and display output is gradually becoming an important factor affecting user experience.

[0003] Traditional rendering mechanisms in current technologies typically rely on a fixed queue order, meaning frames are submitted to the display buffer sequentially according to their rendering completion time. However, rendering time is significantly affected by various factors, making it difficult to precisely control the frame rhythm (i.e., the uniformity of frame rendering intervals). This results in some frames becoming ready too early or too late. When the rendering delay exceeds the display refresh cycle, the monitor may repeatedly output old frames or discard new frames, disrupting the continuity of the image and causing visual stuttering and lag in operation.

[0004] Therefore, there is an urgent need for a frame rendering method that can balance smoothness and stability in order to improve the matching between rendering and display and improve the overall visual experience. Summary of the Invention

[0005] To address the aforementioned technical problems, embodiments of this application provide a frame rendering optimization method and apparatus, an electronic device, and a readable storage medium.

[0006] In a first aspect, one embodiment of this application provides a frame rendering optimization method, which includes: collecting the actual rendering time and completion time of historical frames in the rendering queue; calculating the predicted rendering time and predicted rendering completion time of the next frame using a prediction algorithm; generating a virtual display rhythm sequence based on the actual refresh rate, the virtual display rhythm sequence including multiple time bases with uniform intervals, each time interval being an ideal frame period; calibrating the ideal frame period by combining clock drift offset, the calibrated ideal frame period being synchronized with the hardware display clock; generating an expected display time based on the calibrated frame period, the expected display time being used to represent the ideal output time of the i-th display period; pairing the expected display time with the predicted rendering completion time to generate a time matching table, the time matching table being used to store the weighted cost parameters of each candidate frame and the expected display time; calculating the comprehensive objective function of each candidate frame based on the weighted cost parameters in the time matching table; globally optimizing the comprehensive objective function using a dynamic programming algorithm to generate the optimal matching scheme between the candidate frame and the expected display time; reconstructing the dequeue order of the rendering queue according to the optimal matching scheme, and performing layered processing on the reconstructed rendering sequence output frames within the display period to ensure uniform output frame intervals.

[0007] In conjunction with the first aspect, in certain implementations of the first aspect, a virtual display rhythm sequence is generated based on the actual refresh rate. The virtual display rhythm sequence includes multiple time bases with uniform intervals, each time interval being an ideal frame period. The ideal frame period is calibrated by combining clock drift offset, and the calibrated ideal frame period is synchronized with the hardware display clock. An expected display time is generated based on the calibrated frame period, and the expected display time is used to represent the ideal output time of the i-th display period. This includes: obtaining the actual refresh rate of the vertical synchronization signal and calculating the ideal frame period; generating a virtual display rhythm sequence based on the ideal frame period and the base timestamp; the virtual display rhythm sequence is used to provide a time reference target for subsequent frame scheduling; calculating the clock drift compensation amount by monitoring the vertical synchronization signal deviation, and calibrating the ideal frame period based on the clock drift compensation amount; and generating the expected display time of candidate frames based on the calibrated ideal frame period combined with the virtual display rhythm sequence.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, a comprehensive objective function is calculated for each candidate frame based on the weighted cost parameters in the time-matching table. A dynamic programming algorithm is then used to globally optimize the comprehensive objective function, generating the optimal matching scheme between the candidate frame and the expected display time. The weighted cost parameters include matching cost, movement cost, and user experience cost. This includes: filtering candidate frames that meet the time constraints in the time-matching table; calculating the matching cost for each candidate frame, where the matching cost characterizes the time deviation between the predicted rendering completion time and the expected display time; constructing the movement cost for the candidate frames, where the movement cost characterizes the stability of the rendering queue reordering; defining the user experience cost; and constructing a comprehensive objective function by combining the matching cost and movement cost of the candidate frames, where the comprehensive objective function characterizes the comprehensive state of the candidate frames; and determining the optimal matching scheme based on the comprehensive objective function using a dynamic programming algorithm, where the optimal matching scheme establishes the pairing relationship with the candidate frame and the expected display time that minimizes the global cost.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the dequeue order of the rendering queue is reconstructed according to the optimal matching scheme, and the reconstructed rendering sequence output frames are processed in layers within the display cycle to ensure uniform intervals between output frames. This includes: reconstructing the dequeue order of output frames in the rendering queue according to the priority of candidate frames in the optimal matching scheme; calculating the time difference between the expected display time of the reconstructed output frame and the predicted rendering completion time; processing the output frames in layers according to the difference between the time difference and a tolerance threshold, where the tolerance threshold is used to determine the reasonableness of the expected display time of the output frame; and outputting candidate frames according to the dequeue order of the reconstructed rendering queue to complete frame rendering optimization.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the process of filtering candidate frames that meet the time constraints in the time matching table and calculating the matching cost of each candidate frame includes: filtering candidate frames that meet the time constraints based on the expected display time and the corresponding schedulable time range to form a candidate frame set; calculating the time difference cost of each candidate frame based on the time deviation between the predicted rendering completion time and the expected display time; calculating the scheduling confidence weight of each candidate frame based on the historical average prediction accuracy and the rendering progress of the current frame; and calculating the matching cost of each candidate frame based on the confidence weight and the time difference cost. The matching cost is used to characterize the time deviation between the predicted rendering completion time and the expected display time.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, the output frames are processed in layers based on the difference between the time difference and the tolerance threshold, including: determining the frame label of the output frame based on the difference between the time difference and the tolerance threshold; performing corresponding submission processing on the output frame based on the frame label; the frame label includes early frame, normal frame and late frame; performing a delayed submission mechanism for early frames; performing a frame compensation mechanism for late frames; and submitting normal frames to the display buffer for direct output.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the actual rendering time and completion time of historical frames in the rendering queue are collected, and the predicted rendering time and predicted rendering completion time of the next frame are calculated through a prediction algorithm. It also includes: maintaining the predicted rendering time and actual rendering time of a preset number of historical frames, and calculating the average prediction error in combination with a dynamic error algorithm; adjusting the number of historical frames maintained when the average prediction error exceeds a preset threshold.

[0013] Secondly, this application provides a frame optimization device, comprising: a frame prediction module for acquiring the actual time and completion time of historical frames in a queue, and calculating the prediction time and prediction completion time of the next frame through a prediction algorithm; and a timing calibration module for generating a virtual display rhythm sequence based on the actual refresh rate, the virtual display rhythm sequence including multiple time bases with uniform intervals, each time interval being an ideal frame period, calibrating the ideal frame period in conjunction with a clock drift offset, the calibrated ideal frame period being synchronized with the hardware display clock, and generating an expected display time based on the calibrated frame period, the expected display time being used to represent the ideal time of the i-th display period. The system comprises the following modules: a time output module; a scheduling decision module, which pairs the expected display time with the predicted completion time to generate a time matching table, which stores the weighted cost parameters of each candidate frame with the expected display time; a queue optimization module, which calculates the comprehensive objective function of each candidate frame based on the weighted cost parameters in the time matching table, and performs global optimization of the comprehensive objective function using a dynamic programming algorithm to generate the optimal matching scheme between the candidate frame and the expected display time; and an output control module, which reconstructs the dequeue order of the queue according to the optimal matching scheme and performs layered processing on the reconstructed sequence output frames within the display period to ensure uniform intervals between output frames.

[0014] Thirdly, this application provides an electronic device, including: a processor; and a memory storing computer program instructions, which, when executed by the processor, implement the steps in the frame rendering optimization method mentioned in the first aspect above.

[0015] Fourthly, this application provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor, cause the processor to perform the steps in the frame rendering optimization method mentioned in the first aspect.

[0016] The frame rendering optimization method provided in this application introduces predictive modeling, queue scheduling, and output control mechanisms during the frame rendering process. This achieves reasonable allocation and scheduling of frames, effectively avoiding mismatches between rendering tasks and display timing, thereby improving the continuity and stability of the display process. Furthermore, this application performs targeted optimizations on frame rendering and display synchronization at different execution stages, resulting in a more balanced output screen timing. This significantly reduces frame latency and output jitter, thereby enhancing overall display quality and user interaction experience. Attached Figure Description

[0017] Figure 1 The diagram shown is a flowchart illustrating a frame rendering optimization method provided in an exemplary embodiment of this application.

[0018] Figure 2The diagram shown is a flowchart of a frame rendering optimization method provided in another exemplary embodiment of this application;

[0019] Figure 3 The diagram shown is a flowchart of a frame rendering optimization method provided in another exemplary embodiment of this application;

[0020] Figure 4 The diagram shown is a flowchart of a frame rendering optimization method provided in another exemplary embodiment of this application;

[0021] Figure 5 The diagram shown is a flowchart of a frame rendering optimization method provided in another exemplary embodiment of this application;

[0022] Figure 6 The diagram shown is a flowchart of a frame rendering optimization method provided in another exemplary embodiment of this application;

[0023] Figure 7 The diagram shown is a schematic representation of the frame rendering optimization apparatus provided in an exemplary embodiment of this application.

[0024] Figure 8 The diagram shown is a structural schematic of an electronic device provided in an exemplary embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] The following is combined with Figures 1 to 6 The frame rendering optimization method provided in this application is described in detail.

[0027] Figure 1 The diagram shown is a flowchart illustrating a frame rendering optimization method provided in an exemplary embodiment of this application. Figure 1 As shown in the embodiments of this application, the frame rendering optimization method includes the following steps.

[0028] Step 100: Collect the actual rendering time and completion time of historical frames in the rendering queue, and calculate the predicted rendering time and predicted rendering completion time of the next frame using a prediction algorithm.

[0029] Specifically, obtain the actual rendering time T of the nth frame. actual (Time elapsed from start to finish rendering) and the actual rendering completion time C of the nth frame. actual(n) (absolute timestamp relative to system startup time), sets the historical data window size k (exemplarily, 8-16 frames) to balance prediction accuracy and computational overhead.

[0030] The system maintains a fixed-length circular buffer that records the rendering time sequence of the most recent k frames in real time. Whenever the nth frame is rendered, the oldest data is removed from the buffer.

[0031] Next, based on historical data analysis of rendering time fluctuations, appropriate weight parameters are determined for the prediction algorithm. Specifically, the standard deviation σ of historical rendering times is calculated. render This is used to measure the stability of rendering time, and the calculation formula is:

[0032]

[0033] Among them, T average The average historical rendering completion time is calculated using the following formula:

[0034] ;

[0035] The weight parameter α in the prediction algorithm is dynamically adjusted based on the degree of fluctuation in rendering time. The calculation formula is as follows:

[0036] ;

[0037] It should be understood that when rendering time fluctuations are small, The value is relatively small, relying more on historical averages; when fluctuations are large, It's on the larger side, so pay more attention to the changing trends of the most recent frames.

[0038] An improved weighted moving average algorithm, combined with fluctuation compensation, is used to predict the rendering time of the next frame. An exponentially weighted moving average algorithm is used to calculate the initial predicted rendering time T. base (n):

[0039] ;

[0040] Among them, T base (n) represents the initial predicted rendering time for the nth frame, T actual (n-1) is the actual rendering time of the previous frame, T base (n-1) represents the initial predicted rendering time of the previous frame.

[0041] Next, a fluctuation compensation term is introduced. To improve prediction accuracy, the prediction rendering time T is obtained. render (n):

[0042] ;

[0043] in, This is the trend compensation coefficient (for example, a value of 0.3). The formula for calculating the recent rendering time trend is as follows:

[0044]

[0045] Based on the current time T current And predicted rendering time T render Calculate the expected rendering completion time C(n) for the nth frame:

[0046] .

[0047] Step 102: Generate a virtual display rhythm sequence based on the actual refresh rate, calibrate the ideal frame period by combining the clock drift offset, keep the calibrated ideal frame period synchronized with the hardware display clock, generate the expected display time based on the calibrated frame period, and use the expected display time to represent the ideal output time of the i-th display period.

[0048] For example, the virtual display rhythm sequence includes multiple time bases with uniform intervals, each time interval being an ideal frame period.

[0049] It should be understood that, to address the issue of minor fluctuations in hardware refresh rates, an absolutely uniform and idealized timing rhythm (i.e., a virtual display rhythm sequence) can be generated at the software level. Then, real-time calibration and synchronization are performed by calculating the minute errors (clock drift) between the hardware clock and the virtual display rhythm sequence. Ultimately, it predicts the most precise display time for each future frame (i.e., the expected display moment), allowing the system to plan ahead and ensure that each frame is displayed at the perfect millisecond, achieving an extremely smooth and stutter-free visual experience.

[0050] Step 104: Pair the expected display time with the predicted rendering completion time to generate a time matching table. The time matching table is used to store the weighted cost parameters of each candidate frame and the expected display time.

[0051] Specifically, the expected display time S corrected Pair the predicted rendering completion time C with the predicted time to generate a time matching table (C(n), S). corrected (i)), where S corrected (i) represents the ideal output time for the i-th display cycle.

[0052] Step 106: Based on the weighted cost parameters in the time-matching table, calculate the comprehensive objective function for each candidate frame, and perform global optimization of the comprehensive objective function using a dynamic programming algorithm to generate the optimal matching scheme between the candidate frame and the expected display time.

[0053] Specifically, the weighted cost parameters include matching cost, mobility cost, and user experience cost.

[0054] It should be understood that the dynamic programming algorithm, combined with weighted cost, globally optimizes the comprehensive objective function to find the optimal combination of candidate frames and expected display times, serving as the basis for reconstructing the rendering queue.

[0055] Step 108: Reconstruct the dequeue order of the rendering queue according to the optimal matching scheme, and perform layered processing on the reconstructed rendering sequence output frames within the display cycle to ensure uniform intervals between output frames.

[0056] It should be understood that the task of a graphics card (GPU) is to deliver a series of still images (frames) to the screen and play them continuously at extremely high speeds to create the smooth animations or game visuals you see. "Reconstructing the dequeue order of the rendering queue according to an optimal matching scheme" means that the frames are not mechanically played in the order they were rendered (i.e., the original order of the rendering queue). Instead, the playback order of the frames is rearranged according to an "optimal matching scheme" (e.g., which frame is most ready to be ready in the next refresh cycle). The core purpose is to "fill" each display cycle, avoiding a cycle with no frames to play (causing stuttering) or multiple frames crowded together (causing tearing), thus laying the foundation for a more even output in the next step.

[0057] Next, "layering the reconstructed rendering sequence output frames within the display cycle" means that for this reordered playlist, the entire screen will be broken down into different logical layers. For example, static UI elements (such as health bars and maps) are one layer, relatively static game backgrounds are another, and fast-moving character models are yet another layer. The system can redraw and update only the active layers that have changed, without having to completely redraw everything every frame. This significantly reduces workload and ensures that even in complex scenes, the screen can be prepared within the specified time (display cycle).

[0058] This ensures that every frame is delivered accurately and on time when the screen refreshes, thus presenting you with an extremely stable and smooth visual experience, completely avoiding phenomena such as screen stuttering, jumping, or tearing.

[0059] The frame rendering optimization method provided in this application collects actual rendering data of historical frames, accurately calculates the predicted rendering time and completion time of the next frame using a prediction algorithm, and combines the actual refresh rate and clock drift calibration to ensure that the expected display time conforms to the hardware display rules, thus solving the problem of mismatch between the ideal frame period and the actual hardware clock. Next, by combining a time matching table with a dynamic programming algorithm, optimal pairing of candidate frames and expected display times is achieved, taking into account factors such as time deviation, queue stability, and user experience, thereby improving the scientific nature of scheduling decisions. Finally, the rendering queue is reconstructed and layered processing is performed, directly ensuring uniform output frame intervals, effectively reducing stuttering during display, and providing users with a smoother visual experience.

[0060] Figure 2 The diagram shown is a flowchart illustrating a frame rendering optimization method provided in another exemplary embodiment of this application. Figure 1 This application extends from the embodiments shown. Figure 2 The illustrated embodiment will be described in detail below. Figure 2 The illustrated embodiments and Figure 1 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0061] like Figure 2 As shown, in the frame rendering optimization method provided in this application embodiment, a virtual display rhythm sequence is generated based on the actual refresh rate. The virtual display rhythm sequence includes multiple time bases with uniform intervals. Each time interval is an ideal frame period. The ideal frame period is calibrated by combining the clock drift offset. The calibrated ideal frame period is synchronized with the hardware display clock. The expected display time is generated according to the calibrated frame period. The expected display time is used to represent the ideal output time of the i-th display period. The method includes the following steps.

[0062] Step 200: Obtain the actual refresh rate of the vertical synchronization signal and calculate the ideal frame period. Generate a virtual display rhythm sequence based on the ideal frame period and the reference timestamp. The virtual display rhythm sequence is used to provide a time reference target for subsequent frame scheduling.

[0063] It should be understood that the nominal refresh rate of a monitor However, due to manufacturing errors and environmental factors, the actual refresh rate may have slight deviations.

[0064] Therefore, the system needs to obtain the true refresh rate through vertical synchronization signal sampling, and the calculation formula is as follows:

[0065] ;

[0066] Among them, T vsync(i) represents the actual duration of the i-th vertical synchronization cycle, and m is the number of sampling cycles. For example, m is set to 64 to obtain stable measurement results. Next, the ideal frame period is calculated based on the actual refresh rate. The calculation formula is:

[0067] ;

[0068] Among them, ideal frame period As the reference unit for the virtual clock rhythm, it ensures synchronization with the hardware display clock and characterizes the ideal display cycle of a single frame. Next, the reference timestamp T of the first vertical synchronization signal after system startup is recorded. base As the origin of the virtual clock, all subsequent clock calculations are based on the base timestamp T. base Relative time calculations are performed based on an ideal frame period. and the base timestamp T base Generate virtual display rhythm sequence S ideal (i), the calculation formula is:

[0069] ;

[0070] Where, i∈N + , indicating the frame number, and virtually displaying the rhythm sequence S. ideal (i) represents the ideal display time of the i-th frame.

[0071] It should be understood that this sequence establishes a uniformly spaced time base, with each time interval being exactly [missing information]. .

[0072] Step 202: Calculate the clock drift compensation amount by monitoring the vertical synchronization signal deviation, and calibrate the ideal frame period based on the clock drift compensation amount.

[0073] It should be understood that an offset compensation mechanism is introduced to account for potential clock drift during long-term operation.

[0074] Specifically, monitor the actual vertical synchronization moments of the most recent N frames (for example, N can be set to 128). With virtual reality rhythm sequence S ideal (i) Deviation, calculate clock drift compensation amount The calculation formula is:

[0075] ;

[0076] It should be understood that the system continuously monitors the degree of deviation between the virtual rhythm and the actual vertical synchronization signal.

[0077] Specifically, define the synchronization quality index Q. sync When Qsync A value <0.95 indicates a synchronization deviation exceeding 5%, requiring a clock recalibration process. When a significant deviation is detected, clock drift compensation is considered. The ideal frame period is calibrated using a progressive calibration method, where the synchronization quality index Q... sync With the calibrated ideal frame period The calculation formula is:

[0078] ;

[0079] .

[0080] Step 204: Based on the calibrated ideal frame period combined with the virtual display rhythm sequence, generate the expected display time of the candidate frame.

[0081] Based on the calibrated ideal frame period Combined with virtual display rhythm sequence S ideal (i) Calculate the expected display time S of the candidate frames of the M frames in the future. corrected (i+m), the calculation formula is:

[0082] ;

[0083] For example, M can be set to 16, or it can be any other value.

[0084] The frame rendering optimization method provided in this application constructs a virtual display clock rhythm and generates expected display times, so that the rendering results can be coordinated with the display refresh process, thereby avoiding screen tearing caused by random output of rendering results, and maintaining a stable rhythm of the display process, further improving the consistency of screen output.

[0085] Figure 3 The diagram shown is a flowchart illustrating a frame rendering optimization method provided in another exemplary embodiment of this application. Figure 1 This application extends from the embodiments shown. Figure 3 The illustrated embodiment will be described in detail below. Figure 3 The illustrated embodiments and Figure 1 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0086] like Figure 3 As shown in the embodiment of this application, the frame rendering optimization method calculates the comprehensive objective function of each candidate frame based on the weighted cost parameters in the time matching table, and performs global optimization of the comprehensive objective function by combining dynamic programming algorithm to generate the optimal matching scheme between the candidate frame and the expected display time. The weighted cost parameters include matching cost, motion cost and user experience cost, and include the following steps.

[0087] Step 300: Filter candidate frames that meet the time constraints in the time matching table, and calculate the matching cost for each candidate frame. The matching cost is used to characterize the time deviation between the predicted rendering completion time and the expected display time.

[0088] Specifically, define the set of frame queue states Q at the current moment. current For those belonging to Q current Candidate frames are filtered to select those that meet the time constraints. The filtering criteria are as follows:

[0089] ;

[0090] Among them, R schedulable (i) represents the schedulable time range.

[0091] It should be understood that the frame queue state set describes the candidate frames available in the rendering queue at a specific moment and their predicted completion times, reflecting the rendering resources available to the system at that moment. The time matching table, based on this, assigns candidate frames to target display times, determining the output arrangement of each frame within the display cycle. The relationship between the two can be understood as the connection between input and output; the frame queue state set provides the available frame resources, while the time matching table establishes the optimal correspondence between these resources and target times, ensuring the coordination and consistency of frame rendering and display output. This filtering mechanism ensures that only frames completed within a reasonable time window are considered for a specific expected display time, avoiding unreasonable scheduling decisions.

[0092] Step 302: Construct the move cost of candidate frames. The move cost is used to characterize the stability of the rendering queue reordering.

[0093] It should be understood that a stability penalty term is introduced to avoid system thrashing caused by frequent queue reordering, thus ensuring the stability of the comprehensive objective function.

[0094] Specifically, the cost of moving each candidate frame relative to its original queue position is defined. stability (n,i), the calculation formula is: ;

[0095] Where η is the stability weighting coefficient (value 0.2), pos original (n) represents the position number of frame n in the original queue, pos target (i) represents the ideal queue position corresponding to the expected display time i.

[0096] Step 304: Define the user experience cost. Combine the matching cost and movement cost of the candidate frames to construct a comprehensive objective function. The comprehensive objective function is used to characterize the comprehensive state of the candidate frames.

[0097] Specifically, define the user experience cost (Cost). experience (n, i), the calculation formula is:

[0098] ;

[0099] Where μ is the delay penalty coefficient (exemplarily, it can be 5.0), C(n) is the prediction rendering completion time of the nth candidate frame, and S corrected (i) represents the expected display time for the i-th display cycle. This is combined with the matching cost (Cost) of the candidate frames. weighted (n,i) and movement cost Cost stabilit (n,i), construct the comprehensive objective function Cost total (n,i), the calculation formula is:

[0100] .

[0101] Cost total (n, i) represents the candidate frame n assigned to the expected display time S. corrected (i) The comprehensive cost, which is used in the dynamic specification phase to determine the optimal candidate frame-expected display time mapping relationship.

[0102] Step 306: Based on the comprehensive objective function, the optimal matching scheme is determined by combining dynamic programming algorithm. The optimal matching scheme is used to establish the pairing relationship between candidate frames and expected display times with the minimum global cost.

[0103] Specifically, the state space of the dynamic programming is defined as L consecutive expected display times starting from the current time (for example, L can be 8). The state transition equation is used for each expected display time S. corrected (i) Establish a one-to-one mapping relationship with the candidate frame, and the state transition equation is:

[0104] ;

[0105] Where DP[i][mask] represents the first i expected display times S corrected (i) The minimum total cost when allocating candidate frames, using the set of candidate frames already used as a mask. Then, a pruning strategy based on the upper bound of the cost is adopted, based on the comprehensive objective function Cost. total (n, i), maintain the cost of the currently known optimal solution. best When the cumulative cost of a certain intermediate state exceeds Cost best Yes, terminate the exploration of this branch and obtain the global minimum cost. minThe dynamic programming path is completed. After dynamic programming is completed, the optimal allocation scheme is reconstructed using a backtracking algorithm, starting from the termination state DP[L][mask]. final [Start by tracing the state transition path in reverse, based on the global minimum cost.] min Calculate the optimal candidate frame for each expected display time and allocate the frame accordingly. The calculation formula is as follows:

[0106] .

[0107] The frame rendering optimization method provided in this application establishes an optimization model based on the matching of the rendering prediction time and the target display time, and combines dynamic programming to perform globally optimal frame allocation. This can reduce the invalid waiting or redundant calculation of frames while ensuring the rationality of the timing, thereby improving the utilization efficiency of the rendering queue and the continuity of frame output.

[0108] Figure 4 The diagram shown is a flowchart illustrating a frame rendering optimization method provided in another exemplary embodiment of this application. Figure 1 This application extends from the embodiments shown. Figure 4 The illustrated embodiment will be described in detail below. Figure 4 The illustrated embodiments and Figure 1 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0109] like Figure 4 As shown in the embodiment of this application, the frame rendering optimization method reconstructs the dequeue order of the rendering queue according to the optimal matching scheme, and performs layered processing on the reconstructed rendering sequence output frames within the display period to ensure uniform output frame intervals, including the following steps.

[0110] Step 400: Reconstruct the dequeue order of output frames in the rendering queue according to the priority of candidate frames in the optimal matching scheme.

[0111] Specifically, based on the optimal allocation scheme, the priority of each candidate frame is calculated, and candidate frames with earlier expected display times are assigned higher priority. The calculation formula for the allocation scheme is as follows:

[0112] ;

[0113] Where, pos original (n) represents the position number of candidate frame n in the original queue.

[0114] It should be understood that this priority ensures that the optimized candidate frames are arranged in the order of the expected display time, while the unassigned candidate frames maintain their original relative positions.

[0115] Step 402: Calculate the time difference between the expected display time of the reconstructed output frame and the predicted rendering completion time.

[0116] Specifically, the tolerance threshold δ for display synchronization is determined, and the value of the tolerance threshold δ is:

[0117] ;

[0118] Where g is the tolerance coefficient, which can range from 0.1 to 0.3, representing the allowable time deviation relative to the ideal frame period. The time ratio is then used to calculate the expected display time S of the reconstructed output frame. corrected (i) The time difference between the predicted rendering completion time C(n).

[0119] It should be understood that the tolerance threshold δ is used to determine the rationality of the output timing.

[0120] Step 404: Based on the difference between the time difference and the tolerance threshold, the output frame is processed in layers. The tolerance threshold is used to determine the rationality of the expected display time of the output frame.

[0121] First, the frame label of the output frame is determined based on the difference between the time difference and the tolerance threshold. The frame label includes early frame, normal frame and late frame.

[0122] Specifically, if If so, then the frame tag of the output frame is determined to be an early frame; if If the condition is met, the frame label of the output frame is determined to be a delayed frame; if neither of the above conditions is met, the frame label of the output frame is determined to be a normal frame. The output frame is then processed according to its frame label, specifically including:

[0123] A delayed commit mechanism is implemented for the early frame. Specifically, the delayed commit time T of the early frame is calculated. delay (n), the calculation formula is:

[0124] ;

[0125] Among them, T delay (n) represents the time delay required for the nth frame, S corrected (i) represents the expected display time of the frame, and C(n) represents the prediction completion time of the frame.

[0126] A frame compensation mechanism is implemented for lagging frames. Specifically, the degree of lag T of the lagging frames is calculated. lag (n), the calculation formula is: According to the degree of lag T lag (n) Choosing a compensation strategy: If If so, then copy the content of the previous frame as the transition frame for output; if Then, inter-frame interpolation is performed based on motion vectors to generate interpolated frames. The formula for generating interpolated frames is as follows:

[0127] ;

[0128] Among them, F interp For the generated interpolated frame, F prev For the content of the previous frame, V motion For the estimated motion vector, α is the interpolation coefficient, calculated using the following formula:

[0129] .

[0130] Submit normal frames to the display buffer for direct output.

[0131] Step 406: Output candidate frames according to the dequeue order of the reconstructed rendering queue to complete frame rendering optimization.

[0132] Specifically, for the reconstructed rendering queue, the actual output interval ΔT is calculated. actual (i), the calculation formula is:

[0133] ;

[0134] Where Toutput(i) is the actual output time of the i-th frame.

[0135] The frame rendering optimization method provided in this application can effectively reduce the display instability caused by frame timing deviation by delaying the submission of pre-completed frames and compensating for delayed frames, making the screen output smoother and more stable, ensuring natural transitions between different frames, optimizing the user experience in scenarios with sudden rendering delays, and significantly reducing stuttering perception through mechanisms such as motion compensation interpolation, further improving the viewing experience.

[0136] Figure 5 The diagram shown is a flowchart illustrating a frame rendering optimization method provided in another exemplary embodiment of this application. Figure 3 This application extends from the embodiments shown. Figure 5 The illustrated embodiment will be described in detail below. Figure 5 The illustrated embodiments and Figure 3 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0137] like Figure 5 As shown in the embodiment of this application, the frame rendering optimization method includes the following steps: filtering candidate frames that meet the time constraints in the time matching table and calculating the matching cost of each candidate frame.

[0138] Step 500: Based on the expected display time and the corresponding schedulable time range, candidate frames that meet the time constraints are selected to form a candidate frame set.

[0139] Specifically, based on the calibrated ideal frame period and expected display time S corrected (i) Calculate the expected display time S corrected (i) The corresponding schedulable time range R schedulable Candidate frames that meet the time constraints (i.e., the schedulable time range) are selected to form a candidate frame set, where the schedulable time range is R. schedulable The calculation formula is:

[0140] ;

[0141] It should be understood that the schedulable time range ensures that only candidate frames completed within a reasonable time window will be considered for the corresponding expected display time, avoiding excessively advanced or severely delayed scheduling.

[0142] Step 502: Calculate the time difference cost for each candidate frame based on the time deviation between the predicted rendering completion time and the expected display time of each candidate frame.

[0143] Specifically, the predicted rendering completion time C(n) and the expected display time S for each candidate frame are calculated. corrected (i) time difference The calculation formula is:

[0144] ;

[0145] Next, calculate the cost based on the time difference. weighted (n, i), the calculation formula is:

[0146] .

[0147] Step 504: Calculate the scheduling confidence weight for each candidate frame based on the historical average prediction accuracy and the rendering progress of the current frame.

[0148] Specifically, obtain the historical average prediction accuracy E average Rendering progress P of the current frame peogress (n) and predicted rendering time T render (n), calculate the confidence weight W confidence (n), the calculation formula is:

[0149] ;

[0150] It should be understood that the confidence weight reflects the reliability of the prediction completion time for that frame; the higher the rendering progress and the smaller the historical prediction error, the higher the confidence.

[0151] Step 506: Based on the confidence weight and time difference cost, calculate the matching cost for each candidate frame. The matching cost is used to characterize the time deviation between the predicted rendering completion time and the expected display time.

[0152] Specifically, obtain the confidence weight W. confidence (n) and time difference cost weighted (n,i), calculate the matching cost (Cost) for each candidate frame. temporal (n,i), the calculation formula is:

[0153] ;

[0154] It should be understood that the matching cost function Cost temporal This ensures that frames with high confidence receive higher weight in time matching, thereby improving the reliability of scheduling decisions.

[0155] The frame rendering optimization method provided in this application significantly improves the robustness of scheduling decisions by establishing a confidence weight model. By fusing historical data on rendering progress and prediction errors, the matching weights of candidate frames are dynamically adjusted, enabling the system to intelligently distinguish between high-reliability frames and high-risk frames. This maintains stable scheduling quality even in scenarios with fluctuating predictions, significantly reducing image anomalies caused by misjudgments.

[0156] Figure 6 The diagram shown is a flowchart illustrating a frame rendering optimization method provided in another exemplary embodiment of this application. Figure 1 This application extends from the embodiments shown. Figure 6 The illustrated embodiment will be described in detail below. Figure 6 The illustrated embodiments and Figure 1 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0157] like Figure 6 As shown in the embodiment of this application, the frame rendering optimization method includes collecting the actual rendering time and completion time of historical frames in the rendering queue, calculating the predicted rendering time and predicted rendering completion time of the next frame through a prediction algorithm, and also includes the following steps.

[0158] Step 600: Maintain the predicted rendering time and actual rendering time of a preset number of historical frames, and calculate the average prediction error using a dynamic error algorithm.

[0159] Specifically, after the nth frame is rendered, the prediction error E is calculated. predic t(n) is calculated using the following formula:

[0160] ;

[0161] Maintain the average prediction error E of the most recent h frames (for example, h is 32). average The calculation formula is:

[0162] .

[0163] Step 602: When the average prediction error exceeds a preset threshold, adjust the number of historical frames maintained.

[0164] Specifically, when the average prediction error exceeds a threshold, the historical window size k is increased. If the prediction error for 10 consecutive frames is less than 50% of the threshold, k can be decreased to improve response speed.

[0165] For example, the threshold can be set to 0.1 times the predicted rendering time T. render (n).

[0166] The frame rendering optimization method provided in this application further improves the accuracy of rendering time prediction by maintaining historical frame data and dynamically adjusting prediction errors, providing a more reliable time basis for frame rendering optimization. Specifically, in the historical frame data maintenance stage, the predicted rendering time and actual rendering time of a preset number of historical frames are continuously recorded, providing sufficient data support for prediction error calculation and ensuring the comprehensiveness and objectivity of error assessment. Then, the average prediction error is calculated using a dynamic error algorithm, which can reflect the accuracy of the prediction algorithm in real time and promptly identify prediction deviations. In the historical frame maintenance quantity adjustment stage, the number of historical frames maintained is dynamically adjusted based on the comparison between the average prediction error and a preset threshold. When the prediction error is large, the historical window is increased to obtain more data and improve prediction accuracy; when the prediction error is small, the window is decreased to improve response speed, achieving adaptive optimization of the prediction algorithm. This dynamic adjustment mechanism allows the prediction algorithm to continuously optimize based on actual prediction results, effectively improving the accuracy of rendering time prediction, reducing scheduling errors caused by prediction deviations, and providing a more reliable time basis for subsequent expected display time matching, queue reconstruction, and other stages, further ensuring the overall effect of frame rendering optimization.

[0167] Figure 7 The diagram shown is a structural schematic of a frame rendering optimization apparatus provided in an exemplary embodiment of this application. Figure 7 As shown, the frame rendering optimization device provided in this application embodiment includes: a frame prediction module 700, a timing calibration module 702, a scheduling decision module 704, a queue optimization module 706, and an output control module 708.

[0168] The frame prediction module 700 collects the actual time and completion time of historical frames in the queue, and calculates the predicted time and completion time of the next frame using a prediction algorithm. The timing calibration module 702 generates a virtual display rhythm sequence based on the actual refresh rate. This sequence includes multiple evenly spaced time bases, each interval representing an ideal frame period. The ideal frame period is calibrated using clock drift offset, and the calibrated period is synchronized with the hardware display clock. The expected display time is generated based on this calibrated period, representing the ideal output time for the i-th display period. The scheduling decision module 704 pairs the expected display time with the predicted completion time to generate a timing matching table. This table stores the weighted cost parameters for each candidate frame and the expected display time. The queue optimization module 706 calculates the comprehensive objective function for each candidate frame based on the weighted cost parameters in the timing matching table. It then uses a dynamic programming algorithm to globally optimize the objective function, generating the optimal matching scheme between the candidate frame and the expected display time. The output control module 708 is used to reconstruct the dequeue order of the queue according to the optimal matching scheme, and to perform layered processing on the reconstructed sequence output frames within the display period to ensure uniform output frame intervals.

[0169] Figure 8 The diagram shown is a structural schematic of an electronic device provided in an exemplary embodiment of this application. Figure 8 As shown, the electronic device 800 includes one or more processors 801 and memory 802.

[0170] The processor 801 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 800 to perform desired functions.

[0171] The memory 802 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 801 may execute the program instructions to implement the calibration frame period and / or other desired functions of the various embodiments of this application described above. Various contents, such as history frames, may also be stored in the computer-readable storage medium.

[0172] In one example, the electronic device 800 may also include an input device 803 and an output device 804, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0173] The input device 803 may include, for example, a keyboard, a mouse, etc.

[0174] The output device 804 can output various information to the outside, including rendering queues. The output device 804 may include, for example, a monitor, speakers, a printer, and a communication network and its connected remote output devices, etc.

[0175] Of course, for the sake of simplicity, Figure 8 Only some of the components of the electronic device 800 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 800 may include any other suitable components depending on the specific application.

[0176] In addition to the methods and apparatus described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the frame rendering optimization methods according to various embodiments of this application described above.

[0177] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0178] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the frame rendering optimization methods according to various embodiments of this application described above.

[0179] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0180] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0181] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0182] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0183] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0184] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A frame rendering optimization method, characterized in that, include: Collect the actual rendering time and completion time of historical frames in the rendering queue, and calculate the predicted rendering time and predicted rendering completion time of the next frame using a prediction algorithm. A virtual display rhythm sequence is generated based on the actual refresh rate. The virtual display rhythm sequence includes multiple time bases with uniform intervals. Each time interval is an ideal frame period. The ideal frame period is calibrated by combining clock drift offset. The calibrated ideal frame period is synchronized with the hardware display clock. The expected display time is generated according to the calibrated frame period. The expected display time is used to represent the ideal output time of the i-th display period. A time matching table is generated by pairing the expected display time with the predicted rendering completion time. The time matching table is used to store the weighted cost parameters of each candidate frame with the expected display time. Based on the weighted cost parameters in the time matching table, the comprehensive objective function of each candidate frame is calculated, and the comprehensive objective function is globally optimized by combining a dynamic programming algorithm to generate the optimal matching scheme between the candidate frame and the expected display time. The dequeue order of the rendering queue is reconstructed according to the optimal matching scheme, and the reconstructed rendering sequence output frames are processed in layers within the display cycle to ensure uniform intervals between output frames.

2. The frame rendering optimization method according to claim 1, characterized in that, The virtual display rhythm sequence is generated based on the actual refresh rate. This sequence includes multiple evenly spaced time bases, each interval representing an ideal frame period. The ideal frame period is calibrated using clock drift offset, and the calibrated ideal frame period is synchronized with the hardware display clock. An expected display time is generated based on this calibrated frame period. This expected display time represents the ideal output time for the i-th display period, including: The actual refresh rate of the vertical synchronization signal is obtained and the ideal frame period is calculated. The virtual display rhythm sequence is generated based on the ideal frame period and the reference timestamp. The virtual display rhythm sequence is used to provide a time reference target for subsequent frame scheduling. The clock drift compensation amount is calculated by monitoring the vertical synchronization signal deviation, and the ideal frame period is calibrated based on the clock drift compensation amount; Based on the calibrated ideal frame period and the virtual display rhythm sequence, the expected display time of the candidate frame is generated.

3. The frame rendering optimization method according to claim 1, characterized in that, The weighted cost parameters include matching cost, movement cost, and user experience cost. Based on the weighted cost parameters in the time-matching table, a comprehensive objective function is calculated for each candidate frame. A dynamic programming algorithm is then used to globally optimize the comprehensive objective function to generate the optimal matching scheme between the candidate frame and the expected display time. This includes: Candidate frames that meet the time constraints are filtered from the time matching table, and the matching cost of each candidate frame is calculated. The matching cost is used to characterize the time deviation between the predicted rendering completion time and the expected display time. Construct the movement cost of the candidate frame, which is used to characterize the stability of the rendering queue reordering; Define the user experience cost, and construct a comprehensive objective function by combining the matching cost and movement cost of the candidate frame. The comprehensive objective function is used to characterize the comprehensive state of the candidate frame. Based on the comprehensive objective function, the optimal matching scheme is determined by combining dynamic programming algorithm. The optimal matching scheme is used to establish a pairing relationship between the candidate frame and the expected display time with the minimum global cost.

4. The frame rendering optimization method according to claim 1, characterized in that, The step of reconstructing the dequeue order of the rendering queue according to the optimal matching scheme, and performing layered processing on the reconstructed rendering sequence output frames within the display cycle to ensure uniform output frame intervals, includes: The dequeue order of output frames in the rendering queue is reconstructed based on the priority of candidate frames in the optimal matching scheme. Calculate the time difference between the expected display time of the reconstructed output frame and the predicted rendering completion time; The output frames are processed in layers based on the difference between the time difference and the tolerance threshold. The tolerance threshold is used to determine the rationality of the expected display time of the output frames. The candidate frames are output according to the dequeue order of the reconstructed rendering queue to complete the frame rendering optimization.

5. The frame rendering optimization method according to claim 3, characterized in that, The process of filtering candidate frames that meet the time constraints in the time matching table and calculating the matching cost for each candidate frame includes: Based on the expected display time and the corresponding schedulable time range, candidate frames that meet the time constraints are selected to form a candidate frame set. The time difference cost of each candidate frame is calculated based on the time deviation between the predicted rendering completion time and the expected display time. Based on the historical average prediction accuracy and the rendering progress of the current frame, the scheduling confidence weight of each candidate frame is calculated. Based on the confidence weight and the time difference cost, the matching cost of each candidate frame is calculated. The matching cost is used to characterize the time deviation between the predicted rendering completion time and the expected display time.

6. The frame rendering optimization method according to claim 4, characterized in that, The step of performing layered processing on the output frames based on the difference between the time difference and the tolerance threshold includes: Based on the difference between the time difference and the tolerance threshold, the frame label of the output frame is determined, and the output frame is submitted accordingly based on the frame label. The frame label includes early frame, normal frame and late frame. A delayed submission mechanism is executed on the aforementioned early frame; A frame compensation mechanism is executed for the lagging frames; The normal frame is submitted to the display buffer for direct output.

7. The frame rendering optimization method according to any one of claims 1 to 6, characterized in that, The actual rendering time and completion time of historical frames in the acquisition rendering queue are used to calculate the predicted rendering time and predicted rendering completion time of the next frame through a prediction algorithm, which also includes: Maintain a preset number of historical frames' predicted and actual rendering times, and calculate the average prediction error using a dynamic error algorithm; When the average prediction error exceeds a preset threshold, the number of historical frames maintained is adjusted.

8. A frame rendering optimization device, characterized in that, include: The frame prediction module is used to collect the actual rendering time and completion time of historical frames in the rendering queue, and to calculate the predicted rendering time and predicted rendering completion time of the next frame through a prediction algorithm. The timing calibration module is used to generate a virtual display rhythm sequence based on the actual refresh rate. The virtual display rhythm sequence includes multiple time bases with uniform intervals. Each time interval is an ideal frame period. The ideal frame period is calibrated in combination with the clock drift offset. The calibrated ideal frame period is synchronized with the hardware display clock. The expected display time is generated according to the calibrated frame period. The expected display time is used to represent the ideal output time of the i-th display period. The scheduling decision module is used to pair the expected display time with the predicted rendering completion time to generate a time matching table. The time matching table is used to store the weighted cost parameters of each candidate frame and the expected display time. The queue optimization module is used to calculate the comprehensive objective function of each candidate frame based on the weighted cost parameters in the time matching table, and to perform global optimization of the comprehensive objective function by combining a dynamic programming algorithm to generate the optimal matching scheme between the candidate frame and the expected display time. The output control module is used to reconstruct the dequeue order of the rendering queue according to the optimal matching scheme, and to perform layered processing on the reconstructed rendering sequence output frames within the display cycle to ensure uniform intervals between output frames.

9. An electronic device, characterized in that, include: Processor; and A memory that stores computer program instructions, which, when executed by a processor, cause the processor to perform the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores computer program instructions that, when executed by a processor, cause the processor to perform the steps of the method as described in any one of claims 1 to 7.

Images