Methods, media, products, and computing devices for dynamically adjusting interpolation calculations

Abstract

The present disclosure provides a method, medium, product and computing device for dynamically adjusting interpolation calculation, relating to the field of image and video processing, especially to the field of real-time rendering, and the implementation scheme is as follows: obtaining a feature parameter related to interpolation calculation of a current rendering frame; inputting the feature parameter into a complexity prediction model to obtain a predicted time consumption of interpolation calculation of the current rendering frame before rendering of the current rendering frame starts; selecting an interpolation mode from a plurality of interpolation modes with different calculation accuracies based on the predicted time consumption and a budget time to perform interpolation calculation on the current rendering frame, and recording an actual time consumption of interpolation calculation of the current rendering frame; and updating the complexity prediction model based on a difference between the actual time consumption and the predicted time consumption.

Description

Technical Field

[0001] This disclosure relates to the field of image and video processing, and particularly to the field of real-time rendering, applicable to scenarios such as games and 3D modeling, and especially to methods, media, products and computing devices for dynamically adjusting frame interpolation calculations. Background Technology

[0002] To improve the smoothness of real-time rendering and video game scenes, existing technologies typically use frame interpolation to generate intermediate frames (also known as interpolated frames) between adjacent rendering frames. This makes the output frame rate closer to the display device's refresh rate, thereby reducing screen tearing and stuttering. Traditional frame interpolation processes generally include steps such as motion estimation, occlusion handling, pixel reconstruction, and missing area repair. Its computational complexity largely depends on factors such as the image geometry, local motion speed, and changes in depth occlusion.

[0003] Existing frame interpolation methods typically operate with fixed complexity. This may be sufficient for rendering budgets in simple scenes with minimal motion variation; however, in complex scenes, such as those with high-speed motion, rapid camera transitions, or large-scale occlusion changes, the computational overhead of motion estimation and patching stages increases dramatically. Since the frame interpolation module usually needs to complete within a fixed budget time per frame, exceeding the budget can cause processors, including the graphics processing unit (GPU), to fail to output the interpolated image on time, resulting in frame jitter or accumulated latency.

[0004] Therefore, it is necessary to provide a solution that can dynamically adjust the frame interpolation complexity at runtime based on rendering load and screen motion characteristics, in order to reduce performance pressure in complex scenes and improve frame time stability. Summary of the Invention

[0005] To address the aforementioned technical problems, this disclosure provides a method, medium, computer program product, and computing device for dynamically adjusting frame interpolation calculations.

[0006] According to a first aspect of this disclosure, a method for dynamically adjusting frame interpolation calculation is provided, the method comprising: obtaining feature parameters related to frame interpolation calculation of a current rendering frame; inputting the feature parameters into a complexity prediction model to obtain a predicted time for performing frame interpolation calculation on the current rendering frame before the rendering of the current rendering frame begins; selecting an interpolation mode from multiple interpolation modes with different computational precisions to perform frame interpolation calculation on the current rendering frame based on the predicted time and the budgeted time, and recording the actual time for performing frame interpolation calculation on the current rendering frame; and updating the complexity prediction model based on the difference between the actual time and the predicted time.

[0007] In some embodiments, the characteristic parameters related to the interpolation calculation of the current rendering frame include: the rendering time of the previous rendering frame of the current rendering frame, the actual time spent performing the interpolation calculation on the previous rendering frame of the current rendering frame, the current utilization of the graphics processing unit used for rendering, and the global motion intensity of the previous rendering frame of the current rendering frame.

[0008] In some embodiments, the method further includes: dividing the current rendering frame into multiple regions based on the selected frame interpolation mode; determining the computational complexity of each region based on the motion vectors, depth gradients, and occlusion masks within that region; and fine-tuning the configuration of the frame interpolation computation for each region under the selected frame interpolation mode based on the determined computational complexity of each region.

[0009] In some embodiments, the configuration of frame interpolation computation includes one or more of the following: whether to perform motion field patching; whether to perform occlusion detection; whether to perform optical flow estimation; the number of iterations for optical flow estimation; the resolution of optical flow estimation; and whether to perform motion field reconstruction.

[0010] In some embodiments, when the selection of the interpolation mode involves switching from a mode with lower computational precision to a mode with higher computational precision, the decision to perform the switch is further determined based on the prediction results of the previously rendered frames by the complexity prediction model; and when the selection of the interpolation mode involves switching from a mode with higher computational precision to a mode with lower computational precision, the switch is performed directly.

[0011] In some embodiments, the method further includes: in response to detecting an anomaly, performing frame interpolation calculations using a preset fallback mode until an anomaly exit condition is met, wherein the anomaly includes at least one of the following: the actual time taken for frame interpolation calculations of multiple consecutive rendering frames is higher than the budgeted time; the rate of change of the utilization of the graphics processing unit within a given time period exceeds a preset condition; the change of the occlusion mask between adjacent rendering frames exceeds a predetermined size; or an abnormal distribution occurs in the motion vector field.

[0012] In some embodiments, abnormal exit conditions include the actual time taken for interpolation calculations of multiple consecutive rendering frames being less than the budgeted time and / or the decrease in global motion intensity between adjacent rendering frames exceeding a predetermined size.

[0013] According to a second aspect of this disclosure, a computer-readable storage medium storing instructions is provided, which, when executed by a computer's processor, cause the processor to perform the method according to a first aspect of this disclosure.

[0014] According to a third aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, causes the processor to perform the method according to a first aspect of this disclosure.

[0015] According to a fourth aspect of this disclosure, a computing device is provided, comprising: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the computing device to perform a method according to a first aspect of this disclosure.

[0016] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0017] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of this disclosure. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0018] Figure 1 This is a flowchart illustrating a method according to an exemplary embodiment.

[0019] Figure 2 An example configuration of a computing device that can be used to implement the methods described herein is shown. Detailed Implementation

[0020] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0021] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0022] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0023] Existing frame interpolation algorithms typically operate with a fixed complexity configuration, and their computational load is affected by various factors such as scene motion speed, texture details, and occlusion changes. When the game screen suddenly switches from a low-motion area to a complex scene with high-speed motion or significant occlusion changes, the computational requirements of the motion estimation and patching stages increase significantly. Since the frame interpolation module generally needs to complete calculations within a strict frame time budget, the fixed-complexity execution method is prone to exceeding the budget in these scenarios, leading to unstable frame times, accumulated latency, or noticeable jitter.

[0024] In addition, existing frame interpolation methods are mostly executed globally, which lacks differentiation between the foreground and background of the image, and between static areas and high-speed moving areas, further amplifying the performance pressure in complex scenes.

[0025] Therefore, a mechanism is needed that can perceive changes in rendering load, motion characteristics, and occlusion in real time at runtime, and dynamically adjust multiple computational stages within frame interpolation without changing the rendering pipeline, so that frame interpolation computation can maintain a stable time cost under different scene complexities.

[0026] To address at least the aforementioned issues, this disclosure proposes a two-layer dynamic frame interpolation adjustment method based on complexity model prediction, used to dynamically control the execution complexity of frame interpolation algorithms in video rendering such as games and 3D modeling. This disclosure achieves adaptive scheduling at the frame and spatial levels by establishing a feedback control mechanism for predicting frame interpolation complexity, thereby reducing frame rate fluctuations and latency jitter while ensuring image quality.

[0027] For example, the technical solution provided in this disclosure can perform the following steps before rendering each frame:

[0028] - Runtime feature acquisition: Real-time acquisition of feature parameters related to the complexity of the current frame;

[0029] -Complexity model prediction: Predict computational overhead for different frame interpolation modes using an online updated complexity model;

[0030] - Pattern mapping and selection: Based on forecast results and budget time Select the appropriate frame interpolation mode from the predefined mapping table;

[0031] - Spatial-level allocation: After the global frame interpolation mode is determined, different regions of the image are fine-tuned according to motion characteristics using different calculation modules (e.g., optical flow estimation, motion field reconstruction, occlusion detection, image inpainting, and frame interpolation ratio).

[0032] - Frame interpolation execution and time measurement: Record the actual frame interpolation time and determine the prediction error;

[0033] - Feedback Update: The complexity model is updated online, allowing the predictive power to gradually converge;

[0034] - Exception handling.

[0035] Through the closed-loop mechanism described above, the technical solution provided in this disclosure achieves dynamic adjustment and complexity self-learning of the frame interpolation algorithm. Compared to traditional frame interpolation algorithms that use a fixed budget, the complexity prediction-based mechanism provided in this disclosure allows the computing system to anticipate the performance costs of different strategies and avoid sudden performance fluctuations while ensuring image quality. For example, when the model predicts that the time required for a certain high-precision frame interpolation path may exceed the budgeted time... In such cases, the system will pre-select a lighter frame interpolation strategy to ensure frame time stability.

[0036] First, the complexity prediction mechanism provided in this disclosure can predict the interpolation computation cost of the next frame in advance based on characteristics such as the rendering time, interpolation time, GPU utilization, and motion intensity of the previous frame. By obtaining the predicted value before rendering is executed, the computing system can select a suitable interpolation path in advance, avoiding the instantaneous increase in computational pressure caused by the sudden appearance of complex scenes, and making the interpolation computation more controllable.

[0037] Secondly, by dividing the frame interpolation process into multiple adjustable modules and constructing a mapping table, the computing system can automatically enable or disable computationally intensive operations such as optical flow estimation, motion field repair, occlusion handling, and multi-level interpolation in different load ranges, allowing the frame interpolation complexity to switch rapidly within a finite set. Since the mapping table lookup process does not require large-scale computation, it is suitable for operation within a millisecond-level budget, thereby enabling dynamic adjustment in real-time scenes.

[0038] Furthermore, the technical solution provided in this disclosure introduces a spatial complexity allocation mechanism in addition to the global mode. This allows regions with strong motion, significant depth changes, or unstable occlusion to achieve higher frame interpolation accuracy, while static backgrounds or low-motion areas can be processed using a lightweight approach. Through this localized differentiation strategy, the computing system can utilize computing resources more effectively and improve the overall frame interpolation results without changing the total computing budget.

[0039] Furthermore, by updating model parameters online, the complexity prediction model provided in this disclosure can continuously adapt to the differences brought about by different hardware, game scenarios, and combinations of frame interpolation modes during long-term operation, so that the prediction results gradually approach the true performance characteristics of the system. This adaptive feature gives the entire frame interpolation process stronger generalization ability, without relying on offline training or manual parameter tuning.

[0040] Finally, the anomaly detection and fallback mechanism provided in this disclosure can quickly correct the frame interpolation path and promptly shut down high-cost modules when there are significant scene changes or prediction failures, thereby avoiding frame interpolation time exceeding the budget or severe fluctuations in frame time under extreme conditions. Especially in scenarios with high-speed motion, camera switching, or significant occlusion, the technical solution provided in this disclosure can maintain the continuity and stability of the frame interpolation process.

[0041] The following is combined Figure 1 The embodiments of this disclosure are described in detail.

[0042] Figure 1 This is a flowchart illustrating a method according to an exemplary embodiment.

[0043] like Figure 1 As shown, method 100 includes steps 102, 104, 106, and 108. In some examples, electronic devices may be used to perform these steps. Figure 1 The method illustrated herein. Electronic devices typically possess video decoding, image display, and data communication capabilities, and exemplary include user terminal devices for cloud gaming or remote desktop services, such as personal computers, laptops, tablet computing devices, smartphones, set-top boxes, smart TVs, virtual reality / augmented reality terminals, or in-vehicle infotainment systems, or other terminal devices with display capabilities. The terminal device may integrate a central processing unit (CPU), a graphics processing unit (GPU), a hardware video decoder, a display interface, and a network interface module.

[0044] In step 102, feature parameters related to the interpolation calculation of the current rendered frame are obtained. According to some embodiments, the rendering system used for rendering and interpolation can collect feature parameters related to the interpolation calculation in real time after each frame is rendered, in order to predict the computational load of the interpolation. For example, if the current rendered frame to be rendered and interpolated is the [missing information - likely a specific frame type]... The feature parameters can include the previous rendering frame (the current rendering frame). Rendering time per rendering frame The actual time spent performing frame interpolation calculations on the previous rendered frame. Current utilization of the graphics processing unit (GPU) used for rendering and global motion intensity These characteristic parameters describe the current operating state of the computing system and the changes in the image, and are an important basis for subsequent dynamic adjustment of the frame interpolation mode. It should be understood that, without departing from this disclosure, complexity estimation can be based on more historical or runtime information, such as the rendering time and frame interpolation time of other previously rendered frames. The aforementioned characteristic parameters are merely examples and should not be considered as restrictive requirements on the technical solutions of this disclosure.

[0045] Among them, global motion intensity This reflects the overall dynamic range of the image. To simultaneously consider object motion and camera viewpoint changes, the calculation system combines the motion vector (Game MV) field output by the rendering engine (e.g., Unreal or Vulkan engine) and the camera's pose changes in each frame to calculate this value. Specifically, first, the system obtains the motion vector of each pixel from the rendering engine's geometry buffer (GBuffer). This vector represents the displacement of the pixel on the screen from the previous frame to the current frame. Simultaneously, the computing system calculates the camera's pose changes, including rotational angular velocity. and displacement Based on the image resolution and field of view (FOV), the equivalent pixel displacement caused by camera movement is calculated. This displacement can be achieved through the focal length. The relationship with the camera rotation angle was calculated as follows:

[0046] ,

[0047] Changes in camera pose cause an overall displacement of the entire image. Directly inputting this "global motion" into optical flow estimation can lead the algorithm to misinterpret it as "all pixels are moving," increasing computational burden and producing artifacts. To avoid this, the computational system first subtracts the global displacement caused by camera motion from the motion vector field, retaining only the local changes related to the actual motion of the object. This process yields a local motion field. It can more accurately reflect the real dynamic content in the scene.

[0048] Finally, global motion intensity Defined as the average of the local displacement magnitudes of all pixels:

[0049] ,

[0050] in This refers to the total number of pixels. In this way, the calculation system can use a quantitative metric to reflect the motion complexity of the image. It should be understood that, depending on the specific implementation, the global motion intensity of one or more rendered frames can be used for calculation. In some embodiments, it can be calculated based on the pixel displacement amplitude of the previous rendered frame. In some embodiments, an equivalent global motion intensity can be calculated by averaging the global motion intensities of multiple rendered frames using a sliding window, for example, by weighted averaging based on the proximity of each rendered frame to the current rendered frame in terms of playback order. In some examples, the pixel displacement amplitude used for calculation can also be obtained based on methods such as optical flow estimation.

[0051] In step 104, feature parameters are input into the complexity prediction model to obtain the predicted time for frame interpolation calculation of the current rendering frame before the rendering of the current rendering frame begins. According to some embodiments, after obtaining feature parameters including runtime features of the previous frame, the computing system can use an updatable complexity prediction model to predict the computational overhead and estimated time for frame interpolation calculations under different interpolation strategies. By using this model to estimate the time consumption of each interpolation mode in advance, the most suitable complexity level can be determined before the rendering cycle begins, thus avoiding sudden frame rate drops or GPU overload.

[0052] In some embodiments, complexity prediction can be performed using online-updable models such as linear regression and ridge regression. For example, a complexity prediction model can take the form of linear regression, mapping runtime features and the static costs of different pattern combinations to a prediction time, which can be defined by the following expression:

[0053] ,

[0054] in, This indicates the predicted frame interpolation time. It's the rendering time. It is the time taken to insert the previous frame. It's GPU utilization. It is the overall motion intensity. These are model weight parameters used to reflect the sensitivity of each feature parameter to the time consumption.

[0055] In some examples, to enable the complexity prediction model to provide reasonably reasonable estimates even without historical data, the weight parameters can be adjusted. A simple and easily implemented initialization method is adopted. The computing system can perform a lightweight warm-up on the first few frames (e.g., 5-10 frames) at startup, calculating the average rendering time, GPU utilization, and motion intensity. This data, combined with the reference time of each frame interpolation strategy, generates a set of initial weights. An example initialization configuration is as follows:

[0056] - (Constant term): Set as the typical base time for a lightweight frame interpolation strategy, for example, 0.3 to 0.5 ms, to provide initial bias;

[0057] - (Rendering time correlation): Initialize to a small positive value, such as 0.02 to 0.1, to reflect the mild impact of the rendering stage on frame interpolation cost;

[0058] - (Relevance of frame interpolation time in the previous frame): Initialize it with a medium weight, such as 0.2 to 0.4, so that the model has a certain degree of inter-frame smoothness in the initial stage;

[0059] - (GPU utilization weight): can be set to 0.2 to 0.3, used to express the basic contribution of hardware load to frame interpolation time;

[0060] - (Motion intensity weight): Initialized to a relatively high positive value, such as 0.4 to 0.6, to emphasize that motion complexity is one of the main sources of overhead in frame interpolation computation.

[0061] It should be understood that the above-mentioned weight initialization values ​​are merely examples and should not be considered as restrictive requirements on the technical solutions disclosed herein. The calculation model can be updated online in subsequent runs based on the error between the actual time consumed and the predicted time for each frame interpolation calculation. Continuous calibration is performed to gradually adapt the model to the actual performance of different rendering scenarios and hardware environments. Typically, the model will achieve initial convergence within a few dozen frames, thereby obtaining more stable prediction capabilities.

[0062] According to some embodiments of this disclosure, obtaining the prediction time for frame interpolation calculation of the current rendering frame before the rendering of the current rendering frame begins may include predicting multiple frame interpolation modes with different calculation precisions separately, so as to obtain the prediction time for each frame interpolation mode. In other words, This can be a vector or tensor, where one or more components can be associated with a corresponding frame interpolation mode. In some examples, different prediction models and / or weight parameters can be used for different frame interpolation modes. This allows for a more precise comparison of the matching relationship between different frame interpolation modes and the budget time during the frame interpolation scheduling phase, avoiding overly conservative or overly aggressive decisions due to a single prediction model. Furthermore, modeling and maintaining each frame interpolation mode separately helps improve the stability and timeliness of frame interpolation mode switching, thereby further reducing frame rate fluctuations and improving overall display smoothness while meeting real-time constraints.

[0063] In step 106, based on the predicted time and the budgeted time, an interpolation mode with different computational accuracies is selected from multiple interpolation modes to perform interpolation calculations on the current rendered frame, and the actual time spent performing the interpolation calculation on the current rendered frame is recorded. According to some embodiments of this disclosure, multiple interpolation modes with different computational accuracies can be constructed by dividing the entire interpolation process into several adjustable modules—including but not limited to: optical flow estimation, motion field reconstruction, occlusion detection, image inpainting, and interpolation ratio. By enabling, disabling, or combining the above functional modules according to their computational complexity, multiple interpolation modes with different computational accuracies and computational overheads can be formed, thereby adapting to different trade-offs between image quality and frame time stability requirements.

[0064] In some embodiments, the interpolation time of the current rendered frame is predicted. Then, the computing system can predict the time consumption. With budget time The comparison is performed, and the appropriate frame interpolation mode is selected based on the predicted time interval. Budget time. It can be determined based on one or more factors, such as hardware performance (e.g., GPU performance), latency requirements, and whether the rendering pipeline supports parallel execution.

[0065] In one example, the predicted latency is obtained by mapping runtime features to static costs that combine different patterns. The frame interpolation mode to be selected can be determined using the following lookup table-based method:

[0066] Table 1. Exemplary Complexity Mapping Relationships

[0067]

[0068] In some embodiments, an interpolation mode can be selected from a plurality of interpolation modes based on the prediction time determined for each interpolation mode separately. For example, its prediction time and budget time can be selected. The closest interpolation mode will be used as the interpolation mode.

[0069] According to some embodiments of this disclosure, in some frame interpolation modes, the computational load can be significantly reduced by temporarily disabling optical flow calculation and reducing the number of repair layers, so that the computing system can maintain stable frame time in high dynamic scenes and avoid large jitter.

[0070] For example, when the prediction time is significantly shorter than the budgeted time, the computational system can choose a higher complexity configuration. In this case, optical flow estimation runs with higher accuracy, motion vector inpainting maintains the complete hierarchy, and occlusion detection is also enabled. Due to the lighter GPU load, the computational system can maintain a 2x interpolation rate to maximize image smoothness.

[0071] As prediction time approaches the budgeted time, the computing system can proactively reduce some costly operations. For example, optical flow estimation can stop using the highest resolution and instead use downsampling, while also reducing the number of motion field patching layers. Occlusion detection can remain enabled, but the threshold will be appropriately tightened. The goal of this mode is to maintain a comfortable visual experience while avoiding excessive load. If the prediction load is already close to the budget limit, the computing system can enter a protection mode. In this mode, optical flow calculations can be stopped, and only motion vectors and depth information from the game rendering pipeline are retained to generate interpolated frames using this lighter field data. The patching stage can retain only necessary edge filling, while occlusion detection can be temporarily disabled. To further reduce computational pressure, the computing system can reduce the interpolation factor from 2x to between 1x and 1.5x, keeping the output frame time within a controllable range. This mode is particularly suitable for high-dynamic scenes such as explosion effects and fast camera transitions, effectively avoiding drastic frame time fluctuations.

[0072] When the computing system detects that the load has exceeded the budget, it can enter the lowest complexity fast mode. In this mode, optical flow estimation, image inpainting, and occlusion detection can all be turned off, while only the most basic motion field reconstruction is retained, and the frame interpolation rate is fixed at 1x to prioritize maintaining frame rate stability.

[0073] To avoid frequent switching between different modes, some embodiments of this disclosure also introduce hysteresis logic. In some examples, when the selection of the interpolation mode involves switching from a mode with lower computational precision to a mode with higher computational precision, the decision to perform the switch can be further determined based on the prediction results of the previously rendered frames by the complexity prediction model; while when the selection of the interpolation mode involves switching from a mode with higher computational precision to a mode with lower computational precision, the switch can be performed directly. For example, upgrading (i.e., increasing to a higher complexity) only occurs when the conditions are met for multiple consecutive frames (e.g., 5 to 10 consecutive frames), while downgrading only requires a short trigger time. In this way, the computing system can naturally improve image quality in light-load scenarios while quickly reducing complexity in high-load scenarios, maintaining the smoothness of the overall performance.

[0074] Through the aforementioned mode selection mechanism based on prediction time, the computing system can dynamically trim high-cost calculations in the frame interpolation process according to scene changes, thereby significantly reducing frame time fluctuations while ensuring visual continuity, making frame interpolation more robust in real-time game environments.

[0075] According to some embodiments of this disclosure, after the frame-level mode is determined, the computing system can further divide the screen spatially and assign different interpolation complexities to different regions based on factors such as the motion amplitude, occlusion, and depth changes of each local region. This is because rendered images often exhibit significant local differences. For example, foreground characters, explosion effects, or fast-moving objects may require higher interpolation precision, while background or static areas do not require excessive computation. Performing the same high-complexity interpolation across the entire screen not only wastes GPU resources but also easily leads to sudden increases in frame time, resulting in jitter. Therefore, by adding fine-grained spatial-level control to the frame-level mode, the technical solution of this disclosure enables the computing system to differentiate and process different regions.

[0076] For example, at the beginning of each frame, the computing system can divide the screen into several regular grids, such as 16×9 or 8×8 tile layouts. For each tile, the computing system can estimate the current motion intensity and complexity of the region based on the motion vectors, depth gradients, and occlusion masks from the previous frame. If a tile has a high motion speed, a clear object outline, or significant occlusion changes, the computing system can allocate higher interpolation complexity to that region, such as continuing to use higher-precision motion vector reconstruction, enabling a full patching step, or retaining local optical flow estimation. For background regions that do not change much and are basically static, the computing system can automatically reduce interpolation complexity, such as directly using game motion vectors from previous and next frames, not enabling high-cost patching steps, or even directly copying pixel values ​​from neighboring frames in fast mode. For example, the motion level of a tile or region can be determined based on motion vectors as follows: read the motion vectors from the previous frame; for each tile, calculate the average motion vector magnitude of all pixels within that tile; in response to determining that the average motion vector magnitude of the tile is higher than a preset threshold (e.g., 1.5 times the global average motion vector), identify the tile as a high-motion region. In this way, high-dynamic tiles can receive more computational budget, while low-dynamic regions yield time, thus keeping the total frame time within the budgeted time.

[0077] For example, in some scenarios, the following block types and configurations can be preset:

[0078] 1) High-speed motion of characters: Enable local patching to maintain high motion field accuracy;

[0079] 2) Static background tiles: No patching or occlusion processing is performed; motion vectors or copied pixels are directly used from the rendering engine.

[0080] 3) Smoke effect tiles: Unstable occlusion, partial repairs retained but update range limited;

[0081] 4) Distant vehicle tiles: Fast lateral movement, using a more reliable motion vector source to avoid artifacts.

[0082] In some embodiments, the estimated time can be accumulated based on the priority of the blocks to ensure that the total time does not exceed the budget.

[0083] It is worth noting that this spatial allocation is particularly crucial when frame-level modes tend to employ conservative strategies (e.g., in protected mode). For instance, when global optical flow calculation is disabled, highly motionous regions can still retain a small amount of local patching to avoid noticeable artifacts; while stationary regions can be processed in the lightest possible way, further reducing overall processing time. In lighter, high-precision modes, spatial allocation remains effective because even with all modules enabled, the precision of different regions can still be differentiated as needed, allowing computational resources to be prioritized for regions requiring more detail.

[0084] The entire spatial allocation process works in conjunction with the frame-level mode: the frame-level mode determines the overall complexity level, while the spatial allocation further refines the budget within this overall framework, ensuring that frame interpolation possesses both global consistency and local adaptability. Because the tile allocation strategy is combined with the prediction features of the previous frame, the technical solution provided in this disclosure avoids the phenomenon of "sudden cost spikes" that occurs in traditional global frame interpolation in high-dynamic scenes, resulting in smoother frame interpolation computation time consumption while reducing artifacts and misalignment issues.

[0085] According to some embodiments of this disclosure, spatial complexity can be allocated for different frame interpolation modes through the following operations: dividing the current rendering frame into multiple regions based on the selected frame interpolation mode; determining the computational complexity of each region based on the motion vectors, depth gradients, and occlusion masks within that region; and fine-tuning the configuration of frame interpolation computation for each region under the selected frame interpolation mode based on the determined computational complexity of each region. For example, in high-precision mode, where system resources are relatively abundant, a finer-grained spatial partitioning can be used, such as 16×9 or even 32×18; in balanced mode, since the computing system needs to compromise between image quality and performance, a moderate number of tiles can be used, such as 8×8; in protected mode, a 4×4 or 6×3 tile configuration can be used; and in fast mode, a 2×2 tile configuration can be used, or even tile partitioning can be omitted.

[0086] In some examples, fine-tunable configurations may include: whether to perform image inpainting, whether to perform occlusion detection, whether to perform optical flow estimation, the number of iterations for optical flow estimation, the resolution of optical flow estimation, and whether to perform motion field reconstruction.

[0087] Through this spatial-level dynamic complexity allocation mechanism, the technical solution provided in this disclosure can significantly improve the stability of frame interpolation under the same budget, enabling the system to maintain low frame time fluctuations when facing complex and rapidly changing game scenes, thereby achieving a more stable real-time frame interpolation effect.

[0088] Furthermore, as recognized in this disclosure, the measured actual execution time is not only used to determine whether the current frame meets the budget, but also serves as a crucial input to the complexity prediction model. If there is a significant deviation between the actual and predicted execution time, this deviation reflects errors in the model under certain scenarios or strategy combinations, requiring adjustment through subsequent feedback mechanisms. The system can record the runtime of each major stage in detail during frame interpolation and write this time data, along with the scene features of the current frame, into a runtime buffer for use as input for subsequent complexity prediction model updates and mode selection.

[0089] In step 108, the complexity prediction model is updated based on the difference between the actual time consumption and the predicted time consumption. After completing the frame interpolation calculation for the current rendered frame and obtaining the actual interpolation time consumption, the system can compare the actual time consumption with the predicted time consumption given by the complexity prediction model to obtain the prediction error of the current rendered frame. In some embodiments, in order to make the model gradually conform to the real computational cost under different scenarios during long-term operation, the model parameters can be adjusted online. For example, the weight parameters can be updated using exponential sliding or least squares iterative correction.

[0090] For example, the actual frame interpolation time measured for the current rendered frame. The prediction error can be defined as:

[0091] .

[0092] To allow the model to gradually approximate reality, the system can use this error to perform an incremental update of the weight parameter vector. The update rule can take the form of a simple gradient, and can be written as:

[0093] ,

[0094] in The learning rate is typically a small positive number used to control the magnitude of parameter updates.

[0095] Through these updates, the complexity prediction model can continuously adjust its parameters during operation, gradually adapting to actual hardware performance and scene characteristics. For example, in high-motion, high-occlusion scenes, if the impact of certain features on time consumption increases, this impact will gradually be reflected in the weight changes during the cumulative error updates over multiple frames. As the number of frames increases, the complexity prediction model can reach a stable state, making the predicted time closer to the actual time consumed by the computing system.

[0096] The aforementioned online update mechanism enables the complexity prediction model to automatically adapt to different game content, different screen complexities, and different hardware models without offline training, thereby providing a more accurate reference for subsequent frame-level strategy selection.

[0097] By continuously updating the weight parameters of the complexity prediction model during operation, the technical solution provided in this disclosure enables the model to adapt to real-time performance changes in different scenarios. In this way, the complexity prediction model can be based on the actual frame interpolation time of each frame. Compared with the predicted value The model gradually adjusts its parameters based on the differences between frames, achieving adaptive online learning. After multiple frame iterations, the model gradually converges, ensuring that the prediction results are consistent with the actual performance of the system.

[0098] According to some embodiments of this disclosure, the method for dynamically adjusting frame interpolation also involves anomaly detection and fallback strategies. For example, in addition to the normal prediction and scheduling process, the computing system can also run an anomaly detection logic to handle large load changes or scene abrupt changes that occur within a short period of time. When anomaly conditions are detected, the computing system preferentially switches to a predefined fallback configuration in subsequent frame interpolation calculations. The fallback configuration can typically correspond to a fixed subset of the fast mode: optical flow estimation and multi-layer patching are disabled, only basic motion field reconstruction and color interpolation are retained, the frame interpolation factor is set to 1x, and the image of the previous or next frame is used directly as the output when necessary. The fallback state can continue for a preset number of frames, or until the time taken for several consecutive frames returns to within the budget, after which the system reverts to the normal mode determined by the complexity model and mapping table.

[0099] In some embodiments, the anomaly detection and fallback strategy can be implemented by the following operation: in response to the detection of an anomaly, frame interpolation calculation is performed using a preset fallback strategy until the anomaly exit condition is met, wherein the anomaly includes at least one of the following: the actual time taken for frame interpolation calculation of multiple consecutive rendering frames is higher than the budgeted time, the rate of change of the utilization of the graphics processing unit in a given time period exceeds a preset condition, the change of the occlusion mask between adjacent rendering frames exceeds a predetermined size, or an abnormal distribution occurs in the motion vector field.

[0100] In some embodiments, abnormal exit conditions may include the actual time taken for interpolation calculations in multiple consecutive rendered frames being lower than the budgeted time and / or the decrease in global motion intensity between adjacent rendered frames exceeding a predetermined amount. For example, if the actual interpolation time is lower than the budgeted time for 5 to 10 consecutive frames, the load can be considered to have stabilized. Another example is global motion intensity... If the load has decreased by more than 20% compared to the mean of the previous stable period, it can be considered that the load has returned to stability.

[0101] In summary, the technical solution provided in this disclosure, through a layered, switchable, and adaptive design of the frame interpolation process, enables the computing system to flexibly adjust the frame interpolation complexity according to the actual load, thereby significantly reducing frame time fluctuations and improving the stability of frame interpolation in rendering scenarios, including real-time games. This solution maintains consistent performance under different hardware environments and rendering content, making it suitable for various real-time rendering applications. Furthermore, through anomaly detection and fallback configuration, the frame interpolation process can still simplify its workflow according to clear rules when encountering extreme scenarios or large prediction errors, avoiding uncontrollable computational paths. This mechanism, together with the aforementioned complexity prediction, frame interpolation mode selection, and spatial allocation, constitutes a complete dynamic control closed loop.

[0102] According to one aspect of this disclosure, a computing device is provided, including a memory, at least one processor, and instructions stored in the memory, which, when executed by the at least one processor, cause the computing device to perform the steps of any of the method embodiments described above.

[0103] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the method embodiments described above.

[0104] According to one aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of any of the method embodiments described above.

[0105] In the following text, combined with Figure 2 Illustrative examples describing such computing devices, non-transitory computer-readable storage media, and computer program products.

[0106] Figure 2 An example configuration of a computing device 200 that can be used to implement the methods described herein is shown.

[0107] The computing device 200 can be a variety of different types of devices. Examples of the computing device 200 include, but are not limited to: desktop computers, server computers, laptop or netbook computers, mobile devices (e.g., tablet computers, cellular or other wireless phones (e.g., smartphones), notebook computers, mobile stations), wearable devices (e.g., glasses, watches), entertainment devices (e.g., entertainment appliances, set-top boxes communicatively coupled to a display device, game consoles), televisions or other display devices, automotive computers, and so on.

[0108] The computing device 200 may include at least one processor 202, memory 204, multiple communication interfaces 206, display device 208, other input / output (I / O) devices 210, and one or more mass storage devices 212 capable of communicating with each other, such as via system bus 214 or other suitable connections.

[0109] Processor 202 may be a single processing unit or multiple processing units, and all processing units may include single or multiple computing units or multiple cores. Processor 202 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Among other capabilities, processor 202 may be configured to acquire and execute computer-readable instructions stored in memory 204, mass storage device 212, or other computer-readable media, such as program code of operating system 216, program code of application program 218, program code of other program 220, etc.

[0110] Memory 204 and mass storage device 212 are examples of computer-readable storage media for storing instructions that are executed by processor 202 to perform the various functions described above. For example, memory 204 can generally include both volatile and non-volatile memory (e.g., RAM, ROM, etc.). Furthermore, mass storage device 212 can generally include hard disk drives, solid-state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network-attached storage, storage area networks, etc. Both memory 204 and mass storage device 212 can be collectively referred to herein as memory or computer-readable storage media, and can be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code, which can be executed by processor 202 as a specific machine configured to perform the operations and functions described in the examples herein.

[0111] Multiple programs can be stored on mass storage device 212. These programs include operating system 216, one or more applications 218, other programs 220, and program data 222, and they can be loaded into memory 204 for execution. Examples of such applications or program modules may include, for example, Figure 1 The computer program logic (e.g., computer program code or instructions) of the methods shown and / or other embodiments described herein.

[0112] Although Figure 2 The modules 216, 218, 220, and 222, or portions thereof, are illustrated as being stored in memory 204 of computing device 200; however, modules 216, 218, 220, and 222 may be implemented using any form of computer-readable medium accessible by computing device 200. As used herein, “computer-readable medium” includes at least two types of computer-readable media: computer-readable storage media and communication media.

[0113] Computer-readable storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD, or other optical storage devices, magnetic cassettes, magnetic tapes, disk storage devices or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing device. In contrast, communication media can embody computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms. Computer-readable storage media as defined herein do not include communication media.

[0114] One or more communication interfaces 206 are used for exchanging data with other devices, such as via a network, direct connection, etc. Such communication interfaces can be one or more of the following: any type of network interface (e.g., a network interface card (NIC)), wired or wireless (such as IEEE 802.11 Wireless LAN (WLAN)) wireless interface, Wi-MAX interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth. TMInterfaces include near-field communication (NFC) interfaces. Communication interface 206 can facilitate communication across various network and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, etc. Communication interface 206 can also provide communication with external storage devices (not shown) such as storage arrays, network-attached storage, storage area networks, etc.

[0115] In some examples, a display device 208, such as a monitor, may be included for displaying information and images to the user. Other I / O devices 210 may be devices that receive various inputs from the user and provide various outputs to the user, and may include touch input devices, gesture input devices, cameras, keyboards, remote controls, mice, printers, audio input / output devices, and so on.

[0116] The techniques described herein can be supported by these various configurations of computing device 200 and are not limited to specific examples of the techniques described herein. For example, the functionality can also be implemented wholly or partially on a “cloud” using a distributed system. A cloud includes and / or represents a platform for resources. The platform abstracts the underlying functionality of the cloud’s hardware (e.g., servers) and software resources. Resources may include applications and / or data that can be used when performing computational processing on servers remote from computing device 200. Resources may also include services provided via the Internet and / or via subscriber networks such as cellular or Wi-Fi networks. The platform can abstract resources and functionality to connect computing device 200 to other computing devices. Therefore, the implementation of the functionality described herein can be distributed throughout the cloud. For example, the functionality can be implemented partly on computing device 200 and partly through a platform that abstracts the functionality of the cloud.

[0117] Although the present disclosure has been described and illustrated in detail in the accompanying drawings and the foregoing description, such description and illustration should be considered illustrative and schematic, not restrictive, and the present disclosure is not limited to the disclosed embodiments.

Claims

1. A method for dynamically adjusting frame interpolation calculation, characterized in that, The method includes: Obtain the feature parameters related to the frame interpolation calculation of the current rendered frame; The feature parameters are input into the complexity prediction model to obtain the predicted time for frame interpolation calculation of the current rendering frame before the rendering of the current rendering frame begins. Based on the predicted time and budgeted time, an interpolation mode is selected from multiple interpolation modes with different computational precisions to perform interpolation calculations on the current rendered frame, and the actual time spent performing the interpolation calculations on the current rendered frame is recorded; and The complexity prediction model is updated based on the difference between the actual time consumed and the predicted time consumed.

2. The method according to claim 1, characterized in that, The characteristic parameters related to the interpolation calculation of the current rendering frame include the rendering time of the previous rendering frame, the actual time spent performing interpolation calculation on the previous rendering frame, the current utilization of the graphics processing unit used for rendering, and the global motion intensity of the previous rendering frame.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Based on the selected frame interpolation mode, the current rendering frame is divided into multiple regions; For each region, the computational complexity of that region is determined based on its motion vectors, depth gradients, and occlusion masks; and Based on the determined computational complexity of each region, the configuration of the frame interpolation calculation for each region is fine-tuned under the selected frame interpolation mode.

4. The method according to claim 3, characterized in that, The configuration for frame interpolation calculation includes one or more of the following: Should image retouching be performed? Should occlusion detection be performed? Should optical flow estimation be performed? The number of iterations for optical flow estimation; Resolution of optical flow estimation; Whether to rebuild the sports field.

5. The method according to claim 1 or 2, characterized in that, in: When the selection of the interpolation mode involves switching from a mode with lower computational precision to a mode with higher computational precision, the decision on whether to perform the switch is further determined based on the prediction results of the previously rendered frames by the complexity prediction model. as well as When the selection of the interpolation mode involves switching from a mode with higher computational precision to a mode with lower computational precision, the switching is performed directly.

6. The method according to claim 1 or 2, characterized in that, The method further includes: In response to the detection of an anomaly, frame interpolation calculations are performed using a preset fallback mode until an anomaly exit condition is met, wherein the anomaly includes at least one of the following: The actual time spent on interpolation calculations for multiple consecutive rendering frames is longer than the budgeted time. The rate of change of the utilization rate of the graphics processing unit within a given time period exceeds a preset condition. The occlusion mask changes more than a predetermined size between adjacent rendering frames; or An anomalous distribution appears in the motion vector field.

7. The method according to claim 6, characterized in that, The abnormal exit conditions include the actual time taken for interpolation calculation of multiple consecutive rendering frames being lower than the budget time and / or the decrease in global motion intensity between adjacent rendering frames exceeding a predetermined amount.

8. A computer-readable storage medium having instructions stored thereon, characterized in that, When the instructions are executed by the processor of a computer, the processor performs the method according to any one of claims 1 to 7.

9. A computer program product comprising a computer program that, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 7.

10. A computing device, characterized in that, include: At least one processor; as well as A memory that stores instructions, which, when executed by the at least one processor, cause the computing device to perform the method according to any one of claims 1 to 7.

Images