Interpolation method and related device

Abstract

This application provides a frame interpolation method and related equipment. In this method, it is determined whether the rendering operation for the target frame is completed within a preset time. If so, a predicted image frame is generated through a frame interpolation thread, and this predicted image frame is used as the target frame for display. This scheme can flexibly perform frame interpolation based on actual needs using an independent frame interpolation thread, without having to interpolate frames in integer multiples of the original frame rate. In other words, the target frame rate can be used as a basis for judging each target frame to be displayed. If, according to the target frame rate, an image needs to be displayed on the target frame, but the rendering operation for the target frame is not completed in time (i.e., the rendering operation for the target frame is not completed within the preset time), then frame interpolation needs to be performed through the frame interpolation thread (i.e., generating a predicted image frame through the frame interpolation thread) to ensure that the actual displayed frame rate remains at the target frame rate, thereby improving image stability and enhancing the user experience.

Description

Technical Field

[0001] This application belongs to the field of display processing technology, and in particular relates to a frame interpolation method and related equipment. Background Technology

[0002] With the development of science and technology, in order to improve the user's gaming experience, the screen resolution of electronic devices such as mobile phones and computers is constantly increasing, and game scenes are becoming increasingly diverse. However, high-resolution screens and complex game scenes require electronic devices to have stronger computing and image processing capabilities, which leads to increased power consumption. Furthermore, since the hardware resources of electronic devices are relatively limited, if the hardware resources cannot provide the corresponding computing and image processing capabilities, it will cause problems such as overheating, shortened battery life, and frame drops in games, which will ultimately reduce the user's gaming experience.

[0003] In order to solve the aforementioned game frame dropping problem, frame interpolation technology has been developed. Frame interpolation is a technology that increases the frame rate by inserting new frames between existing frames, thereby reducing game power consumption and improving game smoothness.

[0004] However, even with frame interpolation, the game may still be overloaded due to an overly rich game scene, or the hardware resources of the electronic device may be too limited, resulting in poor rendering of the game scene and thus reducing the user's gaming experience. Summary of the Invention

[0005] This application provides a frame interpolation method and related device, which can flexibly interpolate frames through an independent frame interpolation thread, thereby maintaining the actual displayed frame rate at the target frame rate, improving the stability of the image and enhancing the user experience.

[0006] Firstly, a frame interpolation method is provided. This method includes: determining whether the rendering operation for the target frame is completed within a preset time; if so, generating a predicted image frame through a frame interpolation thread and sending the predicted image frame as the target frame for display. The target frame is the next image frame to be displayed, or the current frame. The preset time is determined based on the previous image frame and the target frame rate; the predicted image frame is generated based on rendering data, which is the data used by the rendering thread during the rendering of the actual image frame; the frame interpolation thread is an independent thread compared to the rendering thread.

[0007] In the above scheme, frame interpolation can be performed flexibly according to actual needs based on an independent frame interpolation thread, rather than having to interpolate frames in integer multiples of the original frame rate. That is, the target frame rate can be used as a basis to determine the target frame to be displayed. If the target frame rate is required to display an image, but the rendering operation for the target frame is not completed in time (i.e., the rendering operation for the target frame is not completed within the preset time), then frame interpolation needs to be performed by the frame interpolation thread (i.e., the frame interpolation thread generates a predicted image frame) to ensure that the actual displayed frame rate is maintained at the target frame rate, thereby improving the stability of the image and enhancing the user experience.

[0008] For example, in one application scenario, when a target frame experiences passive frame drops, the frame interpolation thread can determine that the rendering operation of the target frame did not complete within a preset time, and thus determine to interpolate the target frame. This method can stabilize the actual frame rate to the target frame rate without wasting resources by adding extra frames, thus avoiding increased power consumption and the generation of new dropped frames. As another example, in scenarios with high load, the target frame can be actively dropped to reduce rendering thread blocking. The frame interpolation thread can determine that the rendering operation of the target frame did not complete within a preset time, and thus determine to interpolate the target frame. Since the power consumption of generating predicted image frames through frame interpolation is lower than the power consumption of rendering the actual image frames, this method can achieve the effect of reducing power load while maintaining the same frame rate experience. For example, in scenarios where a higher frame rate is desired (such as when the frame rate is insufficient), an arbitrary target frame rate can be set as needed, instead of setting the target frame rate as an integer multiple of the original frame rate. After setting the actual frame rate to the target frame rate, the frame interpolation thread can determine, based on the rendering instructions and / or synchronization objects, that the rendering operation of the target frame has not been completed within the preset time, i.e., the target frame is a missing frame. Therefore, the frame interpolation thread can fill in the target frame to increase the actual frame rate to the target frame rate.

[0009] In conjunction with the first aspect, in some possible implementations of the first aspect, determining whether the rendering operation for the target frame is completed within a preset time includes: determining whether the rendering instruction for the target frame has been issued within the preset time; and if the rendering instruction has not been issued within the preset time, determining that the rendering operation for the target frame has not been completed within the preset time.

[0010] In the above scheme, the completion of the rendering operation for the target frame within the preset time can be determined by whether the rendering command for the target frame is issued on time. If the rendering command for the target frame has not been issued within the preset time after the previous frame has been rendered or displayed, it is determined that the CPU may be blocked. In this case, frame interpolation rendering operation needs to be performed through a frame interpolation thread. At the same time, the system can also know that the current frame rate problem is caused by CPU blocking, so that targeted resource optimization can be performed.

[0011] In conjunction with the first aspect, in some possible implementations of the first aspect, determining whether the rendering operation for the target frame is completed within a preset time includes: determining whether the synchronization object corresponding to the target frame can be found within the preset time; if the synchronization object corresponding to the target frame cannot be found within the preset time, determining that the rendering operation for the target frame has not been completed within the preset time.

[0012] In the above scheme, the completion time of the rendering operation for the target frame can be determined by whether a synchronization object for the target frame can be found. If a synchronization object is found, it means the target frame has been rendered, and frame interpolation is unnecessary. If no synchronization object is found, it indicates that the GPU may be blocked, requiring frame interpolation rendering via a separate thread. Simultaneously, the system can also identify that the current frame rate issue is caused by GPU blocking, allowing for targeted resource optimization.

[0013] Understandably, in one implementation, it can be first determined whether the rendering command for the target frame has been issued. If not, there is no need to query the corresponding synchronization object, thus saving resources. However, if the rendering command for the target frame has been issued, the corresponding synchronization object can be queried. In this way, it can be determined whether the rendering blockage is caused by the CPU or the GPU, allowing the system to perform targeted resource optimization. The specific process is not limited in this application.

[0014] In conjunction with the first aspect, in some possible implementations of the first aspect, generating a predicted image frame through a frame interpolation thread includes: determining whether the current number of consecutive frame interpolations is greater than or equal to a preset value; and generating a predicted image frame through a frame interpolation thread if the number of consecutive frame interpolations is less than the preset value.

[0015] In the above scheme, the frame interpolation rendering operation will only continue if the number of consecutive interpolated frames is less than the preset value. The purpose of doing this is to reduce the image quality problems such as motion estimation errors and jitter and discontinuity in the final generated image caused by too many consecutive interpolated frames.

[0016] In conjunction with the first aspect, some possible implementations of the first aspect also include: sending and displaying real image frames via a frame interpolation thread.

[0017] In the above scheme, a separate frame interpolation thread is responsible for displaying both the actual image frames and the predicted image frames. By using a separate frame interpolation thread for displaying, the thread can be directly bound to the rendering target surface used by the rendering thread to handle the final display of the content rendered by both threads. This eliminates the need to create a new local window system visible surface or for the two threads to alternately use the same surface, thus reducing unnecessary memory usage or performance overhead.

[0018] In conjunction with the first aspect, in some possible implementations of the first aspect, after sending the predicted image frame as the target frame for display, the method further includes: putting the frame interpolation thread into a sleep waiting state.

[0019] The time taken for the frame interpolation thread to execute the scheme described in the first aspect is usually much shorter than the frame sending interval between two adjacent frames. Therefore, in the current scenario, the frame interpolation thread is idle most of the time. Thus, the frame interpolation thread can enter a sleep waiting state during non-working hours to reduce the CPU time slice occupation and performance and power consumption loss.

[0020] In conjunction with the first aspect, in some possible implementations of the first aspect, the method further includes: determining the time for the frame interpolation thread to enter a sleep waiting state based on the target frame rate and the working duration of the frame interpolation thread between two adjacent frames.

[0021] In conjunction with the first aspect, in some possible implementations of the first aspect, the target frame rate is the initially set frame rate, or the frame rate increased based on the initially set frame rate according to business requirements.

[0022] The solution provided in this application can be applied to scenarios with passive frame drops, scenarios with active frame drops, and scenarios that flexibly improve the frame rate (see details). Figure 6 (corresponding description), therefore, it has a wide range of applications.

[0023] In a second aspect, an electronic device is provided, including a memory and a processor, the memory storing a computer program executable on the processor, wherein when the processor executes the computer program, the electronic device performs the steps of the method as described in any of the first aspects above.

[0024] Thirdly, a computer-readable storage medium is provided that stores a computer program, which, when executed by a processor, implements the steps of the method as described in any of the first aspects above.

[0025] Fourthly, a computer program product is provided that, when run on an electronic device, causes the electronic device to perform the method described in any one of the first aspects.

[0026] Fifthly, a chip system is provided, the chip system including a processor coupled to a memory, the processor executing a computer program stored in the memory to implement the method described in any one of the first aspects above.

[0027] The chip system can be a single chip or a chip module composed of multiple chips.

[0028] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0029] Figure 1 A schematic diagram illustrating the application scenarios provided in the embodiments of this application is shown;

[0030] Figure 2 A schematic diagram of no frame interpolation, single superframe, and multiple superframe is given;

[0031] Figure 3 This is an architecture diagram of a multi-threaded rendering technology;

[0032] Figure 4 The diagram illustrates the operation of the logic thread and the rendering thread in different frame interpolation scenarios.

[0033] Figure 5 The diagram illustrates the application of the frame interpolation thread in single-interpolation and multi-interpolation scenarios.

[0034] Figure 6 This diagram illustrates another application of the frame interpolation thread in single-interpolation and multi-interpolation scenarios.

[0035] Figure 7 An exemplary flowchart of the frame interpolation method 100 provided in an embodiment of this application is shown;

[0036] Figure 8 A schematic diagram illustrating the principle of determining sleep waiting time is shown;

[0037] Figure 9 This illustrates one possible implementation of frame interpolation and display.

[0038] Figure 10 This demonstrates another possible implementation of frame interpolation and display;

[0039] Figure 11 This illustration shows a schematic diagram of the frame interpolation scheme and the display scheme provided in an embodiment of this application.

[0040] Figure 12 A schematic diagram of a dual-queue method provided in an embodiment of this application;

[0041] Figure 13 A schematic diagram of a circular queue method provided in an embodiment of this application;

[0042] Figure 14 This is a hardware architecture diagram provided for an embodiment of this application. Detailed Implementation

[0043] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0044] With the development of technology, various electronic devices are becoming increasingly common, and many of these devices are equipped with screens. When a user uses an application on an electronic device, the device can display the corresponding user interface to the user through the screen. For example, when an electronic device opens a game application based on a user's instruction, it can display real-time game footage. Similarly, when an electronic device opens a live streaming application based on a user's instruction, it can display live video feeds.

[0045] It is understood that the electronic device involved in the embodiments of this application can be any electronic device with a screen, on which an operating system (such as Android) is installed, and various applications can be installed based on the operating system. The electronic device in this application can also be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. As specific examples, the electronic device can be a smartphone, smart TV, wearable device, tablet computer, computer with wireless transceiver function, virtual reality (VR) electronic device, augmented reality (AR) electronic device, wireless terminal in industrial control, wireless terminal in self-driving, wireless terminal in remote medical surgery, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, wireless terminal in smart home, etc. The embodiments of this application do not limit the specific technology or specific device form used in the electronic device.

[0046] When electronic devices display dynamic visuals such as real-time game footage and live video streams, the smoothness and continuity of the image are crucial to the user experience. Dynamic visuals are typically created by electronic devices continuously rendering and rapidly playing frame by frame. Therefore, the higher the number of frames displayed per second on the screen, the smoother the visuals. Related technologies commonly use frame rate to represent the number of frames displayed per second in a video or animation. Frame rate is a key indicator for measuring the smoothness of video or animation playback; a higher frame rate directly affects the user's perception of motion continuity, thus impacting the visual quality and viewing experience of the video or animation. The following section combines... Figure 1 The game scene shown is illustrated as an example.

[0047] During operation, game applications render and display frames of images continuously and rapidly. A single image frame is a static image displayed by the game application. Each static image frame can be considered as composed of scene elements and UI elements. Scene elements constitute the main scene of a frame, which are usually fixed and uncontrollable. UI elements constitute the user interface controls of a frame, which users can typically control by clicking on the screen area where the UI elements are located. Scene elements include, but are not limited to, in-game scenery, game characters, background objects, and special effects. UI image elements include, but are not limited to, control buttons, minimaps, and floating text. In some games, the character's health bar is also included within the UI image. Figure 1 (a) to Figure 1 (d) in the diagram illustrates a game scene where an electronic device displays four consecutive frames on its screen. These four frames, played sequentially, create a coherent dynamic visual. It is understandable that... Figure 1 The provided examples only represent a portion of the images that might be displayed in a game scene. In real-world applications, electronic devices may display more than what's shown. Figure 1 (a) before and / or Figure 1 More images will be displayed after (d) in the text; no limit is specified here.

[0048] For each frame of an image, the game engine in an electronic device performs operations such as scene updates, object position calculations, lighting effect calculations, and collision detection, ultimately generating a complete image. By continuously displaying these images on the screen in a certain order and at a certain rate, dynamic game visuals are created. In other words, by rendering images frame by frame, electronic devices can present vivid and smooth visual effects in games, and the higher the frame rate, the smoother the visuals.

[0049] However, high-resolution screens and complex game scenes require electronic devices to provide more computing and image processing capabilities, leading to increased power consumption. Furthermore, because electronic devices have limited hardware resources, if these resources cannot provide the corresponding computing and image processing capabilities, issues such as overheating, shortened battery life, and frame drops in games will occur, severely degrading the user experience.

[0050] For example in Figure 1 In the game scenario shown, when multiplayer battles occur, factors such as game graphics, physics calculations, and artificial intelligence logic processing become extremely complex, leading to excessive game load. If the electronic device cannot provide the corresponding computing and image processing capabilities, it can cause problems such as overheating, shortened battery life, and potential frame drops. For example, if the electronic device wants to display... Figure 1The four consecutive game frames shown in (a) to (d) are shown in the image. However, due to resource limitations of electronic devices, one of the frames, such as... Figure 1 The second frame shown in (b) failed to display, i.e., a frame drop occurred. Therefore, from the user's perspective, the image displayed on the screen suddenly changed from... Figure 1 (a) in the middle jumps to Figure 1 In section (c), the screen display is not smooth enough, resulting in a poor user experience.

[0051] In related technologies, to solve the problem of frame drops when electronic devices display dynamic images, frame interpolation technology has been developed. Frame interpolation, also known as frame supplementation or superframe, refers to a method of adding a new frame between two frames to increase the frame rate. It can effectively reduce power consumption in games and improve game smoothness. Using frame interpolation technology, only a portion of the actual image frames need to be acquired, and the predicted image frames are calculated based on the actual image frames, thereby increasing the frame rate. Therefore, frame interpolation technology can also be called frame interpolation technique.

[0052] It is understood that the real image frames involved in the embodiments of this application refer to the original image frames that have been normally drawn, rendered, composited, and displayed, and can be simply referred to as real frames or original frames. The concept of real image frames is used to distinguish them from predicted image frames, which refer to newly added image frames obtained by interpolation calculations using one or more real image frames, and can be simply referred to as predicted frames or interpolated frames. Frame interpolation techniques include frame rate conversion (FRC) technology, motion estimate and motion compensation (MeMC) technology, etc.

[0053] It's important to note that frame interpolation typically includes two methods: interpolation and extrapolation. Interpolation generates a new intermediate frame between two known frames; that is, it calculates the intermediate frame based on the two existing frames. This method usually achieves this by interpolating pixel information between existing frames to fill the gaps between them, thus improving the smoothness and coherence of the image. As an example, the process of generating a predicted image frame using interpolation involves: calculating the motion vector between the preceding and following real image frames, and then shifting pixels or pixel blocks in the real image frame based on this motion vector. Typically, pixels or pixel blocks shifted out of the image range are discarded, while pixels shifted into the image range from outside require special processing, such as pixel padding (image completion) or filtering. Extrapolation, on the other hand, generates a new frame outside of an existing frame sequence; that is, it infers future frames based on already generated frames. This method usually requires analyzing and predicting the existing frame sequence to infer the content of future frames. Extrapolation frame technology is commonly used in applications such as video prediction, motion capture, and motion estimation. It can extend existing frame sequences to generate longer video sequences.

[0054] Based on the different interpolation ratios, frame interpolation techniques can be divided into single frame interpolation and multi-frame interpolation.

[0055] Single-frame interpolation is a simple technique that inserts a predicted image frame between two adjacent original frames, making the motion smoother between frames. Its main goal is to increase the frame rate of videos or games, thereby improving visual effects, but it generates relatively few interpolated frames. Multi-frame interpolation, on the other hand, inserts two or more predicted image frames between two adjacent original frames to capture motion in greater detail and improve the visual effects of videos or games. This method may be more effective when dealing with fast motion or requiring higher frame rates, but it also increases the computational burden.

[0056] Figure 2A schematic diagram is provided for no frame interpolation, single-time superframe, and multi-time superframe scenarios. For example, without frame interpolation, the electronic device generates real image frames at 40fps (only a portion of the image frames are shown in the diagram). In this case, the time interval between two real image frames is 25ms. The electronic device uses single-time frame interpolation to generate predicted image frames, uniformly inserting one predicted image frame between every two real image frames. In this case, the time interval between every two image frames (including real and predicted image frames) is 12.5ms, meaning the electronic device refreshes image frames at 80fps. The electronic device uses multi-time frame interpolation to generate predicted image frames, uniformly inserting n (n is a positive integer greater than or equal to 2) predicted image frames between every two real image frames. In this case, the time interval between every two image frames (including real and predicted image frames) is 25 / (n+1)ms, meaning the electronic device refreshes image frames at 40×(n+1)fps, where n is a multiple of the interpolation rate; for example, in a double-time interpolation scenario, n is 2.

[0057] Depend on Figure 2 As shown in the example, frame interpolation technology can make the frame rate of the displayed image increase by a fixed multiple, that is, within a fixed time period, the number of predicted image frames and the number of real image frames are in a fixed proportional relationship.

[0058] In one possible implementation, multi-threaded rendering technology can be used to render both the original image frames and the predicted image frames. Multi-threaded rendering is an optimization technique widely used in games and other scenarios, aiming to improve rendering efficiency through parallel processing. The core idea of ​​this technique is to allocate different rendering tasks to different threads, such as logic threads, rendering threads, and render hardware interface (RHI) threads, thereby improving rendering efficiency.

[0059] Taking a game scenario as an example: To improve the performance and responsiveness of game applications, multiple threads are used to handle different tasks. The logic thread is primarily responsible for handling the core game logic, including player input, game state updates, and AI calculations. The logic thread typically works closely with the main thread to ensure that the game's state and behavior meet expectations. The audio thread handles the game's audio playback and sound effects. The rendering thread is primarily responsible for rendering the game scene, such as drawing game scenes, characters, and special effects. In a game, the rendering thread can be responsible for converting the 3D model of the game scene into a 2D image and displaying it on the screen of an electronic device. In some examples, the rendering thread typically uses graphics libraries or engines for rendering the game scene, such as OpenGL, DirectX, Unity, Unreal Engine, Vulkan, and Metal. These libraries and engines provide various tools and functions that enable the rendering thread to efficiently complete rendering tasks. By using a separate rendering thread, computationally intensive rendering tasks can be separated from the main thread, reducing the main thread's burden and improving the game's rendering performance.

[0060] Figure 3 This is an architecture diagram of a multi-threaded rendering technology, which uses the parallel operation of logic threads and rendering threads as an example. Figure 3 As shown, during the rendering task, the main thread creates two child threads (the logic thread and the rendering thread). The logic thread performs logic initialization, and the rendering thread initializes the rendering context. After the two child threads complete their initialization and all three threads reach the barrier, the logic thread updates resources and generates a queue of rendering commands through steps such as the Mono::init() function callback, swapping rendering queues, and the Mono::Update() function callback. Subsequently, the rendering thread consumes rendering commands and renders the corresponding game screen. Furthermore, after the main thread updates the barrier, the main thread, logic thread, and rendering thread synchronize again via the barrier. Then, subsequent game scene rendering steps are performed.

[0061] Figure 4 This diagram illustrates the operation of the logic thread and rendering thread under different frame interpolation scenarios. Please refer to [link / reference]. Figure 4 In (a) of the example: In scenarios without frame interpolation, the logic thread is responsible for performing logical operations, while the rendering thread is responsible for rendering the actual image frames. The following is an illustrative example of the process (using a game scene as an example):

[0062] During game application operation, after generating a rendering request for real image frame #1, the logic thread performs logical operations. Specifically, the logic thread can perform logical operations through a computing unit, which can be a central processing unit (CPU). These logical operations include responding to input from peripheral devices, processing application message data, and pre-calculating data required for rendering, among other things.

[0063] After the logical operations are completed, the logic thread sends the calculated image resources to the rendering thread for rendering. In a game scene, this rendering thread refers to the game native rendering thread, which is the thread in the game engine responsible for performing rendering operations on the game screen. It is usually an independent thread dedicated to handling graphics rendering-related tasks, including geometry processing, lighting calculations, texture mapping, and shader calculations. The game native rendering thread plays a crucial role in the game engine's rendering pipeline. It is responsible for converting models, textures, effects, etc., in the scene into the final image output. In specific implementations, the rendering thread can perform rendering operations on multiple image data to be rendered through an image processing unit. As one implementation, this image processing unit is used to perform rendering operations on the currently rendered image data. This image processing unit can be a CPU or a graphics processing unit (GPU). That is, the rendering operation in this embodiment can be executed by the CPU or the GPU. In CPU rendering, the game engine uses the CPU's computing power to execute rendering commands and process graphics data. This method is suitable for simple 2D games or scenes with low graphics performance requirements. CPU rendering can use software renderers (such as software implementations of OpenGL), but its efficiency is low and generally cannot meet the needs of complex 3D games. In GPU rendering, the game engine utilizes the parallel computing power of the GPU to execute rendering commands and process graphics data. GPUs have a large number of processing cores and dedicated graphics processing units, enabling them to efficiently perform graphics computation tasks. Modern games widely adopt GPU rendering because it can handle more complex 3D scenes and provide higher rendering performance and visual effects. This application embodiment uses a GPU as the image processing unit.

[0064] After performing the rendering operation, the rendering thread sends the rendered real image frame #1 to the display module for composite display. The game application will continuously generate rendering requests for real image frames #2 and #3. The logic thread and rendering thread execute logical operations and rendering operations in sequence. The specific process is similar to the processing of real image frame #1, and will not be described in detail here.

[0065] Please see Figure 4 In (b) of the example: In the scenario of single frame interpolation, the logic thread is responsible for performing logical operations, and the rendering thread is responsible for performing rendering operations on the real image frame and the predicted image frame. The specific process is illustrated below (using a game scenario as an example): During the game application's operation, after generating a rendering requirement for the real image frame #1, the logic thread performs logical operations. After the logical operations are completed, the logic thread sends the calculated image resources to the rendering thread for rendering. After performing the rendering operation, the rendering thread displays the rendered real image frame #1. Similarly, the rendering thread renders the real image frame #2 in a similar manner. On the other hand, the rendering thread also needs to perform frame interpolation operations, i.e., rendering the predicted image frame #1. In one example, the image resources used to render the predicted image frame #1 include two adjacent image frames, i.e., the real image frame #1 and the real image frame #2. Specifically, for example, the rendering thread performs motion estimation, motion compensation, image completion, and image post-processing operations based on the real image frame #1 and the real image frame #2 to obtain the predicted image frame #1; the specific process is not limited here. The rendered image frame is also rendered in a similar manner to obtain the predicted image frame #2; the specific process is not limited here. It should be understood that the rendering thread should send each image frame to be displayed in the order of real image frame #1, predicted image frame #1, real image frame #2, and predicted image frame #2.

[0066] Please see Figure 4 In (c): In scenarios involving multiple frame interpolation, the logic thread is responsible for performing logical operations, while the rendering thread is responsible for rendering the real image frames and the predicted image frames. The specific implementation process is the same as... Figure 4 The single-interpolation process in (b) is similar, except that in multi-interpolation scenarios, the rendering thread needs to insert multiple predicted image frames between two real image frames. The specific process will not be described in detail here.

[0067] As can be seen from the above example, the logic thread and the rendering thread are responsible for different tasks and work alternately. Furthermore, there is generally only one rendering thread in the system, which is responsible for rendering all real image frames and all predicted image frames. For predicted frames, the resources acquired are primarily from the rendering thread. It should be understood that the power consumption of generating a single real image frame is much greater than the power consumption of generating a single predicted image frame. Therefore, in applications that use frame interpolation to improve the frame rate, single-fold interpolation significantly reduces the load on threads other than the rendering thread, and also reduces the load on the rendering thread to some extent; while multiple-fold interpolation keeps the load on threads other than the rendering thread essentially unchanged, the load on the rendering thread increases significantly.

[0068] It's important to note that during single or multiple frame interpolation, the rendering thread needs to simultaneously render both the actual and predicted image frames. This can lead to excessively long rendering times for the game scene, delaying the normal operation of the logic thread and causing rendering congestion. Alternatively, long execution times for non-rendering threads such as the main thread and logic threads can also cause rendering task congestion. Furthermore, since the rendering thread primarily utilizes the GPU for rendering tasks, heavy GPU load can also lead to rendering congestion. Once rendering is congested, noticeable frame drops and stuttering will appear, and the temperature of the electronic device will rise, significantly reducing the user experience.

[0069] In view of this, embodiments of this application provide a frame interpolation method. In this method, an independent frame interpolation thread is created to handle the rendering of predicted image frames. That is, the rendering of real image frames and the rendering of predicted image frames are decoupled. This allows for flexible frame interpolation based on actual needs, without adhering to a fixed interpolation multiple, thereby increasing the frame rate while reducing power consumption from image rendering. For example, if some real image frames are dropped, the independent frame interpolation thread can be used to perform interpolation. That is, interpolation can be performed only on the real image frames where frames are dropped, thus restoring the frame rate to the original target frame rate. This avoids excessive interpolation leading to increased power consumption and prevents a decrease in user experience due to a drop in frame rate. Furthermore, the frame rate can also be increased through flexible frame interpolation as needed. The independent frame interpolation thread allows for flexible control of the number of predicted image frames inserted, without adhering to a fixed multiple, increasing the frame rate while reducing rendering blocking. The frame interpolation method provided in this application will be described in detail below.

[0070] First, to achieve flexible frame interpolation, this application creates a frame interpolation thread. This thread is independent of the logic thread, rendering thread, and other threads, and is responsible for performing rendering operations on the predicted image frames. Therefore, this frame interpolation thread can also be called an independent thread, an independent rendering thread, or simply a frame interpolation thread; this application does not limit the name of this thread. To facilitate understanding of the function of the frame interpolation thread in this application, the following section combines... Figure 5 and Figure 6 This paper provides a brief introduction to the application of frame interpolation threads in single-interpolation and multi-interpolation scenarios.

[0071] Please see Figure 5 The logic thread, rendering thread, and frame interpolation thread are used to perform logical operations, render real image frames, and render predicted image frames, respectively. As shown in the diagram, the frame interpolation thread is independent of the logic thread and rendering thread, and the task of rendering predicted image frames is no longer performed by the rendering thread, but by the frame interpolation thread.

[0072] Understandable Figure 5 (a) in the image corresponds to a scene with single frame interpolation. Figure 5 (b) in the example corresponds to a scene with multiple frame interpolation. Compare them with... Figure 4 (b) and Figure 4 A comparison with (c) shows that by establishing an independent frame interpolation thread, the frame interpolation thread can share some of the rendering tasks with the rendering thread. This allows the rendering thread more time to handle complex scenes and real-time interactions, reducing stuttering and latency caused by rendering thread overload. Furthermore, when the rendering thread is blocked, if the frame interpolation thread is not blocked, it can take over more rendering tasks. When the rendering thread returns to normal, the frame interpolation thread has already completed part or even all of the rendering work. The rendering thread can then quickly catch up with the rendering progress, merging the rendered original image into the interpolated image rendered by the frame interpolation thread. This maintains the overall rendering continuity and smoothness, ensuring a stable target frame rate, improving the rendering effect of the game screen, and ultimately enhancing the user experience.

[0073] The above combination Figure 5 This paper introduces the application of frame interpolation threads in single-interpolation and multi-interpolation scenarios. However, in both scenarios, the frame interpolation thread needs to perform frame interpolation at a fixed ratio, meaning the number of predicted image frames needs to be an integer multiple of the number of actual image frames. In scenarios where frame rate requirements are not so high, this frame interpolation method may lead to excessive frame interpolation, wasting resources and increasing power consumption. (See also...) Figure 2Example: Assuming the frame rate is 40fps without frame interpolation, in one example, when sporadic frame drops occur, some parts of the screen will stutter. If frame interpolation is used to increase the frame rate and reduce stuttering, even the smallest interpolation (single interpolation) will at least increase the frame rate to 80fps. In another example, the system wants to increase the current frame rate from 40fps to 50fps. However, using integer interpolation can only increase the frame rate from 40fps to 80fps (or higher). Therefore, in some scenarios, using integer interpolation can lead to wasted resources, increased power consumption, and overheating, resulting in more frame drops. Therefore, this application embodiment, based on the independence of the frame interpolation thread, flexibly adjusts the frame interpolation scheme according to actual needs, achieving both stable or increased frame rates while minimizing power consumption increases, thus improving the smoothness and stability of the screen. The following is combined with... Figure 6 Several possible application scenarios are illustrated with examples.

[0074] Figure 6 Figure (a) shows a schematic diagram of the temporal distribution of multiple real image frames rendered by the rendering thread under normal operating conditions. It should be understood that the rendering thread renders and sends multiple image frames according to the target frame rate. For example, if the target frame rate is 60fps, the rendering thread needs to render and send 60 real image frames per second. These real image frames, played continuously, can form a dynamic picture. These multiple real image frames may include, for example,... Figure 1 The four consecutive image frames shown can be played in sequence to form a dynamic game screen.

[0075] Figure 6 (b) shows a schematic diagram of a real image frame in the event of dropped frames. In one possible exemplary scenario, under conditions of heavy rendering load or large load fluctuations, the natively rendered real image frames may experience random small dropped frames (i.e., passive dropped frames), for example in... Figure 6 In (b) of the example, passive frame drops occurred in the actual image frames corresponding to time points 1 and 2, which may cause stuttering in the display of the electronic device. In another possible exemplary scenario, when the rendering load is heavy, power consumption is high, or it is currently impossible to render actual image frames at the target frame rate, the rendering frame rate can be actively limited, allowing the rendering thread to actively reduce the rendering of n frames (i.e., actively drop frames) to reduce rendering thread blocking. For example, the rendering frame rate can be switched from the native frame rate to the target frame rate, where the target frame rate is lower than the native frame rate, such as reducing the rendering frame rate from 60fps to 50fps. Figure 6In (b) of the above, the real image frames corresponding to time points 1 and 2 are actively dropped, that is, the rendering thread is controlled not to execute the rendering task for the real image frames of time points 1 and 2. In this way, the blocking of the rendering thread can be reduced, but it may cause the screen currently displayed by the electronic device to stutter.

[0076] Figure 6 (c) illustrates a schematic diagram of a flexible frame interpolation scheme provided by an embodiment of this application. In an exemplary scenario of passive frame drops, when a real image frame rendered by the rendering thread experiences a passive frame drop at time point 1 / time point 2, the frame interpolation thread supplements the dropped real image frame according to the target frame rate. The frame interpolation thread can supplement the dropped real image frame through frame interpolation rendering, so that the final displayed frame rate is stable and consistent with the target frame rate without fluctuation. It should be understood that the target frame rate here refers to the frame rate originally set by the system or the currently set frame rate. Similarly, in an exemplary scenario of active frame drops, the rendering thread actively reduces the rendering of N frames, such as actively reducing the rendering of real image frames corresponding to time points 1 and 2. In this case, the frame interpolation thread supplements the reduced N frames through frame interpolation rendering, achieving the effect of reducing power consumption load while maintaining the frame rate experience. It should be understood that the target frame rate here refers to the frame rate used before active frame drops or the currently set frame rate by the system.

[0077] Figure 6 (d) illustrates another scheme for flexible frame interpolation using an interpolation thread provided in this application embodiment. In one application scenario, user experience can be improved by increasing the frame rate. In the scheme provided in this application embodiment, since the interpolation thread is an independent thread, it is not necessary to perform frame interpolation in integer multiples according to single or multiple interpolation methods, because single or multiple interpolation methods are not flexible enough and cannot achieve a balance between system power consumption and user experience. For example, it is impossible to increase the frame rate from 60fps to 70fps using single or multiple interpolation methods. However, this embodiment can achieve non-integer multiple interpolation using an interpolation thread. For example, by setting the frame rate, the current frame rate can be increased by any proportion. In specific implementation, the interpolation thread can increase a certain number of frames per second through interpolation rendering, such as increasing 10 frames per second, which can both increase the frame rate and reduce the power consumption brought about by the frame rate increase.

[0078] The above combination Figure 6This paper introduces several exemplary scenarios for flexible frame interpolation using an interpolation thread. As described above, the main purpose of this application is to maintain the actual frame rate at the target frame rate through an independent interpolation thread. Whether due to passive frame drops, active frame drops, or frame rate boosting, the actual number of real image frames may be insufficient. The role of the interpolation thread is to fill these gaps, thereby stabilizing or increasing the frame rate. Therefore, the interpolation thread needs to determine whether and when frame interpolation is necessary. Figure 7 An exemplary flowchart of the frame interpolation method 100 provided in an embodiment of this application is shown.

[0079] The following is combined Figure 7 The steps in the document are illustrated by way of example for method 100.

[0080] S110, The frame interpolation thread determines whether the rendering operation for the target frame is completed within the preset time.

[0081] For example, the frame interpolation thread determines whether a frame interpolation rendering operation needs to be performed. In one implementation, it determines whether the rendering operation for the target frame has been completed within a preset time. The target frame refers to the current frame or the next image frame to be displayed. The temporal position of the target frame can be determined based on the previous frame and the target frame rate, where the previous frame refers to the last image frame that has been displayed. The last image frame that has been displayed may be a real image frame rendered by the rendering thread or a predicted image frame rendered by the frame interpolation thread; this application does not impose any restrictions. The target frame rate refers to the frame rate currently set by the system, which may be the system's native frame rate or a subsequently reset frame rate (i.e., a non-native frame rate); this application does not impose any restrictions. For example, in Figure 6 In the passive frame drop scenario shown in (c), the target frame rate refers to the system's original frame rate setting, or the currently used frame rate; Figure 6 In the scenario of active frame dropping shown in (c), the target frame rate refers to the frame rate used before the active frame dropping occurs; Figure 6 In the scenario of increasing the frame rate shown in (d), the target frame rate refers to the frame rate after increasing the initial frame rate according to business needs, that is, the reset frame rate.

[0082] The display time of the next frame can be determined from the last displayed frame and the target frame rate. If the target frame is rendered on time, the frame interpolation thread determines that no frame drop has occurred; if the target frame is not rendered on time, the frame interpolation thread determines that a frame drop has occurred, or that the current frame rate does not meet the target frame rate. In this case, frame interpolation rendering is performed to generate a predicted image frame to supplement the actual image frame that caused the frame drop, or to increase the actual displayed frame rate to the target frame rate. The following describes one possible implementation of step S110 with reference to S101 to S105.

[0083] S101, The frame interpolation thread determines whether the rendering command for the target frame has been issued.

[0084] For example, the rendering instructions for the target frame are used to trigger the rendering of the scene and / or controls in the target frame. Typically, the CPU sends the instructions to the GPU to trigger the GPU to perform rendering operations on the target frame.

[0085] If the rendering command for the target frame has not been issued within a preset time after the previous frame has been rendered or displayed, it is determined that the CPU may be blocked, and the rendering thread executes step S102.

[0086] It is understandable that the above preset time can be determined based on the time when the previous frame was rendered / the time it was displayed and the target frame rate.

[0087] If the rendering command for the target frame has been successfully issued within a preset time after the previous frame has been rendered or displayed, it means that the CPU has successfully instructed the GPU to perform the rendering operation for the target frame, and the subsequent step S103 can be executed.

[0088] S103. Query the synchronization object corresponding to the current frame.

[0089] For example, if the rendering instructions for the current frame have already been issued, the inserted frame can further query the synchronization object corresponding to the current frame. A synchronization object refers to a class object used for rendering synchronization; for example, for the OpenGL graphics library system, the synchronization object could be `Sync`, and for the Vulkan graphics library system, the synchronization object could be `Fence`. The corresponding synchronization object is created after all rendering instructions for the current frame have been issued.

[0090] S104. Determine whether the synchronization object of the current frame can be found within the preset time.

[0091] For example, once the current frame is rendered, a corresponding synchronization object is generated. Therefore, if the synchronization object for the current frame can be found within a preset time, it can be considered that the current frame has been rendered on time, and the frame interpolation thread does not need to perform frame interpolation. If the synchronization object for the current frame is not found within the preset time, it is determined that the GPU may be blocked, that is, the rendering thread executes step S105, which means that a frame drop has occurred in the target frame.

[0092] The following provides an example of a further solution.

[0093] Optionally, in step S120, the number of consecutive inserted frames is determined to be greater than or equal to a preset value.

[0094] For example, if the inserted frame determines in step S110 that the rendering operation of the target frame has not been completed within a preset time, or if the inserted frame determines in step S110 that the CPU or / or GPU is blocked, or if the inserted frame determines in step S110 that the target frame has experienced frame drops, then before performing the frame interpolation rendering operation, the inserted frame can pre-determine whether the current number of consecutive interpolated frames is greater than or equal to a preset value. Only if the number of consecutive interpolated frames is less than the preset value will the frame interpolation rendering operation continue. The purpose of this is to reduce the possibility of motion estimation errors, jitter, and other image quality problems in the final generated image caused by an excessive number of consecutive interpolated frames.

[0095] In one possible implementation, the frame interpolation thread maintains a counter. Each time the frame interpolation thread generates a predicted image frame through frame interpolation rendering, it increments the counter by 1; each time the rendering thread renders and generates a real image frame, it resets the counter to zero. In this way, the counter value can be used to represent the number of consecutive frames interpolated. Therefore, in step S120, the frame interpolation thread can directly determine whether the counter value is greater than or equal to a preset value. If it is, it abandons frame interpolation and repeats the process in method 100 with the next frame as the target frame; otherwise, it continues to execute step S130.

[0096] S130, The interpolation thread performs interpolation rendering operations.

[0097] For example, if the frame interpolation thread determines that the rendering operation for the target frame has not been completed within a preset time, it means that the actual displayed frame rate cannot reach the target frame rate without frame interpolation. In this case, the frame interpolation thread performs a frame interpolation rendering operation to generate a predicted image frame, which is then inserted into the position of the target frame.

[0098] This application does not limit the specific implementation method of frame interpolation.

[0099] S140. Draw and display the target frame.

[0100] For example, when the frame interpolation thread executes S130, the frame interpolation thread mixes the rendering content of the predicted image frame with the UI and then sends it for display. The specific process is not limited in this application.

[0101] Alternatively, in one possible implementation, the frame interpolation thread can also be used to mix the rendered content of the real image frame with the UI before displaying it; that is, the task of displaying both the predicted image frame and the real image frame is handled by the frame interpolation thread.

[0102] For details on how image frames are displayed, please refer to the subsequent sections. Figures 9-11 The explanation will not be repeated here.

[0103] S150, the frame interpolation thread enters a sleep waiting state.

[0104] For example, after the frame interpolation thread completes step S140, it enters a sleep waiting state before the next frame is displayed.

[0105] The time spent by the frame interpolation thread executing the above scheme (such as steps S110 to S140) is usually much shorter than the frame sending interval between two adjacent frames. Therefore, in the current scenario, the frame interpolation thread is idle most of the time. Thus, the frame interpolation thread can enter a sleep waiting state during non-working hours to reduce CPU time slice occupation and performance / power consumption loss. In one possible implementation, the time for the frame interpolation thread to enter the sleep waiting state is determined based on the target frame rate and the working time of the frame interpolation thread between two adjacent frames. The following section combines... Figure 8 Examples of possible implementations are provided.

[0106] Figure 8 The first and second time points in the timeline represent the display times of two adjacent frames. The time interval between the first and second time points represents the time interval between two adjacent frames, which can be determined based on the target frame rate. For example, if the target frame rate is 40fps, then the time interval between two adjacent frames is 25ms.

[0107] In one possible implementation, the frame insertion thread enters a sleep waiting state during a first time period. This first time period is a fixed period determined based on the time interval between two adjacent frames. It should be shorter than the time interval between adjacent frames, and it should be ensured that within this time interval, the duration excluding the first time period (i.e., the second time period + the third time period in the diagram) should be greater than a preset duration. This ensures that the frame insertion thread has sufficient time to execute its core logic, as described in steps S110 to S140 above. This method allows the frame insertion thread to enter a sleep waiting state during the first time period, reducing its CPU time slice usage and performance / power consumption loss. It also provides sufficient time for the frame insertion thread to execute the solution provided in this application. Since the first time period is a predetermined fixed time, it does not require complex calculations, resulting in high execution efficiency and low resource consumption.

[0108] In another possible implementation, the frame interpolation thread enters a sleep state during the first and third time periods. The third time period is dynamic; its value is obtained by subtracting the first and second time periods from the time interval between two adjacent frames, and it follows the second time period. The second time period refers to the execution period of the core logic, specifically the time from S110 to S140 when the frame interpolation thread executes the code. Since the duration of the second time period is uncertain, the third time period is dynamic. This approach allows the frame interpolation thread to enter a sleep state during the first and third time periods, effectively preventing it from being idle (i.e., not in a sleep state but not executing any tasks), thus minimizing its CPU time slice usage and performance / power consumption.

[0109] In summary, the frame interpolation method provided in this application allows for flexible frame interpolation based on an independent interpolation thread, without necessarily interpolating in integer multiples of the original frame rate. This achieves both frame rate stabilization and improvement without wasting resources or increasing power consumption due to excessive interpolation. For example, when a target frame experiences passive frame drops, the interpolation thread can determine that the rendering operation of the target frame did not complete within a preset time based on rendering instructions and / or synchronization objects, thus determining to interpolate the target frame. This approach stabilizes the actual frame rate to the target frame rate without wasting resources through additional interpolation or increasing power consumption to prevent new frame drops. Furthermore, in scenarios with high load, the target frame can be actively dropped to reduce rendering thread blocking. The interpolation thread can determine that the rendering operation of the target frame did not complete within a preset time based on rendering instructions and / or synchronization objects, thus determining to interpolate the target frame. Since the power consumption of generating predicted image frames through interpolation is lower than that of rendering real image frames, this method achieves the effect of reducing power load while maintaining the frame rate experience. For example, in scenarios where a higher frame rate is desired (such as when the frame rate is insufficient), an arbitrary target frame rate can be set as needed, instead of setting the target frame rate as an integer multiple of the original frame rate. After setting the actual frame rate to the target frame rate, the frame interpolation thread can determine, based on the rendering instructions and / or synchronization objects, that the rendering operation of the target frame has not been completed within the preset time, i.e., the target frame is a missing frame. Therefore, the frame interpolation thread can fill in the target frame to increase the actual frame rate to the target frame rate.

[0110] In one alternative implementation, the frame interpolation thread can also control the number of consecutive frame interpolations. Once the number of consecutive frame interpolations exceeds a preset value, the frame interpolation rendering operation is stopped. This approach can reduce image quality issues such as motion estimation errors and jitter / discontinuity in the final generated image caused by excessive consecutive frame interpolations.

[0111] In one alternative implementation, the frame interpolation thread can also enter a sleep waiting state during idle time. In this way, the CPU time slice occupation and performance and power consumption loss of the frame interpolation thread can be reduced without affecting the normal operation of the frame interpolation thread.

[0112] In the above scheme, flexible frame interpolation is performed through a frame interpolation thread. Since the frame interpolation thread is an independent thread, it needs to be created before implementing the scheme described in this application.

[0113] It should be understood that this application does not limit the specific creation time of the frame interpolation thread; several possible examples are given below.

[0114] In one implementation, the frame interpolation thread can be created after the electronic device is powered on.

[0115] In another implementation, a frame interpolation thread can be created when the electronic device launches the target application. The target application could include, for example, a pre-defined type of game application, which could be a demanding game or a game with high frame rate requirements, such as action games, shooting games, racing games, role-playing games, and strategy games. Pre-defined type of game applications need to ensure smooth game display so that players can make decisions and perform actions more quickly.

[0116] In another implementation, a frame interpolation thread can be created when preset conditions are met. This implementation is illustrated below.

[0117] In one example, the preset conditions include the electronic device launching the target application and the GPU load value exceeding a first load threshold. The GPU load value represents the amount of tasks the GPU is performing or the amount of data it is processing. A higher GPU load value indicates that the GPU is performing more complex tasks or processing a larger amount of data, requiring more computing resources and image processing capabilities to ensure smooth display. Furthermore, the GPU load value typically varies with the complexity of the scene. For example, in a game scene, the GPU load value may increase when there are more characters or objects, more complex lighting effects, or the game requires a higher resolution. In some examples, the first load threshold can be 90%, 95%, etc. That is, if the GPU load value is higher than 90% or 95%, it means that the electronic device meets the preset conditions and can create frame interpolation threads. It should be noted that this application does not limit the specific first load threshold. It should also be noted that the GPU load value can be monitored and measured using various tools and software. In some examples, the GPU load value can be monitored and measured using the task manager built into the electronic device's operating system. In other examples, the GPU load value can also be monitored and measured using third-party graphics processor monitoring software connected to the electronic device. This application does not limit the specific monitoring and measurement methods.

[0118] In another example, the preset conditions include the electronic device launching the target application and the CPU load value exceeding a second load threshold. Here, CPU load value refers to the sum of the number of processes being processed and waiting to be processed by the CPU over a period of time; it can also be understood as the CPU's workload or activity level. The CPU load value depends on multiple factors, including the number of applications running on the electronic device, the number of programs open, and the hardware configuration of the electronic device. In some examples, the second load threshold can be 80%, 85%, etc. That is, if the CPU load value is higher than 80% or 85%, it indicates that the electronic device meets the preset conditions and can create a frame interpolation thread. It should be noted that this application does not limit the specific second load threshold.

[0119] In another example, the preset conditions include the electronic device launching the target application and the target application's real-time frame rate being lower than a frame rate threshold. For example, the frame rate threshold could be 40 FPS (frames per second), 60 FPS, etc. That is, if the electronic device's real-time frame rate is lower than 40 FPS or 60 FPS, it means the electronic device meets the preset conditions and can create an interpolation thread. It should be noted that this application does not limit the specific frame rate threshold.

[0120] It is understood that the above description uses one of three conditions as an example: GPU load value exceeding a first load threshold, CPU load value exceeding a second load threshold, and real-time game frame rate value falling below a frame rate threshold for frame interpolation. In practical applications, frame interpolation conditions can also be multiple of the above three conditions, and this application does not limit this.

[0121] This application does not limit the method of creating the frame interpolation thread. As an example, the frame interpolation thread can be created using the thread library or framework provided by the operating system of the electronic device. For example, the frame interpolation thread can be created using Pthreads (POSIX threads) or thread classes in the C++11 standard library. In some specific implementations, the frame interpolation thread can be created using the following code:

[0122] #include<phread.h>

[0123] intpthread_create(pthread_t*id,pthread_attr_t*attr,void(*fun)(void*),void*arg)

[0124] Furthermore, this application does not limit the frame interpolation method of the interpolation thread, nor the display method of the real image frame and the predicted image frame. The following section combines... Figures 9-11 Examples of possible implementations are provided.

[0125] Figure 9 This illustrates one possible implementation of frame interpolation and display. In this implementation, the frame interpolation thread renders the predicted frame based on rendering data shared by the rendering thread and frame interpolation context data. Furthermore, the rendering thread and the frame interpolation thread are each responsible for displaying their respective rendered image frames; that is, the rendering thread is responsible for displaying the actual image frames, and the frame interpolation thread is responsible for displaying the predicted image frames. A specific implementation is illustrated below.

[0126] On one hand, interpolation context data shared with the rendering thread is created in the interpolation thread. One possible implementation is to create the interpolation context data based on the main context data corresponding to the rendering thread. In this implementation, it is first necessary to identify the rendering thread and its corresponding main context data among the multiple threads corresponding to the current application scenario.

[0127] Taking a game scene as an example: the game's main context data refers to the data recorded by the OpenGL (Open Graphics Library) state machine during rendering, containing all the information and states required for rendering. This includes data such as the color data currently being used for rendering, whether lighting calculations are being performed, and the data of any enabled light sources. It's important to note that game context data is a type of frame rendering data. For example, context data can be one or more of the following: frame buffer object (FBO), vertex array object (VAO), vertex buffer object (VBO), and element buffer object (EBO).

[0128] In a game, there may be multiple threads and multiple context data. Therefore, the rendering thread and the main context data can be identified among multiple threads and multiple context data by recognizing the number of image drawing commands, because the rendering thread and the main context data usually call the most image drawing commands.

[0129] For ease of understanding, the following examples will use OpenGL as the rendering engine for detailed explanation. OpenGL's graphics drawing commands include glDrawArrays (used to extract data from array data to render basic primitives), glDrawElements (used to draw 3D models), and glDrawElementsInstanced (used to specify vertex data using indices and render multiple objects in a single call). Therefore, by intercepting all graphics drawing commands called in the game and counting the number of graphics drawing commands called by each thread, the thread with the most calls is the rendering thread, and its corresponding context data is the main context data.

[0130] Identifying the rendering thread and main context data across multiple threads in a target game can help developers better understand thread and context usage within the game process, and optimize game performance and responsiveness. For example, if identifying the rendering thread and main context data reveals that the rendering thread is overloaded under certain conditions, frame interpolation can be implemented in subsequent steps.

[0131] After determining the main context data, the interpolation context data is further created. Taking the embedded graphics library EGL as an example, the interpolation context data can be created in the interpolation thread using the following code:

[0132] EGLContext eglCreateContext(EGLDisplaydisplay,

[0133] EGLConfig config,

[0134] EGLContext share_context,

[0135] EGLint const*attrib_list);

[0136] It's important to note that when creating interpolated context data, it's crucial to ensure that the interpolated context data shares resources with the main rendering context data; in other words, the interpolated context data must correspond to the game's main context data. Specifically, this can be achieved by specifying `share_context` as the ID of the main rendering context data within the function corresponding to the interpolated context data, thus enabling resource sharing between the main and interpolated context data.

[0137] The purpose of sharing main context data and interpolation context data resources is to improve the performance and efficiency of game rendering. Sharing resources can reduce the overhead of resource copying and repeated loading, thereby improving the game's responsiveness and smoothness. S405: Transfers one or more of the texture data, camera data, and buffer data from the rendering thread to the interpolation thread.

[0138] On the other hand, the rendering thread shares rendering data with the frame interpolation thread. The shared rendering data may include only a portion of the rendering data, and this application does not limit this.

[0139] As an example, this includes, but is not limited to, one or more of the following data: texture data, camera data (shader), and buffer data. It should be noted that texture data, camera data, and buffer data are portions of the frame rendering data. Texture data typically refers to texture resources related to graphics rendering, used to provide surface detail and visual texture. For example, texture data may include the main scene, user interface (UI), depth map, etc. Camera data refers to data describing the lighting conditions of an object's surface during graphics rendering, used to calculate lighting models to determine visual effects such as the object's color, brightness, and shadows. Buffer data refers to a memory area used to store and process data during graphics rendering. In game development and graphics rendering, buffer data is used to store texture data, vertex data, pixel data, etc., for easy reading and writing during the rendering process.

[0140] In some specific implementations, in order to pass texture data from the rendering thread to the frame interpolation thread so that the frame interpolation thread can perform frame interpolation, the following steps need to be performed:

[0141] The first step is to create shared data structures within the rendering thread. These shared data structures will serve as the space for storing texture data in subsequent steps. In this step, the electronic device uses OpenGL or other graphics libraries to create shared data structures, which are then allocated space in the device's memory, awaiting the later storage of texture data.

[0142] The second step is to identify texture data. During game runtime, electronic devices need to be able to identify which resources are the target resources that need to be used as texture data, that is, which texture data needs to be rendered.

[0143] The third step is to copy the texture data into a shared data structure. Once the texture data is detected, the electronic device will copy it into the previously created shared data structure.

[0144] The fourth step is to pass the shared data structure containing texture data to the frame interpolation thread. The filled shared data structure containing texture data is passed from the rendering thread to the frame interpolation thread via global variables, queues, semaphores, or other inter-thread communication mechanisms.

[0145] Understandably, after identifying the texture data in the second step, a unique texture ID (i.e., the sequence number of the frame rendering data) can be assigned to the texture data. This allows the texture ID to be directly passed to the frame interpolation thread via global variables, queues, semaphores, or other inter-thread communication mechanisms, so that the frame interpolation thread can easily and conveniently know which texture data to use for rendering, thereby improving subsequent rendering efficiency.

[0146] It should be noted that the steps of transferring camera data and buffer data from the rendering thread to the frame interpolation thread are similar to those of transferring texture data from the rendering thread to the frame interpolation thread, and will not be repeated here.

[0147] It's important to note that when the rendering thread and the frame interpolation thread simultaneously attempt to access or render the same texture data, camera data, or buffer data, data conflicts, deadlocks, or other unpredictable race conditions may occur. To prevent these race conditions from arising, appropriate synchronization mechanisms are needed to coordinate the operations between the two threads. Therefore, data copying methods, double-queue methods, and circular queue methods can be used to avoid race conditions at their source.

[0148] The data copying method refers to copying all texture data, camera data, or buffer data from the rendering thread to the frame interpolation thread. The rendering thread then uses the texture data, camera data, or buffer data from the rendering thread for rendering, and the frame interpolation thread uses the texture data, camera data, or buffer data from the frame interpolation thread for rendering. This avoids the two threads simultaneously attempting to access or render the same texture data, camera data, or buffer data.

[0149] The advantage of copying data is that it avoids resource contention between the rendering thread and the frame interpolation thread. However, copying data requires copying a large amount of texture data, camera data, or buffer data from the rendering thread to the frame interpolation thread, resulting in significant memory bandwidth overhead, reduced rendering efficiency and performance, and increased rendering power consumption.

[0150] See Figure 12 The figure is a schematic diagram of a dual-queue method provided in an embodiment of this application. The dual-queue method refers to the process where, before the game scene begins rendering, the rendering thread and the frame interpolation thread each create an update queue and a rendering queue, respectively. The rendering thread first renders the current frame (the second frame) and places all rendering instructions for the current frame (the second frame) into the update queue. Simultaneously, the frame interpolation thread is executing all rendering instructions for the previous frame (the first frame).

[0151] After the rendering thread puts all rendering instructions for the current frame (the second frame) into the update queue, it executes the commit command. After the interpolation thread completes all rendering instructions for the previous frame (the first frame), it responds to the commit command by swapping all rendering instructions in the update queue and the rendering queue.

[0152] After exchanging all rendering instructions, the interpolation thread begins interpolating and rendering based on the new instructions. After completing interpolation for the current frame (the second frame), the interpolation thread suspends and waits for the next frame (the third frame). Meanwhile, the rendering thread waits for the logic thread to wake it up. Once the logic thread is awakened, the rendering thread continues rendering the next frame (the third frame) and places all rendering instructions for that frame into the update queue. This process is repeated continuously to maintain multi-threaded rendering.

[0153] The advantage of the dual-queue approach is that the rendering thread and the frame interpolation thread can operate on their own queues separately, without worrying about resource contention caused by two threads accessing the same queue simultaneously. However, the management and synchronization of two queues (the update queue and the rendering queue) increases the complexity of resource management. It's necessary to ensure proper queue swapping and synchronization to avoid data inconsistencies or thread conflicts. Furthermore, thread synchronization operations (such as waiting and waking up) can lead to additional performance overhead. These synchronization operations need to be handled carefully to avoid unnecessary waiting and performance bottlenecks. Moreover, since queues are swapped only at the end of a frame's rendering, there is a possibility of rendering blocking.

[0154] See Figure 13 This figure is a schematic diagram of a circular queue method provided in an embodiment of this application. A circular queue is a linear data structure. The circular queue method refers to using a fixed-size array and two pointers (Head pointer and Tail pointer) to represent the start and end positions of the queue in the frame interpolation thread. Specifically, the Head pointer represents the start position of the queue. Whenever a texture data, camera data, or buffer data is added to the queue, the Head pointer moves forward one position; whenever a texture data, camera data, or buffer data is removed from the queue, the Head pointer moves backward one position. The Tail pointer represents the end position of the queue, i.e., the tail position. Whenever a texture data, camera data, or buffer data is added to the queue, the Tail pointer moves backward one position; whenever a texture data, camera data, or buffer data is removed from the queue, the Tail pointer moves forward one position.

[0155] In a circular queue, when the queue is empty, both the Head and Tail pointers point to the first position in the queue; when the queue is full, both the Head and Tail pointers point to the last position in the queue. The rendering thread continuously adds texture data, camera data, or buffer data to the circular queue. If the circular queue is full, the rendering thread is blocked. The frame interpolation thread continuously retrieves texture data, camera data, or buffer data from the circular queue and performs frame interpolation. If the circular queue is empty, the frame interpolation thread is blocked.

[0156] The main characteristic of a circular queue is that when the Head or Tail pointer reaches the last position of the array, it automatically returns to the first position, resulting in better space utilization. Furthermore, unlike dual queues, circular queues do not require synchronizing the rendering thread and the frame interpolation thread every frame, resulting in shorter latency.

[0157] The following continues... Figure 9 The display method shown is illustrated by example. Figure 9 As shown, the rendering thread is used to display real image frames, while the frame interpolation thread is used to display predicted image frames. In other words, real and predicted image frames are displayed alternately in different threads. It should be understood that the displayed image frames need to be shown through the local window system's visible surface, and there is generally only one surface in the system. In one possible implementation, the rendering thread and the frame interpolation thread alternately use the same surface for display; in another possible implementation, a new surface is created for the frame interpolation thread to display the frames.

[0158] Figure 10 This illustrates another possible implementation of frame interpolation and display. Figure 10 Frame interpolation methods and Figure 9 The frame interpolation method is similar, so it will not be repeated here. Figure 10 In the shown display mechanism, the frame interpolation thread is solely responsible for displaying both the actual image frames and the predicted image frames. It should be understood that because the rendering thread shares the rendering data corresponding to the actual image frames with the frame interpolation thread, the frame interpolation thread can display the actual image frames. Compared to... Figure 9 The display scheme shown uses a separate frame interpolation thread for display. The frame interpolation thread can be directly bound to the rendering target surface used by the rendering thread to be responsible for displaying the final content of the two threads, without creating a new local window system visible surface or having the two threads alternately use the same surface for display, thereby reducing unnecessary memory usage or performance overhead.

[0159] Figure 11 This illustration shows a schematic diagram of the frame interpolation scheme and display scheme provided in an embodiment of this application. It should be understood that... Figure 11 This explanation uses both active and passive frame drop scenarios as examples. Figure 11 The frame interpolation scheme shown is the same as Figure 6 The scheme shown in (c) corresponds to this.

[0160] The rendering thread is responsible for rendering the real image frames and drawing the corresponding main scene and UI content onto several textures, which are then shared with the frame interpolation thread. Figure 11 In the given example, the rendering thread draws the main scene of real image frame 0 in main scene texture 0 and the UI of real image frame 0 in UI texture 0; similarly, the rendering thread draws the main scene of real image frame 1 in main scene texture 1 and the UI of real image frame 1 in UI texture 1. The rendering thread passes the identification information of these textures to the interpolation thread. After receiving the identification information, the interpolation thread can read the content of the texture through the shared context. Then, the interpolation thread mixes the main scene texture and the corresponding UI texture and sends it for display, such as drawing the texture on FB0 of the current surface. Alternatively, in another implementation, the interpolation thread can also use these textures as input for interpolation (intermolecular and extratermolecular) rendering and mixing before sending it for display. It is understood that the shared texture resources used in independent threads are not limited to main scene textures, UI textures, interpolation output textures, etc., and can also include other types of textures, which are not limited in this application.

[0161] Figure 14 This is a schematic diagram of the hardware structure of the electronic device 600 provided in an embodiment of this application. For example... Figure 14 As shown, the electronic device 200 may include a processor 210, a memory 230, and 1 to N displays 220, where N is a positive integer greater than or equal to 1.

[0162] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 200. In other embodiments, the electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0163] Processor 210 may include one or more processing units. In this application, processor 210 includes a central processing unit (CPU) and a graphics processing unit (GPU). The CPU performs logical operations and issues rendering instructions, as detailed in the full description. The GPU renders image frames to be displayed, including real image frames or predicted image frames. Optionally, processor 210 may also include other types of units or modules, such as a modem processor, an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0164] The memory 230 can be used to store computer-executable program code, which includes instructions. The processor 210 executes various functional applications and data processing of the electronic device 200 by running the instructions stored in the memory 230, such as executing the frame interpolation method provided in the embodiments of this application.

[0165] The display screen 220 is used to display images, videos, etc. In some embodiments, the electronic device 200 may include one or N display screens 220, where N is a positive integer greater than 1. In the embodiments of this application, the display screen 220 is used to display application screens, such as displaying dynamic screens by rapidly displaying multiple consecutive image frames.

[0166] Optionally, the electronic device 200 may also include a mobile communication module 240 and a wireless communication module 250. The wireless communication function of the electronic device 200 may be implemented through an antenna (not shown), the mobile communication module 240, the wireless communication module 250, a modem processor, and a baseband processor.

[0167] Antennas are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 200 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antennas can be multiplexed as diversity antennas for a wireless local area network. In some other embodiments, antennas can be used in conjunction with tuning switches.

[0168] The mobile communication module 240 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 200. The mobile communication module 240 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 240 can receive electromagnetic waves via an antenna, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 240 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via the antenna. In some embodiments, at least some functional modules of the mobile communication module 240 may be housed in the processor 210. In some embodiments, at least some functional modules of the mobile communication module 240 and at least some modules of the processor 210 may be housed in the same device.

[0169] The wireless communication module 250 can provide solutions for wireless communication applications on the electronic device 200, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 250 can be one or more devices integrating at least one communication processing module. The wireless communication module 250 receives electromagnetic waves via an antenna, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to the processor 210. The wireless communication module 250 can also receive signals to be transmitted from the processor 210, perform frequency modulation and amplification, and then convert them into electromagnetic waves for radiation via the antenna.

[0170] The methods described in this application can all be implemented in an electronic device 200 having the above-described hardware structure.

[0171] That concludes the introduction to the hardware structure of electronic devices. It is understandable that... Figure 14 The components included in the hardware structure shown do not constitute a specific limitation on the electronic device. The electronic device may have more or fewer components than shown in the figure, may combine two or more components, or may have different component configurations. The various components shown in the figure may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0172] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0173] This application provides a computer program product that, when run on a device, enables the device to perform the steps described in the various method embodiments above.

[0174] This application provides a chip for executing instructions. When the chip is running, it executes the technical solutions described in the above embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0175] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs (DWDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0176] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographic device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0177] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0178] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0179] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0180] It should be understood that the term "embodiment" used throughout the specification means that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, various embodiments throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0181] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

[0182] Furthermore, it should be noted that the various numerical designations used in this application (such as the terms "first," "second," "third," "fourth," and other terminology used in the specification, claims, and accompanying drawings, if any) are merely for descriptive convenience and are not intended to limit the scope of this application. The order of the process numbers does not imply the sequence of execution; the execution order of each process should be determined by its function and internal logic.

[0183] The terms “comprising” and “having”, and any variations thereof, mean “including, but not limited to”, unless otherwise specifically emphasized, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0184] In the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0185] In the various embodiments of this application, unless otherwise specified or logically conflicting, the terminology and / or descriptions between different embodiments are consistent and can be referenced mutually. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships. The specific operational methods in the method embodiments of this application can also be applied to the device embodiments or system embodiments.

Claims

1. A frame interpolation method, characterized in that, The method includes: Determine whether the rendering operation for the target frame is completed within a preset time. The target frame is the next image frame to be displayed. The preset time is determined based on the previous image frame and the target frame rate. If the rendering operation for the target frame is not completed within a preset time, a predicted image frame is generated by a frame interpolation thread. The predicted image frame is generated based on rendering data, which is the data used by the rendering thread in the process of rendering and generating the real image frame. The frame interpolation thread is an independent thread relative to the rendering thread. The predicted image frame is sent as the target frame for display.

2. The method according to claim 1, characterized in that, The determination of whether the rendering operation for the target frame is completed within a preset time includes: Determine whether the rendering command for the target frame has been issued within the preset time. If the rendering instruction is not issued within the preset time, it is determined that the rendering operation for the target frame was not completed within the preset time.

3. The method according to claim 1 or 2, characterized in that, The determination of whether the rendering operation for the target frame is completed within a preset time includes: Determine whether the synchronization object corresponding to the target frame can be found within the preset time period; If the synchronization object corresponding to the target frame cannot be found within the preset time, it is determined that the rendering operation for the target frame was not completed within the preset time.

4. The method according to any one of claims 1 to 3, characterized in that, The generation of predicted image frames via the frame interpolation thread includes: Determine whether the current number of consecutively inserted frames is greater than or equal to a preset value; If the number of consecutive frame interpolations is less than the preset value, the predicted image frame is generated by the frame interpolation thread.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: The actual image frames are displayed through the frame interpolation thread.

6. The method according to any one of claims 1 to 5, characterized in that, After displaying the predicted image frame as the target frame, the method further includes: The frame interpolation thread is put into a sleep waiting state.

7. The method according to claim 6, characterized in that, The method further includes: The time for the frame interpolation thread to enter a sleep waiting state is determined based on the target frame rate and the working time of the frame interpolation thread between two adjacent frames.

8. The method according to any one of claims 1 to 7, characterized in that, The target frame rate is the initially set frame rate, or the frame rate increased based on the initial set frame rate according to business needs.

9. An electronic device, characterized in that, The device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, the electronic device performs the method as described in any one of claims 1-8.

10. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1-8.

11. A computer program product, characterized in that, Includes a computer program, which, when run, causes the method as described in any one of claims 1-8 to be performed.

Images