A method, device and storage medium for inserting frames

Abstract

This invention discloses a frame interpolation method and apparatus. The frame interpolation method includes: displaying the nth game frame; obtaining the view matrix and projection matrix of the (n+1)th game frame; generating a first extrapolated frame based on the view matrix and projection matrix of the (n+1)th game frame; and displaying the first extrapolated frame. Through this solution, since the prediction process of the extrapolated frame references the camera viewpoint of the (n+1)th game frame, the motion trend of objects in the extrapolated frame can be made consistent with the preceding and following game frames, thereby suppressing game screen jitter and improving the user's gaming experience.

Description

Technical Field

[0001] This invention relates to the field of terminal technology, and in particular to a frame interpolation method, device, and storage medium. Background Technology

[0002] To improve the smoothness of animations (such as game visuals), higher frame rates are desired. However, as the frame rate increases, the power consumption of the animation rendering process also increases. Frame interpolation technology can generate predicted frames based on the rendered footage, thereby controlling power consumption while increasing the frame rate. Extrapolation is one type of frame interpolation. In game scenarios, using extrapolation to process game frames results in lower latency and better responsiveness for the user. However, in some scenarios, it can cause screen jitter. Summary of the Invention

[0003] The purpose of this invention is to provide a frame interpolation method and device for reducing game screen jitter caused by extraframe interpolation.

[0004] In a first aspect, the present invention provides a frame interpolation method, comprising: displaying a game frame of the nth frame; obtaining a view matrix and a projection matrix of a game frame of the (n+1)th frame; generating a first extrapolated frame based on the view matrix and projection matrix of the game frame of the (n+1)th frame; and displaying the first extrapolated frame.

[0005] In the above implementation, the nth game frame is the currently displayed game frame. The (n+1)th game frame is the game frame to be rendered; it can be either not yet rendered or not yet rendered. The view matrix and projection matrix can be used to represent the camera's perspective. During the display of the nth game frame, the view matrix and projection matrix of the (n+1)th game frame can be obtained. These matrix values ​​are used to guide the generation of interpolated frames, and the generated interpolated frames are displayed after the nth game frame and before the (n+1)th game frame. Since the prediction process of the interpolated frames references the camera's perspective of the (n+1)th game frame, the motion trend of objects in the interpolated frames can be made consistent with the preceding and following game frames, thereby reducing or even eliminating screen jitter and improving the user's gaming experience.

[0006] In one of the frame interpolation methods described above, in some implementations of the first aspect, obtaining the view matrix and projection matrix of the (n+1)th game frame includes: receiving the view matrix and projection matrix of the (n+1)th game frame sent by the target game application.

[0007] In this implementation, the view matrix and projection matrix of the (n+1)th game frame can be obtained through a pre-arranged protocol with the game application. This allows the game application to proactively send the generated view matrix and projection matrix to the system after generating them. This approach simplifies the process of obtaining the view matrix and projection matrix on the system side, resulting in higher execution efficiency.

[0008] In one of the frame interpolation methods described above, in some implementations of the first aspect, obtaining the view matrix and projection matrix of the (n+1)th game frame includes: listening to a target instruction in the target game application; and in response to listening to the target instruction, obtaining the view matrix and projection matrix of the (n+1)th game frame from the target instruction.

[0009] In this implementation, the view matrix and projection matrix of the (n+1)th game frame can be obtained by actively listening to commands generated by the game application and retrieving the view matrix and projection matrix of the (n+1)th game frame from the specific commands received. This approach allows the solution to be easily applied to any game application on a terminal device, enhancing its practicality.

[0010] In some implementations of the frame interpolation method described above, before listening to the target instructions in the target game application, the method further includes: obtaining a pre-configured target mapping relationship, which describes the mapping relationship between different game applications and interception methods; and determining the interception method corresponding to the target game application based on the target mapping relationship, which describes the target instructions to be listened to and the target information in the target instructions used to represent the view matrix and projection matrix.

[0011] In this implementation, because the instructions for carrying the view matrix and projection matrix, as well as the information in the instructions representing the view matrix and projection matrix, differ across different game applications, the methods for intercepting the view matrix and projection matrix vary for each game application. To address this, this application can pre-configure a mapping relationship between different game applications and interception methods. Therefore, using this mapping relationship, this solution can be combined with any game application on the terminal device, and for any game application, the view matrix and projection matrix of the (n+1)th game frame can be successfully intercepted.

[0012] In some implementations of the frame interpolation method described above, before generating the first extrapolated frame based on the view matrix and projection matrix of the (n+1)th game frame, the method further includes: controlling the cessation of the rendering process for the (n+1)th game frame.

[0013] In this implementation, before executing the process of generating interpolated frames, the rendering process of the (n+1)th game frame is stopped, thus enabling single-threaded processing.

[0014] In some implementations of the first aspect of the frame interpolation method described above, before controlling the cessation of the rendering process for the (n+1)th game frame, the method further includes: detecting that the (n+1)th game frame has started rendering and waiting for the current rendering channel to finish rendering; or detecting that the (n+1)th game frame has not started rendering.

[0015] In this implementation, when using a single-threaded processing method, we first wait for the rendering of the current rendering channel of the (n+1)th game frame to finish, or first determine that the (n+1)th game frame has not yet started rendering, and then control the rendering of the (n+1)th game frame to stop. This can prevent rendering errors caused by the interruption of the rendering progress of a single rendering channel.

[0016] In some implementations of the frame interpolation method described above, after displaying the first interpolated frame, the method further includes: controlling the resumption of the rendering process for the (n+1)th game frame.

[0017] In one implementation of the frame interpolation method described above, generating a first extrapolated frame based on the view matrix and projection matrix of the (n+1)th game frame includes: determining the motion vector of a static object between the (n+1)th game frame and the nth game frame based on the view matrix and projection matrix of the (n+1)th game frame; and generating the first extrapolated frame based on the motion vector of the static object between the (n+1)th game frame and the nth game frame.

[0018] In this implementation, since static objects do not move, their positions in the frame will change significantly when the camera view changes abruptly due to user touch input or other reasons. Therefore, when using extrapolation for frame interpolation, the resulting jitter in the static object model is quite noticeable. To address this, this application uses the view matrix and projection matrix of the (n+1)th game frame to calculate the motion vectors of the static objects between the (n+1)th and nth game frames. Extrapolated frames are then generated based on these motion vectors, allowing the motion trend of the static objects in the (n+1)th game frame to guide the generation of the extrapolated frames. This ensures that even if the camera view changes abruptly in the (n+1)th game frame, the motion trend of the static objects in the extrapolated frames remains consistent with that in the (n+1)th game frame, thus suppressing screen jitter.

[0019] In one implementation of the frame interpolation method described above, determining the motion vector of a static object between the (n+1)th game frame and the nth game frame based on the view matrix and projection matrix of the (n+1)th game frame includes: calculating the clipping coordinates of the static object in the (n+1)th game frame based on the view matrix and projection matrix of the (n+1)th game frame; and determining the motion vector of the static object between the (n+1)th game frame and the nth game frame based on the clipping coordinates of the static object in the (n+1)th game frame.

[0020] In one implementation of the frame interpolation method described above, the clipping coordinates of static objects in the (n+1)th game frame are calculated based on the view matrix and projection matrix of the (n+1)th game frame, including: calculating the clipping coordinates of static objects in the (n+1)th game frame according to the following formula: Among them, Coord NDCFrameN+1 Coord represents the clipping coordinates of static objects in the (n+1)th game frame. NDCFrameN Let (VP) be the clipping coordinates of a static object in the nth game frame. FrameN+1 Let (VP) be the view matrix and projection matrix of the (n+1)th game frame. FrameN These are the view matrix and projection matrix corresponding to the nth game frame.

[0021] Secondly, this technical solution provides an electronic device, including: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the device, cause the device to perform the method of the first aspect or any possible implementation thereof.

[0022] Thirdly, the present invention also provides a computer-readable storage medium storing program code for execution by a device, the program code including instructions for performing the method in the first aspect or any possible implementation thereof. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of a scenario for the frame interpolation method provided in an embodiment of this application;

[0024] Figure 2 This is another scenario illustration of the frame interpolation method provided in the embodiments of this application;

[0025] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0026] Figure 4This is a schematic diagram of a spatial transformation relationship of the frame interpolation method provided in the embodiments of this application;

[0027] Figure 5 This is a schematic flowchart of the frame interpolation method provided in the embodiments of this application;

[0028] Figure 6 This is a schematic diagram of the motion vector calculation method provided in the embodiments of this application;

[0029] Figure 7 This is a schematic diagram of a method for generating motion vectors of dynamic objects provided in an embodiment of this application;

[0030] Figure 8 This is a schematic diagram of another method for generating motion vectors of dynamic objects provided in the embodiments of this application;

[0031] Figure 9 This is a schematic diagram of a method for generating motion vectors of static objects provided in an embodiment of this application;

[0032] Figure 10 This is another structural schematic diagram of the electronic device provided in the embodiments of this application;

[0033] Figure 11 This is another schematic flowchart of the frame interpolation method provided in the embodiments of this application;

[0034] Figure 12 This is another schematic flowchart of the frame interpolation method provided in the embodiments of this application;

[0035] Figure 13 This is another schematic flowchart of the frame interpolation method provided in the embodiments of this application. Detailed Implementation

[0036] Currently, the variety of game applications is increasing, and game scenarios are becoming more complex. At the same time, users' demands for smooth gameplay are constantly rising, requiring games to also have high frame rates. Against this backdrop, the memory consumption and power consumption of rendering game visuals during gameplay are increasing, easily leading to frame drops or device overheating, thus impacting the user experience.

[0037] Frame interpolation technology can solve the above problems. Specifically, frame interpolation technology can predict subsequent frames based on the already rendered game frames of the game application, thereby generating predicted frames for display. By using frame interpolation technology, new game frames can be added to the original game screen, which can not only improve the frame rate, but also reduce power consumption during the frame interpolation process, thus alleviating problems such as frame drops or device overheating.

[0038] Based on the method of frame interpolation, frame interpolation techniques can be divided into two types: interpolation and extrapolation.

[0039] Figure 1 Schematic diagrams illustrating the implementation logic of interpolation and extrapolation are provided. Figure 1 As shown, the game's native rendering logic refers to the logic by which the game application generates game frames without frame interpolation. Based on this native rendering logic, the game application sequentially generates and displays real frames A, B, C, and D. It can be understood that the real frames described in this application refer to the game frames generated by the game application itself.

[0040] Interpolation logic refers to the generation logic of each interpolated frame when using interpolation for frame interpolation processing. Based on interpolation logic, interpolated frame A is predicted based on real frame A and real frame B, and interpolated frame A is displayed in the display node corresponding to real frame B in the game's native rendering logic. Real frame B is displayed after the display node corresponding to real frame B in the game's native rendering logic. It can be understood that interpolation generates interpolated frames based on the (N-1)th frame of the game and the Nth frame of the game. The (N-1)th frame of the game is the previous frame, and the Nth frame of the game is the current frame. The generated interpolated frame is displayed in the display node of the Nth frame of the game, while the Nth frame of the game is displayed after the interpolated frame. In other words, based on the implementation of interpolation, during game execution, at time N, the actual game screen is the Nth frame, but the user does not see the Nth frame at this time, but rather the interpolated frame. The user can only see the Nth frame, the actual game screen, after time N. It's understandable that interpolation causes a delay in the game screen displayed to the user, resulting in poor responsiveness during gameplay. In other words, when the user swipes the screen, the screen response is slow, leading to a poor user experience, especially in shooting and competitive games, where reduced responsiveness significantly diminishes the gaming experience.

[0041] Extrapolation logic refers to the generation logic of each interpolated frame when using extrapolation for frame interpolation processing. Based on extrapolation logic, interpolated frame B is predicted based on real frame A and real frame B, and interpolated frame B is displayed at the display node between real frame B and real frame C in the game's native rendering logic. In other words, extrapolation also predicts and generates interpolated frames based on the (N-1)th and Nth frames of the game. However, unlike interpolation, in extrapolation, the generated interpolated frames are displayed between the display nodes of the Nth and N+1th frames of the game, where the N+1th frame refers to the next frame. Based on the implementation of extrapolation, the game displays the actual Nth frame at time N, the actual N+1th frame at time N+1, and the interpolated frames between time N and N+1. It's understandable that with interpolation, users see the actual game screen first, then the interpolated frame. Therefore, compared to interpolation, interpolation reduces game screen latency and has less impact on responsiveness. However, because the interpolated frame is displayed between time N and time N+1, and the interpolation prediction process doesn't reference the game screen displayed at time N+1, in some scenarios, the motion trend of objects in the interpolated frame may not match the motion trend of objects in the game screen at time N+1, resulting in screen jitter.

[0042] For example, consider a scenario where user touch input changes. Consider touch input jitter: Suppose a user initiates (or stops) a touch operation after frame N. As the user's finger presses (or leaves) the screen, the number of touch points between the finger and the screen gradually increases (or decreases), causing jitter in the touch information received by the screen. Therefore, compared to frames N-1 and N, the camera perspective in frame N+1 will change abruptly, and the movement of objects in frame N+1 will also change accordingly. Here, "camera perspective" is a technical term used in the gaming industry to represent the player's viewpoint of the game world presented on the screen. Since interpolated frames are predicted from frames N-1 and N, the movement of objects in interpolated frames may not be consistent with that in frame N+1, resulting in screen jitter. For example, if a user changes the touch input: Suppose that after frame N, the user rotates or stops rotating the camera view by touch input, the movement trend of objects in the game screen of frame N+1 will also change. Similarly, the movement of objects in the interpolated frame may not be consistent with the game screen of frame N+1, resulting in screen jitter.

[0043] In another scenario, the game scene changes. For example, a game character encounters an obstacle while moving, causing a change in their motion state, or the camera's view suddenly zooms in or out when the game character encounters an obstacle while moving. In this case, similar to the above situation, after the game scene changes abruptly, the interpolated frame predicted based on the (N-1)th frame of the game scene before the change and the Nth frame of the game scene after the change may not be consistent with the movement of objects in the N+1th frame of the game scene after the change, resulting in screen jitter.

[0044] For ease of understanding, such as Figure 2 As shown, real frames refer to game frames generated by the game application itself, while interpolated frames refer to interpolated frames predicted based on real frames using interpolation methods. (Reference) Figure 2 From real frame 1 to real frame 2, the user controls the game character to rotate to the left. Correspondingly, the relative position between the game character and static objects in the scene (such as roads, trees, etc.) gradually changes. After real frame 2 is displayed, the user controls the game character to stop rotating. Consequently, the relative position between the character and static objects in the scene no longer changes. Therefore, the relative position between the game character and static objects in real frame 3 is the same as in real frame 2 and remains unchanged. However, interpolated frame 2 is predicted based on real frames 1 and 2. Therefore, in interpolated frame 2, the game character continues its leftward rotation. The game character continues to rotate to the left relative to real frame 2, causing interpolated frame 2 to lose continuity with real frame 3, resulting in screen jitter.

[0045] As can be seen, while interpolation offers lower latency and less impact on game responsiveness, it can cause screen jitter in certain scenarios. Therefore, this application is proposed to address the screen jitter issue caused by interpolation during application.

[0046] Figure 3 A schematic diagram of the structure of the electronic device 100 provided in an embodiment of this application is shown.

[0047] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a fingerprint sensor 180H, a pressure sensor 180A, a touch sensor 180K, an ambient light sensor 180L, a proximity light sensor 180G, a temperature sensor 180J, a distance sensor 180F, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer 180E, a bone conduction sensor 180M, etc.

[0048] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0049] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0050] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0051] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0052] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0053] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 180K, charger, flash, camera 193, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0054] The MIPI interface can be used to connect the processor 110 and the display screen 194. The processor 110 and the display screen 194 communicate through the DSI interface to realize the display function of the electronic device 100.

[0055] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0056] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering, such as the rendering of game scenes in this application. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0057] Display screen 194 is used to display game graphics, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0058] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0059] Internal memory 121 can be used to store computer executable program code, which includes instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as frame interpolation), etc. The data storage area may store data created during the use of electronic device 100 (such as rendering status information, frame data, etc.). In addition, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of electronic device 100 by running instructions stored in internal memory 121 and / or instructions stored in memory disposed in the processor.

[0060] Touch sensor 180K, also known as a "touch device," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In some embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194. In a game scenario, electronic device 100 can change the game viewpoint, zoom in or out of the game screen, etc., based on the touch operation detected by touch sensor 180K.

[0061] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 about three axes (i.e., the x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used in motion-sensing game scenarios.

[0062] The following embodiments of this application will be used to illustrate having Figure 3 Taking the electronic device with the structure shown as an example, the frame interpolation method provided in the embodiments of this application will be specifically described.

[0063] The method provided in this application can be used to perform frame interpolation processing on the screen displayed during gameplay, and can improve the frame rate while alleviating the problem of screen jitter caused by frame interpolation. In this application embodiment, a target game application is used as an example for illustration; the target game application can be any game application within an electronic device.

[0064] The game world space of the target game application can contain multiple object models, such as game characters, birds, houses, trees, etc. Each object model has its own spatial coordinates in different spaces. For example, it has its own world coordinates in the game world space, and its own local coordinates in the model space. The MVP matrix can be used to transform the spatial coordinates of object models in different spaces.

[0065] Specifically, such as Figure 4 As shown, the MVP matrix consists of the model matrix (M), the view matrix (V), and the projection matrix (P). The M matrix is ​​used to convert the local coordinates of an object's vertices in local space to world coordinates in world space. The V matrix is ​​used to convert the world coordinates of an object's vertices in world space to view coordinates in view space. The P matrix is ​​used to convert the view coordinates of an object's vertices in view space to clip coordinates in clip space. Model space is the space formed by establishing a coordinate system with a point on the model as the origin. World space refers to the game world space. View space refers to the game world space observed from the camera's perspective. Clip space is the space obtained by projecting a transformation onto view space.

[0066] Meanwhile, object models within the game world can be divided into two categories: static object models and dynamic object models. Static object models are stationary; their local coordinates in model space remain unchanged, and their position in the game world remains constant, such as house models and tree models. Dynamic object models, on the other hand, are in motion; their local coordinates in model space change constantly, and their position in the game world changes constantly, such as various animal models and character models.

[0067] For static object models, when the camera's perspective changes abruptly due to factors such as changes in user touch input, the position of the static object in the frame will change significantly. Therefore, when using extrapolation for frame interpolation, the resulting jitter in the static object model in the frame is quite noticeable. The specific principle behind this jitter can be found in the preceding explanation.

[0068] For dynamic object models, because they move according to specific motion trends and their motion states are relatively consistent, changes in user touch input or other factors that cause sudden shifts in the camera's perspective have a relatively small impact on the consistency of their motion states. Therefore, when using extrapolation for frame interpolation, the resulting jitter in the dynamic object models in the image is weak, and the impact on the user's perceptual experience is minimal.

[0069] Therefore, in this embodiment of the application, the frame interpolation method provided is mainly used to reduce the jitter of static object models in the game screen caused by extrapolation.

[0070] Figure 5 This is a schematic flowchart of the frame interpolation method provided in the embodiments of this application, such as... Figure 5 As shown, the frame interpolation method provided in this application includes:

[0071] 101, displays the nth game frame.

[0072] 102. Obtain the view matrix and projection matrix of the (n+1)th game frame.

[0073] 103. Generate the first extrapolation frame based on the view matrix and projection matrix of the (n+1)th game frame.

[0074] 104, displaying the first extrapolated frame.

[0075] In this embodiment, after displaying the nth game frame, an interpolated frame can be calculated. The process of generating the interpolated frame mainly consists of four steps: image preprocessing, calculating motion vector (MV), calculating the interpolated image, and post-processing of the interpolated image.

[0076] In the image preprocessing step, the electronic device can identify the image using the acquired image data and other relevant data, such as marking dynamic objects, static objects, and the frame interpolation method of pixels or pixel blocks. For example, the depth value and stencil value of the game scene can be obtained by intercepting a pre-drawn depth stencil. Then, by analyzing the game rendering characteristics offline, the stencil value can be used to distinguish between dynamic and static objects in the scene.

[0077] Motion vectors refer to the distance a pixel or pixel block moves between adjacent frames in an image; their value is equal to the difference in pixel coordinates between adjacent frames. For example... Figure 6 As shown, assume the coordinates of the pixel in the (n-1)th frame are... The coordinates of the same pixel in the nth frame are Therefore, the motion vector of this pixel between the two frames is

[0078] In the step of calculating motion vectors, since the jitter of the dynamic object model in the image caused by extrapolation is relatively weak, in this embodiment of the application, for dynamic objects, the electronic device can obtain the motion vector of the dynamic object between the nth game frame and the (n-1)th game frame. Specifically, as shown... Figure 7As shown, for example, optical flow can be used to calculate the motion vector of dynamic objects by inputting the main scene texture data and moving / static object identification information of the nth and (n-1)th game frames. Alternatively, as... Figure 8 As shown, a vertex calculation method can be used to intercept the vertex cache data of dynamic objects in the nth and (n-1)th game frames, and then, based on the dynamic objects between the two frames (such as... Figure 8 The motion vector is determined by the projection pixel difference of the vertex cache data corresponding to the same vertex of the figure and the bird shown.

[0079] For static objects, electronic devices can obtain their motion vectors according to the method provided in the embodiments of this application to reduce the jitter of static object models caused by extrapolation frames.

[0080] The method for determining the motion vector of a static object in the embodiments of this application will be described in detail below.

[0081] In this embodiment of the application, unlike dynamic objects, for static objects, the electronic device can calculate the motion vector of the static object between the nth game frame and the (n+1)th game frame. Here, the nth game frame is the currently displayed game frame. In practical applications, when displaying the nth game frame, the (n+1)th game frame is usually in a state awaiting rendering. This state can be either a state where rendering has not yet begun or a state where rendering has not yet been completed.

[0082] Specifically, the motion vector between the nth game frame and the (n+1)th game frame is equal to the difference between the pixel coordinates in the (n+1)th game frame and the pixel coordinates in the nth game frame. Here, the pixel coordinates in the nth game frame are known, while the pixel coordinates in the (n+1)th game frame are unknown, as they are in a state to be rendered. Therefore, to determine the motion vector of a static object between the nth and (n+1)th game frames, the electronic device first needs to obtain the pixel coordinates of the static object model in the (n+1)th game frame.

[0083] Specifically, as explained above, the MVP matrix can be used to transform the local coordinates of an object in model space into clipping coordinates in clipping space. Based on this, the following equation can be obtained:

[0084] Coord NDCFrameN =Local Coordinate FrameN *(MVP) FrameN

[0085] Coord NDCFrameN+1 =Local Coordinate FrameN+1 *(MVP) FrameN+1

[0086] Among them, CoordNDCFrameN LocalCoordinate is the clipping coordinate of the pixel of the static object model in the nth game frame. FrameN Let be the local coordinates of a pixel in the static object model in the nth game frame (MVP). FrameN The MVP matrix corresponding to the nth game frame; Coord NDCFrameN+1 Local Coordinate is the clipping coordinate of the pixel of the static object model in the (n+1)th game frame. FrameN+1 Let be the local coordinates of the pixel point of the static object model in the (n+1)th game frame, (MVP). FrameN+1 This is the MVP matrix corresponding to the (n+1)th game frame.

[0087] Since the position of a static object model remains unchanged in model space and in game world space, therefore, in the above formula, Local Coordinate FrameN *(M) FrameN The value of Local Coordinate FrameN+1 *(M) FrameN+1 The values ​​of are equal. Therefore, based on this, we can obtain the following formula:

[0088]

[0089] Because in the above formula, Coord NDCFrameN With (VP) FrameN Since all of these are known quantities, in this embodiment of the application, it is only necessary to obtain the VP matrix corresponding to the (n+1)th game frame to calculate the clipping coordinates of the static object model pixels in the (n+1)th game frame.

[0090] Specifically, before rendering begins for the (n+1)th game frame, the target game application generates the VP matrix corresponding to that game frame. Therefore, in this embodiment, the VP matrix already generated by the target game application can be obtained after the target game application generates the VP matrix for the (n+1)th game frame. Then, the electronic device can determine the motion vector of static objects between the nth and (n+1)th game frames based on the obtained VP matrix.

[0091] First, the clipping coordinates of the static object model pixels in the (n+1)th game frame can be calculated using the formula above.

[0092] Then, the clipping coordinates of the static object model pixels in the (n+1)th game frame and the clipping coordinates of the static object model pixels in the nth game frame can be converted into screen coordinates, respectively. The screen coordinates are defined as follows: uv= (u, v, depth), where u and v are the normalized coordinates of the pixel's clipping coordinates, and depth is the depth map data. Clipping coordinates can be converted to screen coordinates using Coord. uv =(Coord) NDC +1)*0.5 where, Coord uv Coord is the screen coordinate. NDC For the clipping coordinates.

[0093] Finally, the difference between the screen coordinates of the two can be used to obtain the motion vector of the static object between the nth game frame and the (n+1)th game frame.

[0094] In other words, such as Figure 9 As shown in the embodiment of this application, based on the VP matrix corresponding to the (n+1)th game frame, the known VP matrix corresponding to the nth game frame, and the known depth map data corresponding to the nth game frame, the motion vector of the static object between the nth game frame and the (n+1)th game frame can be obtained.

[0095] Furthermore, in the step of calculating the interpolated image, the pixels or pixel blocks in the image can be moved or blended using the obtained motion vectors and other relevant data to form a preliminary interpolated image. It should be noted that in this embodiment, the motion vector of the static object used is the motion vector between the nth game frame and the (n+1)th game frame, while the motion vector of the dynamic object used can be the motion vector between the previous game frame and the nth game frame.

[0096] Furthermore, in the post-processing steps of the interpolated image, post-processing operations such as hole filling and filtering can be performed on the preliminary interpolated image to form the final interpolated image.

[0097] After the extrapolated frame is generated, the extrapolated frame image can be displayed. Then, the electronic device can display the rendered (n+1)th frame of the game screen.

[0098] Through the above technical solution, the interpolated frame is displayed between the current real game screen and the next real game screen. The prediction process of the interpolated frame refers to the position change trend of static objects in the next real game screen. Therefore, the continuity between the interpolated frame and the two real game screens can be improved, thereby increasing the frame rate while suppressing the jitter of the game screen and improving the user's gaming experience.

[0099] In another embodiment of this application, the method provided in the embodiment of this application is described in conjunction with the software structure of the electronic device 100.

[0100] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100.

[0101] Figure 10 This is a software structure block diagram of the electronic device 100 according to an embodiment of this application.

[0102] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into the application layer (APP), the application framework layer (Framework), the native layer (Native), and the kernel layer (Kernel) from top to bottom.

[0103] like Figure 10 As shown, the application layer may contain at least one game application, including the target game application.

[0104] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0105] The local layer may include a frame interpolation unit, a generation unit, and an execution unit. In practical applications, for example, a CPU may be used to implement all the functions of the generation unit provided in the embodiments of this application, and a GPU may be used to implement all the functions of the execution unit provided in the embodiments of this application.

[0106] In one possible implementation, the frame interpolation unit belongs to the system side, while the generation unit and execution unit belong to the target game application side. Based on this implementation, for any game application, regardless of whether it supports frame interpolation, the application can implement the frame interpolation method provided in this embodiment through interaction with the electronic device system, thereby achieving external frame interpolation processing of the game screen, improving the game frame rate while suppressing screen jitter.

[0107] Alternatively, in another implementation, the frame interpolation unit, generation unit, and execution unit can all be deployed on the target game application side. Based on this implementation, the game application itself can perform extra-frame interpolation processing on the game screen by executing the frame interpolation method provided in this application embodiment without the participation of the electronic device system side, thereby improving the game frame rate while suppressing game screen jitter.

[0108] Alternatively, the frame interpolation unit, generation unit, and execution unit can all be deployed on the system side.

[0109] This application does not impose any limitations on this.

[0110] The kernel layer can include a display driver, which can be used to control the display screen to show the rendered game graphics.

[0111] based on Figure 10 Taking the electronic device with the structure shown as an example, where the frame interpolation unit is deployed on the system side and the generation and execution units are deployed on the target game application side, the specific implementation process of the above frame interpolation method can be as follows: The frame interpolation unit on the system side obtains the VP matrix of the (n+1)th game frame from the generation unit on the target game application side. Then, based on the VP matrix of the (n+1)th game frame, the frame interpolation unit calculates the motion vector of the static object between the nth game frame and the (n+1)th game frame. Finally, the frame interpolation unit generates an extraframe based on the motion vector of the static object between the nth game frame and the (n+1)th game frame and sends it for display.

[0112] In the following embodiments of this application, based on Figure 10 The electronic device with the structure shown is used as an example to further explain the specific implementation of the above frame interpolation method, with the frame interpolation unit deployed on the system side and the generation unit and execution unit deployed on the target game application side.

[0113] Figure 11 This is another illustrative flowchart of the frame interpolation method provided in the embodiments of this application, such as... Figure 11 As shown, the frame interpolation method provided in this application includes:

[0114] 201. The execution unit on the target game application side renders the nth game frame to obtain the nth game frame to be displayed.

[0115] 202, The execution unit on the target game application side sends the nth game frame.

[0116] 203. The frame interpolation unit on the system side receives the view matrix and projection matrix of the (n+1)th game frame sent by the generation unit on the target game application side.

[0117] In this embodiment of the application, the method for the frame interpolation unit on the system side to obtain the VP matrix of the (n+1)th game frame may be to wait for the generation unit on the target game application side to actively send the VP matrix of the (n+1)th game frame.

[0118] Specifically, for any game frame during game execution, the generation unit on the target game application side can generate the data required for rendering the game frame, including the VP matrix. Furthermore, the generation unit can send data and instructions matching a pre-defined task node to the execution unit. Upon receiving relevant instructions, the execution unit can complete the operations indicated by those instructions based on the received data, including frame rendering, thereby obtaining the rendered game frame and displaying it.

[0119] In this embodiment, after generating the VP matrix, the generation unit on the target game application side can actively send the generated VP matrix to the frame interpolation unit on the system side. In this implementation, after the execution unit on the target game application side finishes rendering the nth game frame and sends it to the display, the frame interpolation unit on the system side only needs to wait to receive the VP matrix of the (n+1)th game frame. The implementation is simple and has high execution efficiency.

[0120] 204. The frame interpolation unit on the system side controls the execution unit on the target game application side to stop the rendering process of the (n+1)th game frame.

[0121] In this embodiment of the application, when using a single-threaded processing method, the frame interpolation unit on the system side can control the execution unit on the target game application side to pause the rendering process of the (n+1)th game frame before calculating the extrapolated frame.

[0122] In practical applications, the generation unit can also be used to configure specific rendering state parameters during frame rendering. These rendering state parameters are then used by the execution unit to perform rendering operations. The values ​​of the rendering state parameters change in real time during frame rendering. When the execution unit controlling the target game application pauses the rendering process for the (n+1)th game frame, the frame interpolation unit can save the rendering state parameters of the generation unit at the current moment. This prevents the loss of the rendering state parameters at the corresponding moment after the rendering process for the (n+1)th game frame is resumed.

[0123] In another implementation, a multi-threaded processing approach can be adopted. In this approach, after obtaining the VP matrix of the (n+1)th game frame, the interpolation unit on the system side can calculate the extrapolated frame based on the VP matrix. At the same time, the execution unit on the target game application side can execute the rendering process of the (n+1)th game frame after receiving the relevant instructions from the generation unit.

[0124] 205. The frame interpolation unit on the system side determines the motion vector of the static object between the nth game frame and the n+1th game frame based on the view matrix and projection matrix of the (n+1)th game frame.

[0125] 206. The interpolation unit on the system side generates extrapolated frames and controls the display of extrapolated frames based on the motion vectors of static objects between the nth game frame and the (n+1)th game frame.

[0126] The frame interpolation unit on the system side can generate an extrapolated frame based on the VP matrix of the (n+1)th game frame and control the extrapolated frame to be displayed. The specific method for generating the extrapolated frame can be referred to the foregoing embodiments of this application, and will not be repeated here.

[0127] 207. The frame interpolation unit on the system side controls the execution unit on the target game application side to resume the rendering process for the (n+1)th game frame.

[0128] 208. The execution unit on the target game application side renders the (n+1)th game frame and controls the display of the rendered game screen.

[0129] In a single-threaded processing mode, after controlling the delivery of interpolated frames, the interpolation unit on the system side can reconfigure the rendering state parameters of the generation unit based on the saved rendering state parameters. Then, the interpolation unit on the system side can control the target game application to resume the rendering process for the (n+1)th game frame. The target game application can then continue rendering the (n+1)th game frame according to the aforementioned rendering state parameters. Once the (n+1)th game frame is rendered, the execution unit on the target game application side can control the delivery of the (n+1)th game frame for display.

[0130] When using a multi-threaded processing approach, after controlling the delivery of interpolated frames, you can wait for the target game application to finish rendering the (n+1)th game frame and then control the delivery of the (n+1)th game frame.

[0131] In the above technical solution, the system can obtain the true motion trend between the current frame and the next frame based on the VP matrix of the next game frame. Based on this true motion trend, it can predict the interpolated frames between the two frames, thereby greatly improving the accuracy of the prediction results and suppressing screen jitter. Simultaneously, the target game application actively sends the VP matrix to the system, which simplifies the process of obtaining the VP matrix on the system side and improves the execution efficiency of the solution.

[0132] Figure 12 This is another illustrative flowchart of the frame interpolation method provided in the embodiments of this application, such as... Figure 12 As shown, the frame interpolation method provided in this application includes:

[0133] 301, The execution unit on the target game application side renders the nth game frame to obtain the nth game frame to be displayed.

[0134] 302, The execution unit on the target game application side sends the nth game frame.

[0135] 303, The frame interpolation unit on the system side listens for the target instructions sent to the execution unit by the target game application-side generation unit.

[0136] 304. In response to the detection of the target instruction, the frame interpolation unit on the system side obtains the view matrix and projection matrix of the (n+1)th game frame from the target instruction.

[0137] In this embodiment of the application, the method for the frame interpolation unit on the system side to obtain the VP matrix of the (n+1)th game frame can be to actively intercept from the target game application side.

[0138] Specifically, in practical applications, after the generation unit on the target game application side generates the VP matrix of the (n+1)th game frame, it sends a target instruction to the execution unit. This target instruction may carry target information representing the VP matrix. However, for different game applications, the target instruction to be intercepted and the target information representing the VP matrix within the target instruction are different. Therefore, in this embodiment, a mapping relationship between different game applications and interception methods can be pre-generated. The interception method can be used to describe the target instruction to be intercepted and the target information representing the VP matrix within the target instruction.

[0139] Based on the above explanation, after the execution unit on the target game application side completes rendering the nth game frame and sends it to the display, the generation unit on the target game application side begins generating the data required to render the (n+1)th game frame. At this time, the frame interpolation unit on the system side can use the above mapping relationship to determine the target instruction to be intercepted corresponding to the current game application, as well as the target information in the target instruction used to represent the VP matrix, based on the currently running game application. Then, the frame interpolation unit on the system side can use to listen for the target instructions sent by the generation unit to the execution unit. When the target instruction is detected, the frame interpolation unit on the system side can obtain the target information used to represent the VP matrix from the target instruction according to the above mapping relationship.

[0140] 305. The frame interpolation unit on the system side controls the execution unit on the target game application side to stop the rendering process of the (n+1)th game frame.

[0141] Similar to the aforementioned embodiments, in the case of using a single-threaded processing method, before generating the interpolated frame, the interpolation unit on the system side can first control the execution unit on the target game application side to pause the rendering process of the (n+1)th game frame. At the same time, the rendering state parameters at the current moment can be saved, thereby preventing the loss of the rendering state parameters at the corresponding moment after the rendering process of the (n+1)th game frame is resumed.

[0142] Furthermore, in the aforementioned embodiments, after generating the VP matrix, the generation unit on the target game application side actively sends the VP matrix to the frame interpolation unit on the system side. That is, in the aforementioned embodiments, the moment the generation unit generates the VP matrix is ​​the moment the generation unit sends the VP matrix to the system side. Therefore, when the system side obtains the VP matrix of the (n+1)th game frame, the generation unit on the target game application side has not yet sent the command to start rendering to the execution unit, and the rendering process for the (n+1)th game frame has not yet begun.

[0143] In this embodiment, unlike the previous embodiments, the system actively intercepts the VP matrix from the target instruction sent by the generation unit to the execution unit. Due to differences in game applications, in some game applications, the target instruction containing the VP matrix may be an instruction to start rendering. In this case, when the system intercepts the VP matrix, the execution unit may have already received the instruction to start rendering sent by the generation unit and started the rendering process.

[0144] In practical applications, the rendering process of game frames includes rendering several rendering channels sequentially. For any given rendering channel, to prevent errors in the rendering process, interruptions should be avoided as much as possible. Therefore, in this embodiment, when using a single-threaded processing method, before the execution unit on the target game application side suspends the rendering process of the (n+1)th game frame, the frame interpolation unit on the system side can also detect the rendering progress of the (n+1)th game frame.

[0145] Specifically, such as Figure 13 As shown, in step 401, the frame interpolation unit on the system side can detect whether the rendering process of the (n+1)th game frame has been started.

[0146] If the rendering process for the (n+1)th game frame has been initiated, in step 402, the frame interpolation unit can wait for the rendering process of the current rendering channel to finish. Then, after the rendering process of the current rendering channel finishes, in step 403, the frame interpolation unit can determine whether the current rendering channel is the last rendering channel. If it is determined that the current rendering channel is the last rendering channel, it means that the rendering of the (n+1)th game frame has been completed. At this point, if the resource data of the (n+1)th game frame is not stored after rendering, it will result in the loss of the resource data, causing the (n+1)th game frame to fail to display correctly after the interpolated frame is displayed. Therefore, in this embodiment, if it is determined that the current rendering channel is the last rendering channel, in step 404, the generated resource data of the (n+1)th game frame can be saved to a preset storage area. This resource data can be used by the target game application to control the display of the (n+1)th game frame. Furthermore, the frame interpolation unit on the system side can control the execution unit on the target game application side to pause the rendering process of the (n+1)th game frame. If it is determined that the current rendering channel is not the last rendering channel, it means that the rendering of the (n+1)th game frame has not yet been completed. At this time, the execution unit on the target game application side can be directly controlled to stop the rendering process of the (n+1)th game frame, and the rendering state parameters at the current moment are saved.

[0147] If it is determined that the rendering process of the (n+1)th game frame has not yet started, the execution unit on the target game application side can be directly controlled to stop the rendering process of the (n+1)th game frame, and at the same time, the rendering state parameters at the current moment are saved.

[0148] When using a multi-threaded processing approach, after obtaining the VP matrix of the (n+1)th game frame, the system-side frame interpolation unit can calculate the extrapolated frame based on this VP matrix. Simultaneously, the execution unit on the target game application side can execute the rendering process for the (n+1)th game frame. This improves the execution efficiency of the solution.

[0149] 306. The frame interpolation unit on the system side determines the motion vector of the static object between the nth game frame and the n+1th game frame based on the view matrix and projection matrix of the (n+1)th game frame.

[0150] 307. The frame interpolation unit on the system side generates and displays extrapolated frames based on the motion vectors of static objects between the nth game frame and the (n+1)th game frame.

[0151] After pausing the rendering process of the (n+1)th game frame on the target game application side, the system side can generate an interpolated frame based on the obtained VP matrix of the (n+1)th game frame and control the display of the interpolated frame. The specific method for generating the interpolated frame can be referred to the foregoing embodiments of this application, and will not be repeated here.

[0152] 308. The frame interpolation unit on the system side controls the execution unit on the target game application side to resume the rendering process for the (n+1)th game frame.

[0153] 309, the execution unit renders the (n+1)th game frame and controls the display of the rendered game screen.

[0154] In a single-threaded processing mode, after controlling the delivery of interpolated frames, the system-side interpolation unit can reconfigure the rendering state parameters of the generation unit based on the saved rendering state parameters. Furthermore, the system-side interpolation unit can control the target game application to resume the rendering process for the (n+1)th game frame. Once the (n+1)th game frame is rendered, the execution unit on the target game application side can control the delivery of the (n+1)th game frame.

[0155] When using a multi-threaded processing approach, after controlling the delivery of interpolated frames, you can wait for the target game application to finish rendering the (n+1)th game frame and then deliver it for display.

[0156] In the above technical solution, the system can obtain the true motion trend between the current frame and the next frame based on the VP matrix of the next game frame. Based on this true motion trend, it can predict the interpolated frame between the two frames, thereby greatly improving the accuracy of the prediction results and reducing screen jitter. Simultaneously, the system actively intercepts the VP matrix of the next game frame from the target game application, making this solution applicable to various game applications.

[0157] It is understandable that in the implementation where the frame interpolation unit, generation unit, and execution unit are all deployed on the target game application side, after the target game application side generates the VP matrix of the (n+1)th game frame, unlike the above implementation, the target game application side does not need to send the VP matrix to the system side, nor does the system side need to intercept the VP matrix from the target game application side. Instead, the target game application side determines the motion vector of the static object between the nth game frame and the (n+1)th game frame based on the VP matrix of the (n+1)th game frame, and generates the interpolated frame and controls the display of the interpolated frame based on the motion vector of the static object between the nth game frame and the (n+1)th game frame.

[0158] The specific implementation of the target game application side determining the motion vector of the static object between the nth game frame and the n+1th game frame based on the VP matrix of the (n+1)th game frame, and generating the interpolated frame based on the motion vector of the static object between the nth game frame and the n+1th game frame, can still refer to the above embodiment, and will not be repeated here.

[0159] It should be understood that in this application, the term "unit" can be implemented in software and / or hardware, and is not specifically limited thereto. For example, a "unit" can be a software program, a hardware circuit, or a combination of both that implements the above-described functions. The hardware circuit may include application-specific integrated circuits (ASICs), electronic circuits, a processor (e.g., a shared processor, a proprietary processor, or a group processor, etc.) and memory for executing one or more software or firmware programs, combined logic circuits, and / or other suitable components supporting the described functions. Whether a function is implemented in hardware or in a manner driven by computer software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments.

[0160] The module division in this embodiment is illustrative and represents only one logical functional division. In actual implementation, other division methods are possible. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into a single processing module. The integrated modules described above can be implemented in hardware.

[0161] This application also provides an electronic device, which includes a storage medium and a central processing unit. The storage medium may be a non-volatile storage medium, and a computer-executable program is stored in the storage medium. The central processing unit is connected to the non-volatile storage medium and executes the computer-executable program to implement the above-described frame interpolation method.

[0162] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the various steps of the frame interpolation method of this application.

[0163] This application also provides a computer program product containing instructions that, when run on a computer or any at least one processor, causes the computer to execute the various steps of the frame interpolation method of this application.

[0164] This application also provides a chip, including a processor and a data interface. The processor reads instructions stored in a memory through the data interface to execute corresponding operations and / or processes performed by the frame interpolation method provided in this application.

[0165] Optionally, the chip further includes a memory connected to the processor via a circuit or wire, the processor being used to read and execute computer programs stored in the memory. Further optionally, the chip includes a communication interface to which the processor is connected. The communication interface is used to receive data and / or information that needs to be processed, the processor obtaining the data and / or information from the communication interface and processing the data and / or information. The communication interface can be an input / output interface.

[0166] The memory can be read-only memory (ROM), other types of static storage devices that can store static information and instructions, random access memory (RAM), or other types of dynamic storage devices that can store information and instructions. It can also be electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices. Alternatively, it can be any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.

[0167] In this embodiment, "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0168] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0169] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0170] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0171] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method of inserting frames, characterized by, include: Displays the nth game frame; Obtain the view matrix and projection matrix of the (n+1)th game frame; Based on the view matrix and projection matrix of the (n+1)th game frame, generate the first interpolated frame; The first interpolated frame is displayed before the (n+1)th game frame is displayed; The step of generating the first extrapolated frame based on the view matrix and projection matrix of the (n+1)th game frame includes: Based on the view matrix and projection matrix of the (n+1)th game frame, determine the motion vector of the static object between the (n+1)th game frame and the nth game frame; The first extrapolated frame is generated based on the motion vectors of static objects between the (n+1)th game frame and the nth game frame, and the motion vectors of dynamic objects between the (n-1)th game frame and the nth game frame.

2. The method of claim 1, wherein, The process of obtaining the view matrix and projection matrix of the (n+1)th game frame includes: Receive the view matrix and projection matrix of the (n+1)th game frame sent by the target game application.

3. The method according to claim 1, characterized in that, The process of obtaining the view matrix and projection matrix of the (n+1)th game frame includes: Listen for target commands in the target game application; In response to the detected target instruction, the view matrix and projection matrix of the (n+1)th game frame are obtained from the target instruction.

4. The method according to claim 3, characterized in that, Before monitoring the target instructions in the target game application, the method further includes: Obtain a pre-configured target mapping relationship, which describes the mapping relationship between different game applications and interception methods; Based on the target mapping relationship, the interception method corresponding to the target game application is determined. The interception method is used to describe the target instruction to be monitored and the target information in the target instruction used to represent the view matrix and projection matrix.

5. The method according to claim 1, characterized in that, Before generating the first interpolated frame based on the view matrix and projection matrix of the (n+1)th game frame, the method further includes: Control the cessation of the rendering process for the (n+1)th game frame.

6. The method according to claim 5, characterized in that, Before the control stops the rendering process for the (n+1)th game frame, the method further includes: Detection indicates that the (n+1)th game frame has started rendering; wait for the current rendering pass to finish rendering; or... Rendering of the (n+1)th game frame has not been started.

7. The method according to claim 5 or 6, characterized in that, After displaying the first extrapolated frame, the method further includes: Control the resumption of the rendering process for the (n+1)th game frame.

8. The method according to claim 1, characterized in that, The step of determining the motion vector of the static object between the (n+1)th game frame and the nth game frame based on the view matrix and projection matrix of the (n+1)th game frame includes: Based on the view matrix and projection matrix of the (n+1)th game frame, calculate the clipping coordinates of the static objects in the (n+1)th game frame. Based on the clipping coordinates of the static objects in the (n+1)th game frame, determine the motion vector of the static objects between the (n+1)th game frame and the nth game frame.

9. The method according to claim 8, characterized in that, The step of calculating the clipping coordinates of static objects in the (n+1)th game frame based on the view matrix and projection matrix of the (n+1)th game frame includes: Calculate the clipping coordinates of static objects in the (n+1)th game frame using the following formula: ； Among them, the The clipping coordinates of the static object in the (n+1)th game frame are given. Let the clipping coordinates of the static object in the nth game frame be [the coordinates of the clipping coordinates of the object]. The view matrix and projection matrix of the (n+1)th game frame are given. These are the view matrix and projection matrix corresponding to the nth game frame.

10. An electronic device, characterized in that, include: One or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs including instructions that, when executed by the device, cause the device to perform the method as described in any one of claims 1-9.

11. A storage medium, characterized in that, The storage medium stores program instructions that, when run on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-9.

Images