Prediction frame generation method, terminal device, and storage medium

Abstract

Embodiments of the present application provide a kind of prediction frame generation method, terminal equipment and storage medium, involve terminal technical field.In game scene, terminal equipment can obtain 3D scene image and UI image corresponding to each real frame from two game rendering instructions respectively, and according to game rendering instruction, obtain general information such as depth image, world matrix and projection matrix of game virtual camera, then utilize these information to do pixel offset to 3D scene image, then fill blank area in 3D scene image using AI image repair technology, then superimpose UI image on repaired 3D scene image, to generate prediction frame.The scheme is lower to the requirement of operation ability, can be run in real time on terminal equipment such as mobile phone with limited operation ability, and the prediction frame generated finally is closer to the original quality of game.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a predictive frame generation method, terminal device, and storage medium. Background Technology

[0002] Currently, game scene rendering has increasingly higher requirements for screen smoothness and image clarity, which relies on higher screen refresh rates and graphics rendering resolutions, posing a severe challenge to the chip power consumption load of terminal devices.

[0003] To reduce chip power consumption, various frame prediction schemes have been proposed based on the similarity of two consecutive rendered frames in a game. However, these schemes have several problems. For example, one scheme involves semi-transparent blending of the two frames using motion vectors as weights, but the predicted frame obtained using this scheme is prone to image quality issues such as ghosting. Another scheme involves a personal computer (PC) generating a complete predicted frame based on artificial intelligence (AI) technology, but this scheme has high computational and bandwidth requirements and cannot run in real time on terminal devices with limited computing power, such as mobile phones. Summary of the Invention

[0004] This application provides a prediction frame generation method, a terminal device, and a storage medium, which solves the technical problem of how to generate prediction frames with near-native game image quality on terminal devices with limited computing power.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] In a first aspect, embodiments of this application provide a method for generating a predicted frame. This method may include:

[0007] Obtain the user interface image UI0, the first scene image Color0, the first depth image Depth0, the first projection matrix ProjectionMatrix0, and the first world matrix ViewMatrix0 from the first rendering instruction stream; obtain the second scene image Color1, the second depth image Depth1, the second projection matrix ProjectionMatrix1, and the second world matrix ViewMatrix1 from the second rendering instruction stream; based on the first depth image Depth0, the first projection matrix ProjectionMatrix0, the first world matrix ViewMatrix0, the second depth image Depth1, the second projection matrix ProjectionMatrix1, and the second world matrix ViewMatrix1... The process involves: identifying static and dynamic pixels in the first scene image Color0; determining the motion vector of the static pixels based on their pixel coordinates and the reprojection coordinates after reprojection; identifying the matching pixel of the dynamic pixels in the second scene image Color1; determining the motion vector of the dynamic pixels based on their pixel coordinates and the matching pixel coordinates; moving the static pixels based on their motion vectors and the dynamic pixels based on their motion vectors to obtain an intermediate image Color_NewFrame; filling the blank areas of the intermediate image Color_NewFrame with pixels using an AI model to obtain a scene prediction image Pixel_Color; and overlaying the user interface image UI0 onto the scene prediction image Pixel_Color to obtain a prediction frame.

[0008] In the above scheme, by obtaining the scene image and UI image corresponding to each real frame from two game rendering instructions, as well as general information such as depth image, world matrix and projection matrix of the game virtual camera, this information can be used to perform pixel-by-pixel offset of static and dynamic pixels in the 3D scene image, thereby solving the physical distortion problem of the block-by-block offset scheme and the ghosting problem of the image blending scheme. In addition, an AI model is used to fill the blank areas after pixel-by-pixel offset, making the final generated predicted frame image closer to the original image quality of the game. Unlike the PC-based AI model that generates the entire frame, the AI ​​model in this application only repairs the blank areas of the image, which has lower computational requirements and can run in real time on terminal devices with limited computing power, such as mobile phones.

[0009] In one possible implementation, the first and second rendering instruction streams are game rendering instruction streams. The game rendering instruction streams can include various drawing instructions, such as scene drawing instructions, UI drawing instructions, and image display instructions. Scene drawing instructions can be used to draw 3D scene images such as in-game scenery, characters, special effects, and skills; UI drawing instructions can be used to draw UI images such as control buttons, minimaps, and floating text; and image display instructions are used to place the drawn image data into a specified location in the operating system to complete frame display.

[0010] In one possible implementation, a game frame can consist of 3D scene images and UI images. The 3D scene image is also called a color image. For example, the 3D scene image may include in-game scenery, game characters, background objects, special effects, and skills, while the UI image may include control buttons, a minimap, floating text, user avatars, energy pack controls, and health bars. The UI image sits on top of the 3D scene image. It should be understood that both the game characters in the 3D scene image and the rendering control buttons in the UI image are objects within the game frame.

[0011] In one possible implementation, the pixel coordinates of the 3D scene image and the UI image have a one-to-one correspondence, with the UI image positioned above the 3D scene image. If an object in the UI image has the same pixel coordinates as an object in the 3D scene image, the object in the UI image may visually occlude the object in the 3D scene image. It should be understood that separating the 3D scene image and the UI image from the rendering instruction stream facilitates scene prediction of the 3D scene image later.

[0012] In one possible implementation, based on a first depth image Depth0, a first projection matrix ProjectionMatrix0, a first world matrix ViewMatrix0, a second depth image Depth1, a second projection matrix ProjectionMatrix1, and a second world matrix ViewMatrix1, static and dynamic pixels in a first scene image Color0 are determined, including:

[0013] For each pixel in the first scene image Color0, perform the following steps: Based on the first depth image Depth0, the first projection matrix ProjectionMatrix0, and the first world matrix ViewMatrix0, determine the first world space coordinates corresponding to the first pixel coordinates of the first scene image Color0; based on the second projection matrix ProjectionMatrix1 and the second world matrix ViewMatrix1, reproject the first world space coordinates onto the pixel coordinate system to obtain the second pixel coordinates; based on the second depth image Depth1, the second projection matrix ProjectionMatrix1, and the second world matrix ViewMatrix1, determine the second world space coordinates corresponding to the second pixel coordinates. For any pixel in the first scene image Color0: if the first world space coordinates and the second world space coordinates of a pixel are the same, then the pixel is marked as a static pixel; or, if the first world space coordinates and the second world space coordinates of a pixel are different, then the pixel is marked as a dynamic pixel.

[0014] It should be understood that for the same pixel: if the depth in depth image Depth0 is different from the depth in depth image Depth1, then after performing the coordinate transformation as described above, the pixel will correspond to two different positions in the world coordinate system O-XYZ. This means that the pixel at coordinates (w0, h0) on the first scene image Color0 has a changed position in the world coordinate system. This pixel belongs to a dynamic object (such as a running person), and can therefore be marked as Mask = 1. If the depth in depth image Depth0 is the same as the depth in depth image Depth1, then after performing the coordinate transformation as described above, the pixel will correspond to the same position in the world coordinate system O-XYZ. This means that the pixel at coordinates (w0, h0) on the 3D scene image Color0 has no changed position in the world coordinate system. This pixel belongs to a static object (such as a stationary house), and can therefore be marked as Mask = 0. After marking all pixels in the first scene image Color0, the masks for all pixels can be output onto a single mask image, thus generating a mask image for marking static and dynamic pixels. The pixel coordinates of the mask image correspond one-to-one with the pixel coordinates of the first scene image Color0.

[0015] In one possible implementation, determining the motion vector of a static pixel based on its pixel coordinates and the reprojected coordinates after reprojection can include: determining the world space coordinates corresponding to the static pixel coordinates (w0, h0) of the first scene image Color0 based on the first depth image Depth0, the first projection matrix ProjectionMatrix0, and the first world matrix ViewMatrix0; reprojecting the world space coordinates corresponding to the static pixel coordinates (w0, h0) onto the pixel coordinate system based on the second projection matrix ProjectionMatrix1 and the second world matrix ViewMatrix1 to obtain the reprojected coordinates (w1, h1); and determining [(w1-w0)*0.5, (h1-h0)*0.5] as the motion vector of the static pixel.

[0016] It should be understood that when a pixel is a static pixel, the motion vector of the static pixel can be calculated by reprojection based on the characteristics of the static pixel.

[0017] In one possible implementation, the matching pixel of the dynamic pixel is determined in the second scene image Color1. Based on the pixel coordinates of the dynamic pixel and the pixel coordinates of the matching pixel, the motion vector of the dynamic pixel is determined. This may include: obtaining the brightness and feature vector corresponding to the dynamic pixel coordinates (w0', h0') of the first scene image Color0; searching for the matching pixel (w1', h1') in the second scene image Color1 based on the brightness and feature vector corresponding to the dynamic pixel coordinates (w0', h0'); and determining [(w1'-w0')*0.5, (h1'-h0')*0.5] as the motion vector of the dynamic pixel.

[0018] For example, the matching pixels mentioned above can satisfy the following conditions:

[0019] Condition 1: The absolute value of the first difference is less than the absolute value of the second difference. The first difference is the difference between the brightness corresponding to the dynamic pixel coordinate (w0', h0') and the brightness corresponding to the matching pixel (w1', h1'). The second difference is the difference between the brightness corresponding to the dynamic pixel coordinate (w0', h0') and the brightness corresponding to other pixel coordinates of the second scene image Color1.

[0020] Condition 2: The first feature distance is less than the second feature distance. The first feature distance is the feature distance between the feature vector corresponding to the dynamic pixel coordinates (w0', h0') and the feature vector corresponding to the matching pixel (w1', h1'). The second feature distance is the feature distance between the feature vector corresponding to the dynamic pixel coordinates (w0', h0') and the feature vector corresponding to the other pixel coordinates of the second scene image Color1.

[0021] It should be understood that when a pixel is a dynamic pixel, based on the characteristics of dynamic pixels, the motion vector of the dynamic pixel can be calculated by searching for the best matching pixel.

[0022] In one possible implementation, moving static pixels based on motion vectors of static pixels and moving dynamic pixels based on motion vectors of dynamic pixels to obtain an intermediate image Color_NewFrame includes: creating an intermediate image Color_NewFrame; moving each static pixel to the intermediate image Color_NewFrame one by one based on the motion vector of each static pixel in the first scene image Color0; and moving each dynamic pixel to the intermediate image Color_NewFrame one by one based on the motion vector of each dynamic pixel in the first scene image Color0.

[0023] Specifically, after creating the intermediate image Color_NewFrame, you can assign a value of 0.0 to the RGB channel of the intermediate image Color_NewFrame and a value of 0.5 to the A channel of the intermediate image Color_NewFrame.

[0024] Accordingly, based on the motion vector of each static pixel in the first scene image Color0, each static pixel is moved sequentially into the intermediate image Color_NewFrame, including: determining the pixel coordinates corresponding to each static pixel in the intermediate image Color_NewFrame based on the motion vector of each static pixel in the first scene image Color0; assigning a first value to the RGB channel of the pixel coordinates corresponding to each static pixel in the intermediate image Color_NewFrame, and assigning a value of 1.0 to the A channel of the pixel coordinates corresponding to each static pixel in the intermediate image Color_NewFrame. Here, the first value is the RGB channel pixel value of each static pixel in the first scene image Color0.

[0025] Accordingly, based on the motion vector of each dynamic pixel in the first scene image Color0, each dynamic pixel is moved sequentially into the intermediate image Color_NewFrame. This includes: determining the pixel coordinates corresponding to each dynamic pixel in the intermediate image Color_NewFrame based on the motion vector of each dynamic pixel in the first scene image Color0; assigning a second value to the RGB channel of the pixel coordinates corresponding to each dynamic pixel in the intermediate image Color_NewFrame, and assigning a value of 1.0 to the A channel of the pixel coordinates corresponding to each dynamic pixel in the intermediate image Color_NewFrame. Here, the second value is the RGB channel pixel value of each dynamic pixel in the first scene image Color0.

[0026] In one possible implementation, the method may further include: if multiple pixels move to the same pixel coordinate in the intermediate image Color_NewFrame, then, based on the depth of each pixel in the first depth image Depth0, transform the same pixel coordinate to the camera coordinate system to obtain the camera coordinate of each pixel; calculate the distance from the camera coordinate of each pixel to the camera origin based on the camera coordinate of each pixel; and assign a value to the RGB channel of the same pixel coordinate based on the RGB channel pixel value of the pixel corresponding to the smallest distance among the multiple distances. It should be understood that after offsetting all pixels in the first scene image Color0, multiple pixels in the first scene image Color0 may move to the same position in the intermediate image Color_NewFrame, causing pixel occlusion at that position. Therefore, the pixel corresponding to the position closest to the camera origin can be calculated as the pixel finally displayed in the intermediate image Color_NewFrame.

[0027] In one possible implementation, the scene prediction image Pixel_Color is obtained by filling the blank areas of the intermediate image Color_NewFrame with pixels using an AI model. This can include: generating a target edge image edge_map based on a first scene image Color0 and a second scene image Color1; and inputting the target edge image edge_map and the intermediate image Color_NewFrame into the AI ​​model to generate the scene prediction image Pixel_Color. The scene prediction image Pixel_Color is the image obtained by the AI ​​model after filling the blank areas of the intermediate image Color_NewFrame with pixels using the target edge image edge_map.

[0028] It should be understood that traditional AI models are typically loaded onto devices with high computing power, such as PCs. By inputting real frames into the AI ​​model, a complete predicted frame can be obtained. However, traditional AI models have complex algorithms and high requirements for computation and bandwidth, making them unsuitable for real-time operation on terminal devices with limited computing power, such as mobile phones. Unlike traditional AI models, the AI ​​model provided in this application is only used to repair blank areas in an image, requiring less computing power and can run in real-time on terminal devices with limited computing power, such as mobile phones. Furthermore, by predicting data missing areas, the generated 3D scene prediction image Pixel_Color is made closer to the original image.

[0029] In one possible implementation, generating a target edge image edge_map based on a first scene image Color0 and a second scene image Color1 includes: extracting a first edge image edge0_warp from the first scene image Color0 and extracting a second edge image edge1_warp from the second scene image Color1; warping the first edge image edge0_warp in pixel blocks and outputting the warped result of the first edge image edge0_warp to a first warped edge image; warping the second edge image edge1_warp in pixel blocks and outputting the warped result of the second edge image edge1_warp to a second warped edge image; and generating the target edge image edge_map based on the A channel label of the intermediate image Color_NewFrame and the first warped edge image and the second warped edge image.

[0030] Specifically, the first edge image edge0_warp, the second edge image edge1_warp, the first distorted edge image, and the second distorted edge image are single-channel images.

[0031] Accordingly, warping the first edge image edge0_warp in units of pixel blocks and outputting the warping result of the first edge image edge0_warp to the first warped edge image may include: determining the pixel coordinates corresponding to the pixels in the first edge image edge0_warp in the first warped edge image according to the motion vectors corresponding to each vertex of each grid in the 2D mesh model; and collecting pixel values ​​from the first edge image edge0_warp according to the texture coordinates corresponding to the pixel coordinates in the first warped edge image to obtain the first warped edge image with the warping result of the first edge image edge0_warp saved.

[0032] Accordingly, warping the second edge image edge1_warp in units of pixel blocks and outputting the warping result of the second edge image edge1_warp to the second warped edge image may include: determining the pixel coordinates corresponding to the pixels in the second edge image edge1_warp in the second warped edge image according to the opposite vector of the motion vector corresponding to each vertex of each grid in the 2D mesh model; and acquiring pixel values ​​from the second edge image edge1_warp according to the texture coordinates corresponding to the pixel coordinates in the second warped edge image to obtain the second warped edge image with the warping result of the second edge image edge1_warp saved.

[0033] Accordingly, based on the A-channel marker of the intermediate image, the target edge image is generated according to the first distorted edge image and the second distorted edge image, including: if the A-channel marker of a pixel coordinate in the intermediate image Color_NewFrame is 0.5, then the corresponding pixel coordinate in the target edge image edge_map is filled with pixel data from the second edge image edge1_warp; or, if the A-channel marker of a pixel coordinate in the intermediate image Color_NewFrame is 1, then the corresponding pixel coordinate in the target edge image edge_map is filled with pixel data from the first edge image edge0_warp. It should be understood that if A = 0.5 for a pixel coordinate in the intermediate image Color_NewFrame, the edge image edge1_warp can be used to fill the blank area. If A ≠ 0.5 for a pixel coordinate in the intermediate image Color_NewFrame, pixel data from the edge image edge0_warp can be used to fill the blank area, thereby generating a complete edge image, which facilitates edge recognition by the AI ​​model later.

[0034] In one possible implementation, the first rendering instruction stream is acquired before the second rendering instruction stream. After overlaying the user interface image onto the scene prediction image Pixel_Color to obtain the prediction frame, the method may further include: sequentially displaying the first frame, the prediction frame, and the second frame. The first frame is a real image frame generated according to the first rendering instruction stream, and the second frame is a real image frame generated according to the second rendering instruction stream.

[0035] It should be understood that, based on the first and second rendering instruction streams, not only can two real frames be generated, but a predicted frame can also be inserted between the two real frames. Since the computational resource requirements for generating the predicted frame are less than the resource requirements for rendering real frames in a game application, the resource consumption of the terminal device is reduced.

[0036] Secondly, this application provides an apparatus comprising units for performing the method described in the first aspect above. This apparatus corresponds to performing the prediction frame generation method described in the first aspect above. For a detailed description of the units within this apparatus, please refer to the description in the first aspect above; for brevity, it will not be repeated here.

[0037] Thirdly, this application provides a terminal device including a memory and one or more processors. The memory stores computer program code, which includes computer instructions. When the computer instructions are invoked by the processor, the terminal device executes the predictive frame generation method provided in any of the first aspects.

[0038] Fourthly, this application provides a computer-readable storage medium. The computer-readable storage medium includes computer instructions. When the computer instructions are executed on a terminal device, the terminal device performs the predictive frame generation method provided by the first aspect and any possible implementation thereof.

[0039] Fifthly, this application provides a computer program product. When the computer program product is run on a computer, it causes the computer to execute the predictive frame generation method provided by the first aspect and any possible implementation thereof.

[0040] Sixthly, this application provides a chip system. The chip system includes one or more interface circuits and one or more processors. The interface circuits and processors are interconnected via lines. The chip system can be applied to a terminal device including a communication module and a memory. The interface circuits are used to receive signals from the terminal device's memory and send the received signals to the processor, the signals including computer instructions stored in the memory. When the processor invokes the computer instructions, the terminal device can execute the predictive frame generation method provided by the first aspect and any possible implementation thereof.

[0041] It is understood that the beneficial effects achieved by the apparatus of the second aspect, the terminal device of the third aspect, the computer-readable storage medium of the fourth aspect, the computer program product of the fifth aspect, and the chip system of the sixth aspect can be referred to the beneficial effects of the first aspect and any possible implementation thereof, which will not be repeated here. Attached Figure Description

[0042] Figure 1 A schematic diagram of the hardware structure of a mobile phone provided in an embodiment of this application;

[0043] Figure 2 This is a schematic diagram of the architecture of a terminal device provided in an embodiment of this application;

[0044] Figure 3 A schematic diagram illustrating the rendering and display process of game video frames provided in an embodiment of this application;

[0045] Figure 4A A flowchart illustrating a prediction frame generation method provided in an embodiment of this application;

[0046] Figure 4B A flowchart illustrating another prediction frame generation method provided in an embodiment of this application;

[0047] Figure 4C A flowchart illustrating another prediction frame generation method provided in this application embodiment;

[0048] Figure 5AA schematic diagram illustrating the change of pixel coordinates based on the movement of virtual objects in a game scene, provided as an embodiment of this application;

[0049] Figure 5B A schematic diagram illustrating the change of pixel coordinates based on the movement of a virtual camera in a game scene, as provided in an embodiment of this application;

[0050] Figure 6 This is a schematic diagram illustrating the separation of 3D scene images and UI images from the rendering instruction stream, as provided in an embodiment of this application.

[0051] Figure 7 A schematic diagram of various coordinate systems provided for embodiments of this application;

[0052] Figures 8A-8C A schematic diagram illustrating the determination of static and dynamic pixels provided in an embodiment of this application;

[0053] Figure 9 A schematic diagram illustrating the generation of a mask image provided in an embodiment of this application;

[0054] Figure 10 A schematic diagram of pixel-by-pixel offset based on motion vector provided for an embodiment of this application;

[0055] Figure 11 This is a schematic diagram illustrating the movement of multiple pixels to the same position in an intermediate image, as provided in an embodiment of this application.

[0056] Figure 12 A schematic diagram illustrating the occlusion of multiple pixels at the same position in the intermediate image, provided as an embodiment of this application;

[0057] Figure 13 A schematic diagram of a 2D mesh model provided in an embodiment of this application;

[0058] Figures 14A-14C This is a schematic diagram illustrating the acquisition of an edge image corresponding to an intermediate image, provided in an embodiment of this application.

[0059] Figure 15 A schematic diagram illustrating the repair of blank areas in an image based on an AI model, as provided in an embodiment of this application.

[0060] Figure 16 This is a schematic diagram illustrating the overlay of a UI image onto a 3D scene prediction image, provided as an embodiment of this application.

[0061] Figure 17 This is a schematic diagram illustrating the frame-by-frame playback of game footage by a terminal device provided in an embodiment of this application. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this application, but not all embodiments.

[0063] In the description of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. In the description of this application, "and / or" is merely a way of describing the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone.

[0064] In the specification and claims of this application, the terms "first" and "second," etc., are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a specific order of the objects. For example, "first image" and "second image," etc., are used to distinguish different images, rather than to describe a specific order of the images. In the embodiments of this application, "a plurality of" refers to two or more.

[0065] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, phrases such as "in some embodiments" appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0066] To facilitate understanding of this application, some of the terms used in this application will be explained below.

[0067] 1. Frame rate refers to the number of frames generated per second for display during application runtime. Terminal devices typically involve processes such as drawing, rendering, and compositing to generate frames for display. The unit of frame rate is frames per second (fps). Frame drawing refers to drawing images for the display interface; frame rendering involves coloring or adding 3D effects to the drawn images; and frame compositing is the process of combining at least one rendered view into the display interface.

[0068] 2. Refresh rate, also known as screen refresh rate, refers to the number of times the screen refreshes its image per second. Each refresh represents displaying one frame. Typically, the refresh rate depends on fixed hardware parameters. The more powerful the hardware of the terminal device, the higher the supported refresh rate, and the smoother the screen display. The unit of refresh rate is Hertz (Hz).

[0069] 3. Inter-frame prediction technology: This refers to using the correlation between frames in the temporal domain to predict the pixels of intermediate frames using pixels from neighboring real frames, thereby effectively removing temporal redundancy. Inter-frame prediction technology is generally well-established in the video playback field. For example, it involves weighted mixing of two video frames or using optical flow to calculate motion vectors for pixel blurring.

[0070] This application primarily relates to inter-frame prediction in the gaming field. Specifically, when running game applications on terminal devices, the frame rate can reach 60fps or even 100fps, consuming significant system resources and causing problems such as rapid battery drain, battery overheating, or system lag. Research shows that the resource consumption of generating predicted frames using inter-frame prediction technology is less than the resource consumption of rendering real frames in a game application. Therefore, reducing the number of real frames generated by game applications and using inter-frame prediction technology to generate predicted frames can ensure the total number of game frames while also reducing the power consumption load on the chip.

[0071] However, game-rendered images have significantly higher clarity and sharpness than video. In game scenes, predicted frames must not only approach or achieve the clarity and sharpness of real frames, but also ensure the accuracy of the image content and prevent distortion of objects in the image. Therefore, traditional video inter-frame prediction techniques used in video playback cannot meet the requirements for generating predicted game frames.

[0072] Frame prediction techniques for game scenarios typically fall into several categories: One type involves semi-transparent blending of two consecutive frames using motion vectors as weights. However, this method often results in image quality issues such as ghosting. Another type loads an AI model onto a PC, inputting real frames into the model to obtain a complete predicted frame. However, AI models are complex, requiring significant computation and bandwidth, making them unsuitable for real-time operation on mobile devices with limited computing power. A third type employs block-based motion compensation, first finding the optimal estimation block within the coded block to minimize the residual between the predicted and current blocks, then calculating the motion vector (mv) of the current block. Based on this motion vector, the estimated value of the current frame is obtained. However, block-by-block offsetting can easily lead to distortion of objects in the image. Therefore, a streamlined and efficient algorithm design, along with a robust framework, is essential for applying frame prediction techniques to mobile devices with limited computing power.

[0073] In view of the above problems, this application provides a method for predictive frame generation. In this method, the terminal device can obtain the three-dimensional (3D) scene image and user interface (UI) image corresponding to each real frame from two game rendering instructions, as well as general information such as depth image, world matrix and projection matrix of the game virtual camera. Then, it uses this information to perform pixel-by-pixel offset on the 3D scene image, thereby solving the physical distortion problem of the block-by-block offset scheme and the ghosting problem of the image blending scheme. Afterwards, the terminal device can fill the blank areas after pixel-by-pixel offset based on an AI model, so that the final generated predictive frame image is closer to the original image quality of the game. It should be understood that, unlike the scheme of generating the entire frame by a PC-based AI model, the AI ​​model of this application only repairs the blank areas of the image, which has lower requirements for computing power and can run in real time on terminal devices with limited computing power, such as mobile phones.

[0074] In some embodiments, the aforementioned terminal device is also referred to as a terminal or user equipment (UE). For example, a terminal device can be a mobile phone, smart screen, smart TV, tablet computer, wearable device, virtual reality (VR) device, augmented reality (AR) device, or other terminal device with limited computing power. It should be understood that the prediction frame generation method provided in this application can also be applied to terminal devices with sufficient computing power, such as PCs. This application does not impose any limitations on the specific type of terminal device. Regardless of the type of terminal device to which the prediction frame generation method provided in this application is applied, the computational resource requirement for generating prediction frames is less than the resource requirement for rendering real frames in a game application, thereby reducing the resource consumption of the terminal device.

[0075] To better understand the embodiments of this application, the structure of the terminal device of the embodiments of this application is described below.

[0076] Take a mobile phone as an example as the terminal device. Figure 1 This is a schematic diagram of the hardware structure of a mobile phone 100 provided in an embodiment of this application.

[0077] like Figure 1 As shown, the mobile phone 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a SIM card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, and a bone conduction sensor 180M, etc.

[0078] The processor 110 may include one or more processing units, such as a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), an image signal processor (ISP), a neural network processing unit (NPU), a video codec, a modem, and / or a baseband processor.

[0079] The CPU is the final execution unit for information processing and program execution. Its main tasks include processing instructions, executing operations, controlling timing, and processing data. The CPU may include a controller, an arithmetic logic unit (ALU), a cache memory, and a bus for connecting these components. The controller is the nerve center and command center of the mobile phone 100. The controller can generate operation control signals based on the instruction opcode and timing signals to control instruction fetching and execution. The ALU refers to the component that performs various arithmetic and logical operations. The cache memory can store instructions or data that the processor 110 has just used or is recurring.

[0080] A GPU, also known as a graphics processing unit, visual processor, or display chip, is a microprocessor for graphics and image processing. The GPU connects to the display (I / O), processing unit (AP), and CPU. GPUs can perform complex mathematical and geometric calculations, floating-point operations, parallel computing, and graphics rendering. GPUs reduce the graphics card's reliance on the CPU, especially in 3D graphics processing. GPUs employ core technologies such as cubic environment mapping and vertex blending, thus performing some of the work that would otherwise be done by the CPU.

[0081] NPU stands for Neural Network (NN) Computing Processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs can enable applications such as intelligent cognition in mobile phones.

[0082] In this embodiment, the predicted frame generation method is primarily executed by the GPU. For example, the GPU can obtain the 3D scene image and UI image corresponding to each real frame from two game rendering instructions, as well as general information such as depth image, world matrix and projection matrix of the game virtual camera. Using this information, it generates a mask image marking moving and static pixels, calculates the motion vector for each pixel based on the mask image, performs pixel-by-pixel offset on the 3D scene image based on the motion vector, and then calls the NPU's neural network model to perform image inpainting on the blank areas after pixel-by-pixel offset, ultimately generating the predicted frame image. Detailed descriptions of the following embodiments are provided and will not be repeated here.

[0083] The internal memory 121 can be used to store executable program code, including instructions. The processor 110 executes various functional applications and data processing of the mobile phone 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. The program storage area can store the operating system and at least one application (APP) required for a function, such as a game application. The data storage area can store configuration files for each APP, as well as data created during the use of the mobile phone 100.

[0084] Mobile phone 100 can achieve audio functions through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and AP, etc. For example, playing game background sounds, real-person dialogue, etc.

[0085] The display screen 194 includes a signal processing module and a display panel. It is used to display images, videos, etc., such as displaying a game interface.

[0086] It is understood that the hardware structure illustrated in the embodiments of this application does not constitute a specific limitation on the mobile phone 100. In other embodiments, the mobile phone 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. Figure 1 The components shown can be implemented in hardware, software, or a combination of both.

[0087] The software system of a terminal device can adopt a layered architecture, event-driven architecture, microkernel architecture, or microservice architecture. This embodiment of the invention uses the layered architecture of the Android system as an example to illustrate the software architecture of the terminal device.

[0088] Figure 2 A schematic diagram of the architecture of a terminal device provided in an embodiment of this application is shown. Figure 2As shown, terminal devices can adopt a layered architecture, dividing the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the software layers of the software architecture are divided from top to bottom as follows: application (APP) layer, application framework (FW) layer, system library (FWK LIB) layer, and kernel layer. This software architecture runs on top of the hardware layer, which may include GPUs, NPUs, and displays, etc.

[0089] The application layer, also known as the application layer, can include a series of application packages. For example, the application layer may include game applications, camera applications, calendar applications, phone applications, and settings applications. When these application packages are run, they can access the various service modules provided by the application framework layer through application programming interfaces (APIs) and execute corresponding intelligent business logic.

[0090] The application framework layer provides APIs and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example... Figure 2 As shown, the application framework layer can include a graphics manager, a video codec, a window manager, a resource manager, and a notification manager. The graphics manager is responsible for calling interfaces in the graphics library to complete drawing, rendering, and compositing the interface; the video codec is responsible for video encoding and decoding; the window manager manages all windows in the system; the resource manager manages system resources; and the notification manager manages notification messages in the phone's top status bar.

[0091] The system library can include multiple functional modules, such as a surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), and a 2D graphics engine (e.g., SGL). The surface manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The media libraries support playback and recording of various common audio and video formats, as well as still image files. The media libraries support various audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG. The 3D graphics processing libraries are used to implement 3D graphics drawing, image rendering, compositing, and layer processing. The 2D graphics engine is the drawing engine for 2D graphics.

[0092] Within the system libraries, the Android Runtime comprises the core libraries and the virtual machine. The Android Runtime is responsible for the scheduling and management of the Android system. The core libraries consist of two parts: one part contains the functionalities that Java calls, and the other part comprises the core Android libraries. The application layer and application framework layer run within the virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used for managing object lifecycles, stack management, thread management, security and exception management, and garbage collection, among other functions.

[0093] The kernel layer is the layer between hardware and software, belonging to the lower layers of the Android system. The kernel layer can contain various driver interfaces, such as GPU drivers, NPU drivers, CPU drivers, and display drivers. It can also contain power management, process / thread management, and other functionalities.

[0094] It should be noted that although the embodiments of this application are illustrated using the Android system as an example, the basic principles are also applicable to terminal devices based on operating systems such as iOS or Windows.

[0095] Figure 3 The rendering and display process of game video frames provided in the embodiments of this application is illustrated.

[0096] Game applications run on the operating system of the terminal device. Within the game scene, the application modifies function pointers of the rendering API to generate the game's rendering instruction stream, such as by hooking the game's rendering instruction stream. This rendering instruction stream is then passed to the GPU via the graphics manager and GPU driver. The GPU obtains functions through the rendering instructions and, based on the characteristics of the game's rendering instructions, identifies the game's rendering instruction stream.

[0097] A rendering instruction stream is a sequence of drawing instructions (rendering instruction sequences). Typically, one rendering instruction stream is used to draw one frame of an image. Specifically, the GPU can execute each drawing instruction in the rendering instruction stream to draw the various objects in a frame of an image. The collection of objects generated from the rendering instruction stream constitutes a frame of an image. A frame of an image is a static image displayed by a game application. Each static frame of an image can be composed of 3D scene images and UI images, where the 3D scene image is also called a color image. For example, the 3D scene image may include in-game scenery, game characters, background objects, special effects, and skills, while the UI image may include control buttons (such as run buttons and skill buttons), a minimap, floating text, user avatars, energy pack controls, and health bars, etc. The UI image is located on top of the 3D scene image. It should be understood that both the game characters in the 3D scene image and the rendering control buttons in the UI image are objects within the game frame. Furthermore, through the rendering instruction stream, general information such as the world matrix and projection matrix of the virtual camera corresponding to a frame of an image, as well as the depth image corresponding to a frame of an image, can also be obtained.

[0098] The game frames in this application embodiment can be divided into two types:

[0099] One type is the real frame generated based on a rendering instruction stream. In a game scene, the operating system sends rendering instruction streams to the GPU sequentially. The GPU generates a real frame corresponding to each rendering instruction stream according to the order in which these streams are received. One rendering instruction stream is used to draw one real frame.

[0100] Another type is the predicted frame generated based on two sequentially acquired rendering instruction streams. "Two sequentially acquired rendering instruction streams" means that rendering instruction stream 1 is acquired first, followed by rendering instruction stream 2. The GPU does not acquire any other rendering instruction streams between acquiring rendering instruction stream 1 and acquiring rendering instruction stream 2. That is, the real frame N-1 obtained through rendering instruction stream 1 and the real frame N+1 obtained through rendering instruction stream 2 are two adjacent real frames. Specifically, the GPU can acquire the two rendering instruction streams sequentially and separate the 3D scene image and UI image corresponding to a real frame from each rendering instruction stream (for example, separating the scene image Color0 and user interface image UI0 corresponding to real frame N-1 from rendering instruction stream 1, and separating the scene image Color1 and user interface image UI1 corresponding to real frame N+1 from rendering instruction stream 2), as well as general information such as the depth image, the world matrix of the game's virtual camera, and the projection matrix corresponding to each real frame. Then, the GPU can use this information to generate a mask image that marks moving and static pixels, calculate the motion vector for each pixel based on the mask image, and then perform pixel-by-pixel offset on the 3D scene image corresponding to a certain real frame based on the motion vector. Afterwards, the GPU can call the NPU's neural network model to perform image inpainting on the blank areas after pixel-by-pixel offset, and then overlay the inpainted 3D scene image with the UI image corresponding to this real frame to generate the final predicted frame N.

[0101] Continue to refer to Figure 3 After obtaining the real frame N-1 through rendering instruction stream 1, the real frame N+1 through rendering instruction stream 2, and the predicted frame N through rendering instruction streams 1 and 2, the GPU can send these frames into the display queue in the order of real frame N-1, predicted frame N, and real frame N+1. Then, the display screen plays the real frame N-1, predicted frame N, and real frame N+1 in sequence.

[0102] The above Figure 3 This document only describes the process of generating one prediction frame using two rendering command streams. In actual game scenarios, the number of rendering command streams may be greater than two. The above process can be used to generate prediction frames for any two rendering command streams acquired sequentially. It should be understood that since the resource consumption of generating prediction frames using inter-frame prediction technology is less than the resource consumption of rendering real frames in the game application, reducing the number of real frames generated by the game application and using inter-frame prediction technology to generate prediction frames not only ensures the total number of game frames but also reduces the power consumption load on the chip.

[0103] The following is combined Figures 4A to 17 The prediction frame generation method provided in the embodiments of this application is illustrated by example.

[0104] Figures 4A-4CThe diagram illustrates three methods for predicting frame generation. The differences between these three diagrams are: Figure 4A The specific flow of this method is shown. Figure 4B The data flow relationships between the various images involved in this method are shown. Figure 4C The specific flow of the method and the data flow relationships between the various images are illustrated. It should be noted that, as... Figures 4A-4C The execution entity in the predicted frame generation method shown can be a terminal device, a GPU within the terminal device, or other software or hardware functional modules within the terminal device. The following example uses a GPU as the execution entity for the predicted frame generation method, but this does not limit the scope of this application.

[0105] For example, in a game scene, the GPU can continuously obtain rendering instruction streams from the game application. A rendering instruction stream instructs the generation of a real frame. Whenever a rendering instruction stream is obtained, such as rendering instruction stream 1 or rendering instruction stream 2, the GPU can perform steps 1 and 2 below for that rendering instruction stream.

[0106] Step 1: The GPU separates the 3D scene image and UI image corresponding to a real frame from the rendering instruction stream.

[0107] The aforementioned 3D scene image refers to a color image containing the 3D scene but not UI pixels, i.e., an RGB image.

[0108] For example, the aforementioned rendering instruction stream can be hooked into the game's rendering instruction stream. The GPU can identify the various instructions contained within the hooked game rendering instruction stream by using a rendering instruction acquisition function, based on the characteristics of the game's rendering instruction stream. Hooking the game rendering instruction stream generally includes multiple drawing instructions, such as scene drawing instructions, UI drawing instructions, and image display instructions. Scene drawing instructions can be used to draw 3D scene images such as in-game scenery, characters, effects, and skills; UI drawing instructions can be used to draw UI images such as control buttons, minimaps, and floating text; and image display instructions are used to place the drawn image data into a specified location in the operating system (such as a framebuffer) to complete frame display.

[0109] The pixel coordinates of the 3D scene image and the UI image have a one-to-one correspondence, and the UI image is located on top of the 3D scene image. If an object in the UI image has the same pixel coordinates as an object in the 3D scene image, the object in the UI image may visually occlude the object in the 3D scene image. Objects in the 3D scene image may change their pixel coordinates for two reasons: one is that objects in the game scene, such as characters, vehicles, or birds, may move, such as... Figure 5AAs shown, in the world coordinate system O-XYZ corresponding to the game scene, the initial position of the virtual character is coordinate 1. When the virtual character runs, it moves to coordinate 2. In this case, based on the virtual character's movement, the position of the virtual character in the 3D scene image of real frame N-1 and real frame N+1 changes. Another reason is that the virtual camera in the game scene shakes, that is, the perspective of the virtual character corresponding to the game player changes, such as... Figure 5B As shown, in the world coordinate system O-XYZ corresponding to the game scene, the virtual character's position is always at coordinate position 1 in the world coordinate system O-XYZ. However, the virtual camera moves from position P1 to position P2, and the orientation of the virtual camera changes. When the position and orientation of the virtual camera change, the perspective of the virtual character corresponding to the game player also changes. In this case, the position of the virtual character in the 3D scene image of real frame N-1 and the 3D scene image of real frame N+1 moves.

[0110] When an object in the UI image has the same pixel coordinates as an object in the 3D scene image, regardless of the reason for changing the pixel coordinates, if the object in the 3D scene image is moved according to steps 3-5 below, the object in the UI image will move along with the object in the 3D scene image. However, the object in the UI image may not actually move, or the motion vectors of the object in the UI image and the object in the 3D scene image may be different. This will result in a serious image quality problem in the final predicted frame. Therefore, the GPU needs to separate the 3D scene image and the UI image corresponding to a real frame from a rendering instruction stream.

[0111] Specifically, for each game rendering instruction in a rendering instruction stream, the GPU can perform the following operations: Based on the characteristics of a game rendering instruction, it identifies whether it is a UI drawing instruction. For example, UI drawing instructions may include resource binding instructions, state setting instructions, and drawing instructions. If it is a UI drawing instruction, the output of these instructions is directed to a newly created image. This newly created image can be used to store UI pixels, and therefore, this newly created image can be called a UI image. If it is a scene drawing instruction, the original rendering result is changed to a newly created image to store a 3D scene image that does not contain UI pixels. In this way, the GPU separates the UI image and the 3D scene image from the rendering instruction stream.

[0112] For example, such as Figure 6As shown, the GPU first obtains rendering instruction stream 1, which instructs the generation of a real frame N-1, and separates the 3D scene image Color0 and UI image UI0 from rendering instruction stream 1. If UI image UI0 is superimposed on 3D scene image Color0, a complete real frame N-1 is obtained. Then, the GPU can obtain rendering instruction stream 2, which instructs the generation of a real frame N+1, and separate the 3D scene image Color1 and UI image UI1 from rendering instruction stream 2. If UI image UI1 is superimposed on 3D scene image Color1, a complete real frame N+1 is obtained.

[0113] Step 2: The GPU obtains the depth image, the world matrix of the virtual camera, and the projection matrix corresponding to a real frame based on the rendering instruction stream.

[0114] The GPU can identify the rendering instruction sequence used for model rendering based on the characteristics of the game's rendering instructions, and determine the glBindBuffer instruction preceding the model rendering drawcall instruction. The glBindBuffer instruction contains data related to the virtual camera. The GPU can retrieve the virtual camera's world matrix and projection matrix from the buffer corresponding to the glBindBuffer instruction. Furthermore, the GPU can also obtain a depth image from the rendering instruction stream.

[0115] It should be noted that the 3D scene image obtained by the GPU in step 1 and the depth image obtained in step 2 have the same image content; for example, both images contain people, trees, and mountains. The difference between the two images is that the 3D scene image is a three-channel (RGB) color image, while the depth image is a single-channel image. The depth image is similar to a grayscale image, and each pixel value of the depth image can be used to represent the distance between the virtual camera and the object. The depth image reflects the geometry of the visible surface of the object.

[0116] To facilitate understanding of the world matrix and projection matrix of the virtual camera mentioned above, the following will combine... Figure 7 The pixel coordinate system, image coordinate system, normalized device coordinate (NDC), camera coordinate system, and world coordinate system involved in the embodiments of this application will be described.

[0117] The pixel coordinate system o-wh and the image coordinate system o-xy are two-dimensional coordinate systems. Typically, images are stored as arrays in terminal devices, where the value of each element (i.e., pixel) represents the brightness of that image point. The origin of the pixel coordinate system o-wh is located at the top left corner of the image, with the w-axis pointing horizontally to the right and the h-axis pointing vertically downwards. Each pixel coordinate (w, h) represents the column and row number of a pixel in the array, respectively; therefore, (w, h) are the coordinates of the pixel coordinate system o-wh in pixels. The origin of the image coordinate system o-xy is located at the center of the image (called the principal point), with the x-axis parallel to the w-axis and the y-axis parallel to the h-axis. Coordinates in the pixel coordinate system o-wh and the image coordinate system o-xy can be converted between each other.

[0118] Assuming the pixel coordinates in the pixel coordinate system o-wh are (w0, h0), and the physical dimensions of each pixel on the x-axis and y-axis are dx and dy, respectively, the relationship between the pixel coordinate system o-wh and the image coordinate system o-xy is as follows:

[0119]

[0120] The normalized device coordinate system o-x'y'z' is a three-dimensional coordinate system. In the normalized device coordinate system o-x'y'z', the values ​​of x', y', and z' take values ​​in the interval [-1, 1], where z' represents the depth value of the image, and x' and y' are obtained by normalizing the coordinates (x, y) in the image coordinate system o-xy, respectively. Through viewport transform, the normalized device coordinate system o-x'y'z' can be transformed into the image coordinate system o-xy or the pixel coordinate system o-wh. Taking a 512*512 image frame as an example, the pixel coordinates of the lower right corner point in the pixel coordinate system o-wh are (512, 512). After coordinate transformation, the coordinates of this point in the normalized device coordinate system o-x'y'z' are (1, 1, z'), where z' represents the depth value at that coordinate.

[0121] The camera coordinate system O0-X0Y0Z0 is a three-dimensional coordinate system. Within this system, the virtual camera is a virtual camera set up for the game scene, not a true camera. In a game scene, if multiple players are roaming the game world, each player's position and line of sight will differ, resulting in different frames displayed on each player's device. Assuming a coordinate system is created with a player as the origin, this system can be called the camera coordinate system or the view space coordinate system. (Refer to...) Figure 7The origin O0 of the camera coordinate system is the optical center (projection center) of the virtual camera. The X0 and Y0 axes are parallel to the x and y axes of the camera's image plane, respectively. The Z0 axis is the optical axis of the virtual camera and is perpendicular to the camera's image plane. oO0 is the focal length of the virtual camera. It should be understood that each player's position and line of sight in the game world are different; therefore, the camera coordinate system O0-X0Y0Z0 established with each player's corresponding virtual camera as the center will also be different.

[0122] The world coordinate system O-XYZ is a three-dimensional coordinate system introduced to describe the positions of various virtual cameras. In a game scene, virtual objects such as characters, tables, rooms, animals, and vehicles all have a specific location. The position of a virtual object in the game world is defined as its coordinates in the world coordinate system, called world space coordinates. For example, the world coordinate system O-XYZ can be a three-dimensional coordinate system established with the Earth's center as the origin. It should be understood that different players have different positions and viewing directions in the world coordinate system O-XYZ, which causes the position and orientation of the virtual camera in the world coordinate system O-XYZ to also differ. Coordinates in the camera coordinate system O0-X0Y0Z0 can be converted to coordinates in the world coordinate system O-XYZ through translation and rotation.

[0123] The world matrix, also known as the extrinsic parameter matrix, transforms the object coordinates in the world coordinate system O-XYZ into the camera coordinate system O0-X0Y0Z0. In other words, when the virtual camera's viewpoint is used as the origin in the game scene, the world matrix transforms all world space coordinates into observation coordinates relative to the virtual camera's position and orientation.

[0124] For example, the world matrix can be represented as:

[0125]

[0126] Here, (Rx,Ry,Rz) can represent the X-axis direction vector of the virtual camera, (Ux,Uy,Uz) can represent the Y-axis direction vector of the virtual camera, (Dx,Dy,Dz) can represent the line-of-sight direction and Z-axis of the virtual camera, and (Px,Py,Pz) can represent the world space coordinates of the virtual camera.

[0127] The projection matrix, also known as the intrinsic parameter matrix, transforms the object coordinates in the camera coordinate system O0-X0Y0Z0 into a normalized device coordinate system o-x'y'z'. During rendering, a 3D scene is typically projected onto the screen (i.e., the camera plane) as a 2D image. Specifically, all vertex data is first transformed from the camera coordinate system O0-X0Y0Z0 to the clip space. Then, by dividing by the w component in the clip coordinate system, the transformed clip coordinates are converted back to the normalized device coordinate system o-x'y'z'. Here, the camera coordinate system O0-X0Y0Z0 is a right-handed coordinate system, and the normalized device coordinate system o-x'y'z' is a left-handed coordinate system.

[0128] For example, suppose the projection matrix is ​​the matrix of a general conical section, in which the coordinates of the four vertices of a conical section are represented as (r, t, -n), (r, b, -n), (f, b, -n), (f, t, -n), then the projection matrix can be represented as:

[0129]

[0130] Assume the GPU receives rendering instruction stream 1 and rendering instruction stream 2 sequentially. Rendering instruction stream 1 can be used to generate real frame N-1, and rendering instruction stream 2 can be used to generate real frame N+1. Following steps 1 and 2 above, the GPU can obtain the 3D scene image Color0, UI image UI0, depth image Depth0, virtual camera world matrix ViewMatrix0, and virtual camera projection matrix ProjectionMatrix0 corresponding to real frame N-1, as well as the 3D scene image Color1, UI image UI1, depth image Depth1, virtual camera world matrix ViewMatrix1, and virtual camera projection matrix ProjectionMatrix1 corresponding to real frame N+1. Then, the GPU can generate predicted frame N based on these parameters according to steps 3-7 below.

[0131] Step 3: The GPU generates mask images that mark dynamic and static pixels based on the depth image of real frame N-1, the world matrix and projection matrix of the virtual camera, and the depth image of real frame N+1, the world matrix and projection matrix of the virtual camera.

[0132] For example, the GPU can generate the mask image through the following steps (1) to (3).

[0133] (1) Based on the depth image Depth0 of the real frame N-1, the world matrix ViewMatrix0 of the virtual camera of the real frame N-1, and the projection matrix ProjectionMatrix0, calculate the world space coordinates (x0, y0, z0) corresponding to each pixel coordinate (w0, h0) of the 3D scene image Color0.

[0134] Specifically, assuming the coordinates (w0, h0) of any pixel in the 3D scene image Color0 of real frame N-1 are known, and the depth image Depth0 of real frame N-1 is also known, the GPU can first convert the coordinates (w0, h0) of any pixel in the 3D scene image Color0 into image coordinates (x0, y0) in the image coordinate system o0-x0y0. Then, based on the depth image Depth0 of real frame N-1, the GPU calculates the normalized coordinates (x', y', z') in the normalized device coordinate system o0-x'y'z'. Then, using the inverse of the projection matrix ProjectionMatrix0, the GPU obtains the camera coordinates in the camera coordinate system O0-X0Y0Z0. Finally, using the inverse of the world matrix ViewMatrix0, the GPU obtains the world space coordinates (x0, y0, z0) in the world coordinate system O-XYZ. Figure 8A As shown, for any pixel coordinate in the 3D scene image Color0, the world space coordinates (x0, y0, z0) corresponding to a pixel coordinate (w0, h0) can be obtained by referring to the above steps.

[0135] As an example, the transformation relationship from coordinates (x', y', z') in the normalized device coordinate system o0-x'y'z' to world space coordinates (x0, y0, z0) in the world coordinate system O-XYZ is as follows:

[0136]

[0137] Here, ViewMatrix0 represents the world matrix of the virtual camera in real frame N-1, and ProjectionMatrix0 represents the projection matrix of the virtual camera in real frame N-1. The specific implementation of the world matrix ViewMatrix0 and the projection matrix ProjectionMatrix0 can be found in the detailed description of step 2 above, and will not be repeated here.

[0138] (2) For the world space coordinates (x0, y0, z0) of each pixel in the 3D scene image Color0 of the real frame N-1 calculated in step (1) above, the world matrix ViewMatrix1 and projection matrix ProjectionMatrix1 of the virtual camera of the real frame N+1 are further used to reproject to the pixel coordinate system, so as to obtain the reprojected pixel coordinates (w1, h1).

[0139] Specifically, after calculating the world space coordinates (x0, y0, z0) of each pixel in the 3D scene image Color0 of the real frame N-1 through step (1) above, the GPU can first use the virtual camera's world matrix ViewMatrix1 to calculate the coordinates in the camera coordinate system O1-X1Y1Z1, and then use the virtual camera's projection matrix ProjectionMatrix1 to calculate the coordinates (x', y', z') in the normalized device coordinate system o1-x'y'z'. Then, the coordinates (x', y', z') in the normalized device coordinate system o1-x'y'z' are transformed to the image coordinate system o1-x1y1, and then converted to the pixel coordinates (w1, h1) in the pixel coordinate system. Figure 8B As shown, after reprojecting the world space coordinates (x0, y0, z0) corresponding to each pixel obtained by step (1), the reprojected pixel coordinates (w1, h1) can be obtained.

[0140] As an example, the transformation relationship from world space coordinates (x0, y0, z0) in world coordinate system O-XYZ to coordinates (x', y', z') in normalized device coordinate system o-x'y'z' is as follows:

[0141]

[0142] Here, ProjectionMatrix1 represents the projection matrix of the virtual camera in real frame N+1, and ViewMatrix1 represents the world matrix of the virtual camera in real frame N+1. The specific implementation of the world matrix ViewMatrix1 and the projection matrix ProjectionMatrix1 can be found in the detailed description of step 2 above, and will not be repeated here.

[0143] (3) For each pixel coordinate (w1, h1) calculated in step (2) above and after reprojection, the world space coordinate (x1, y1, z1) corresponding to each pixel coordinate (w1, h1) is further calculated based on the depth image Depth1 of the real frame N+1, the world matrix ViewMatrix1 of the virtual camera of the real frame N+1, and the projection matrix ProjectionMatrix1.

[0144] Specifically, after calculating the reprojected pixel coordinates (w1, h1) of each pixel through step (2) above, the GPU can first convert the pixel coordinates (w1, h1) into the coordinates (x1, y1) of the image coordinate system o1-x1y1, and then calculate the normalized coordinates (x1, y1) of the image coordinates (x1, y1) in the normalized device coordinate system o1-x'y'z' based on the depth image Depth1 of the real frame N+1. Then, the inverse matrix of the projection matrix ProjectionMatrix1 can be used to obtain the coordinates in the camera coordinate system O1-X1Y1Z1, and then the inverse matrix of the world matrix ViewMatrix1 can be used to obtain the world space coordinates (x1, y1, z1) in the world coordinate system O-XYZ. Thus, as Figure 8C As shown, the world space coordinates (x1, y1, z1) corresponding to the reprojected pixel coordinates (w1, h1) were obtained.

[0145] As an example, the transformation relationship from coordinates (x', y', z') in the normalized device coordinate system o1-x'y'z' to world space coordinates (x1, y1, z1) in the world coordinate system O-XYZ is as follows:

[0146]

[0147] Here, ViewMatrix1 represents the world matrix of the virtual camera in real frame N+1, and ProjectionMatrix1 represents the projection matrix of the virtual camera in real frame N+1. The specific implementation of the world matrix ViewMatrix1 and the projection matrix ProjectionMatrix1 can be found in the detailed description of step 2 above, and will not be repeated here.

[0148] It should be understood that, for the same pixel: if the depth in depth image Depth0 is different from the depth in depth image Depth1, then... Figure 8C As shown, after performing coordinate transformations according to steps (1) to (3) above, the pixel will correspond to two different positions in the world coordinate system O-XYZ, that is, the world space coordinates (x0, y0, z0) and the world space coordinates (x1, y1, z1) will be different. Alternatively, if the depth in the depth image Depth0 is the same as the depth in the depth image Depth1, then after performing coordinate transformations according to steps (1) to (3) above, the pixel will correspond to the same position in the world coordinate system O-XYZ, that is, the world space coordinates (x0, y0, z0) and the world space coordinates (x1, y1, z1) will be the same.

[0149] If the following three conditions are met simultaneously: x0 = x1, y0 = y1, z0 = z1, then the pixel at pixel coordinates (w0, h0) on the 3D scene image Color0 has not changed its position in the world coordinate system. This pixel is a pixel of a static object (such as a stationary house), and thus the pixel is marked as Mask = 0.

[0150] If any of the above three conditions are not met, such as x0≠x1, y0≠y1, or z0≠z1, then the position of the pixel at pixel coordinates (w0, h0) on the 3D scene image Color0 changes in the world coordinate system. This pixel is a pixel of a dynamic object (such as a running person), and thus the pixel is marked as Mask=1.

[0151] For example, Figure 9 A schematic diagram of the mask image generated from the 3D scene image Color0 is shown. Assume that the running figure is a dynamic object, and other objects such as trees and mountains are static objects. Each small square represents a pixel, and each object consists of multiple pixels. For each pixel in the 3D scene image Color0, the GPU can determine whether a pixel is a pixel of a dynamic object or a pixel of a static object according to steps (1) to (3) above. Then, the pixels of dynamic objects are marked as Mask = 1, and the pixels of static objects are marked as Mask = 0. After marking all pixels in the 3D scene image Color0, the GPU can output the Masks of all pixels to a mask image, thus generating a mask image for marking dynamic and static pixels. It should be understood that the pixel coordinates of the mask image and the pixel coordinates of the 3D scene image Color0 have a one-to-one correspondence.

[0152] Step 4: Based on the pixel markers of the mask image obtained in Step 3, the GPU calculates the motion vector of each pixel in the 3D scene image Color0 of the real frame N-1 obtained in Step 1.

[0153] For example, for each pixel in the mask image, the GPU can specifically perform the following operations:

[0154] If a pixel in the mask image is marked as 0, it means that the pixel in the 3D scene image Color0 is a pixel of a static object (referred to as a static pixel). The GPU can calculate the motion vector of the static pixel by reprojecting the coordinates.

[0155] As an example, if the reprojection coordinates corresponding to each pixel coordinate are obtained by referring to steps (1) to (2) in step 3 above, for static pixels marked Mask=0 in the mask image, the GPU can filter out the reprojection coordinates (w1, h1) corresponding to each static pixel coordinate (w0, h0) from the reprojection coordinates of all pixels, and then use [(w1-w0)*0.5, (h1-h0)*0.5] to calculate the motion vector of the static pixel.

[0156] As another example, the GPU can also perform the reprojection process again for static pixels marked Mask=0 in the mask image, instead of based on the reprojection coordinates obtained in steps (1) to (2) of step 3 above. For example, the GPU can calculate the world space coordinates (x0, y0, z0) corresponding to each static pixel coordinate (w0, h0) based on the depth image Depth0 of the real frame N-1, the world matrix ViewMatrix0 of the virtual camera in the real frame N-1, and the projection matrix ProjectionMatrix0; then, using the world matrix ViewMatrix1 and the projection matrix ProjectionMatrix1 of the virtual camera in the real frame N+1, the world space coordinates (x0, y0, z0) of the static pixel are reprojected onto the pixel coordinate system to obtain a new pixel coordinate (w1, h1). After calculating the reprojection coordinates (w1, h1) corresponding to each static pixel coordinate (w0, h0), the GPU can use [(w1-w0)*0.5, (h1-h0)*0.5] to calculate the motion vector of the static pixel.

[0157] For example, suppose the two-dimensional coordinates of a static pixel in the 3D scene image Color0 are (w0,h0)=(255,255), and the two-dimensional coordinates of the static pixel after reprojection are (w1,h1)=(255,269), then the motion vector of the static pixel is: [(255-255)*0.5,(269-255)*0.5]=[0,7].

[0158] If a pixel in a mask image is marked as 1, it means that the pixel in the 3D scene image Color0 is a pixel of a dynamic object (referred to as a dynamic pixel). The GPU can calculate the motion vector of the dynamic pixel by searching for the best matching pixel.

[0159] Specifically, the GPU can obtain the brightness Intensity0 corresponding to the coordinates (w0, h0) of a dynamic pixel in the 3D scene image Color0, and the feature vector Special0 corresponding to that dynamic pixel. Then, the GPU can find the best matching pixel in the 3D scene image Color1 corresponding to the dynamic pixel as follows: calculate the brightness Intensity1 of each pixel in the 3D scene image Color1 and the feature vector Special1 corresponding to that pixel. Through iterative search, it traverses each pixel in the 3D scene image Color1 to find the pixel (w1, h1) with the smallest absolute value of (Intensity0 – Intensity1) and the smallest feature distance between the feature vector Special0 and the feature vector Special1. The pixel (w1, h1) is the best matching pixel. Then, the GPU can use [(w1-w0)*0.5, (h1-h0)*0.5] to calculate the motion vector of the dynamic pixel. Among them, feature vectors Special0 and Special1 are used to characterize the brightness and geometric features of pixels in the image. They are pixels with obvious and unique features in the image, such as table corners, wall corners, and clothing patterns. Usually, the main parameters of the feature vector are the color and depth of the current pixel and its neighboring pixels within a certain range. For example, assuming that the pixel coordinates of a dynamic pixel in the 3D scene image Color0 are (w0, h0) = (255, 257), and the pixel coordinates of the best matching pixel in the 3D scene image Color1 are (w1, h1) = (259, 239), then the motion vector of the dynamic pixel is: [(259-255)*0.5, (239-257)*0.5] = [2, -9].

[0160] After calculating the motion vector for each pixel in the 3D scene image Color0 according to the steps described above, the GPU can output these motion vectors to a motion vector image mv. This motion vector image mv can be used for pixel-by-pixel offset in step 5 below.

[0161] Step 5: Based on the motion vectors in the motion vector image mv obtained in Step 4, the GPU performs pixel-by-pixel offset on each pixel in the 3D scene image Color0 of the real frame N-1 obtained in Step 1, generating an intermediate image Color_NewFrame containing blank areas.

[0162] For example, Figure 10 This diagram illustrates how each pixel in the 3D scene image Color0 is offset pixel-by-pixel to the intermediate image Color_NewFrame.

[0163] First, the GPU can create a new four-channel color image, Color_NewFrame (called the intermediate image), and initialize the pixel value of each pixel in the intermediate image, Color_NewFrame.rgba, to (0.0, 0.0, 0.0, 0.5). This intermediate image, Color_NewFrame, can be used to store pixel values ​​after pixel-by-pixel offset. Specifically, Color_NewFrame.rgb = (0.0, 0.0, 0.0) indicates that the RGB values ​​of the intermediate image, Color_NewFrame, are all 0.0, i.e., black; Color_NewFrame.a = 0.5 indicates that the A channel of the intermediate image, Color_NewFrame, has a value of 0.5, which will be used for pixel value prediction later, i.e., pixel repair.

[0164] It should be noted that in this embodiment, the RGB three channels refer to the R channel, G channel, and B channel, where R channel represents red, G channel represents green, and B channel represents blue. The intensity values ​​of the RGB three channels can range from 0 to 255, or the intensity values ​​of the RGB three channels can be normalized to 0.0 to 1.0. The A channel refers to the alpha channel, which represents the transparency and semi-transparency of an image. Typically, the value of the A channel ranges from 0.0 to 1.0.

[0165] Then, the GPU can perform a pixel-by-pixel offset on each pixel Color0.rgba in the 3D scene image Color0 based on the motion vector calculated in step 4 above. As an example, such as... Figure 10 As shown, the GPU can first move static pixels one by one towards the intermediate image Color_NewFrame based on the motion vectors of the static pixels calculated in step 4, and then move dynamic pixels one by one towards the intermediate image Color_NewFrame based on the motion vectors of the dynamic pixels calculated in step 4, thus obtaining the intermediate image Color_NewFrame that stores the offset pixel values. As another example, the GPU can also start from the top left corner of the 3D scene image Color0 and move each pixel row by row and column by column based on the motion vectors calculated in step 4, that is, move both static and dynamic pixels simultaneously.

[0166] For each pixel Color0.rgba in the 3D scene image Color0, assuming the 2D coordinates of a pixel are (w0, h0), the motion vector of the pixel at (w0, h0) in the motion vector image mv is obtained as mv0. The GPU can move the pixel coordinates (w0, h0) according to the motion vector mv0 to obtain the new 2D coordinates (w1, h1). Then, the GPU can reassign the pixel value Color_NewFrame.rgba at coordinates (w1, h1) in the intermediate image Color_NewFrame to:

[0167] Color_NewFrame.rgb=Color0.rgb;

[0168] Color_NewFrame.a = 1.0.

[0169] Here, `Color_NewFrame.rgb = Color0.rgb` means that the RGB three-channel color at coordinates (w1, h1) of the intermediate image `Color_NewFrame` is assigned the same RGB three-channel color at pixel coordinates (w0, h0) of the 3D scene image `Color0`. `Color_NewFrame.a = 1.0` means that the A channel at coordinates (w1, h1) of the intermediate image `Color_NewFrame` is set to 1, and no further pixel value prediction is needed, i.e., no repair is required for this pixel.

[0170] It should be noted that the embodiments in this application illustrate the use of a pixel coordinate with an A-channel value of 0.5 to indicate that the coordinate needs repair, and a pixel coordinate with an A-channel value of 1 to indicate that the coordinate does not need repair. This does not limit the scope of this application. In actual implementation, a pixel coordinate with an A-channel value of 1 can also indicate that the pixel at that coordinate needs repair, and a pixel coordinate with an A-channel value of 0.5 can indicate that the pixel at that coordinate does not need repair.

[0171] In accordance with Figure 10Taking pixel-by-pixel offset as an example, when an object (such as a tree) in the 3D scene image Color0 is a static object, even if the virtual camera is displaced and rotated, the coordinates of each static pixel in the world coordinate system remain unchanged. Therefore, after each static pixel moves according to its motion vector, it is unlikely that multiple static pixels will move to the same position, resulting in pixel occlusion, nor is it likely that there will be blank areas due to no pixels moving over. However, when an object (such as a running person) in the 3D scene image Color0 is a dynamic object, the motion vectors corresponding to each dynamic pixel of the dynamic object may be different. This may cause multiple dynamic pixels to move to the same position (w1, h1) in the intermediate image Color_NewFrame, resulting in pixel occlusion at position (w1, h1). In some positions, no pixels may move over, causing the pixel values ​​at these positions to retain their initial values ​​(0.0, 0.0, 0.0, 0.5). Based on the description of the above embodiments, when the A channel value of certain regions in the intermediate image Color_NewFrame is 0.5, it indicates that pixel value prediction needs to be performed on these regions subsequently. These regions are the blank areas that need to be filled with pixels later.

[0172] Due to edge characteristics and algorithmic reasons, the motion vectors in the edge region of a dynamic object and the motion vectors in the adjacent static region may deviate significantly. When the motion vectors corresponding to the various dynamic pixels of a dynamic object are different, multiple dynamic pixels in the edge region may move to the same position, making it more likely that blank areas will appear in the edge region of the dynamic object. Of course, in some cases, blank areas may also appear in non-edge regions of the dynamic object, such as joints and facial areas, which are not limited in the embodiments of this application.

[0173] It should be noted that the "blank area" mentioned above refers to certain areas in the intermediate image Color_NewFrame where pixel values ​​have not been moved from the 3D scene image Color0 and still retain the initial values ​​(0.0,0.0,0.0,0.5). Pixel filling is required later, but this does not mean that the area is white.

[0174] The following describes how to solve the pixel occlusion problem that occurs when multiple pixels in the 3D scene image Color0 move to the same position in the intermediate image Color_NewFrame.

[0175] The first possible implementation

[0176] like Figure 11As shown, at the initial moment, the coordinates of the four adjacent pixels in the intermediate image Color_NewFrame are (w1,h1), (w2,h1), (w1,h2), and (w2,h2), respectively, and the Color_NewFrame.rgba of these four pixels is initialized to (0.0,0.0,0.0,0.5). The coordinates of the four adjacent pixels in the 3D scene image Color0 are (w1,h1), (w2,h1), (w1,h2), and (w2,h2), respectively, and the Color_NewFrame.rgba of these four pixels are (r1,g1,b1,a1), (r2,g2,b2,a2), (r3,g3,b3,a3), and (r4,g4,b4,a4). Pixel 1 at coordinates (w1, h1) is moved using motion vector mv(1, 1), pixel 2 at coordinates (w2, h1) is moved using motion vector mv(0, 1), and pixel 3 at coordinates (w1, h2) is moved using motion vector mv(1, 0). All three pixels eventually move to the middle image's Color_NewFrame at coordinates (w2, h2), resulting in pixel occlusion at that location. Conversely, if no pixels move from (w1, h1), (w2, h1), or (w1, h2) in the middle image's Color_NewFrame, these pixels retain their initial values ​​(0.0, 0.0, 0.0, 0.5), remaining as blank areas.

[0177] Assuming Figure 11 Pixels 1, 2, and 3 are moved to the same position P(w0, h0) in the intermediate image Color_NewFrame, with pixel 1 corresponding to depth l1, pixel 2 to depth l2, and pixel 3 to depth l3. For example... Figure 12As shown in the figure, the origin O0 of the camera coordinate system O0-X0Y0Z0 is the optical center of the virtual camera. The GPU can, according to the description of step 3 above, successively convert the coordinates (w2, h2) to the camera coordinate system O0-X0Y0Z0 at depths of l1, l2, or l3, so as to obtain three positions P1, P2, and P3 in the camera coordinate system O0-X0Y0Z0. Then, the GPU can calculate the distance L1 from position P1 to the origin O0 of the camera coordinate system O0-X0Y0Z0, the distance L2 from position P2 to the origin O0 of the camera coordinate system O0-X0Y0Z0, and the distance L3 from position P3 to the origin O0 of the camera coordinate system O0-X0Y0Z0. Since L1 < L2 < L3 and position P1 is the closest to the origin O0, the GPU can use the pixel value Color0.rgba1 of pixel 1 corresponding to position P1 as the pixel value at the coordinate (w2, h2) finally displayed on the intermediate image Color_NewFrame.

[0178] Among them, the above distances L1, L2, and L3 can be obtained through the following relational expressions:

[0179] L1 = sqrt(x1 * x1 + y1 * y1 + z1 * z1);

[0180] L2 = sqrt(x2 * x2 + y2 * y2 + z2 * z2);

[0181] L3 = sqrt(x3 * x3 + y3 * y3 + z3 * z3).

[0182] Among them, sqrt() is the square root function, which is used to calculate the modulus length of a position in the camera coordinate system O0-X0Y0Z0, that is, the distance from a position in the camera coordinate system O0-X0Y0Z0 to the origin O0.

[0183] It should be noted that the above first possible implementation manner is an example for illustration by using the camera coordinate system to calculate the distances from each pixel point to the origin O0 of the camera coordinate system O0-X0Y0Z0, and it does not limit this application.

[0184]

[0185] Since the coordinates of the two coordinate systems can be mutually converted, the GPU can also use the world coordinate system to calculate the distances from each pixel point to the origin O0 of the world coordinate system O0-X0Y0Z0, and use the pixel value Color0.rgba1 of the pixel point 1 closest to the origin of the world coordinate system O0-X0Y0Z0 as the pixel value at the coordinate (w2, h2) finally displayed on the intermediate image Color_NewFrame. It should be understood that whether using the world coordinate system or the camera coordinate system, the final calculation result is the same pixel point.

[0186] The second possible implementation method

[0187] The GPU calculates the world space coordinates (x0, y0, z0) corresponding to each pixel coordinate (w0, h0) of the 3D scene image Color0 based on the depth image Depth0 of the N-1 real frames, the world matrix ViewMatrix0 of the virtual camera of the N-1 real frames, and the projection matrix ProjectionMatrix0. Then, using the world space coordinates (x0, y0, z0), it calculates the magnitude distance = sqrt(x0*x0 + y0*y0 + z0*z0) corresponding to each pixel coordinate.

[0188] After offsetting all pixels in the 3D scene image Color0, if multiple pixels in the 3D scene image Color0 move to the same position (w1, h1) in the intermediate image Color_NewFrame, the GPU can compare the magnitude distance of these multiple pixels and use the pixel value of the pixel coordinate corresponding to the smallest magnitude distance as the final pixel value displayed in the intermediate image Color_NewFrame.

[0189] It should be noted that regardless of whether the first or second possible implementation method is used, the final calculation result is the same pixel.

[0190] Furthermore, the two possible implementations described above are illustrative examples of moving three pixels to the same position P(w0,h0) in the intermediate image Color_NewFrame and selecting one pixel from the three pixels as the final pixel displayed at position P(w0,h0) in the intermediate image Color_NewFrame. These examples do not limit the scope of this application. In actual implementation, the above method can also be used to select one pixel from two, four, or more pixels moved to the same position as the final pixel displayed at that position in the intermediate image Color_NewFrame.

[0191] Step 6: The GPU generates an edge image edge_map based on the 3D scene image Color0 and 3D scene image Color1 obtained in Step 1. Then, the edge image edge_map and the intermediate image Color_NewFrame obtained in Step 5 are input into the AI ​​model to obtain the 3D scene prediction image Pixel_Color.

[0192] For example, the GPU can specifically generate a 3D scene prediction image Pixel_Color by the following (a) to (e).

[0193] (a) Use the edge detection algorithm to extract the edge image edge0 of 3D scene image Color0 and the edge image edge1 of 3D scene image Color1.

[0194] Among them, edge image edge0 and edge image edge1 are both single-channel images, i.e., grayscale images.

[0195] The edge images described above are obtained by extracting edges from the original image using an edge detection algorithm. For example, edge detection algorithms can include the Canny algorithm, differential operator method, Laplacian Gaussian operator method, fitting method, neural network method, wavelet transform method, wavelet packet decomposition method, or morphological method, etc.

[0196] Taking the Canny algorithm as an example of edge detection, the GPU first uses a Gaussian filter to smooth the 3D scene image Color0 / Color1 to filter out noise. Then, it calculates the gradient intensity and direction of each pixel in the image. After that, it uses a non-maximum suppression method to eliminate the spurious response brought by edge detection. Then, it uses a double-threshold detection method to determine the real edges and potential edges. Finally, it suppresses isolated weak edges and retains the real edges, thus obtaining the edge image edge0 / edge1.

[0197] (b) Create two new edge images edge0_warp and edge1_warp.

[0198] Among them, edge image edge0_warp and edge image edge1_warp are both single-channel images, i.e., grayscale images.

[0199] The edge image edge0_warp can be used to store the warped result of the edge image edge0.

[0200] The edge image edge1_warp can be used to store the warped result of the edge image edge1.

[0201] (c) Perform image warping on edge image edge0 in units of pixel blocks, and output the warping result of edge image edge0 to edge image edge0_warp; and perform image warping on edge image edge1 in units of pixel blocks, and output the warping result of edge image edge1 to edge image edge1_warp.

[0202] The aforementioned pixel block can be composed of multiple pixels.

[0203] As an example, a pixel block can consist of N*N pixels, where N is an integer greater than or equal to 2. For example, N = 8. According to the description in step 5 above, when offsetting the real image pixel by pixel, multiple pixels may move to the same position in the intermediate image Color_NewFrame, causing pixel occlusion and resulting in blank areas. It should be understood that the image distortion of the edge image in step (c) above, performed on a pixel block basis, can solve the pixel occlusion problem, thus facilitating image repair of blank areas in subsequent steps.

[0204] This application provides a 2D mesh model. This 2D mesh model can be an offline mesh model that is manually constructed by a person.

[0205] As an example, Figure 13 A 2D mesh model is shown. This 2D mesh model consists of p*q rectangular meshes, where p represents the number of rows and q represents the number of columns. Each rectangular mesh represents a pixel block, for example, composed of 8*8 pixels. That is, the 2D mesh model consists of 8m*8n pixels. It should be noted that the pixel coordinates of the 2D mesh model have a one-to-one correspondence with the pixel coordinates in the 3D scene images Color0, Color1, edge images edge0, edge images edge1, edge images edge0_warp, and edge images edge1_warp. Assume the pixel coordinates of the four vertices of a certain rectangular mesh are (0,0), (7,0), (0,7), and (7,7) respectively, and the motion vectors of these four vertices determined in step 4 above are mv1 = (0,2), mv2 = (2,1), mv3 = (1,1), and mv4 = (1,2). The GPU can move the pixel coordinates of the four vertices according to the motion vector corresponding to the pixel coordinates of each vertex, thus obtaining the pixel coordinates of the four new vertices as (0,2), (9,1), (1,8), and (8,9). The GPU can then concatenate the pixel coordinates of the four new vertices to obtain the following: Figure 13 The new mesh is shown in the dashed box. This new mesh is then rasterized, and the coordinates of each pixel in the new mesh are calculated using a linear interpolation method. Since the texture coordinates (ux, uy) of each vertex remain unchanged before and after the movement, the pixel data before the movement can be assigned to the pixel coordinates after the movement based on the texture coordinates (ux, uy).

[0206] It should be noted that, Figure 13This diagram illustrates image warping of a rectangular grid consisting of 8x8 pixels in a 2D mesh model. It should be understood that for any rectangular grid in a 2D mesh model, the image warping can be performed by moving the pixel coordinates of the four vertices of the rectangular grid, as described above. After warping all rectangular grids, a new edge image is obtained.

[0207] The following is combined Figure 14A and Figure 14B This paper provides an example illustrating the specific process of rendering edge images edge0_warp and edge1_warp using a 2D mesh model.

[0208] Figure 14A A schematic diagram is shown showing how edge image edge0 is obtained by warping edge image edge0.

[0209] like Figure 14A As shown, the GPU can first extract the edge image edge0 from the 3D scene image Color0. The edge image edge0 is a single-channel image. Then, the GPU can move each vertex according to the motion vector mv0 corresponding to each vertex vertex0 in the 2D mesh model to form a new warped mesh. After that, each new mesh is rasterized, and the pixel coordinates of each pixel in the new mesh are linearly interpolated. Each pixel coordinate is used to determine the position of a pixel in the edge image edge0_warp. Assuming that the pixel coordinates of a pixel before the movement are (vx, vy), the pixel coordinates after the movement according to the motion vector (mv.x, mv.y) are (vx_n, vy_n) = (vx + mv.x, vy + mv.y). Since the texture coordinates (ux, uy) of a pixel remain unchanged before and after the movement, i.e., (ux_n, uy_n) = (ux, uy), the GPU can collect pixel data from the pixel coordinates (vx, vy) of the edge image edge0 based on the texture coordinates (ux, uy) corresponding to the moved pixel coordinates (vx_n, vy_n), and output the collected pixel data to the pixel coordinates (vx_n, vy_n) of the edge image edge0_warp, finally obtaining the edge image edge0_warp that saves the distortion result of the edge image edge0.

[0210] Figure 14B A schematic diagram is shown showing how edge image edge1 is warped to obtain edge image edge1_warp.

[0211] like Figure 14BAs shown, the GPU can first extract the edge image edge1 from the 3D scene image Color1. Edge image edge1 is a single-channel image. Then, the GPU can move each vertex according to the motion vector mv0*(-1.0) corresponding to each vertex vertex0 in the 2D mesh model to form a new distorted mesh. After that, each new mesh is rasterized, and the pixel coordinates of each pixel in the new mesh are linearly interpolated. The pixel coordinates of each pixel are used to determine the position of a pixel in the edge image edge1_warp. Assuming that the pixel coordinates before the movement are (vx, vy), the pixel coordinates after the movement according to the motion vector (-mv.x, -mv.y) are (vx_n, vy_n) = (vx-mv.x, vy-mv.y). Since the texture coordinates (ux, uy) of a pixel remain unchanged before and after the movement, i.e., (ux_n, uy_n) = (ux, uy), the GPU can collect pixel data from the pixel coordinates (vx, vy) of the edge image edge1 based on the texture coordinates (ux, uy) corresponding to the moved pixel coordinates (vx_n, vy_n), and output the collected pixel data to the pixel coordinates (vx_n, vy_n) of the edge image edge1_warp, finally obtaining the edge image edge1_warp that saves the distortion result of the edge image edge1.

[0212] In the above embodiment, the GPU distorts the edge images edge0 and edge1 on a pixel block basis. This effectively prevents multiple pixels from moving to the same position and ensures the integrity of the edge images.

[0213] (d) Based on the A channel label of the intermediate image Color_NewFrame, output the edge images edge0_warp and edge1_warp to the final edge image edge_map.

[0214] For example, the GPU can create an edge image `edge_map`. The pixel coordinates of the intermediate image `Color_NewFrame`, the edge image `edge0_warp`, the edge image `edge1_warp`, and the edge image `edge_map` have a one-to-one correspondence. The GPU can determine the padding value for the corresponding pixel coordinates in the edge image `edge_map` based on the specific value of the A channel of each pixel coordinate in the intermediate image `Color_NewFrame`.

[0215] If A = 0.5 for a pixel coordinate in the intermediate image Color_NewFrame, then the corresponding pixel coordinate in the edge image edge_map is filled using the pixel data from the edge image edge1_warp. That is, the blank area is filled using the edge image edge1_warp.

[0216] If the pixel coordinate A in the intermediate image Color_NewFrame is not equal to 0.5, then the corresponding pixel coordinate in the edge image edge_map is filled with the pixel data of the edge image edge0_warp. It should be understood that filling with the pixel data of the edge image edge0_warp generates a complete edge image, making it easier for AI models to recognize edges.

[0217] For example, in the case of Figure 14A Obtain the edge image edge0_warp, through Figure 14B After obtaining the edge image edge1_warp, as follows Figure 14C As shown, in the final edge image edge_map, the regions corresponding to the regions where A≠0.5 in the intermediate image Color_NewFrame are filled with the data of the edge image edge0_warp, and the regions corresponding to the regions where A=0.5 in the intermediate image Color_NewFrame are filled with the data of the edge image edge1_warp.

[0218] After outputting edge images edge0_warp and edge1_warp to the final edge image edge_map, edge_map is a grayscale image. In some embodiments, the GPU can directly input the single-channel edge image edge_map into the AI ​​model, and then the AI ​​model converts the single-channel edge image edge_map into an RGB image. In other embodiments, the GPU can first convert the single-channel edge image edge_map into an RGB image, and then input the converted RGB image into the AI ​​model.

[0219] (e) Input the edge image edge_map obtained in (d) and the intermediate image Color_NewFrame obtained in step 5 into the NPU's AI model. After the AI ​​model fills the blank areas of the intermediate image Color_NewFrame with pixels, the GPU obtains the 3D scene prediction image Pixel_Color from the NPU.

[0220] The AI ​​model mentioned above can be a neural network model, such as an AI image inpainting model.

[0221] For example, the terminal device can pre-train a neural network model and load that model onto an NPU used to execute neural network algorithms. Figure 15 As shown, the GPU can input the edge image `edge_map` obtained through step (d) and the intermediate image `Color_NewFrame` obtained through step 5 into the AI ​​model. The AI ​​model performs convolution and deconvolution calculations on the edge image `edge_map` and the intermediate image `Color_NewFrame` to inpaint the areas to be filled in the intermediate image `Color_NewFrame`, thereby generating a complete 3D scene prediction image `Pixel_Color`. Then, the NPU returns the 3D scene prediction image `Pixel_Color` output by the AI ​​model to the GPU.

[0222] Based on the description of the above embodiments, traditional AI models are typically loaded onto devices with high computing power, such as PCs. By inputting real frames into the AI ​​model, a complete predicted frame can be obtained. However, traditional AI models have complex algorithms and high requirements for computation and bandwidth, making them unsuitable for real-time operation on terminal devices with limited computing power, such as mobile phones. Unlike traditional AI models, the AI ​​model provided in this application is only used to repair blank areas of an image, has lower computational requirements, and can run in real-time on terminal devices with limited computing power, such as mobile phones. Furthermore, by predicting data-missing areas, the generated 3D scene prediction image Pixel_Color is made closer to the original image.

[0223] Step 7: The GPU overlays the UI image UI0 of the real frame N-1 obtained in Step 1 onto the 3D scene prediction image Pixel_Color obtained in Step 6 to generate the final prediction frame image Pixel_Final, which is the prediction frame N.

[0224] For example, such as Figure 16 As shown, the GPU can perform alpha blending (i.e. A-channel blending) between the UI image UI0 of the real frame N-1 generated by step 1 and the 3D scene prediction image Pixel_Color generated by step 6 to generate the final prediction frame image Pixel_Final, i.e., prediction frame N.

[0225] The rules for alpha mixing can be:

[0226] Pixel_Final.rgb=Pixel_Color.rgb*(1.0–Pixel_UI.a)+Pixel_UI.rgb*Pixel_UI.a;

[0227] Pixel_Final.a = 1.0.

[0228] Wherein, Pixel_Final.rgb is the RGB three-channel pixel of the predicted frame image Pixel_Final, and Pixel_Final.a is the A-channel pixel of the predicted frame image Pixel_Final. That is, Pixel_Final.rgba is the RGBA four-channel pixel of the predicted frame image Pixel_Final.

[0229] Pixel_Color.rgb represents the RGB three-channel pixels of the 3D scene prediction image Pixel_Color, and Pixel_Final.a represents the A-channel pixels of the 3D scene prediction image Pixel_Color. In other words, Pixel_Color.rgba represents the RGBA four-channel pixels of the 3D scene prediction image Pixel_Color.

[0230] Pixel_UI.rgb represents the RGB three-channel pixels of the UI image UI0 in real frame N-1, and Pixel_UI.a represents the A-channel pixels of the UI image UI0 in real frame N-1. In other words, Pixel_UI.rgba represents the RGBA four-channel pixels of the UI image UI0 in real frame N-1. It should be noted that Pixel_UI.a can be obtained from the rendering instruction stream of real frame N-1.

[0231] It should be understood that alpha blending can achieve a transparent effect in the UI. The degree of transparency can be controlled by Pixel_UI.a. When Pixel_UI.a = 0 for a pixel in the UI, it means that the pixel is completely transparent and will not obscure the 3D scene prediction image Pixel_Color located below the UI; while when Pixel_UI.a = 1 for a pixel in the UI, it means that the pixel is completely opaque and will obscure the 3D scene prediction image Pixel_Color located below the UI.

[0232] Based on the description of the above embodiments, the GPU can generate real frames N-1 based on the first obtained rendering instruction stream 1, generate real frames N+1 based on the subsequently obtained rendering instruction stream 2, and generate predicted frames N based on rendering instruction stream 1 and rendering instruction stream 2. The GPU can then send these frames into the display queue in the order of real frames N-1, predicted frames N, and real frames N+1. Then, as... Figure 17 As shown, the terminal device's display screen sequentially plays the real frame N-1, the predicted frame N, and the real frame N+1.

[0233] In the predicted frame generation method provided in this application, the GPU can obtain the 3D scene image and UI image corresponding to each real frame from two game rendering instructions, as well as general information such as depth image, world matrix and projection matrix of game virtual camera, and then use this information to perform pixel-by-pixel offset on the 3D scene image, thereby solving the physical distortion problem of the block offset scheme and the ghosting problem of the image blending scheme; then the GPU can use AI image inpainting technology to fill the blank area after pixel-by-pixel offset, so that the final generated predicted frame image is closer to the original image quality of the game.

[0234] This application also provides a terminal device, including a processor coupled to a memory, the processor being used to execute computer programs or instructions stored in the memory, so that the terminal device implements the methods in the above embodiments.

[0235] This application also provides a computer-readable storage medium storing computer instructions. When the computer-readable storage medium is run on a computer, it causes the computer to perform the methods described above. The computer instructions can be stored in the computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or can include one or more data storage devices such as servers or data centers that can be integrated with the medium. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, or magnetic tape), an optical medium, or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0236] This application also provides a computer program product, which includes computer program code that, when run on a computer, causes the computer to perform the methods described in the above embodiments.

[0237] This application also provides a chip coupled to a memory. This chip is used to read and execute computer programs or instructions stored in the memory to perform the methods described in the above embodiments. The chip can be a general-purpose processor or a special-purpose processor. It should be noted that the chip can be implemented using one or more field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gate logic, discrete hardware components, any other suitable circuits, or any combination of circuits capable of performing the various functions described throughout this application.

[0238] The terminal device, computer-readable storage medium, computer program product, and chip provided in the embodiments of this application are all used to execute the methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects corresponding to the methods provided above, and will not be repeated here.

[0239] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0240] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0241] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0242] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0243] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for generating a prediction frame, characterized in that, The method includes: Obtain the user interface image, the first scene image, the first depth image, the first projection matrix, and the first world matrix from the first rendering instruction stream; Obtain the second scene image, the second depth image, the second projection matrix, and the second world matrix from the second rendering instruction stream; Based on the first depth image, the first projection matrix, the first world matrix, the second depth image, the second projection matrix, and the second world matrix, determine the static pixels and dynamic pixels in the first scene image; Based on the pixel coordinates of the static pixel and the reprojection coordinates after reprojection of the static pixel, the motion vector of the static pixel is determined; and, in the second scene image, the matching pixel of the dynamic pixel is determined, and the motion vector of the dynamic pixel is determined based on the pixel coordinates of the dynamic pixel and the pixel coordinates of the matching pixel. The static pixel is moved based on the motion vector of the static pixel, and the dynamic pixel is moved based on the motion vector of the dynamic pixel to obtain an intermediate image; The blank areas of the intermediate image are filled with pixels based on the AI ​​model to obtain the scene prediction image; The user interface image is superimposed on the scene prediction image to obtain a prediction frame.

2. The method according to claim 1, characterized in that, The step of determining static and dynamic pixels in the first scene image based on the first depth image, the first projection matrix, the first world matrix, the second depth image, the second projection matrix, and the second world matrix includes: Perform the following steps on each pixel in the first scene image: Based on the first depth image, the first projection matrix, and the first world matrix, determine the first world space coordinates corresponding to the first pixel coordinates of the first scene image; based on the second projection matrix and the second world matrix, reproject the first world space coordinates onto the pixel coordinate system to obtain the second pixel coordinates; based on the second depth image, the second projection matrix, and the second world matrix, determine the second world space coordinates corresponding to the second pixel coordinates. For any pixel in the first scene image: If the first world space coordinates and the second world space coordinates of a pixel are the same, then the pixel is marked as a static pixel; or, if the first world space coordinates and the second world space coordinates of a pixel are different, then the pixel is marked as a dynamic pixel.

3. The method according to claim 2, characterized in that, The method further includes: The labeling results for each pixel are output to the mask image.

4. The method according to claim 1, characterized in that, Determining the motion vector of the static pixel based on its pixel coordinates and the reprojected coordinates after reprojection includes: Based on the first depth image, the first projection matrix, and the first world matrix, determine the world space coordinates corresponding to the static pixel coordinates (w0, h0) of the first scene image; Based on the second projection matrix and the second world matrix, the world space coordinates corresponding to the static pixel coordinates (w0, h0) are reprojected onto the pixel coordinate system to obtain the reprojected coordinates (w1, h1). [(w1-w0)*0.5, (h1-h0)*0.5] is determined as the motion vector of the static pixel.

5. The method according to claim 1, characterized in that, The step of determining the matching pixel of the dynamic pixel in the second scene image, and determining the motion vector of the dynamic pixel based on the pixel coordinates of the dynamic pixel and the pixel coordinates of the matching pixel, includes: Obtain the brightness and feature vector corresponding to the dynamic pixel coordinates (w0', h0') of the first scene image; Based on the brightness and feature vector corresponding to the dynamic pixel coordinates (w0', h0'), the matching pixel (w1', h1') is searched in the second scene image; [(w1'-w0')*0.5, (h1'-h0')*0.5] is determined as the motion vector of the dynamic pixel.

6. The method according to claim 5, characterized in that, The matched pixels satisfy the following conditions: The absolute value of the first difference is less than the absolute value of the second difference. The first difference is the difference between the brightness corresponding to the dynamic pixel coordinates (w0', h0') and the brightness corresponding to the matching pixel (w1', h1'). The second difference is the difference between the brightness corresponding to the dynamic pixel coordinates (w0', h0') and the brightness corresponding to other pixel coordinates of the second scene image. The first feature distance is less than the second feature distance. The first feature distance is the feature distance between the feature vector corresponding to the dynamic pixel coordinates (w0', h0') and the feature vector corresponding to the matching pixel (w1', h1'). The second feature distance is the feature distance between the feature vector corresponding to the dynamic pixel coordinates (w0', h0') and the feature vector corresponding to other pixel coordinates of the second scene image.

7. The method according to any one of claims 1 to 6, characterized in that, The process of moving the static pixel based on the motion vector of the static pixel and moving the dynamic pixel based on the motion vector of the dynamic pixel to obtain an intermediate image includes: Create an intermediate image; Based on the motion vector of each static pixel in the first scene image, each static pixel is moved one by one into the intermediate image; Based on the motion vector of each dynamic pixel in the first scene image, each dynamic pixel is moved one by one into the intermediate image.

8. The method according to claim 7, characterized in that, After creating the intermediate image, the method further includes: The RGB channel of the intermediate image is assigned a value of 0.0, and the A channel of the intermediate image is assigned a value of 0.

5. The step of moving each static pixel sequentially into the intermediate image based on the motion vector of each static pixel in the first scene image includes: Based on the motion vector of each static pixel in the first scene image, the pixel coordinates corresponding to each static pixel are determined in the intermediate image; the RGB channel of the pixel coordinates corresponding to each static pixel in the intermediate image is assigned a first value, and the A channel of the pixel coordinates corresponding to each static pixel in the intermediate image is assigned a value of 1.0; wherein, the first value is the RGB channel pixel value of each static pixel in the first scene image; The step of moving each dynamic pixel sequentially into the intermediate image based on the motion vector of each dynamic pixel in the first scene image includes: Based on the motion vector of each dynamic pixel in the first scene image, the pixel coordinates corresponding to each dynamic pixel are determined in the intermediate image; the RGB channel of the pixel coordinates corresponding to each dynamic pixel in the intermediate image is assigned a second value, and the A channel of the pixel coordinates corresponding to each dynamic pixel in the intermediate image is assigned a value of 1.0; wherein, the second value is the RGB channel pixel value of each dynamic pixel in the first scene image.

9. The method according to claim 8, characterized in that, The method further includes: If multiple pixels move to the same pixel coordinate in the intermediate image, then based on the depth of each pixel in the first depth image, the same pixel coordinate is transformed to the camera coordinate system to obtain the camera coordinate of each pixel in the multiple pixels; Based on the camera coordinates of each pixel in the plurality of pixels, calculate the distance from the camera coordinates of each pixel in the plurality of pixels to the camera origin; The RGB channel of the same pixel coordinate is assigned a value based on the RGB channel pixel value of the pixel corresponding to the smallest distance among multiple distances.

10. The method according to any one of claims 1 to 6, characterized in that, The step of filling blank areas in the intermediate image with pixels based on an AI model to obtain a scene prediction image includes: Generate a target edge image based on the first scene image and the second scene image; The target edge image and the intermediate image are input into the AI ​​model to generate the scene prediction image; wherein, the scene prediction image is the image obtained by the AI ​​model after filling the blank areas of the intermediate image with pixels using the target edge image.

11. The method according to claim 10, characterized in that, The step of generating a target edge image based on the first scene image and the second scene image includes: Extract a first edge image from the first scene image, and extract a second edge image from the second scene image; The first edge image is distorted in units of pixel blocks, and the distortion result of the first edge image is output to the first distorted edge image; The second edge image is distorted in units of pixels, and the distortion result of the second edge image is output to the second distorted edge image; Based on the A-channel label of the intermediate image, the target edge image is generated according to the first distorted edge image and the second distorted edge image.

12. The method according to claim 11, characterized in that, The step of distorting the first edge image in units of pixel blocks and outputting the distortion result of the first edge image to a first distorted edge image includes: Based on the motion vectors corresponding to each vertex of each grid in the 2D mesh model, the pixel coordinates corresponding to the pixels in the first distorted edge image are determined in the first edge image. Based on the texture coordinates corresponding to the pixel coordinates in the first distorted edge image, pixel values ​​are collected from the first edge image to obtain the first distorted edge image that has saved the distortion result of the first edge image.

13. The method according to claim 11, characterized in that, The step of distorting the second edge image in units of pixel blocks and outputting the distortion result of the second edge image to a second distorted edge image includes: In the second distorted edge image, the pixel coordinates corresponding to the pixels in the second edge image are determined according to the opposite vector of the motion vector corresponding to each vertex of each grid in the 2D mesh model. Based on the texture coordinates corresponding to the pixel coordinates in the second distorted edge image, pixel values ​​are collected from the second edge image to obtain the second distorted edge image that has saved the distortion result of the second edge image.

14. The method according to claim 11, characterized in that, The first edge image, the second edge image, the first distorted edge image, and the second distorted edge image are single-channel images.

15. The method according to any one of claims 11 to 14, characterized in that, The step of generating the target edge image based on the A-channel labeling of the intermediate image and the first distorted edge image and the second distorted edge image includes: If the A channel of a pixel coordinate in the intermediate image is marked as 0.5, then the corresponding pixel coordinate in the target edge image is filled with pixel data from the second edge image; or, If the A channel of a pixel coordinate in the intermediate image is marked as 1, then the corresponding pixel coordinate in the target edge image is filled with the pixel data of the first edge image.

16. The method according to any one of claims 1 to 6, characterized in that, The first rendering instruction stream is obtained before the second rendering instruction stream; After superimposing the user interface image onto the scene prediction image to obtain a prediction frame, the method further includes: The first frame, the predicted frame, and the second frame are displayed sequentially. The first frame is a real image frame generated according to the first rendering instruction stream, and the second frame is a real image frame generated according to the second rendering instruction stream.

17. A terminal device, characterized in that, The device includes a processor and a memory coupled to the processor; wherein the memory stores instructions which the processor invokes to cause the terminal device to perform the prediction frame generation method as described in any one of claims 1 to 16.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a terminal device, causes the terminal device to perform the prediction frame generation method as described in any one of claims 1 to 16.

Images