Predicted frame generation method, terminal device, and storage medium

Abstract

Embodiments of the present application relate to the technical field of terminals, and provide a predicted frame generation method, a terminal device, and a storage medium. In a game scene, a terminal device respectively acquire from two game rendering instructions a 3D scene image and a UI image corresponding to each real frame, and acquire general information such as depth images, and world matrices and projection matrices of a virtual game camera according to the game rendering instructions; then, the information is used to perform pixel-by-pixel offset on the 3D scene image; a blank area in the 3D scene image is filled by using AI image restoration technology; and then the UI image is superposed on the restored 3D scene image to generate a predicted frame. The scheme requires low computing power, such that the scheme can be carried out in real time on a terminal device having limited computing power such as a mobile phone, and the finally generated predicted frame is closer to the native image quality of a game.

Description

Prediction frame generation method, terminal device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on August 22, 2023, with application number 202311072985.4 and application name “Prediction Frame Generation Method, Terminal Device and Storage Medium”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to a prediction frame generation method, a terminal device, and a storage medium. Background Art

[0003] At present, game scene rendering has increasingly higher requirements for picture smoothness and picture clarity, which relies on higher screen refresh rates and graphics rendering resolutions, making terminal devices face severe challenges in chip power consumption load.

[0004] To reduce the power consumption of chips, various frame prediction schemes have been proposed based on the similarity between two frames rendered in succession by the game. However, these schemes have many problems. For example, one scheme is to perform a semi-transparent blend of the two frames using motion vectors as weights. However, the resulting predicted frame is prone to image quality issues such as ghosting. For another example, a personal computer (PC) generates a complete predicted frame based on artificial intelligence (AI) technology. However, this scheme requires a lot of computing power and bandwidth, and cannot run in real time on terminal devices with limited computing power, such as mobile phones.

[0005] Summary of the Invention

[0006] The present application provides a prediction frame generation method, a terminal device, and a storage medium, which solve the technical problem of how to generate prediction frames with a quality close to the original game image on a terminal device with limited computing power.

[0007] To achieve the above objectives, this application adopts the following technical solutions:

[0008] In a first aspect, an embodiment of the present application provides a method for generating a prediction frame. The method may include:

[0009] Get the user interface image UI0, the first scene image Color0, the first depth image Depth0, the first projection matrix ProjectionMatrix0 and the first world matrix ViewMatrix0 from the first rendering instruction stream; get the second scene image Color1, the second depth image Depth1, the second projection matrix ProjectionMatrix1 and the second world matrix ViewMatrix1 from the second rendering instruction stream; based on the first depth image Depth0, the first projection matrix ProjectionMatrix0, the first world matrix ViewMatrix0, the second depth image Depth1, the second projection matrix ProjectionMatrix1 and the second world matrix ViewMatrix1 , determine the static pixels and dynamic pixels in the first scene image Color0; determine the motion vector of the static pixels according to the pixel coordinates of the static pixels and the reprojected coordinates after reprojecting the static pixels; determine the matching pixels of the dynamic pixels in the second scene image Color1, and determine the motion vector of the dynamic pixels according to the pixel coordinates of the dynamic pixels and the pixel coordinates of the matching pixels; move the static pixels based on the motion vector of the static pixels, and move the dynamic pixels based on the motion vector of the dynamic pixels to obtain the intermediate image Color_NewFrame; fill the blank area of ​​the intermediate image Color_NewFrame with pixels based on the AI ​​model to obtain the scene prediction image Pixel_Color; superimpose the user interface image UI0 on the scene prediction image Pixel_Color to obtain a predicted frame.

[0010] In the above scheme, by obtaining the scene image and UI image corresponding to each real frame from the two game rendering instructions, as well as obtaining general information such as the depth image, the world matrix and the projection matrix of the game virtual camera, this information can be used to offset the static pixels and dynamic pixels of the 3D scene image pixel by pixel, thereby solving the physical distortion problem existing in the block-by-block offset scheme and the ghosting problem existing in the image mixing scheme. In addition, the AI ​​model is used to fill the blank areas after the pixel-by-pixel offset, so that the final generated predicted frame image is closer to the native picture quality of the game. Different from the scheme that generates the entire frame based on the PC-based AI model, the AI ​​model of this application only repairs the blank areas of the image, has low requirements on computing power, and can run in real time on terminal devices with limited computing power such as mobile phones.

[0011] In one possible implementation, the first rendering instruction stream and the second rendering instruction stream are game rendering instruction streams. The game rendering instruction stream may include multiple drawing instructions, such as scene drawing instructions, UI drawing instructions, and image display instructions. Scene drawing instructions may be used to draw 3D scene images such as in-game scenery, characters, special effects, and skills; UI drawing instructions may be used to draw UI images such as control buttons, mini-maps, and floating text; and image display instructions are used to place the drawn image data into a specified location of the operating system to complete frame display.

[0012] In one possible implementation, a real game frame may consist of a 3D scene image and a UI image. The 3D scene image is also referred to as a color image. For example, the 3D scene image may include in-game scenery, game characters, background objects, special effects, and skills, while the UI image may include control buttons, a mini-map, floating text, user avatars, energy pack controls, and health bars. The UI image is layered on top of the 3D scene image. It should be understood that both the game characters in the 3D scene image and the rendered control buttons in the UI image are objects within the game frame.

[0013] In one possible implementation, the pixel coordinates of the 3D scene image and the pixel coordinates of the UI image are in a one-to-one correspondence, and the UI image is located above the 3D scene image. If the pixel coordinates of an object in the UI image are the same as those of an object in the 3D scene image, the object in the UI image may cause visual occlusion of the object in the 3D scene image. It should be understood that by separating the 3D scene image and the UI image from the rendering instruction stream, it is easier to perform scene prediction on the 3D scene image later.

[0014] In one possible implementation, determining static pixels and dynamic pixels in the first scene image Color0 based on the first depth image Depth0, the first projection matrix ProjectionMatrix0, the first world matrix ViewMatrix0, the second depth image Depth1, the second projection matrix ProjectionMatrix1, and the second world matrix ViewMatrix1 includes:

[0015] For each pixel in the first scene image Color0, the following steps are performed: based on the first depth image Depth0, the first projection matrix ProjectionMatrix0, and the first world matrix ViewMatrix0, the first world space coordinate corresponding to the first pixel coordinate of the first scene image Color0 is determined; based on the second projection matrix ProjectionMatrix1 and the second world matrix ViewMatrix1, the first world space coordinate is reprojected to the pixel coordinate system to obtain the second pixel coordinate; based on the second depth image Depth1, the second projection matrix ProjectionMatrix1, and the second world matrix ViewMatrix1, the second world space coordinate corresponding to the second pixel coordinate is determined. For any pixel in the first scene image Color0: if the first world space coordinate and the second world space coordinate of a pixel are the same, then the pixel is marked as a static pixel; otherwise, if the first world space coordinate and the second world space coordinate of a pixel are different, then the pixel is marked as a dynamic pixel.

[0016] It should be understood that for the same pixel: if the depth in the depth image Depth0 is different from the depth in the depth image Depth1, then after the coordinate transformation is performed according to the above steps, the pixel will correspond to two different positions in the world coordinate system O-XYZ, representing that the position of the pixel at the pixel coordinate (w0, h0) on the first scene image Color0 in the world coordinate system has changed, and the pixel is a pixel of a dynamic object (such as a running person), so the pixel can be marked as Mask = 1. If the depth in the depth image Depth0 is the same as the depth in the depth image Depth1, then after the coordinate transformation is performed according to the above steps, the pixel will correspond to the same position in the world coordinate system O-XYZ, representing that the position of the pixel at the pixel coordinate (w0, h0) on the 3D scene image Color0 in the world coordinate system has not changed, and the pixel is a pixel of a static object (such as a stationary house), so the pixel can be marked as Mask = 0. After marking all pixels in the first scene image Color0, the masks of all pixels can be output to a mask image, thereby generating a mask image for marking moving and static pixels. The pixel coordinates of the mask image correspond one-to-one with the pixel coordinates of the first scene image Color0.

[0017] In one possible implementation, determining the motion vector of the static pixel based on the pixel coordinates of the static pixel and the reprojected coordinates after reprojecting the static pixel can include: determining the world space coordinates corresponding to the static pixel coordinates (w0, h0) of the first scene image Color0 based on the first depth image Depth0, the first projection matrix ProjectionMatrix0 and the first world matrix ViewMatrix0; reprojecting the world space coordinates corresponding to the static pixel coordinates (w0, h0) to the pixel coordinate system based on the second projection matrix ProjectionMatrix1 and the second world matrix ViewMatrix1 to obtain the reprojected coordinates (w1, h1); and determining [(w1-w0)*0.5, (h1-h0)*0.5] as the motion vector of the static pixel.

[0018] It should be understood that when a pixel is a static pixel, the motion vector of the static pixel can be calculated by reprojection based on the characteristics of the static pixel.

[0019] In one possible implementation, determining a matching pixel of a dynamic pixel in the second scene image Color1, and determining a motion vector of the dynamic pixel based on the pixel coordinates of the dynamic pixel and the pixel coordinates of the matching pixel may include: obtaining the brightness and feature vector corresponding to the dynamic pixel coordinates (w0', h0') of the first scene image Color0; searching for a matching pixel (w1', h1') in the second scene image Color1 based on the brightness and feature vector corresponding to the dynamic pixel coordinates (w0', h0'); and determining [(w1'-w0')*0.5, (h1'-h0')*0.5] as the motion vector of the dynamic pixel.

[0020] For example, the matching pixels may satisfy the following conditions:

[0021] Condition 1: The absolute value of the first difference is less than the absolute value of the second difference. The first difference is the difference between the brightness corresponding to the dynamic pixel coordinate (w0', h0') and the brightness corresponding to the matching pixel (w1', h1'). The second difference is the brightness corresponding to the dynamic pixel coordinate (w0', h0') and the brightness corresponding to other pixel coordinates of the second scene image Color1.

[0022] Condition 2: The first feature distance is less than the second feature distance. The first feature distance is the feature distance between the feature vector corresponding to the dynamic pixel coordinate (w0', h0') and the feature vector corresponding to the matching pixel (w1', h1'). The second feature distance is the feature distance between the feature vector corresponding to the dynamic pixel coordinate (w0', h0') and the feature vector corresponding to other pixel coordinates of the second scene image Color1.

[0023] It should be understood that when a pixel is a dynamic pixel, based on the characteristics of the dynamic pixel, the motion vector of the dynamic pixel can be calculated by searching for the best matching pixel.

[0024] In one possible implementation, static pixels are moved based on motion vectors of the static pixels, and dynamic pixels are moved based on motion vectors of the dynamic pixels to obtain an intermediate image Color_NewFrame, including: creating an intermediate image Color_NewFrame; moving each static pixel one by one to the intermediate image Color_NewFrame based on the motion vector of each static pixel in the first scene image Color0; and moving each dynamic pixel one by one to the intermediate image Color_NewFrame based on the motion vector of each dynamic pixel in the first scene image Color0.

[0025] Specifically, after the intermediate image Color_NewFrame is created, the RGB channels of the intermediate image Color_NewFrame may be assigned a value of 0.0, and the A channel of the intermediate image Color_NewFrame may be assigned a value of 0.5.

[0026] Accordingly, based on the motion vector of each static pixel in the first scene image Color0, each static pixel is moved one by one to the intermediate image Color_NewFrame, including: determining the pixel coordinates corresponding to each static pixel in the intermediate image Color_NewFrame based on the motion vector of each static pixel in the first scene image Color0; assigning the RGB channels of the pixel coordinates corresponding to each static pixel in the intermediate image Color_NewFrame to a first value, and assigning the A channel of the pixel coordinates corresponding to each static pixel in the intermediate image Color_NewFrame to 1.0. The first value is the RGB channel pixel value of each static pixel in the first scene image Color0.

[0027] Accordingly, based on the motion vector of each dynamic pixel in the first scene image Color0, each dynamic pixel is moved one by one to the intermediate image Color_NewFrame, including: determining the pixel coordinates corresponding to each dynamic pixel in the intermediate image Color_NewFrame based on the motion vector of each dynamic pixel in the first scene image Color0; assigning the RGB channels of the pixel coordinates corresponding to each dynamic pixel in the intermediate image Color_NewFrame to a second value, and assigning the A channel of the pixel coordinates corresponding to each dynamic pixel in the intermediate image Color_NewFrame to 1.0. The second value is the RGB channel pixel value of each dynamic pixel in the first scene image Color0.

[0028] In one possible implementation, the method may further include: if multiple pixels are moved to the same pixel coordinates of the intermediate image Color_NewFrame, converting the same pixel coordinates to a camera coordinate system based on the depth of each of the multiple pixels in the first depth image Depth0 to obtain the camera coordinates of each of the multiple pixels; calculating the distance from the camera coordinates of each of the multiple pixels to the camera origin based on the camera coordinates of each of the multiple pixels; and assigning values ​​to the RGB channels of the same pixel coordinates based on the RGB channel pixel values ​​of a pixel corresponding to the minimum distance among the multiple distances. It should be understood that after completing the shifting of all pixels in the first scene image Color0, multiple pixels in the first scene image Color0 may move to the same position in the intermediate image Color_NewFrame, resulting in pixel occlusion problems at that position. Therefore, the pixel corresponding to the position closest to the camera origin can be calculated as the pixel ultimately displayed in the intermediate image Color_NewFrame.

[0029] In one possible implementation, pixel-filling the blank areas of the intermediate image Color_NewFrame based on the AI ​​model to obtain the scene prediction image Pixel_Color may include: generating a target edge image edge_map based on the first scene image Color0 and the second scene image Color1; and inputting the target edge image edge_map and the intermediate image Color_NewFrame into the AI ​​model to generate the scene prediction image Pixel_Color. The scene prediction image Pixel_Color is the image obtained by the AI ​​model using the target edge image edge_map to fill the blank areas of the intermediate image Color_NewFrame with pixels.

[0030] It should be understood that traditional AI models are usually loaded on devices with high computing power, such as PCs. By inputting the real frame into the AI ​​model, a complete predicted frame can be obtained. However, the algorithms of traditional AI models are complex, and the computational and bandwidth requirements are high, making them unable to run in real time on terminal devices with limited computing power, such as mobile phones. Unlike traditional AI models, the AI ​​model provided in this application is only used to repair blank areas of the image, and has low computing power requirements. It can run in real time on terminal devices with limited computing power, such as mobile phones. In addition, by predicting the areas where data is missing, the generated 3D scene predicted image Pixel_Color is closer to the result of the original image.

[0031] In a possible implementation, generating a target edge image edge_map based on the first scene image Color0 and the second scene image Color1 includes: extracting a first edge image edge0_warp from the first scene image Color0, and extracting a second edge image edge1_warp from the second scene image Color1; warping the first edge image edge0_warp in units of pixel blocks, and outputting the warped result of the first edge image edge0_warp to a first warped edge image; warping the second edge image edge1_warp in units of pixel blocks, and outputting the warped result of the second edge image edge1_warp to a second warped edge image; and generating a target edge image edge_map based on the first warped edge image and the second warped edge image based on the A channel flag of the intermediate image Color_NewFrame.

[0032] Specifically, the first edge image edge0_warp, the second edge image edge1_warp, the first warped edge image, and the second warped edge image are single-channel images.

[0033] Accordingly, warping the first edge image edge0_warp in units of pixel blocks and outputting the warped result of the first edge image edge0_warp to the first warped edge image may include: determining, in the first warped edge image, pixel coordinates corresponding to pixels in the first edge image edge0_warp according to motion vectors corresponding to respective vertices of each mesh in the 2D mesh model; and collecting pixel values ​​from the first edge image edge0_warp according to texture coordinates corresponding to the pixel coordinates in the first warped edge image, to obtain the first warped edge image in which the warped result of the first edge image edge0_warp is stored.

[0034] Accordingly, warping the second edge image edge1_warp in units of pixel blocks and outputting the warped result of the second edge image edge1_warp to a second warped edge image may include: determining, in the second warped edge image, pixel coordinates corresponding to pixels in the second edge image edge1_warp according to opposite vectors of motion vectors corresponding to respective vertices of each mesh in the 2D mesh model; and acquiring pixel values ​​from the second edge image edge1_warp according to texture coordinates corresponding to the pixel coordinates in the second warped edge image, to obtain a second warped edge image that has stored the warped result of the second edge image edge1_warp.

[0035] Accordingly, based on the A channel mark of the intermediate image, the target edge image is generated according to the first warped edge image and the second warped edge image, including: if the A channel mark of a pixel coordinate in the intermediate image Color_NewFrame is 0.5, then the corresponding pixel coordinate of the target edge image edge_map is filled with the pixel data of the second edge image edge1_warp; or, if the A channel mark of a pixel coordinate in the intermediate image Color_NewFrame is 1, then the corresponding pixel coordinate of the target edge image edge_map is filled with the pixel data of the first edge image edge0_warp. It should be understood that if A=0.5 of a pixel coordinate in the intermediate image Color_NewFrame, the edge image edge1_warp can be used to fill the blank area. If A≠0.5 of a pixel coordinate in the intermediate image Color_NewFrame, the pixel data of the edge image edge0_warp can be used to fill, thereby generating a complete edge image, which is convenient for the subsequent AI model to recognize edges.

[0036] In one possible implementation, the first rendering instruction stream is obtained before the second rendering instruction stream. After superimposing the user interface image on the scene prediction image Pixel_Color to obtain a predicted frame, the method may further include: sequentially displaying the first frame, the predicted frame, and the second frame. The first frame is a real image frame generated according to the first rendering instruction stream, and the second frame is a real image frame generated according to the second rendering instruction stream.

[0037] It should be understood that, based on the first rendering instruction stream and the second rendering instruction stream, not only can two real frames be generated, but a predicted frame can also be inserted between the two real frames. Because the computing resources required to generate the predicted frame are less than the resources required for the game application to render the real frame, this reduces the resource consumption of the terminal device.

[0038] In a second aspect, the present application provides an apparatus comprising units for executing the method of the first aspect. The apparatus may correspond to executing the predicted frame generation method described in the first aspect. For a description of the units in the apparatus, please refer to the description of the first aspect above and will not be repeated here for the sake of brevity.

[0039] In a third aspect, the present application provides a terminal device comprising a memory and one or more processors. The memory is configured to store computer program code, which includes computer instructions. When the computer instructions are invoked by the processor, the terminal device executes the predicted frame generation method provided in any one of the first aspects.

[0040] In a fourth aspect, the present application provides a computer-readable storage medium. The computer-readable storage medium includes computer instructions. When the computer instructions are executed on a terminal device, the terminal device executes the predicted frame generation method provided in the first aspect and any possible implementation thereof.

[0041] In a fifth aspect, the present application provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the prediction frame generation method provided in the first aspect and any possible implementation thereof.

[0042] In a sixth aspect, the present application provides a chip system. The chip system includes one or more interface circuits and one or more processors. The interface circuits and processors are interconnected via a circuit. The chip system can be applied to a terminal device including a communication module and a memory. The interface circuit is configured to receive a signal from the memory of the terminal device and transmit the received signal to the processor, the signal including a computer instruction stored in the memory. When the processor invokes the computer instruction, the terminal device can execute the predicted frame generation method provided in the first aspect and any possible implementation thereof.

[0043] It can be understood that the beneficial effects that can be achieved by the second aspect apparatus, the third aspect terminal device, the fourth aspect computer-readable storage medium, the fifth aspect computer program product, and the sixth aspect chip system provided above can be referred to as the beneficial effects in the first aspect and any possible implementation thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] FIG1 is a schematic diagram of the hardware structure of a mobile phone provided in an embodiment of the present application;

[0045] FIG2 is a schematic diagram of the architecture of a terminal device provided in an embodiment of the present application;

[0046] FIG3 is a schematic diagram of a rendering and display process of a game video frame provided in an embodiment of the present application;

[0047] FIG4A is a schematic diagram of a flow chart of a method for generating a prediction frame according to an embodiment of the present application;

[0048] FIG4B is a schematic diagram of a flow chart of another method for generating a prediction frame according to an embodiment of the present application;

[0049] FIG4C is a schematic flow chart of another method for generating a prediction frame according to an embodiment of the present application;

[0050] FIG5A is a schematic diagram of changing pixel coordinates based on movement of a virtual object in a game scene according to an embodiment of the present application;

[0051] FIG5B is a schematic diagram of changing pixel coordinates based on movement of a virtual camera in a game scene according to an embodiment of the present application;

[0052] FIG6 is a schematic diagram of separating a 3D scene image and a UI image from a rendering instruction stream according to an embodiment of the present application;

[0053] FIG7 is a schematic diagram of various coordinate systems provided in embodiments of the present application;

[0054] 8A-8C are schematic diagrams of determining static pixels and dynamic pixels according to an embodiment of the present application;

[0055] FIG9 is a schematic diagram of generating a mask image according to an embodiment of the present application;

[0056] FIG10 is a schematic diagram of pixel-by-pixel shifting based on motion vectors provided in an embodiment of the present application;

[0057] FIG11 is a schematic diagram of moving multiple pixels to the same position in an intermediate image according to an embodiment of the present application;

[0058] FIG12 is a schematic diagram showing occlusion of multiple pixels at the same position in an intermediate image according to an embodiment of the present application;

[0059] FIG13 is a schematic diagram of a 2D grid model provided in an embodiment of the present application;

[0060] 14A-14C are schematic diagrams of obtaining an edge image corresponding to an intermediate image according to an embodiment of the present application;

[0061] FIG15 is a schematic diagram of repairing a blank area of ​​an image based on an AI model according to an embodiment of the present application;

[0062] FIG16 is a schematic diagram of superimposing a UI image onto a 3D scene prediction image according to an embodiment of the present application;

[0063] Figure 17 is a schematic diagram of a terminal device playing a game screen frame by frame provided by an embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0065] In the description of this application, unless otherwise specified, " / " means or. For example, A / B can mean A or B. In the description of this application, "and / or" is simply a way to describe the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0066] In the specification and claims of this application, the terms "first" and "second" are used to distinguish different objects or to distinguish different processing of the same object, rather than to describe a specific order of objects. For example, the terms "first image" and "second image" are used to distinguish different images, rather than to describe a specific order of images. In the embodiments of this application, "plurality" refers to two or more.

[0067] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present invention. Thus, phrases such as "in some embodiments" appearing in different places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0068] To facilitate understanding of this application, some of the terms used in this application are explained below.

[0069] 1. Frame rate refers to the number of frames generated per second by an application during runtime. The generation of frames for display by a terminal device typically involves drawing, rendering, and compositing. The unit of frame rate is frames per second (fps). Frame drawing refers to drawing images for the display interface, frame rendering involves coloring the drawn images or adding 3D effects, and frame compositing involves compositing at least one rendered view into the display interface.

[0070] 2. Refresh rate, also known as screen refresh rate, refers to the number of times the screen refreshes per second. Each refresh represents one frame. Typically, refresh rate is determined by fixed hardware parameters. More powerful hardware supports higher refresh rates, resulting in smoother screen display. Refresh rate is measured in Hertz (Hz).

[0071] 3. Inter-frame prediction: This technology leverages temporal correlation between frames, using pixels from adjacent real frames to predict pixels in intermediate frames, effectively removing temporal redundancy. Inter-frame prediction is a well-established technology in video playback, for example, by weighted blending of two frames or using optical flow to calculate motion vectors for pixel blurring.

[0072] This application mainly relates to inter-frame prediction in the field of gaming. Specifically, when a terminal device runs a game application, the frame rate of the game application can reach 60fps or even 100fps, which occupies a large amount of system resources, causing the terminal device to experience problems such as fast power consumption, battery heating or system lag. Research shows that the resource consumption of using inter-frame prediction technology to generate predicted frames is less than the resource consumption of the game application to render real frames. Therefore, reducing the number of real frames generated by the game application and using inter-frame prediction technology to generate predicted frames can ensure the total amount of game frames and reduce the power consumption load of the chip.

[0073] However, game-rendered images are significantly clearer and sharper than video. In gaming scenarios, predicted frames must not only approach or even match the clarity and sharpness of real frames, but also ensure the accuracy of the image content and prevent distortion of objects in the image. Therefore, traditional inter-frame prediction technology used in video playback cannot meet the requirements for generating predicted frames for gaming.

[0074] Generally, there are several main categories of frame prediction technologies for gaming scenarios: one type of frame prediction technology involves semi-transparently blending two consecutive frames using motion vectors as weights. However, the resulting predicted frame is prone to image quality issues such as ghosting. Another type of frame prediction technology involves loading an AI model on a PC and feeding the real frame into the AI ​​model to obtain a complete predicted frame. However, the AI ​​model's algorithm is complex, requiring high computational effort and bandwidth, making it impossible to run in real time on mobile phones and other devices with limited computing power. Yet another type of frame prediction technology uses block-based motion compensation technology, which first searches for the best estimated block in the coding block to minimize the residual between the predicted block and the current block, calculates the motion vector (mv) of the current block, and then obtains an estimated value for the current frame based on the motion vector of the current block. However, block-by-block offsets can easily cause distortion of objects in the image. Therefore, a streamlined and efficient algorithm design and a robust framework structure are necessary for the application of frame prediction technology to mobile phones and other devices with limited computing power.

[0075] In view of the above problems, the present application provides a method for generating a predicted frame. In this method, the terminal device can obtain the three-dimensional (3D) scene image and user interface (UI) image corresponding to each real frame from two game rendering instructions, as well as general information such as the depth image, the world matrix (world matrix) and the projection matrix (projection matrix) of the game virtual camera, and then use this information to offset the 3D scene image pixel by pixel, thereby solving the physical distortion problem existing in the block-by-block offset scheme and the ghosting problem existing in the image mixing scheme; after that, the terminal device can fill the blank area after the pixel-by-pixel offset based on the AI ​​model, so that the final generated predicted frame image is closer to the native image quality of the game. It should be understood that, unlike the scheme of generating the entire frame based on the PC-based AI model, the AI ​​model of the present application only repairs the blank area of ​​the image, has low requirements on computing power, and can run in real time on terminal devices with limited computing power such as mobile phones.

[0076] In some embodiments, the terminal device is also referred to as a terminal (terminal) or user equipment (UE). For example, the terminal device may be a terminal device with limited computing power, such as a mobile phone, a smart screen, a smart TV, a tablet computer (Pad), a wearable device, a virtual reality (VR) device, an augmented reality (AR) device, etc. It should be understood that the prediction frame generation method provided in the present application can also be applied to terminal devices with sufficient computing power, such as PCs. The embodiments of the present application do not impose any restrictions on the specific type of terminal device. Regardless of the type of terminal device to which the prediction frame generation method provided in the present application is applied, the amount of computing resources required to generate the prediction frame is less than the amount of resources required for the game application to render the real frame, thereby reducing the resource consumption of the terminal device.

[0077] In order to better understand the embodiments of the present application, the structure of the terminal device of the embodiments of the present application is introduced below.

[0078] Taking a mobile phone as an example, FIG1 is a schematic diagram of the hardware structure of a mobile phone 100 provided in an embodiment of the present application.

[0079] As shown in Figure 1, the mobile phone 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a SIM card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, and a bone conduction sensor 180M, etc.

[0080] The processor 110 may include one or more processing units. For example, the processor 110 may include a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), an image signal processor (ISP), a neural-network processing unit (NPU), a video codec, a modem, and / or a baseband processor.

[0081] The CPU is the final execution unit for information processing and program execution. Its primary tasks include processing instructions, executing operations, controlling time, and processing data. The CPU can include a controller, an arithmetic unit (ALU), a cache memory, and a bus connecting these components. The controller is the nerve center and command center of the mobile phone 100. Based on instruction opcodes and timing signals, the controller generates operational control signals to control instruction fetching and execution. The ALU is the component that performs various arithmetic and logical operations. The cache memory can store instructions or data that have just been used or are recycled by the processor 110.

[0082] The GPU, also known as a display core, visual processor, or display chip, is a microprocessor for graphics and image processing. The GPU connects the display 194, the AP, and the CPU. The GPU can perform complex mathematical and geometric calculations, floating-point operations, parallel computing, and graphics rendering. The GPU reduces graphics card reliance on the CPU, especially when processing 3D graphics. The GPU utilizes core technologies such as cubic environment material mapping and vertex blending, allowing the GPU to perform some of the work previously performed by the CPU.

[0083] The NPU is a neural network (NN) computing processor that rapidly processes input information and can continuously self-learn by drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain. The NPU can enable applications such as intelligent cognition in the mobile phone 100.

[0084] In an embodiment of the present application, the prediction frame generation method is mainly performed by the GPU. For example, the GPU can obtain the 3D scene image and UI image corresponding to each real frame, as well as general information such as the depth image, the world matrix and projection matrix of the game virtual camera from two game rendering instructions, and use this information to generate a mask image that marks the dynamic and static pixels, and calculate the motion vector of each pixel (pixel) based on the mask image, and then perform pixel-by-pixel offset on the 3D scene image based on the motion vector. Then, the GPU calls the neural network model of the NPU to perform image repair on the blank area after the pixel-by-pixel offset, and finally generates a predicted frame image. Please refer to the specific description of the following embodiment, which will not be repeated here.

[0085] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the mobile phone 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system and an application (APP) required for at least one function, such as a game application. The data storage area can store configuration files of each APP, as well as data created during the use of the mobile phone 100.

[0086] The mobile phone 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the AP, for example, playing game background sounds, real-person conversations, etc.

[0087] The display screen 194 includes a signal processing module and a display panel, and is used to display images, videos, etc., such as displaying a game interface.

[0088] It should be understood that the hardware structure illustrated in the embodiments of the present application does not constitute a specific limitation on the mobile phone 100. In other embodiments, the mobile phone 100 may include more or fewer components than shown, or may combine or separate certain components, or have different component arrangements. The components shown in Figure 1 may be implemented in hardware, software, or a combination of software and hardware.

[0089] The software system of the terminal device can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, or a micro-service architecture. In the embodiment of the present invention, the Android system with a layered architecture is used as an example to exemplify the software architecture of the terminal device.

[0090] Figure 2 shows a schematic diagram of the architecture of a terminal device provided in an embodiment of the present application. As shown in Figure 2, the terminal device can adopt a layered architecture to divide the software into several layers, each layer having a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the software layers of the software structure are divided from top to bottom into: application (APP) layer, application framework (FW) layer, system library (FWK LIB) and kernel layer. The above software architecture runs on top of the hardware layer, which may include GPU, NPU and display screen, etc.

[0091] The application layer, also known as the application layer, can include a series of application packages. For example, the application layer may include game applications, camera applications, calendar applications, phone applications, and settings applications. When these application packages are running, they can access the various service modules provided by the application framework layer through the application programming interface (API) and execute corresponding intelligent services.

[0092] The application framework layer provides APIs and a programming framework for applications in the application layer. The application framework layer includes predefined functions. As shown in Figure 2, the application framework layer can include a graphics manager, video codec, window manager, resource manager, and notification manager. The graphics manager is responsible for calling interfaces in the graphics library to complete drawing, rendering, and compositing the interface. The video codec is responsible for video encoding and decoding. The window manager manages all system windows. The resource manager is responsible for system resources. The notification manager is responsible for managing notification messages in the status bar at the top of the phone.

[0093] The system library can include multiple functional modules, such as a surface manager, media libraries, a 3D graphics processing library (e.g., OpenGL ES), and a two-dimensional (2D) graphics engine (e.g., SGL). The surface manager is used to manage the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing. The 2D graphics engine is a drawing engine for 2D drawing.

[0094] Within the system libraries, the Android Runtime includes the core libraries and a virtual machine. The Android Runtime is responsible for scheduling and management of the Android system. The core libraries consist of two parts: one for the Java language's callable functions and the other for the Android core libraries. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files in the application layer and application framework layer as binary files. The virtual machine is responsible for performing functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0095] The kernel layer is the layer between hardware and software, belonging to the bottom layer of the Android system. The kernel layer can include various driver interfaces, such as GPU drivers, NPU drivers, CPU drivers, and display drivers. The kernel layer also includes power management, processes / threads, etc.

[0096] It should be noted that although the embodiments of the present application are described using the Android system as an example, its basic principles are also applicable to terminal devices based on operating systems such as iOS or Windows.

[0097] FIG3 shows the rendering and display process of the game video frame provided by an embodiment of the present application.

[0098] Game applications run on the terminal device's operating system. In a game scenario, the game application modifies the rendering API's function pointers to generate the game's rendering instruction stream, such as by hooking the game's rendering instruction stream. This rendering instruction stream is then passed to the GPU via the graphics manager and GPU driver. The GPU uses the rendering instruction acquisition function to identify the game's rendering instruction stream based on its characteristics.

[0099] A rendering instruction stream is an instruction stream consisting of multiple drawing instructions (rendering instruction sequences). Typically, a rendering instruction stream is used to draw a single frame of image. Specifically, the GPU can execute each drawing instruction in the rendering instruction stream to render each object in a single frame of image. The collection of objects generated according to the rendering instruction stream can constitute a single frame of image. A single frame of image is a single static image displayed by a game application. Each static image can be composed of, for example, a 3D scene image and a UI image, where a 3D scene image is also referred to as a color image. For example, a 3D scene image can include in-game scenery, game characters, background objects, special effects, and skills, while a UI image can include control buttons (such as a run button and skill buttons), a mini-map, floating text, a user avatar, energy pack controls, and a health bar. The UI image is located on top of the 3D scene image. It should be understood that both the game characters in the 3D scene image and the rendering control buttons in the UI image are objects in the game frame. In addition, the rendering instruction stream can also be used to obtain general information such as the world matrix and projection matrix of the virtual camera corresponding to a single frame of image, as well as the depth image corresponding to the single frame of image.

[0100] The game frames in the embodiment of the present application can be divided into two types:

[0101] One type is a real frame generated from a rendering instruction stream. In a game scenario, the operating system sends rendering instruction streams to the GPU in sequence. The GPU generates a real frame corresponding to each rendering instruction stream based on the order in which these rendering instruction streams are received. One rendering instruction stream is used to draw one real frame.

[0102] The other type is a predicted frame generated based on two rendering instruction streams obtained successively. "Two rendering instruction streams obtained successively" means that rendering instruction stream 1 is obtained first, and then rendering instruction stream 2 is obtained. After obtaining rendering instruction stream 1 and before obtaining rendering instruction stream 2, the GPU has not obtained other rendering instruction streams. That is, the real frame N-1 obtained by rendering instruction stream 1 and the real frame N+1 obtained by rendering instruction stream 2 are two adjacent real frames. Specifically, the GPU can obtain two rendering instruction streams successively, and separate the 3D scene image and UI image corresponding to a real frame from each rendering instruction stream (for example, the scene image Color0 and the user interface image UI0 corresponding to the real frame N-1 are separated from the rendering instruction stream 1, and the scene image Color1 and the user interface image UI1 corresponding to the real frame N+1 are separated from the rendering instruction stream 2), as well as obtain general information such as the depth image corresponding to each real frame, the world matrix and the projection matrix of the game virtual camera. The GPU then uses this information to generate a mask image that marks moving and still pixels. It then calculates the motion vector for each pixel based on the mask image and uses this motion vector to perform a pixel-by-pixel offset on the 3D scene image corresponding to a real frame. The GPU then uses the NPU's neural network model to inpaint the blank areas after the pixel-by-pixel offset and overlay the inpainted 3D scene image with the UI image corresponding to the real frame to generate the final predicted frame N.

[0103] Continuing with Figure 3 , after real frame N-1 is obtained from rendering instruction stream 1, real frame N+1 is obtained from rendering instruction stream 2, and predicted frame N is obtained from rendering instruction streams 1 and 2, the GPU can enter these frames into the display queue in the order of real frame N-1, predicted frame N, and real frame N+1. The display then plays real frame N-1, predicted frame N, and real frame N+1 in sequence.

[0104] Figure 3 above only illustrates the process of generating a predicted frame using two rendering instruction streams. In actual gaming scenarios, the number of rendering instruction streams may be greater than two. For any two rendering instruction streams acquired successively, a predicted frame can be generated using the same process. It should be understood that because the resource consumption of generating predicted frames using inter-frame prediction technology is less than the resource consumption of rendering real frames by the game application, by reducing the number of real frames generated by the game application and using inter-frame prediction technology to generate predicted frames, not only can the total number of game frames be guaranteed, but the power consumption load of the chip can also be reduced.

[0105] The following is an example description of the prediction frame generation method provided in the embodiment of the present application with reference to Figures 4A to 17.

[0106] Figures 4A-4C show schematic diagrams of three prediction frame generation methods. The difference between these three figures is that Figure 4A shows the specific process of the method, Figure 4B shows the data flow relationship between the various images involved in the method, and Figure 4C shows the specific process of the method and the data flow relationship between the various images. It should be noted that the execution subject in the prediction frame generation method shown in Figures 4A-4C can be a terminal device, a GPU in the terminal device, or other software or hardware functional modules in the terminal device. The following example uses a GPU as the execution subject of the prediction frame generation method, which does not limit the present application.

[0107] For example, in a gaming scenario, the GPU can continuously receive rendering instruction streams from the gaming application. Each rendering instruction stream indicates the generation of a real frame. Whenever a rendering instruction stream is received, such as rendering instruction stream 1 or rendering instruction stream 2, the GPU can execute steps 1 and 2 below for that rendering instruction stream.

[0108] Step 1: The GPU separates the 3D scene image and UI image corresponding to a real frame from the rendering instruction stream.

[0109] The 3D scene image mentioned above refers to a color image that does not contain UI pixels but contains the 3D scene, that is, an RGB image.

[0110] For example, the above-mentioned rendering instruction stream may be a Hook game rendering instruction stream. The GPU may identify the various instructions contained in the Hook game rendering instruction stream through the rendering instruction acquisition function according to the characteristics of the game's rendering instruction stream. The Hook game rendering instruction stream generally includes a variety of drawing instructions, such as scene drawing instructions, UI drawing instructions, and image display instructions. Among them, scene drawing instructions can be used to draw 3D scene images such as scenery, characters, special effects, and skills in the game, UI drawing instructions can be used to draw UI images such as control buttons, small maps, and floating text, and image display instructions are used to put the drawn image data into a specified location of the operating system (such as framebuffer) to complete the frame display.

[0111] The pixel coordinates of the 3D scene image and the pixel coordinates of the UI image are in a one-to-one correspondence, and the UI image is located on top of the 3D scene image. If an object in the UI image has the same pixel coordinates as an object in the 3D scene image, the object in the UI image may cause visual occlusion of the object in the 3D scene image. Objects in a 3D scene image may change pixel coordinates for two reasons: one reason is that objects such as characters, vehicles, or birds in the game scene move. As shown in FIG5A , in the world coordinate system O-XYZ corresponding to the game scene, the initial position of the virtual character is coordinate position 1. When the virtual character runs, it moves to coordinate position 2. In this case, based on the running of the virtual character, the position of the virtual character in the 3D scene image of the real frame N-1 and the 3D scene image of the real frame N+1 moves. Another reason is that the virtual camera in the game scene shakes, that is, the perspective of the virtual character corresponding to the game player changes. As shown in FIG5B , in the world coordinate system O-XYZ corresponding to the game scene, the position of the virtual character is always at coordinate position 1 of the world coordinate system O-XYZ, but the virtual camera moves from position P1 to position P2, and the orientation of the virtual camera changes. When the position and orientation of the virtual camera change, the perspective of the virtual character corresponding to the game player also changes. In this case, the position of the virtual character in the 3D scene image of the real frame N-1 and the 3D scene image of the real frame N+1 moves.

[0112] When the pixel coordinates of an object in the UI image are the same as those of an object in the 3D scene image, no matter what the above reasons are for changing the pixel coordinates, if the object in the 3D scene image is moved according to steps 3 to 5 below, the object in the UI image will move with the object in the 3D scene image, but in fact the object in the UI image may not move, or the motion vectors of the object in the UI image and the object in the 3D scene image are different, which will cause the final predicted frame to have more serious image quality problems. Therefore, the GPU needs to separate the 3D scene image and UI image corresponding to a real frame from a rendering instruction stream.

[0113] Specifically, for each game rendering instruction in a rendering instruction stream, the GPU can perform the following operations: Based on the characteristics of a game rendering instruction, identify whether it is a UI drawing instruction. For example, a UI drawing instruction can include resource binding instructions, state setting instructions, and drawing instructions; if it is a UI drawing instruction, direct the output of these instructions to a newly created image. This newly created image can be used to store UI pixels, so this newly created image can be called a UI image; if it is a scene drawing instruction, change the original rendering result to a newly created image for storing a 3D scene image that does not contain UI pixels. In this way, the GPU separates the UI image and the 3D scene image from the rendering instruction stream.

[0114] For example, as shown in Figure 6, the GPU first obtains the rendering instruction stream 1 for instructing the generation of real frame N-1, and separates the 3D scene image Color0 and UI image UI0 of real frame N-1 from the rendering instruction stream 1. If the UI image UI0 is superimposed on the 3D scene image Color0, a complete real frame N-1 is obtained. Then, the GPU can obtain the rendering instruction stream 2 for instructing the generation of real frame N+1, and separate the 3D scene image Color1 and UI image UI1 of real frame N+1 from the rendering instruction stream 2. If the UI image UI1 is superimposed on the 3D scene image Color1, a complete real frame N+1 is obtained.

[0115] Step 2: The GPU obtains the depth image, world matrix and projection matrix of the virtual camera corresponding to a real frame according to the rendering instruction stream.

[0116] Based on the characteristics of the game's rendering instructions, the GPU can identify the rendering instruction sequence for model rendering and determine the glBindBuffer instruction that precedes the model rendering drawcall instruction. The glBindBuffer instruction contains data related to the virtual camera. The GPU obtains the virtual camera's world matrix and projection matrix from the buffer corresponding to the glBindBuffer instruction. In addition, the GPU can also obtain a depth image from the rendering instruction stream.

[0117] It should be noted that the 3D scene image obtained by the GPU in step 1 and the depth image obtained in step 2 have the same image content; for example, both images contain people, trees, and mountains. The difference between the two images is that the 3D scene image is a three-channel (RGB) color image, while the depth image is a single-channel image. A depth image is similar to a grayscale image, and each pixel value in the depth image can be used to represent the distance between the virtual camera and the object. The depth image reflects the geometry of the object's visible surface.

[0118] To facilitate understanding of the world matrix and projection matrix of the virtual camera, the pixel coordinate system, image coordinate system, normalized device coordinate system (NDC), camera coordinate system, and world coordinate system involved in the embodiment of the present application are described below with reference to FIG7 .

[0119] The pixel coordinate system o-wh and the image coordinate system o-xy are two-dimensional coordinate systems. Typically, images are stored in arrays on terminal devices, and the value of each element (i.e., pixel) in the array represents the brightness of the image point. The origin of the pixel coordinate system o-wh is located in the upper left corner of the image, with the w axis pointing horizontally to the right and the h axis pointing vertically downward. Each pixel coordinate (w, h) represents the column and row number of a pixel in the array, respectively. Therefore, (w, h) is the coordinate of the pixel coordinate system o-wh in pixels. The origin of the image coordinate system o-xy is located at the center of the image (called the principal point of the image), with the x axis parallel to the w axis and the y axis parallel to the h axis. Coordinates in the pixel coordinate system o-wh and the image coordinate system o-xy can be converted to each other.

[0120] Assume that the pixel coordinates in the pixel coordinate system o-wh are (w0, h0), and the physical sizes of each pixel on the x-axis and y-axis are dx and dy. The relationship between the pixel coordinate system o-wh and the image coordinate system o-xy is as follows:

[0121] The normalized device coordinate system o-x'y'z' is a three-dimensional coordinate system. In the normalized device coordinate system o-x'y'z', the values ​​of x', y', and z' are in the interval [-1, 1], where z' represents the depth value of the image. x' and y' are obtained by normalizing the coordinates (x, y) in the image coordinate system o-xy, respectively. Through the viewport transform, the normalized device coordinate system o-x'y'z' can be transformed into the image coordinate system o-xy or the pixel coordinate system o-wh. Taking a 512*512 image as an example, the pixel coordinates of the lower right corner of the pixel coordinate system o-wh are (512, 512). After the coordinate transformation, the coordinates of this point in the normalized device coordinate system o-x'y'z' are (1, 1, z'), where z' represents the depth value of this coordinate.

[0122] The camera coordinate system O0-X0Y0Z0 is a three-dimensional coordinate system. In the camera coordinate system O0-X0Y0Z0, the virtual camera is a virtual camera set for the game scene, not a real camera. In the game scene, if there are multiple players roaming in the game world, since each player's position and line of sight in the game world are different, the frame images displayed by the terminal device used by each player are also different. Assuming that a coordinate system is created with a certain player as the origin, this coordinate system can be called a camera coordinate system or an observation space coordinate system. Referring to Figure 7, the origin O0 of the camera coordinate system is the optical center (projection center) of the virtual camera, the X0 axis and the Y0 axis are parallel to the x-axis and y-axis of the camera plane, the Z0 axis is the optical axis of the virtual camera, and the Z0 axis is perpendicular to the camera plane. oO0 is the focal length of the virtual camera. It should be understood that each player's position and line of sight in the game world are different, so the camera coordinate system O0-X0Y0Z0 established with the virtual camera corresponding to each player as the center will also be different.

[0123] The world coordinate system O-XYZ is a three-dimensional coordinate system introduced to describe the position of each virtual camera. In the game scene, virtual objects such as characters, tables, rooms, animals, and vehicles all have a specific position in the game. The position of the virtual object in the game world is defined as the coordinate in the world coordinate system, which is called the world space coordinate. For example, the world coordinate system O-XYZ can be a three-dimensional coordinate system established with the center of the earth as the origin. It should be understood that different players have different positions and sight directions in the world coordinate system O-XYZ, which makes the position and orientation of the virtual camera in the world coordinate system O-XYZ also different. The coordinates in the camera coordinate system O0-X0Y0Z0 can be converted into coordinates in the world coordinate system O-XYZ through translation and rotation.

[0124] The world matrix, also known as the extrinsic matrix, converts object coordinates in the world coordinate system O-XYZ into those described by the camera coordinate system O0-X0Y0Z0. In other words, when the virtual camera's perspective is used as the origin of the game scene, the world matrix transforms all world space coordinates into observation coordinates relative to the virtual camera's position and orientation.

[0125] For example, the world matrix can be expressed as:

[0126] Among them, (Rx, Ry, Rz) can represent the X-axis direction vector of the virtual camera, (Ux, Uy, Uz) can represent the Y-axis direction vector of the virtual camera, (Dx, Dy, Dz) represent the line of sight direction and Z-axis of the virtual camera. (Px, Py, Pz) can represent the world space coordinates of the virtual camera.

[0127] The projection matrix, also known as the intrinsic parameter matrix, is used to convert the object coordinates in the camera coordinate system O0-X0Y0Z0 into descriptions using the standardized device coordinate system o-x'y'z'. When rendering, a three-dimensional scene is usually projected onto the screen (i.e., the camera plane) to become a two-dimensional image. Specifically, all vertex data is first converted from the camera coordinate system O0-X0Y0Z0 to the clipping space, and then the converted clipping coordinates are transformed into the standardized device coordinate system o-x'y'z' by dividing by the w component in the clipping coordinate system. Among them, the camera coordinate system O0-X0Y0Z0 is a right-handed coordinate system, and the standardized device coordinate system o-x'y'z' is a left-handed coordinate system.

[0128] For example, assuming that the projection matrix is ​​a matrix of a general frustum, in which the coordinates of the four vertices of a cone section are expressed as (r, t, -n), (r, b, -n), (f, b, -n), (f, t, -n), the projection matrix can be expressed as:

[0129] Assume that the GPU obtains rendering instruction stream 1 and rendering instruction stream 2 in sequence. Rendering instruction stream 1 can be used to generate real frame N-1, and rendering instruction stream 2 can be used to generate real frame N+1. The GPU can follow steps 1 and 2 above to obtain the 3D scene image Color0, UI image UI0, depth image Depth0, virtual camera's world matrix ViewMatrix0, and virtual camera's projection matrix ProjectionMatrix0 corresponding to real frame N-1, as well as the 3D scene image Color1, UI image UI1, depth image Depth1, virtual camera's world matrix ViewMatrix1, and virtual camera's projection matrix ProjectionMatrix1 corresponding to real frame N+1. Then, the GPU can follow steps 3 to 7 below to generate predicted frame N based on these parameters.

[0130] Step 3: The GPU generates a mask image that marks dynamic pixels and static pixels based on the depth image of the real frame N-1, the world matrix and projection matrix of the virtual camera, and the depth image of the real frame N+1, the world matrix and projection matrix of the virtual camera.

[0131] Exemplarily, the GPU may generate a mask image through the following steps (1) to (3).

[0132] (1) Based on the depth image Depth0 of the real frame N-1, the world matrix ViewMatrix0 and the projection matrix ProjectionMatrix0 of the virtual camera of the real frame N-1, calculate the world space coordinates (x0, y0, z0) corresponding to each pixel coordinate (w0, h0) of the 3D scene image Color0.

[0133] Specifically, assuming that the coordinates (w0, h0) of any pixel in the 3D scene image Color0 of the real frame N-1 and the depth image Depth0 of the real frame N-1 are known, the GPU can first convert the coordinates (w0, h0) of any pixel in the 3D scene image Color0 into the image coordinates (x0, y0) of the image coordinate system o0-x0y0, and then calculate the normalized coordinates (x', y', z') of the image coordinates (x0, y0) in the standardized device coordinate system o0-x'y'z' based on the depth image Depth0 of the real frame N-1, and then use the inverse matrix of the projection matrix ProjectionMatrix0 to obtain the camera coordinates in the camera coordinate system O0-X0Y0Z0, and then use the inverse matrix of the world matrix ViewMatrix0 to obtain the world space coordinates (x0, y0, z0) in the world coordinate system O-XYZ. As shown in FIG8A , for any pixel coordinate in the 3D scene image Color0 , the world space coordinate (x0, y0, z0) corresponding to the pixel coordinate (w0, h0) can be obtained by referring to the above steps.

[0134] As an example, the conversion relationship from the coordinates (x', y', z') of the normalized device coordinate system o0-x'y'z' to the world space coordinates (x0, y0, z0) of the world coordinate system O-XYZ is as follows:

[0135] Where ViewMatrix0 represents the world matrix of the virtual camera of real frame N-1, and ProjectionMatrix0 represents the projection matrix of the virtual camera of real frame N-1. The specific implementation of world matrix ViewMatrix0 and projection matrix ProjectionMatrix0 can refer to the specific description of step 2 above and will not be repeated here.

[0136] (2) For the world space coordinates (x0, y0, z0) corresponding to each pixel in the 3D scene image Color0 of the real frame N-1 calculated in step (1) above, the world matrix ViewMatrix1 and projection matrix ProjectionMatrix1 of the virtual camera of the real frame N+1 are further used to reproject them to the pixel coordinate system, thereby obtaining the reprojected pixel coordinates (w1, h1).

[0137] Specifically, after calculating the world space coordinates (x0, y0, z0) corresponding to each pixel in the 3D scene image Color0 of the real frame N-1 through step (1), the GPU can first use the virtual camera's world matrix ViewMatrix1 to calculate the coordinates in the camera coordinate system O1-X1Y1Z1, and then use the virtual camera's projection matrix ProjectionMatrix1 to calculate the coordinates (x', y', z') in the standardized device coordinate system o1-x'y'z'. The coordinates (x', y', z') in the standardized device coordinate system o1-x'y'z' are then converted to the image coordinate system o1-x1y1, and then converted to the pixel coordinates (w1, h1) in the pixel coordinate system. As shown in Figure 8B, after reprojecting the world space coordinates (x0, y0, z0) corresponding to each pixel calculated through step (1), the reprojected pixel coordinates (w1, h1) can be obtained.

[0138] As an example, the conversion relationship from the world space coordinates (x0, y0, z0) of the world coordinate system O-XYZ to the coordinates (x', y', z') of the normalized device coordinate system o-x'y'z' is as follows:

[0139] Here, ProjectionMatrix1 represents the projection matrix of the virtual camera for real frame N+1, and ViewMatrix1 represents the world matrix of the virtual camera for real frame N+1. The specific implementation of world matrix ViewMatrix1 and projection matrix ProjectionMatrix1 can be referred to the detailed description of step 2 above and will not be repeated here.

[0140] (3) For each pixel coordinate (w1, h1) calculated and reprojected in step (2) above, the world space coordinate (x1, y1, z1) corresponding to each pixel coordinate (w1, h1) is further calculated based on the depth image Depth1 of the real frame N+1, the world matrix ViewMatrix1 of the virtual camera of the real frame N+1, and the projection matrix ProjectionMatrix1.

[0141] Specifically, after calculating the pixel coordinates (w1, h1) of each pixel after reprojection through step (2) above, the GPU can first convert the pixel coordinates (w1, h1) to the coordinates (x1, y1) of the image coordinate system o1-x1y1, and then calculate the coordinates (x', y', z') of the image coordinates (x1, y1) in the standardized device coordinate system o1-x'y'z' after normalization based on the depth image Depth1 of the real frame N+1. Then, the inverse matrix of the projection matrix ProjectionMatrix1 can be used to obtain the coordinates in the camera coordinate system O1-X1Y1Z1, and then the inverse matrix of the world matrix ViewMatrix1 can be used to obtain the world space coordinates (x1, y1, z1) in the world coordinate system O-XYZ. In this way, as shown in Figure 8C, the world space coordinates (x1, y1, z1) corresponding to the reprojected pixel coordinates (w1, h1) are obtained.

[0142] As an example, the conversion relationship from the coordinates (x', y', z') of the normalized device coordinate system o1-x'y'z' to the world space coordinates (x1, y1, z1) of the world coordinate system O-XYZ is as follows:

[0143] Where ViewMatrix1 represents the world matrix of the virtual camera for real frame N+1, and ProjectionMatrix1 represents the projection matrix of the virtual camera for real frame N+1. The specific implementation of world matrix ViewMatrix1 and projection matrix ProjectionMatrix1 can be referred to the detailed description of step 2 above and will not be repeated here.

[0144] It should be understood that for the same pixel: if the depth in the depth image Depth0 is different from the depth in the depth image Depth1, then as shown in FIG8C , after the coordinate transformation is performed according to steps (1) to (3) above, the pixel will correspond to two different positions in the world coordinate system O-XYZ, that is, the world space coordinates (x0, y0, z0) and the world space coordinates (x1, y1, z1) are different. Alternatively, if the depth in the depth image Depth0 is the same as the depth in the depth image Depth1, then after the coordinate transformation is performed according to steps (1) to (3) above, the pixel will correspond to the same position in the world coordinate system O-XYZ, that is, the world space coordinates (x0, y0, z0) and the world space coordinates (x1, y1, z1) are the same.

[0145] If the following three conditions x0=x1, y0=y1, z0=z1 are met at the same time, then the position of the pixel at the pixel coordinate (w0, h0) on the 3D scene image Color0 in the world coordinate system has not changed, and the pixel is a pixel of a static object (such as a stationary house), and the pixel is marked as Mask=0.

[0146] If any of the above three conditions is not met, such as x0≠x1, y0≠y1 or z0≠z1, then the position of the pixel at the pixel coordinate (w0,h0) on the 3D scene image Color0 in the world coordinate system changes, and the pixel is a pixel of a dynamic object (such as a running person), and the pixel is marked as Mask=1.

[0147] For example, FIG9 shows a schematic diagram of a mask image generated based on the 3D scene image Color0. Assume that the running person is a dynamic object, and other objects such as trees and mountains are static objects. A small square represents a pixel, and each object is composed of multiple pixels. For each pixel in the 3D scene image Color0, the GPU can determine whether a pixel is a pixel of a dynamic object or a pixel of a static object according to steps (1) to (3) above, and then mark the pixel of the dynamic object as Mask=1 and the pixel of the static object as Mask=0. After completing the marking of all pixels in the 3D scene image Color0, the GPU can output the Mask of all pixels to a mask image, thereby generating a mask image for marking dynamic and static pixels. It should be understood that the pixel coordinates of the mask image are in a one-to-one correspondence with the pixel coordinates of the 3D scene image Color0.

[0148] Step 4: The GPU calculates the motion vector of each pixel in the 3D scene image Color0 of the real frame N-1 obtained in step 1 based on the pixel labels of the mask image obtained in step 3.

[0149] For example, for each pixel in the mask image, the GPU may perform the following operations:

[0150] If the pixel mark of a pixel in the mask image is 0, it means that the pixel in the 3D scene image Color0 is a pixel of a static object (referred to as a static pixel for short), and the GPU can calculate the motion vector of the static pixel by reprojecting the coordinates.

[0151] As an example, when referring to (1) to (2) in the above step 3 and obtaining the reprojection coordinates corresponding to each pixel coordinate, for the static pixels marked Mask = 0 in the mask image, the GPU can filter out the reprojection coordinates (w1, h1) corresponding to each static pixel coordinate (w0, h0) from the reprojection coordinates of all pixels, and then use [(w1-w0)*0.5, (h1-h0)*0.5] to calculate the motion vector of the static pixel.

[0152] As another example, the GPU may not use the reprojected coordinates obtained from steps (1) to (2) in step 3 above, but instead re-perform the reprojection process for the static pixels marked Mask = 0 in the mask image. For example, the GPU may calculate the world space coordinates (x0, y0, z0) corresponding to each static pixel coordinate (w0, h0) based on the depth image Depth0 of the real frame N-1, the world matrix ViewMatrix0 of the virtual camera of the real frame N-1, and the projection matrix ProjectionMatrix0; then, using the world matrix ViewMatrix1 and projection matrix ProjectionMatrix1 of the virtual camera of the real frame N+1, the world space coordinates (x0, y0, z0) of the static pixel are reprojected to the pixel coordinate system, thereby obtaining a new pixel coordinate (w1, h1). After calculating the reprojected coordinates (w1, h1) corresponding to each static pixel coordinate (w0, h0), the GPU may use [(w1-w0)*0.5, (h1-h0)*0.5] to calculate the motion vector of the static pixel.

[0153] For example, assuming that the two-dimensional coordinates of a static pixel in the 3D scene image Color0 are (w0, h0) = (255, 255), and the two-dimensional coordinates of the static pixel after reprojection are (w1, h1) = (255, 269), then the motion vector of the static pixel is: [(255-255)*0.5, (269-255)*0.5] = [0, 7].

[0154] If the pixel mark of a pixel in the mask image is 1, it means that the pixel in the 3D scene image Color0 is a pixel of a dynamic object (referred to as a dynamic pixel for short), and the GPU can calculate the motion vector of the dynamic pixel by searching for the best matching pixel.

[0155] Specifically, the GPU can obtain the brightness Intensity0 corresponding to the coordinates (w0, h0) of a dynamic pixel in the 3D scene image Color0, as well as the feature vector Special0 corresponding to the dynamic pixel. The GPU can then search for the best matching pixel corresponding to the dynamic pixel in the 3D scene image Color1 according to the following method: Calculate the brightness Intensity1 of each pixel in the 3D scene image Color1 and the feature vector Special1 corresponding to the pixel. Through an iterative search, it traverses each pixel in the 3D scene image Color1 in turn to find the pixel (w1, h1) with the minimum absolute value of (Intensity0–Intensity1) and the minimum feature distance between the feature vector Special0 and the feature vector Special1. The pixel (w1, h1) is the best matching pixel. The GPU can then calculate the motion vector of the dynamic pixel using [(w1-w0)*0.5, (h1-h0)*0.5]. Among them, the eigenvectors Special0 and Special1 are used to characterize the brightness and geometric features of pixels in the image. They are pixels with obvious unique features in the image, such as table corners, wall corners, and clothing patterns. Typically, the main parameters of the eigenvector are the color and depth of the current pixel and the surrounding adjacent pixels within a certain range. For example, assuming that the pixel coordinates of a dynamic pixel in the 3D scene image Color0 are (w0,h0)=(255,257), and the pixel coordinates of the best matching pixel in the 3D scene image Color1 are (w1,h1)=(259,239), then the motion vector of the dynamic pixel is: [(259-255)*0.5, (239-257)*0.5]=[2,-9].

[0156] After calculating the motion vectors for each pixel in the 3D scene image Color0 according to the above steps, the GPU can output these motion vectors to a motion vector image mv. This motion vector image mv can be used for pixel-by-pixel offset in step 5 below.

[0157] Step 5: Based on the motion vector in the motion vector image mv obtained in step 4, the GPU performs a pixel-by-pixel offset on each pixel in the 3D scene image Color0 of the real frame N-1 obtained in step 1 to generate an intermediate image Color_NewFrame containing a blank area.

[0158] Exemplarily, FIG10 shows a schematic diagram of offsetting each pixel in the 3D scene image Color0 to the intermediate image Color_NewFrame pixel by pixel.

[0159] First, the GPU can create a new four-channel color image Color_NewFrame (called the intermediate image) and initialize the pixel value Color_NewFrame.rgba of each pixel in the intermediate image Color_NewFrame to (0.0, 0.0, 0.0, 0.5). This intermediate image Color_NewFrame can be used to store the pixel values ​​after pixel-by-pixel offset. Among them, Color_NewFrame.rgb = (0.0, 0.0, 0.0) means that the values ​​of the RGB channels of the intermediate image Color_NewFrame are all 0.0, that is, black; Color_NewFrame.a = 0.5, which means that the value of the A channel of the intermediate image Color_NewFrame is 0.5. The pixel value needs to be predicted later, that is, the pixel needs to be repaired.

[0160] It should be noted that in the embodiment of the present application, the RGB three channels refer to the R channel, the G channel, and the B channel, where the R channel represents red, the G channel represents green, and the B channel represents blue. The intensity values ​​of the RGB three channels can be 0 to 255, or the intensity values ​​of the RGB three channels can be normalized to 0.0 to 1.0. The A channel refers to the alpha channel, which refers to the transparency and translucency of an image. Typically, the value range of the A channel is 0.0 to 1.0.

[0161] Then, the GPU can perform a pixel-by-pixel offset on each pixel Color0.rgba in the 3D scene image Color0 based on the motion vector calculated in step 4 above. As an example, as shown in FIG10 , the GPU can first move the static pixels one by one to the intermediate image Color_NewFrame based on the motion vector of the static pixel calculated in step 4, and then move the dynamic pixels one by one to the intermediate image Color_NewFrame based on the motion vector of the dynamic pixel calculated in step 4, thereby obtaining the intermediate image Color_NewFrame that stores the offset pixel values. As another example, the GPU can also start from the upper left corner of the 3D scene image Color0 and move each pixel row by row and column by column based on the motion vector calculated in step 4 above, that is, move the static pixels and the dynamic pixels at the same time.

[0162] For each pixel Color0.rgba in the 3D scene image Color0, assuming that the two-dimensional coordinates of a pixel are (w0, h0), the motion vector of the pixel at (w0, h0) in the motion vector image mv is mv0. The GPU can move the pixel coordinates (w0, h0) according to the motion vector mv0 to obtain the new two-dimensional coordinates (w1, h1). Then, the GPU can reassign the pixel value Color_NewFrame.rgba at the coordinates (w1, h1) of the intermediate image Color_NewFrame to:

[0163] Color_NewFrame.rgb=Color0.rgb;

[0164] Color_NewFrame.a=1.0.

[0165] Here, Color_NewFrame.rgb = Color0.rgb, which assigns the RGB three-channel color at coordinate (w1, h1) of the intermediate image Color_NewFrame to the RGB three-channel color at coordinate (w0, h0) of the pixel in the 3D scene image Color0. Color_NewFrame.a = 1.0, which sets the A channel at coordinate (w1, h1) of the intermediate image Color_NewFrame to 1. This means that no subsequent pixel value prediction is required, meaning that no inpainting is performed on that pixel.

[0166] It should be noted that the embodiment of the present application is described by taking the example of setting the value of the A channel of a pixel coordinate to 0.5 to indicate that the coordinate needs to be repaired, and setting the value of the A channel of a pixel coordinate to 1 to indicate that the coordinate does not need to be repaired, which does not limit the present application. In actual implementation, the value of the A channel of a pixel coordinate can also be set to 1 to indicate that the pixel at the coordinate needs to be repaired, and the value of the A channel of a pixel coordinate can be set to 0.5 to indicate that the pixel at the coordinate does not need to be repaired.

[0167] Take the pixel-by-pixel offset method shown in Figure 10 as an example. When an object in the 3D scene image Color0 (such as a tree) is static, even if the virtual camera is displaced and rotated, the coordinates of each static pixel of the static object in the world coordinate system remain unchanged. Therefore, after each static pixel is moved one by one according to the motion vector, pixel occlusion caused by multiple static pixels moving to the same position is unlikely to occur, and blank areas caused by no pixels moving are also unlikely to occur. However, when an object in the 3D scene image Color0 (such as a running person) is dynamic, the motion vectors corresponding to each dynamic pixel of the dynamic object may be different. As a result, multiple dynamic pixels may move to the same position (w1, h1) in the intermediate image Color_NewFrame, resulting in pixel occlusion at position (w1, h1). At some positions, no pixels may move, causing the pixel values ​​at these positions to retain the initialization values ​​(0.0, 0.0, 0.0, 0.5). In combination with the description of the above embodiment, when the A channel value of certain areas in the intermediate image Color_NewFrame is 0.5, it means that pixel value prediction needs to be performed on these areas later. These areas are blank areas that need to be filled with pixels later.

[0168] Due to edge characteristics and algorithmic reasons, the motion vectors in the edge areas of dynamic objects and the motion vectors of the static areas adjacent to the edge areas may deviate significantly. When the motion vectors corresponding to the dynamic pixels of a dynamic object are different, multiple dynamic pixels in the edge area may move to the same position, making it more likely that blank areas will appear in the edge areas of the dynamic object. Of course, in some cases, blank areas may also appear in non-edge areas of dynamic objects, such as joints and facial areas, but this is not limited to this in the embodiments of the present application.

[0169] It should be noted that the above-mentioned "blank area" refers to certain areas in the intermediate image Color_NewFrame where no pixel values ​​are moved from the 3D scene image Color0 and the initialization values ​​(0.0, 0.0, 0.0, 0.5) are retained. Pixel filling is required later, but it does not mean that the area is white.

[0170] The following describes how to solve the pixel occlusion problem that occurs when multiple pixels in the 3D scene image Color0 are moved to the same position in the intermediate image Color_NewFrame.

[0171] The first possible implementation:

[0172] As shown in FIG. 11, at the initial moment, the coordinates of four adjacent pixel points in the intermediate image Color_NewFrame are (w1, h1), (w2, h1), (w1, h2), and (w2, h2) in sequence, and the Color_NewFrame.rgba of these four pixel points are all initialized to (0.0, 0.0, 0.0, 0.5). The coordinates of four adjacent pixel points in the 3D scene image Color0 are (w1, h1), (w2, h1), (w1, h2), and (w2, h2) in sequence, and the Color_NewFrame.rgba of these four pixel points are (r1, g1, b1, a1), (r2, g2, b2, a2), (r3, g3, b3, a3), and (r4, g4, b4, a4) in sequence. Pixel 1 with pixel coordinate (w1, h1) is moved based on the motion vector mv(1, 1), pixel 2 with pixel coordinate (w2, h1) is moved based on the motion vector mv(zero, 1), and pixel 3 with pixel coordinate (w1, h2) is moved based on the motion vector mv(1, zero). Eventually, these three pixels are all moved to the pixel coordinate (w2, h2) of the intermediate image Color_NewFrame, resulting in pixel occlusion at the pixel coordinate (w2, h2). Additionally, if no pixels are moved to (w1, h1), (w2, h1), and (w1, h2) in the intermediate image Color_NewFrame, the pixels at these three positions still retain the initialization value (0.0, 0.0, 0.0, 0.5), which is a blank area.

[0173] Suppose pixel 1, pixel 2, and pixel 3 shown in FIG. 11 are moved to the same position P(w0, h0) in the intermediate image Color_NewFrame, and the depth corresponding to pixel 1 is l1, the depth corresponding to pixel 2 is l2, and the depth corresponding to pixel 3 is l3. As shown in FIG. 12, the origin O0 of the camera coordinate system O0-X0Y0Z0 is the optical center of the virtual camera. The GPU can refer to the description of step 3 above and convert the coordinate (w2, h2) to the camera coordinate system O0-X0Y0Z0 in sequence according to the depth l1, depth l2, or depth l3, thereby obtaining three positions P1, P2, and P3 in the camera coordinate system O0-X0Y0Z0. Then, the GPU can calculate the distance L1 from position P1 to the origin O0 of the camera coordinate system O0-X0Y0Z0, the distance L2 from position P2 to the origin O0 of the camera coordinate system O0-X0Y0Z0, and the distance L3 from position P3 to the origin O0 of the camera coordinate system O0-X0Y0Z0. Since L1 < L2 < L3 and position P1 is the closest to the origin O0, the GPU can use the pixel value Color0.rgba1 of pixel 1 corresponding to position P1 as the pixel value to be finally displayed at the coordinate (w2, h2) of the intermediate image Color_NewFrame.

[0174] The distances L1, L2, and L3 can be obtained by the following equations:

[0175] L1=sqrt(x1*x1+y1*y1+z1*z1);

[0176] L2=sqrt(x2*x2+y2*y2+z2*z2);

[0177] L3=sqrt(x3*x3+y3*y3+z3*z3).

[0178] Among them, sqrt() is a square root function, which is used to calculate the modulus of a position in the camera coordinate system O0-X0Y0Z0, that is, the distance from a position in the camera coordinate system O0-X0Y0Z0 to the origin O0.

[0179] It should be noted that the above-mentioned first possible implementation method is illustrated by using the camera coordinate system to calculate the distance between each pixel point and the origin O0 of the camera coordinate system O0-X0Y0Z0, which does not limit the present application. Since coordinates of the two world coordinate systems can be converted to each other, the GPU can also use the world coordinate system to calculate the distance between each pixel and the origin O0 of the world coordinate system O0-X0Y0Z0, and use the pixel value Color0.rgba1 of the pixel 1 closest to the origin of the world coordinate system O0-X0Y0Z0 as the pixel value of the coordinate (w2,h2) displayed in the intermediate image Color_NewFrame. It should be understood that whether the world coordinate system or the camera coordinate system is used, the final calculation result is the same pixel point.

[0180] The second possible implementation is:

[0181] The GPU calculates the world space coordinates (x0, y0, z0) corresponding to each pixel coordinate (w0, h0) of the 3D scene image Color0 based on the depth image Depth0 of the real frame N-1, the world matrix ViewMatrix0 of the virtual camera of the real frame N-1, and the projection matrix ProjectionMatrix0. Then, using the world space coordinates (x0, y0, z0), the modulus distance corresponding to each pixel coordinate is calculated as distance = sqrt(x0*x0+y0*y0+z0*z0).

[0182] After completing the offset of all pixels in the 3D scene image Color0, if multiple pixels in the 3D scene image Color0 are moved to the same position (w1, h1) of the intermediate image Color_NewFrame, the GPU can compare the modulus distances of these multiple pixels and use the pixel value of the pixel coordinate corresponding to the minimum modulus distance as the pixel value finally displayed in the intermediate image Color_NewFrame.

[0183] It should be noted that, no matter whether the first possible implementation method or the second possible implementation method is adopted, the final calculation result is the same pixel point.

[0184] In addition, the two possible implementations described above are illustrative examples of moving three pixels to the same position P(w0, h0) in the intermediate image Color_NewFrame and selecting one pixel from the three pixels as the pixel ultimately displayed at position P(w0, h0) in the intermediate image Color_NewFrame, and do not limit the present application. In actual implementation, the above method can also be used to select one pixel from two, four, or more pixels moved to the same position as the pixel ultimately displayed at that position in the intermediate image Color_NewFrame.

[0185] Step 6. After the GPU generates the edge image edge_map based on the 3D scene images Color0 and Color1 obtained in step 1, the edge image edge_map and the intermediate image Color_NewFrame obtained in step 5 are input into the AI ​​model to obtain the 3D scene prediction image Pixel_Color.

[0186] Exemplarily, the GPU may generate the 3D scene prediction image Pixel_Color through the following (a) to (e).

[0187] (a) Use the edge detection algorithm to extract the edge image edge0 of the 3D scene image Color0 and the edge image edge1 of the 3D scene image Color1.

[0188] Among them, the edge image edge0 and the edge image edge1 are both single-channel images, that is, grayscale images.

[0189] The edge image is obtained by extracting edges from the original image using an edge detection algorithm. For example, the edge detection algorithm may be a Canny algorithm, a differential operator method, a Laplace Gaussian operator method, a fitting method, a neural network method, a wavelet transform method, a wavelet packet decomposition method, or a morphological method.

[0190] Taking the Canny algorithm as an example, the GPU can first use a Gaussian filter to smooth the 3D scene images Color0 / Color1 to remove noise. It then calculates the gradient strength and direction of each pixel in the image. It then uses non-maximum suppression to eliminate spurious responses introduced by edge detection. It then uses a double-threshold detection method to determine true and potential edges. Finally, it suppresses isolated weak edges and retains true edges, resulting in the edge images edge0 / edge1.

[0191] (b) Create two new edge images edge0_warp and edge1_warp.

[0192] Among them, the edge image edge0_warp and the edge image edge1_warp are both single-channel images, that is, grayscale images.

[0193] The edge image edge0_warp can be used to save the warping result of the edge image edge0.

[0194] The edge image edge1_warp can be used to save the warping result of the edge image edge1.

[0195] (c) performing image warping on edge image edge0 in units of pixel blocks, and outputting the warped result of edge image edge0 to edge image edge0_warp; and performing image warping on edge image edge1 in units of pixel blocks, and outputting the warped result of edge image edge1 to edge image edge1_warp.

[0196] The above pixel block may be composed of multiple pixel points.

[0197] As an example, a pixel block can be composed of N*N pixels. Wherein, N is an integer greater than or equal to 2. For example, N=8. According to the description of step 5 above, when the real image is offset pixel by pixel in units of pixels, multiple pixels may move to the same position of the intermediate image Color_NewFrame, resulting in pixel occlusion, resulting in blank areas. It should be understood that the above (c) warping the edge image in units of pixel blocks can solve the problem of pixel occlusion, thereby facilitating image repair of blank areas in subsequent steps.

[0198] The present application provides a 2D grid model. The 2D grid model can be an offline grid model manually constructed.

[0199] As an example, Figure 13 shows a 2D grid model. The 2D grid model consists of p*q rectangular grids, where p represents the number of rows of the rectangular grid in the 2D grid model, and q represents the number of columns of the rectangular grid in the 2D grid model. For each rectangular grid, one rectangular grid represents a pixel block, for example, consisting of 8*8 pixels. In other words, the 2D grid model consists of 8m*8n pixels. It should be noted that the pixel coordinates of the 2D grid model are in one-to-one correspondence with the pixel coordinates in the 3D scene image Color0, the 3D scene image Color1, the edge image edge0, the edge image edge1, the edge image edge0_warp, and the edge image edge1_warp. Suppose the pixel coordinates of the four vertices of a rectangular mesh are (0,0), (7,0), (0,7), and (7,7), and the motion vectors for these four vertex pixel coordinates determined in step 4 above are mv1 = (0,2), mv2 = (2,1), mv3 = (1,1), and mv4 = (1,2). The GPU can move the pixel coordinates of these four vertices according to the motion vector corresponding to each vertex pixel coordinate, thereby obtaining the pixel coordinates of the four new vertices, (0,2), (9,1), (1,8), and (8,9). The GPU can connect the pixel coordinates of the four new vertices in sequence to obtain the new mesh shown in the dashed box in Figure 13. The new mesh is then rasterized and the linear interpolation method is used to calculate the coordinates of each pixel in the new mesh. Since the texture coordinates (ux,uy) of each vertex before and after the movement remain unchanged, the pixel data before the movement can be assigned to the pixel coordinates after the movement based on the texture coordinates (ux,uy).

[0200] It should be noted that Figure 13 only illustrates the image warping of a rectangular grid consisting of 8*8 pixels in a 2D grid model. It should be understood that the above description can be applied to any rectangular grid in the 2D grid model by moving the pixel coordinates of its four vertices to perform image warping. After warping all rectangular grids, a new edge image is obtained.

[0201] 14A and 14B , the specific process of rendering the edge image edge0_warp and the edge image edge1_warp using the 2D mesh model will be described with examples.

[0202] FIG. 14A is a schematic diagram showing how the edge image edge0 is warped to obtain the edge image edge0_warp.

[0203] As shown in Figure 14A, the GPU can first extract the edge image edge0 from the 3D scene image Color0. The edge image edge0 is a single-channel image. Then, the GPU can move each vertex according to the motion vector mv0 corresponding to each vertex vertex0 in the 2D mesh model to form a warped new mesh. After that, each new mesh is rasterized and the pixel coordinates of each pixel in the new mesh are linearly interpolated. Each pixel coordinate is used to determine the position of a pixel in the edge image edge0_warp. Assume that the pixel coordinates of a pixel before moving are (vx, vy), and the pixel coordinates after moving according to the motion vector (mv.x, mv.y) are (vx_n, vy_n) = (vx+mv.x, vy+mv.y). Since the texture coordinates (ux, uy) of a pixel point remain unchanged before and after movement, that is, (ux_n, uy_n) = (ux, uy), the GPU can collect pixel data from the pixel coordinates (vx, vy) of the edge image edge0 according to the texture coordinates (ux, uy) corresponding to the pixel coordinates (vx_n, vy_n) after movement, and output the collected pixel data to the pixel coordinates (vx_n, vy_n) of the edge image edge0_warp, and finally obtain the edge image edge0_warp that saves the distortion result of the edge image edge0.

[0204] FIG14B is a schematic diagram showing how the edge image edge1 is warped to obtain the edge image edge1_warp.

[0205] As shown in Figure 14B, the GPU can first extract the edge image edge1 from the 3D scene image Color1. The edge image edge1 is a single-channel image. Then, the GPU can move each vertex according to the motion vector mv0*(-1.0) corresponding to each vertex vertex0 in the 2D mesh model to form a distorted new mesh, and then rasterize each new mesh and linearly interpolate the pixel coordinates of each pixel in the new mesh. The pixel coordinates of each pixel are used to determine the position of a pixel in the edge image edge1_warp. Assume that the pixel coordinates of a pixel before moving are (vx, vy), and the pixel coordinates after moving according to the motion vector (-mv.x, -mv.y) are (vx_n, vy_n) = (vx-mv.x, vy-mv.y). Since the texture coordinates (ux, uy) of a pixel point remain unchanged before and after movement, that is, (ux_n, uy_n) = (ux, uy), the GPU can collect pixel data from the pixel coordinates (vx, vy) of the edge image edge1 according to the texture coordinates (ux, uy) corresponding to the pixel coordinates (vx_n, vy_n) after movement, and output the collected pixel data to the pixel coordinates (vx_n, vy_n) of the edge image edge1_warp, finally obtaining the edge image edge1_warp that saves the warping result of the edge image edge1.

[0206] In the above embodiment, the GPU performs image distortion on the edge image edge0 and the edge image edge1 in units of pixel blocks, which can effectively prevent multiple pixels from moving to the same position and ensure the integrity of the edge image.

[0207] (d) Based on the A channel label of the intermediate image Color_NewFrame, the edge image edge0_warp and the edge image edge1_warp are output to the final edge image edge_map.

[0208] For example, the GPU can create an edge image edge_map, where the pixel coordinates of the intermediate image Color_NewFrame, the edge image edge0_warp, the edge image edge1_warp, and the edge image edge_map are in a one-to-one correspondence. The GPU can determine the fill value of the corresponding pixel coordinate in the edge image edge_map based on the specific value of the A channel of each pixel coordinate in the intermediate image Color_NewFrame.

[0209] If A=0.5 at a pixel coordinate in the intermediate image Color_NewFrame, the pixel data of the edge image edge1_warp is used to fill the corresponding pixel coordinate in the edge image edge_map. That is, the edge image edge1_warp is used to fill the blank area.

[0210] If A≠0.5 for a pixel in the intermediate image Color_NewFrame, the corresponding pixel in the edge image edge_map is filled with pixel data from the edge image edge0_warp. Filling with pixel data from the edge image edge0_warp generates a complete edge image, making it easier for the AI ​​model to recognize edges.

[0211] For example, after obtaining the edge image edge0_warp through Figure 14A and the edge image edge1_warp through Figure 14B, as shown in Figure 14C, in the final edge image edge_map, the area corresponding to the area of ​​the intermediate image Color_NewFrame where A≠0.5 is filled with the data of the edge image edge0_warp, and the area corresponding to the area of ​​the intermediate image Color_NewFrame where A=0.5 is filled with the data of the edge image edge1_warp.

[0212] After edge images edge0_warp and edge images edge1_warp are output to the final edge image edge_map, the edge image edge_map is a grayscale image. In some embodiments, the GPU can directly input the single-channel edge image edge_map into the AI ​​model, which then converts the single-channel edge image edge_map into an RGB image. In other embodiments, the GPU can first convert the single-channel edge image edge_map into an RGB image, and then input the converted RGB image into the AI ​​model.

[0213] (e) The edge image edge_map obtained in (d) and the intermediate image Color_NewFrame obtained in step 5 are input to the AI ​​model of the NPU. After the AI ​​model fills the blank areas of the intermediate image Color_NewFrame with pixels, the GPU obtains the 3D scene prediction image Pixel_Color from the NPU.

[0214] Among them, the above-mentioned AI model can be a neural network model, such as an AI image restoration (AI inpainting) model.

[0215] Exemplarily, the terminal device can pre-train a neural network model and load the neural network model onto the NPU for executing the neural network algorithm. As shown in Figure 15, the GPU can input the edge image edge_map obtained by the above (d) and the intermediate image Color_NewFrame obtained by the above step 5 into the AI ​​model. The AI ​​model performs convolution and deconvolution on the edge image edge_map and the intermediate image Color_NewFrame to achieve image repair of the area to be filled in the intermediate image Color_NewFrame, thereby generating a complete 3D scene prediction image Pixel_Color. Then, the NPU returns the 3D scene prediction image Pixel_Color output by the AI ​​model to the GPU.

[0216] In conjunction with the description of the above embodiments, traditional AI models are usually loaded on devices with high computing power, such as PCs. By inputting the real frame into the AI ​​model, a complete predicted frame can be obtained. However, the algorithms of traditional AI models are complex, and the requirements for computing power and bandwidth are high, making them unable to run in real time on terminal devices with limited computing power, such as mobile phones. Unlike traditional AI models, the AI ​​model provided in this application is only used to repair blank areas of images, and has low requirements for computing power. It can run in real time on terminal devices with limited computing power, such as mobile phones. In addition, by predicting areas with missing data, the generated 3D scene predicted image Pixel_Color is closer to the result of the original image.

[0217] Step 7: The GPU superimposes the UI image UI0 of the real frame N-1 obtained in step 1 onto the 3D scene prediction image Pixel_Color obtained in step 6 to generate the final prediction frame image Pixel_Final, that is, the prediction frame N.

[0218] For example, as shown in FIG16 , the GPU may perform alpha blending (i.e., A channel blending) of the UI image UI0 of the real frame N-1 generated by the above step 1 and the 3D scene prediction image Pixel_Color generated by the above step 6 to generate the final prediction frame image Pixel_Final, i.e., the prediction frame N.

[0219] Among them, the rules of alpha blending can be:

[0220] Pixel_Final.rgb=Pixel_Color.rgb*(1.0–Pixel_UI.a)+Pixel_UI.rgb*Pixel_UI.a;

[0221] Pixel_Final.a=1.0.

[0222] Among them, Pixel_Final.rgb is the RGB three-channel pixel of the predicted frame image Pixel_Final, Pixel_Final.a is the A channel pixel of the predicted frame image Pixel_Final, that is, Pixel_Final.rgba is the RGBA four-channel pixel of the predicted frame image Pixel_Final.

[0223] Pixel_Color.rgb is the RGB three-channel pixel of the 3D scene prediction image Pixel_Color, Pixel_Final.a is the A channel pixel of the 3D scene prediction image Pixel_Color, that is, Pixel_Color.rgba is the RGBA four-channel pixel of the 3D scene prediction image Pixel_Color.

[0224] Pixel_UI.rgb is the RGB three-channel pixel of UI image UI0 in real frame N-1, and Pixel_UI.a is the A channel pixel of UI image UI0 in real frame N-1. That is, Pixel_UI.rgba is the RGBA four-channel pixel of UI image UI0 in real frame N-1. It should be noted that Pixel_UI.a can be obtained from the rendering instruction stream of real frame N-1.

[0225] It should be understood that the transparency effect of the UI interface can be achieved through alpha blending. The degree of transparency can be controlled by Pixel_UI.a. When Pixel_UI.a=0 for a pixel point in the UI interface, it means that the pixel point is completely transparent and will not block the 3D scene prediction image Pixel_Color located at the lower layer of the UI interface; and when Pixel_UI.a=1 for a pixel point in the UI interface, it means that the pixel point is completely opaque and will block the 3D scene prediction image Pixel_Color located at the lower layer of the UI interface.

[0226] In conjunction with the description of the above embodiment, the GPU can generate real frame N-1 based on the rendering instruction stream 1 obtained first, generate real frame N+1 based on the rendering instruction stream 2 obtained later, and generate predicted frame N based on rendering instruction streams 1 and 2. The GPU can enter these frames into the queue to be displayed in the order of real frame N-1, predicted frame N, and real frame N+1. Then, as shown in Figure 17, the display screen of the terminal device plays real frame N-1, predicted frame N, and real frame N+1 in sequence.

[0227] In the above-mentioned predicted frame generation method provided in the present application, the GPU can obtain the three-dimensional scene image and UI image corresponding to each real frame from two game rendering instructions, as well as general information such as the depth image, the world matrix and projection matrix of the game virtual camera, and then use this information to perform pixel-by-pixel offset on the 3D scene image, thereby solving the physical distortion problem existing in the block-by-block offset scheme and the ghosting problem existing in the image mixing scheme; the GPU can then use AI image restoration technology to fill the blank area after the pixel-by-pixel offset, so that the final generated predicted frame image is closer to the native picture quality of the game.

[0228] An embodiment of the present application further provides a terminal device, including a processor, the processor being coupled to a memory, and the processor being configured to execute a computer program or instruction stored in the memory, so that the terminal device implements the methods in the above embodiments.

[0229] The embodiment of the present application also provides a computer-readable storage medium having computer instructions stored therein. When the computer-readable storage medium is run on a computer, the computer is caused to execute the method shown above. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more media that can be integrated. The available medium can be a magnetic medium (e.g., a floppy disk, hard disk or tape), an optical medium or a semiconductor medium (e.g., a solid state drive (SSD)), etc.

[0230] An embodiment of the present application further provides a computer program product, which includes computer program code. When the computer program code runs on a computer, the computer executes the methods in the above embodiments.

[0231] The present application also provides a chip that is coupled to a memory and is used to read and execute computer programs or instructions stored in the memory to perform the methods in the above embodiments. The chip can be a general-purpose processor or a dedicated processor. It should be noted that the chip can be implemented using the following circuits or devices: one or more field programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gate logic, discrete hardware components, any other suitable circuits, or any combination of circuits capable of performing the various functions described throughout this application.

[0232] The terminal device, computer-readable storage medium, computer program product and chip provided in the above-mentioned embodiments of the present application are all used to execute the method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects corresponding to the method provided above, and will not be repeated here.

[0233] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0234] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0235] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0236] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0237] The above content is only a specific embodiment of this application, but the scope of protection of this application is not limited to this. Any changes or replacements within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A prediction frame generation method, characterized in that: The method comprises: Obtaining a user interface image, a first scene image, a first depth image, a first projection matrix, and a first world matrix from a first rendering instruction stream; Obtain a second scene image, a second depth image, a second projection matrix, and a second world matrix from a second rendering instruction stream; determining static pixels and dynamic pixels in the first scene image based on the first depth image, the first projection matrix, the first world matrix, the second depth image, the second projection matrix, and the second world matrix; Determine the motion vector of the static pixel according to the pixel coordinates of the static pixel and the reprojected coordinates after reprojecting the static pixel; and determine the matching pixel of the dynamic pixel in the second scene image, and determine the motion vector of the dynamic pixel according to the pixel coordinates of the dynamic pixel and the pixel coordinates of the matching pixel; Move the static pixel based on the motion vector of the static pixel, and move the dynamic pixel based on the motion vector of the dynamic pixel to obtain an intermediate image; Filling pixels in the blank area of ​​the intermediate image based on the AI ​​model to obtain a scene prediction image; The user interface image is superimposed on the scene prediction image to obtain a prediction frame.

2. The method according to claim 1, characterized in that The determining, based on the first depth image, the first projection matrix, the first world matrix, the second depth image, the second projection matrix, and the second world matrix, static pixels and dynamic pixels in the first scene image comprises: Perform the following steps for each pixel in the first scene image: Determine, based on the first depth image, the first projection matrix, and the first world matrix, a first world space coordinate corresponding to a first pixel coordinate of the first scene image; reproject the first world space coordinate to a pixel coordinate system based on the second projection matrix and the second world matrix to obtain a second pixel coordinate; determine, based on the second depth image, the second projection matrix, and the second world matrix, a second world space coordinate corresponding to the second pixel coordinate; For any pixel in the first scene image: If the first world space coordinate and the second world space coordinate of a pixel are the same, the pixel is marked as a static pixel; or if the first world space coordinate and the second world space coordinate of the pixel are different, the pixel is marked as a dynamic pixel.

3. The method according to claim 2, characterized in that The method further comprises: Output the labeling result of each pixel to the mask image.

4. The method according to any one of claims 1 to 3, characterized in that The determining the motion vector of the static pixel according to the pixel coordinates of the static pixel and the reprojected coordinates after reprojecting the static pixel comprises: Determining world space coordinates corresponding to static pixel coordinates (w0, h0) of the first scene image based on the first depth image, the first projection matrix, and the first world matrix; Based on the second projection matrix and the second world matrix, reproject the world space coordinates corresponding to the static pixel coordinates (w0, h0) to a pixel coordinate system to obtain the reprojected coordinates (w1, h1); [(w1-w0)*0.5, (h1-h0)*0.5] is determined as the motion vector of the static pixel.

5. The method according to any one of claims 1 to 3, characterized in that The step of determining a matching pixel of the dynamic pixel in the second scene image, and determining a motion vector of the dynamic pixel according to the pixel coordinates of the dynamic pixel and the pixel coordinates of the matching pixel, comprises: Acquire brightness and feature vectors corresponding to the dynamic pixel coordinates (w0', h0') of the first scene image; According to the brightness and feature vector corresponding to the dynamic pixel coordinates (w0', h0'), search in the second scene image The matching pixel (w1', h1'); [(w1'-w0')*0.5, (h1'-h0')*0.5] is determined as the motion vector of the dynamic pixel.

6. The method according to claim 5, characterized in that The matching pixels satisfy the following conditions: The absolute value of the first difference is smaller than the absolute value of the second difference, the first difference is the difference between the brightness corresponding to the dynamic pixel coordinate (w0', h0') and the brightness corresponding to the matching pixel (w1', h1'), and the second difference is the brightness corresponding to the dynamic pixel coordinate (w0', h0') and the brightness corresponding to other pixel coordinates of the second scene image; and The first feature distance is smaller than the second feature distance. The first feature distance is the feature distance between the feature vector corresponding to the dynamic pixel coordinate (w0', h0') and the feature vector corresponding to the matching pixel (w1', h1'). The second feature distance is the feature distance between the feature vector corresponding to the dynamic pixel coordinate (w0', h0') and the feature vector corresponding to other pixel coordinates of the second scene image.

7. The method according to any one of claims 1 to 6, characterized in that The step of moving the static pixel based on the motion vector of the static pixel and moving the dynamic pixel based on the motion vector of the dynamic pixel to obtain an intermediate image comprises: Creating an intermediate image; Based on the motion vector of each static pixel in the first scene image, move each static pixel one by one to the intermediate image; Based on the motion vector of each dynamic pixel in the first scene image, each dynamic pixel is moved one by one to the intermediate image.

8. The method according to claim 7, characterized in that After creating the intermediate image, the method further comprises: Assign the RGB channels of the intermediate image to 0.0, and assign the A channel of the intermediate image to 0.5; The step of moving each static pixel one by one to the intermediate image based on the motion vector of each static pixel in the first scene image comprises: Based on the motion vector of each static pixel in the first scene image, determine the pixel coordinates corresponding to each static pixel in the intermediate image; assign the RGB channels of the pixel coordinates corresponding to each static pixel in the intermediate image to a first value, and assign the A channel of the pixel coordinates corresponding to each static pixel in the intermediate image to 1.0; wherein the first value is the RGB channel pixel value of each static pixel in the first scene image; The step of moving each dynamic pixel one by one to the intermediate image based on the motion vector of each dynamic pixel in the first scene image comprises: Based on the motion vector of each dynamic pixel in the first scene image, the pixel coordinates corresponding to each dynamic pixel are determined in the intermediate image; the RGB channels of the pixel coordinates corresponding to each dynamic pixel in the intermediate image are assigned a second value, and the A channel of the pixel coordinates corresponding to each dynamic pixel in the intermediate image is assigned 1.0; wherein the second value is the RGB channel pixel value of each dynamic pixel in the first scene image.

9. The method according to claim 8, characterized in that The method further comprises: If a plurality of pixels are moved to the same pixel coordinate of the intermediate image, converting the same pixel coordinate to a camera coordinate system according to the depth of each pixel of the plurality of pixels in the first depth image to obtain the camera coordinate of each pixel of the plurality of pixels; According to the camera coordinates of each pixel in the plurality of pixels, the camera coordinates of each pixel in the plurality of pixels are calculated to The distance to the camera origin; The RGB channels of the same pixel coordinates are assigned values ​​according to the RGB channel pixel value of a pixel corresponding to the minimum distance among the multiple distances.

10. The method according to any one of claims 1 to 9, characterized in that Filling pixels in the blank area of ​​the intermediate image based on the AI ​​model to obtain a scene prediction image includes: generating a target edge image according to the first scene image and the second scene image; The target edge image and the intermediate image are input into the AI ​​model to generate the scene prediction image; wherein the scene prediction image is an image obtained by the AI ​​model using the target edge image to fill pixels in the blank area of ​​the intermediate image.

11. The method according to claim 10, characterized in that The step of generating a target edge image according to the first scene image and the second scene image includes: Extracting a first edge image from the first scene image, and extracting a second edge image from the second scene image; Warping the first edge image in units of pixel blocks, and outputting the warping result of the first edge image to a first warped edge image; warping the second edge image in units of pixel blocks, and outputting the warping result of the second edge image to a second warped edge image; The target edge image is generated according to the first warped edge image and the second warped edge image based on the A channel label of the intermediate image.

12. The method according to claim 11, characterized in that The step of distorting the first edge image in units of pixel blocks and outputting the distortion result of the first edge image to a first distorted edge image includes: Determining, in the first warped edge image, pixel coordinates corresponding to pixels in the first edge image according to motion vectors corresponding to respective vertices of each mesh in the 2D mesh model; According to the texture coordinates corresponding to the pixel coordinates in the first warped edge image, pixel values ​​are collected from the first edge image to obtain the first warped edge image which stores the warping result of the first edge image.

13. The method according to claim 11, characterized in that The step of distorting the second edge image in units of pixel blocks and outputting the distortion result of the second edge image to a second distorted edge image includes: Determining, in the second warped edge image, pixel coordinates corresponding to pixels in the second edge image according to the inverse vector of the motion vector corresponding to each vertex of each mesh in the 2D mesh model; According to the texture coordinates corresponding to the pixel coordinates in the second distorted edge image, pixel values ​​are collected from the second edge image to obtain the second distorted edge image which stores the distortion result of the second edge image.

14. The method according to claim 11, characterized in that The first edge image, the second edge image, the first warped edge image, and the second warped edge image are single-channel images.

15. The method according to any one of claims 11 to 14, characterized in that The step of generating the target edge image based on the A channel mark of the intermediate image and the first warped edge image and the second warped edge image comprises: If the A channel mark of a pixel coordinate in the intermediate image is 0.5, the corresponding pixel coordinate in the target edge image is filled with the pixel data of the second edge image; or If the A channel mark of a pixel coordinate in the intermediate image is 1, the corresponding pixel coordinate in the target edge image is filled with the pixel data of the first edge image.

16. The method according to any one of claims 1 to 15, characterized in that The first rendering instruction stream is obtained before the second rendering instruction stream; After superimposing the user interface image on the scene prediction image to obtain a prediction frame, the method further includes: Displaying the first frame, the predicted frame and the second frame in sequence; The first frame is a real image frame generated according to the first rendering instruction stream, and the second frame is a real image frame generated according to the second rendering instruction stream.

17. A terminal device, characterized in that: It comprises a processor and a memory coupled to the processor; wherein the memory stores instructions, and the processor calls the instructions so that the terminal device executes the prediction frame generation method as described in any one of claims 1 to 16.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on a terminal device, the terminal device executes the prediction frame generation method according to any one of claims 1 to 16.

Images