Prediction frame generation method, terminal equipment and storage medium

Abstract

The embodiment of the invention provides a prediction frame generation method, terminal equipment and a storage medium, and relates to the technical field of terminals. In a game scene, terminal equipment can respectively acquire a 3D scene image and a UI image corresponding to each real frame from two game rendering instructions, acquire general information such as a depth image, a world matrix and a projection matrix of a game virtual camera and the like according to the game rendering instructions, and then perform pixel-by-pixel offset on the 3D scene image by using the information. And then filling a blank area in the 3D scene image by adopting an AI image restoration technology, and then superposing a UI image on the restored 3D scene image so as to generate a prediction frame. According to the scheme, the requirement for the operational capability is low, real-time operation can be carried out on terminal equipment with limited operational capability such as a mobile phone, and the finally generated prediction frame is closer to the original image quality of the game.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a prediction frame generation method, a terminal device and a storage medium. Background Art

[0002] At present, game scene rendering has increasingly higher requirements for picture smoothness and picture clarity, which relies on higher screen refresh rates and graphics rendering resolutions, posing severe challenges to terminal devices in terms of chip power consumption load.

[0003] In order to reduce the power consumption load of the chip, the related technologies have proposed a variety of frame prediction schemes based on the similarity of two frames of images rendered successively by the game, but these schemes have many problems. For example, one type of scheme is to perform semi-transparent blending of two frames of images with motion vectors as weights, but the predicted frames obtained by blending using this scheme are prone to image quality problems such as ghosting. For another example, another type of scheme is that a personal computer (PC) generates a complete predicted frame based on artificial intelligence (AI) technology, but this scheme has high requirements for computing power and bandwidth and cannot run in real time on terminal devices with limited computing power such as mobile phones. Summary of the invention

[0004] The present application provides a prediction frame generation method, a terminal device and a storage medium, which solve the technical problem of how to generate a prediction frame with a quality close to the original game image on a terminal device with limited computing power.

[0005] In order to achieve the above objectives, this application adopts the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a method for generating a prediction frame. The method may include:

[0007] Get the user interface image UI0, the first scene image Color0, the first depth image Depth0, the first projection matrix ProjectionMatrix0 and the first world matrix ViewMatrix0 from the first rendering instruction stream; get the second scene image Color1, the second depth image Depth1, the second projection matrix ProjectionMatrix1 and the second world matrix ViewMatrix1 from the second rendering instruction stream; based on the first depth image Depth0, the first projection matrix ProjectionMatrix0, the first world matrix ViewMatrix0, the second depth image Depth1, the second projection matrix ProjectionMatrix1 and the second world matrix ViewMatrix1 , determine the static pixels and dynamic pixels in the first scene image Color0; determine the motion vector of the static pixels according to the pixel coordinates of the static pixels and the reprojection coordinates after reprojecting the static pixels; determine the matching pixels of the dynamic pixels in the second scene image Color1, and determine the motion vector of the dynamic pixels according to the pixel coordinates of the dynamic pixels and the pixel coordinates of the matching pixels; move the static pixels based on the motion vector of the static pixels, and move the dynamic pixels based on the motion vector of the dynamic pixels to obtain the intermediate image Color_NewFrame; fill the blank area of ​​the intermediate image Color_NewFrame with pixels based on the AI ​​model to obtain the scene prediction image Pixel_Color; superimpose the user interface image UI0 on the scene prediction image Pixel_Color to obtain a predicted frame.

[0008] In the above scheme, by obtaining the scene image and UI image corresponding to each real frame from two game rendering instructions, as well as obtaining general information such as the depth image, the world matrix and projection matrix of the game virtual camera, this information can be used to offset the static pixels and dynamic pixels of the 3D scene image pixel by pixel, thereby solving the physical distortion problem existing in the block-by-block offset scheme and the ghosting problem existing in the image mixing scheme. In addition, the AI ​​model is used to fill the blank area after the pixel-by-pixel offset, so that the final generated predicted frame image is closer to the native picture quality of the game. Different from the scheme that generates the entire frame based on the PC-based AI model, the AI ​​model of this application only repairs the blank area of ​​the image, has low requirements on computing power, and can run in real time on terminal devices with limited computing power such as mobile phones.

[0009] In a possible implementation, the first rendering instruction stream and the second rendering instruction stream are game rendering instruction streams. The game rendering instruction stream may include a variety of drawing instructions, such as scene drawing instructions, UI drawing instructions, and image display instructions. Among them, the scene drawing instructions can be used to draw 3D scene images such as scenery, characters, special effects and skills in the game, the UI drawing instructions can be used to draw UI images such as control buttons, small maps and floating text, and the image display instructions are used to put the drawn image data into the specified location of the operating system to complete the frame display.

[0010] In a possible implementation, a real game frame may be composed of a 3D scene image and a UI image, etc. Among them, the 3D scene image is also called a color image. For example, the 3D scene image may include in-game scenery, game characters, background objects, special effects and skills, etc., and the UI image may include control buttons, a small map, floating text, a user avatar, an energy pack control and a health bar, etc. The UI image is located on the upper layer of the 3D scene image. It should be understood that both the game characters in the 3D scene image and the rendering control buttons in the UI image are objects in the game frame.

[0011] In a possible implementation, the pixel coordinates of the 3D scene image and the pixel coordinates of the UI image are in a one-to-one correspondence, and the UI image is located on the upper layer of the 3D scene image. If the pixel coordinates of an object in the UI image are the same as those of an object in the 3D scene image, the object in the UI image may cause visual occlusion to the object in the 3D scene image. It should be understood that by separating the 3D scene image and the UI image from the rendering instruction stream, it is convenient to perform scene prediction on the 3D scene image later.

[0012] In a possible implementation, based on the first depth image Depth0, the first projection matrix ProjectionMatrix0, the first world matrix ViewMatrix0, the second depth image Depth1, the second projection matrix ProjectionMatrix1, and the second world matrix ViewMatrix1, determining static pixels and dynamic pixels in the first scene image Color0 includes:

[0013] The following steps are performed for each pixel in the first scene image Color0: based on the first depth image Depth0, the first projection matrix ProjectionMatrix0 and the first world matrix ViewMatrix0, the first world space coordinates corresponding to the first pixel coordinates of the first scene image Color0 are determined; based on the second projection matrix ProjectionMatrix1 and the second world matrix ViewMatrix1, the first world space coordinates are reprojected to the pixel coordinate system to obtain the second pixel coordinates; based on the second depth image Depth1, the second projection matrix ProjectionMatrix1 and the second world matrix ViewMatrix1, the second world space coordinates corresponding to the second pixel coordinates are determined. For any pixel in the first scene image Color0: if the first world space coordinates and the second world space coordinates of a pixel are the same, then the pixel is marked as a static pixel; or, if the first world space coordinates and the second world space coordinates of a pixel are different, then the pixel is marked as a dynamic pixel.

[0014] It should be understood that for the same pixel point: if the depth in the depth image Depth0 is different from the depth in the depth image Depth1, then after the coordinate transformation is performed according to the above steps, the pixel point will correspond to two different positions in the world coordinate system O-XYZ, representing that the position of the pixel at the pixel coordinate (w0, h0) on the first scene image Color0 in the world coordinate system has changed, and the pixel is a pixel of a dynamic object (such as a running person), so the pixel can be marked as Mask = 1. If the depth in the depth image Depth0 is the same as the depth in the depth image Depth1, then after the coordinate transformation is performed according to the above steps, the pixel point will correspond to the same position in the world coordinate system O-XYZ, representing that the position of the pixel at the pixel coordinate (w0, h0) on the 3D scene image Color0 in the world coordinate system has not changed, and the pixel is a pixel of a static object (such as a stationary house), so the pixel can be marked as Mask = 0. After marking all pixels in the first scene image Color0, the Mask of all pixels can be output to a mask image, thereby generating a mask image for marking dynamic and static pixels. The pixel coordinates of the mask image are in one-to-one correspondence with the pixel coordinates of the first scene image Color0.

[0015] In a possible implementation, determining the motion vector of the static pixel according to the pixel coordinates of the static pixel and the reprojected coordinates after reprojecting the static pixel can include: determining the world space coordinates corresponding to the static pixel coordinates (w0, h0) of the first scene image Color0 based on the first depth image Depth0, the first projection matrix ProjectionMatrix0 and the first world matrix ViewMatrix0; reprojecting the world space coordinates corresponding to the static pixel coordinates (w0, h0) to the pixel coordinate system based on the second projection matrix ProjectionMatrix1 and the second world matrix ViewMatrix1 to obtain the reprojected coordinates (w1, h1); and determining [(w1-w0)*0.5, (h1-h0)*0.5] as the motion vector of the static pixel.

[0016] It should be understood that when a pixel is a static pixel, the motion vector of the static pixel can be calculated by reprojection based on the characteristics of the static pixel.

[0017] In a possible implementation, matching pixels of dynamic pixels are determined in the second scene image Color1, and the motion vector of the dynamic pixels is determined according to the pixel coordinates of the dynamic pixels and the pixel coordinates of the matching pixels. This may include: obtaining the brightness and feature vector corresponding to the dynamic pixel coordinates (w0', h0') of the first scene image Color0; searching for matching pixels (w1', h1') in the second scene image Color1 according to the brightness and feature vector corresponding to the dynamic pixel coordinates (w0', h0'); and determining [(w1'-w0')*0.5, (h1'-h0')*0.5] as the motion vector of the dynamic pixel.

[0018] Exemplarily, the above matching pixels may satisfy the following conditions:

[0019] Condition 1: The absolute value of the first difference is smaller than the absolute value of the second difference. The first difference is the difference between the brightness corresponding to the dynamic pixel coordinates (w0', h0') and the brightness corresponding to the matching pixel (w1', h1'). The second difference is the brightness corresponding to the dynamic pixel coordinates (w0', h0') and the brightness corresponding to other pixel coordinates of the second scene image Color1.

[0020] Condition 2: The first feature distance is less than the second feature distance. The first feature distance is the feature distance between the feature vector corresponding to the dynamic pixel coordinate (w0', h0') and the feature vector corresponding to the matching pixel (w1', h1'). The second feature distance is the feature distance between the feature vector corresponding to the dynamic pixel coordinate (w0', h0') and the feature vector corresponding to other pixel coordinates of the second scene image Color1.

[0021] It should be understood that when a pixel is a dynamic pixel, based on the characteristics of the dynamic pixel, the motion vector of the dynamic pixel can be calculated by searching for the best matching pixel.

[0022] In a possible implementation, static pixels are moved based on motion vectors of the static pixels, and dynamic pixels are moved based on motion vectors of the dynamic pixels to obtain an intermediate image Color_NewFrame, including: creating an intermediate image Color_NewFrame; based on the motion vector of each static pixel in the first scene image Color0, moving each static pixel one by one to the intermediate image Color_NewFrame; based on the motion vector of each dynamic pixel in the first scene image Color0, moving each dynamic pixel one by one to the intermediate image Color_NewFrame.

[0023] Specifically, after the intermediate image Color_NewFrame is created, the RGB channels of the intermediate image Color_NewFrame may be assigned a value of 0.0, and the A channel of the intermediate image Color_NewFrame may be assigned a value of 0.5.

[0024] Accordingly, based on the motion vector of each static pixel in the first scene image Color0, each static pixel is moved one by one to the intermediate image Color_NewFrame, including: based on the motion vector of each static pixel in the first scene image Color0, determining the pixel coordinates corresponding to each static pixel in the intermediate image Color_NewFrame; assigning the RGB channels of the pixel coordinates corresponding to each static pixel in the intermediate image Color_NewFrame to the first value, and assigning the A channel of the pixel coordinates corresponding to each static pixel in the intermediate image Color_NewFrame to 1.0. The first value is the RGB channel pixel value of each static pixel in the first scene image Color0.

[0025] Accordingly, based on the motion vector of each dynamic pixel in the first scene image Color0, each dynamic pixel is moved one by one to the intermediate image Color_NewFrame, including: based on the motion vector of each dynamic pixel in the first scene image Color0, determining the pixel coordinates corresponding to each dynamic pixel in the intermediate image Color_NewFrame; assigning the RGB channels of the pixel coordinates corresponding to each dynamic pixel in the intermediate image Color_NewFrame to the second value, and assigning the A channel of the pixel coordinates corresponding to each dynamic pixel in the intermediate image Color_NewFrame to 1.0. The second value is the RGB channel pixel value of each dynamic pixel in the first scene image Color0.

[0026] In a possible implementation, the method may further include: if multiple pixels move to the same pixel coordinate of the intermediate image Color_NewFrame, then according to the depth of each pixel in the first depth image Depth0, convert the same pixel coordinate to the camera coordinate system to obtain the camera coordinate of each pixel in the multiple pixels; according to the camera coordinate of each pixel in the multiple pixels, calculate the distance from the camera coordinate of each pixel in the multiple pixels to the camera origin; according to the RGB channel pixel value of a pixel corresponding to the minimum distance in the multiple distances, assign the RGB channel of the same pixel coordinate. It should be understood that after completing the offset of all pixels in the first scene image Color0, multiple pixels in the first scene image Color0 may move to the same position of the intermediate image Color_NewFrame, resulting in a pixel occlusion problem at the position, so the pixel point corresponding to the position closest to the camera origin can be calculated as the pixel point finally displayed in the intermediate image Color_NewFrame.

[0027] In a possible implementation, filling pixels in the blank area of ​​the intermediate image Color_NewFrame based on the AI ​​model to obtain the scene prediction image Pixel_Color may include: generating a target edge image edge_map according to the first scene image Color0 and the second scene image Color1; inputting the target edge image edge_map and the intermediate image Color_NewFrame into the AI ​​model to generate the scene prediction image Pixel_Color. The scene prediction image Pixel_Color is an image obtained by the AI ​​model using the target edge image edge_map to fill pixels in the blank area of ​​the intermediate image Color_NewFrame.

[0028] It should be understood that traditional AI models are usually loaded on devices with high computing power such as PCs. By inputting the real frame into the AI ​​model, a complete predicted frame can be obtained. However, the algorithm of the traditional AI model is complex, and the demand for computing power and bandwidth is high, and it cannot be run in real time on terminal devices with limited computing power such as mobile phones. Different from the traditional AI model, the AI ​​model provided in this application is only used to repair the blank areas of the image, and has low requirements for computing power. It can be run in real time on terminal devices with limited computing power such as mobile phones. In addition, by predicting the data missing area, the generated 3D scene prediction image Pixel_Color is closer to the result of the original image.

[0029] In a possible implementation, a target edge image edge_map is generated according to a first scene image Color0 and a second scene image Color1, including: extracting a first edge image edge0_warp from the first scene image Color0, and extracting a second edge image edge1_warp from the second scene image Color1; warping the first edge image edge0_warp in units of pixel blocks, and outputting the warping result of the first edge image edge0_warp to a first warped edge image; warping the second edge image edge1_warp in units of pixel blocks, and outputting the warping result of the second edge image edge1_warp to a second warped edge image; and generating a target edge image edge_map according to the first warped edge image and the second warped edge image based on an A channel mark of an intermediate image Color_NewFrame.

[0030] Specifically, the first edge image edge0_warp, the second edge image edge1_warp, the first warped edge image, and the second warped edge image are single-channel images.

[0031] Accordingly, warping the first edge image edge0_warp in units of pixel blocks and outputting the warping result of the first edge image edge0_warp to the first warped edge image may include: determining, in the first warped edge image, pixel coordinates corresponding to pixels in the first edge image edge0_warp according to motion vectors corresponding to vertices of each mesh in the 2D mesh model; and acquiring pixel values ​​from the first edge image edge0_warp according to texture coordinates corresponding to the pixel coordinates in the first warped edge image, to obtain a first warped edge image that has saved the warping result of the first edge image edge0_warp.

[0032] Accordingly, warping the second edge image edge1_warp in units of pixel blocks and outputting the warping result of the second edge image edge1_warp to the second warped edge image may include: determining, in the second warped edge image, pixel coordinates corresponding to pixels in the second edge image edge1_warp according to opposite vectors of motion vectors corresponding to respective vertices of each mesh in the 2D mesh model; and acquiring pixel values ​​from the second edge image edge1_warp according to texture coordinates corresponding to the pixel coordinates in the second warped edge image, to obtain a second warped edge image having saved the warping result of the second edge image edge1_warp.

[0033] Accordingly, based on the A channel mark of the intermediate image, the target edge image is generated according to the first warped edge image and the second warped edge image, including: if the A channel mark of a pixel coordinate in the intermediate image Color_NewFrame is 0.5, then the corresponding pixel coordinate of the target edge image edge_map is filled with the pixel data of the second edge image edge1_warp; or, if the A channel mark of a pixel coordinate in the intermediate image Color_NewFrame is 1, then the corresponding pixel coordinate of the target edge image edge_map is filled with the pixel data of the first edge image edge0_warp. It should be understood that if A=0.5 of a pixel coordinate in the intermediate image Color_NewFrame, the edge image edge1_warp can be used to fill the blank area. If A≠0.5 of a pixel coordinate in the intermediate image Color_NewFrame, the pixel data of the edge image edge0_warp can be used to fill, thereby generating a complete edge image, which is convenient for the later AI model to recognize the edge.

[0034] In a possible implementation, the first rendering instruction stream is obtained before the second rendering instruction stream. After the user interface image is superimposed on the scene prediction image Pixel_Color to obtain the predicted frame, the method may further include: displaying the first frame, the predicted frame, and the second frame in sequence. The first frame is a real image frame generated according to the first rendering instruction stream, and the second frame is a real image frame generated according to the second rendering instruction stream.

[0035] It should be understood that according to the first rendering instruction stream and the second rendering instruction, not only two real frames can be generated, but also a predicted frame can be inserted between the two real frames. Since the computing resource requirement for generating the predicted frame is less than the resource requirement for the game application to render the real frame, the resource consumption of the terminal device is reduced.

[0036] In a second aspect, the present application provides a device, the device comprising a unit for executing the method in the first aspect. The device may correspond to executing the prediction frame generation method described in the first aspect. For the relevant description of the units in the device, please refer to the description of the first aspect, which will not be repeated here for the sake of brevity.

[0037] In a third aspect, the present application provides a terminal device, the terminal device comprising a memory and one or more processors. The memory is used to store computer program code, and the computer program code comprises computer instructions. When the computer instructions are called by the processor, the terminal device executes the prediction frame generation method provided in any one of the first aspects.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium. The computer-readable storage medium includes computer instructions. When the computer instructions are executed on a terminal device, the terminal device executes the prediction frame generation method provided in the first aspect and any possible implementation thereof.

[0039] In a fifth aspect, the present application provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the prediction frame generation method provided in the first aspect and any possible implementation thereof.

[0040] In a sixth aspect, the present application provides a chip system. The chip system includes one or more interface circuits and one or more processors. The interface circuit and the processor are interconnected by a line. The chip system can be applied to a terminal device including a communication module and a memory. The interface circuit is used to receive a signal from the memory of the terminal device and send the received signal to the processor, the signal including a computer instruction stored in the memory. When the processor calls the computer instruction, the terminal device can execute the prediction frame generation method provided in the first aspect and any possible implementation thereof.

[0041] It can be understood that the beneficial effects that can be achieved by the above-mentioned apparatus of the second aspect, the terminal device of the third aspect, the computer-readable storage medium of the fourth aspect, the computer program product of the fifth aspect, and the chip system of the sixth aspect can be referred to as the beneficial effects in the first aspect and any possible implementation method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of the hardware structure of a mobile phone provided in an embodiment of the present application;

[0043] Figure 2 A schematic diagram of the architecture of a terminal device provided in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of a rendering and display process of a game video frame provided in an embodiment of the present application;

[0045] Figure 4A A flowchart of a method for generating a prediction frame provided in an embodiment of the present application;

[0046] Figure 4B A schematic diagram of a flow chart of another prediction frame generation method provided in an embodiment of the present application;

[0047] Figure 4C A flowchart of another method for generating a prediction frame provided in an embodiment of the present application;

[0048] Figure 5AA schematic diagram of changing pixel coordinates based on the movement of a virtual object in a game scene provided by an embodiment of the present application;

[0049] Figure 5B A schematic diagram of changing pixel coordinates based on the movement of a virtual camera in a game scene provided by an embodiment of the present application;

[0050] Figure 6 A schematic diagram of separating a 3D scene image and a UI image from a rendering instruction stream provided in an embodiment of the present application;

[0051] Figure 7 Schematic diagrams of various coordinate systems provided in embodiments of the present application;

[0052] Figure 8A-8C A schematic diagram of determining static pixels and dynamic pixels provided in an embodiment of the present application;

[0053] Fig. 9 A schematic diagram of generating a mask image provided by an embodiment of the present application;

[0054] Fig.10 A schematic diagram of pixel-by-pixel shifting based on motion vectors provided in an embodiment of the present application;

[0055] Fig.11 A schematic diagram of moving multiple pixels to the same position of an intermediate image provided by an embodiment of the present application;

[0056] Fig.12 A schematic diagram of occlusion of multiple pixels at the same position of an intermediate image provided by an embodiment of the present application;

[0057] Fig.13 A schematic diagram of a 2D grid model provided in an embodiment of the present application;

[0058] Figure 14A-Figure 14C A schematic diagram of obtaining an edge image corresponding to an intermediate image provided in an embodiment of the present application;

[0059] Fig.15 A schematic diagram of repairing a blank area of ​​an image based on an AI model provided in an embodiment of the present application;

[0060] Fig.16 A schematic diagram of superimposing a UI image onto a 3D scene prediction image provided in an embodiment of the present application;

[0061] Fig.17 A schematic diagram of a terminal device playing a game screen frame by frame provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0063] In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B. In the description of this application, "and / or" is only a way to describe the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0064] The terms "first" and "second" in the specification and claims of the present application are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects. For example, the first image and the second image are used to distinguish different images, rather than to describe a specific order of images. In the embodiments of the present application, "plurality" refers to two or more.

[0065] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the phrases "in some embodiments" etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0066] To facilitate understanding of this application, some of the terms used in this application are explained below.

[0067] 1. Frame rate refers to the number of frames to be displayed generated per second by the application when it is running. The terminal device usually needs to go through the processes of drawing, rendering and synthesis to generate the frames to be displayed. The unit of frame rate is frames per second (fps). Among them, frame drawing refers to the drawing of the picture of the display interface, frame rendering is to color the drawn picture or add 3D effects, etc., and frame synthesis is the process of synthesizing at least one rendered view into the interface to be displayed.

[0068] 2. Refresh rate, also known as screen refresh rate, refers to the number of times the screen refreshes per second. Each refresh represents the display of one frame. Usually, the refresh rate depends on the fixed parameters of the hardware. The more powerful the hardware function of the terminal device, the higher the refresh rate it supports, and the smoother the screen display. The unit of refresh rate is Hertz (Hz).

[0069] 3. Inter-frame prediction technology: It refers to using the correlation of the frame time domain to use the pixels of the adjacent real frame to predict the pixels of the intermediate frame, so as to effectively remove the temporal redundancy. Generally, in the field of video playback, inter-frame prediction technology is relatively mature, for example, weighted mixing of two video frames, or using optical flow to calculate motion vectors for pixel blurring.

[0070] The present application mainly relates to inter-frame prediction in the field of games. Specifically, when a terminal device runs a game application, the frame rate of the game application can reach 60fps or even 100fps, which occupies a large amount of system resources, causing the terminal device to have problems such as fast power consumption, battery heating or system freeze. Studies have shown that the resource consumption of using inter-frame prediction technology to generate predicted frames is less than the resource consumption of game applications to render real frames. Therefore, reducing the number of real frames generated by game applications and using inter-frame prediction technology to generate predicted frames can ensure the total amount of game frames and reduce the power consumption load of the chip.

[0071] However, the clarity and sharpness of images rendered by games are significantly higher than those of videos. In game scenarios, the predicted frames must not only approach or reach the clarity and sharpness of the real frames, but also ensure the correctness of the picture content and prevent deformation of objects in the image. Therefore, the traditional video inter-frame prediction technology used in the field of video playback cannot meet the needs of generating game prediction frames.

[0072] Generally, there are several categories of frame prediction technologies for game scenes: one category of frame prediction technology is to perform semi-transparent mixing of two consecutive frames of images with motion vectors as weights, but the predicted frames obtained by mixing using this technology are prone to image quality problems such as ghosting; another category of frame prediction technology is to load an AI model on a PC and input the real frame into the AI ​​model to obtain a complete predicted frame, but the algorithm of the AI ​​model is complex, and the demand for computing power and bandwidth is high, and it cannot be run in real time on terminal devices with limited computing power such as mobile phones; another category of frame prediction technology is to use block-based motion compensation technology, first find the best estimated block in the coding block, so that the residual between the predicted block and the current block is as small as possible, and calculate the motion vector (motion vector, mv) of the current block, and then obtain the estimated value of the current frame based on the motion vector of the current block, but the block-by-block offset is likely to cause distortion of objects in the picture. Therefore, a concise and efficient algorithm design and a robust framework structure have become necessary conditions for the application of frame prediction technology to terminal devices with limited computing power such as mobile phones.

[0073] In view of the above problems, the present application provides a method for generating a predicted frame. In this method, the terminal device can obtain the three-dimensional (3D) scene image and user interface (UI) image corresponding to each real frame from two game rendering instructions, as well as obtain the depth image, the world matrix (worldmatrix) and projection matrix (projection matrix) of the game virtual camera and other general information, and then use this information to offset the 3D scene image pixel by pixel, thereby solving the physical distortion problem existing in the block-by-block offset scheme and the ghosting problem existing in the image mixing scheme; after that, the terminal device can fill the blank area after pixel-by-pixel offset based on the AI ​​model, so that the predicted frame image finally generated is closer to the original picture quality of the game. It should be understood that, unlike the scheme of generating the entire frame based on the PC-based AI model, the AI ​​model of the present application only repairs the blank area of ​​the image, has low requirements on computing power, and can run in real time on terminal devices with limited computing power such as mobile phones.

[0074] In some embodiments, the terminal device is also referred to as a terminal or user equipment (UE). For example, the terminal device may be a mobile phone, a smart screen, a smart TV, a tablet computer (Pad), a wearable device, a virtual reality (VR) device, an augmented reality (AR) device, and other terminal devices with limited computing power. It should be understood that the prediction frame generation method provided in the present application can also be applied to terminal devices with sufficient computing power such as PCs. The embodiments of the present application do not impose any restrictions on the specific type of terminal device. Regardless of the type of terminal device to which the prediction frame generation method provided in the present application is applied, the amount of computing resources required to generate the prediction frame is less than the amount of resources required for the game application to render the real frame, thereby reducing the resource consumption of the terminal device.

[0075] In order to better understand the embodiments of the present application, the structure of the terminal device of the embodiments of the present application is introduced below.

[0076] Take the terminal device as an example, which is a mobile phone. Figure 1 A schematic diagram of the hardware structure of a mobile phone 100 provided in an embodiment of the present application.

[0077] like Figure 1 As shown, the mobile phone 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a SIM card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, and a bone conduction sensor 180M, etc.

[0078] The processor 110 may include one or more processing units. For example, the processor 110 may include a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), an image signal processor (ISP), a neural-network processing unit (NPU), a video codec, a modem, and / or a baseband processor.

[0079] The CPU is the final execution unit for information processing and program execution. Its main tasks include processing instructions, executing operations, controlling time, and processing data. The CPU may include a controller, an arithmetic unit, a cache memory, and a bus for connecting these components. The controller is the nerve center and command center of the mobile phone 100. The controller can generate an operation control signal based on the instruction opcode and timing signal to complete the control of fetching and executing instructions. The arithmetic unit refers to a component that performs various arithmetic and logical operations. The cache memory can store instructions or data that the processor 110 has just used or circulated.

[0080] GPU, also known as display core, visual processor or display chip, is a microprocessor for graphics and image processing. GPU connects display screen 194, AP and CPU. GPU can be used to perform complex mathematical and geometric calculations, floating-point operations, parallel calculations, and can also be used for graphics rendering. GPU reduces the graphics card's dependence on the CPU, especially when processing 3D graphics. GPU uses core technologies such as cubic environment material mapping and vertex blending, so GPU performs some of the work originally done by the CPU.

[0081] NPU is a neural network (NN) computing processor, which can quickly process input information and continuously self-learn by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain. NPU can realize applications such as intelligent cognition of the mobile phone 100.

[0082] In the embodiment of the present application, the prediction frame generation method is mainly performed by the GPU. For example, the GPU can obtain the 3D scene image and UI image corresponding to each real frame from two game rendering instructions, as well as general information such as the depth image, the world matrix and projection matrix of the game virtual camera, etc., and use this information to generate a mask image marking dynamic and static pixels, and calculate the motion vector of each pixel (pixel) according to the mask image, and then offset the 3D scene image pixel by pixel based on the motion vector, and then the GPU calls the neural network model of the NPU to perform image repair on the blank area after the pixel-by-pixel offset, and finally generates a predicted frame image. The specific description of the following embodiment can be referred to, which will not be repeated here.

[0083] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the mobile phone 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application (application, APP) required for at least one function, such as a game application, etc. The data storage area can store configuration files of each APP, as well as data created during the use of the mobile phone 100.

[0084] The mobile phone 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the AP, etc. For example, the background sound of the game, the real person conversation, etc. can be played.

[0085] The display screen 194 includes a signal processing module and a display panel, and is used to display images, videos, etc., such as displaying a game interface.

[0086] It is understandable that the hardware structure illustrated in the embodiment of the present application does not constitute a specific limitation on the mobile phone 100. In other embodiments, the mobile phone 100 may include more or fewer components than shown, or combine some components, or separate some components, or arrange the components differently. Figure 1 The components shown may be implemented in hardware, software or a combination of software and hardware.

[0087] The software system of the terminal device can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, or a micro-service architecture. The embodiment of the present invention takes the Android system of a layered architecture as an example to exemplify the software architecture of the terminal device.

[0088] Figure 2 The schematic diagram of the architecture of the terminal device provided in the embodiment of the present application is shown. Figure 2As shown, the terminal device can adopt a layered architecture to divide the software into several layers, each with clear roles and division of labor. The layers communicate through software interfaces. In some embodiments, the software layers of the software structure are divided from top to bottom into: application (APP) layer, application framework (FW) layer, system library (FWK LIB) and kernel layer. The above software architecture runs on the hardware layer, which may include GPU, NPU and display screen, etc.

[0089] The application layer, also called the application layer, can include a series of application packages. For example, the application layer can include game applications, camera applications, calendar applications, phone applications, and settings applications. When these application packages are run, they can access various service modules provided by the application framework layer through the application programming interface (API) and execute corresponding intelligent services.

[0090] The application framework layer provides API and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 2 As shown, the application framework layer may include a graphics manager, a video codec, a window manager, a resource manager, and a notification manager. Among them, the graphics manager is responsible for calling the interface in the graphics library to complete drawing, rendering, and synthesizing the interface, the video codec is responsible for video encoding and decoding, the window manager manages all windows in the system, the resource manager is responsible for system resources, and the notification manager is responsible for managing the prompt messages in the status bar at the top of the mobile phone.

[0091] The system library can include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (e.g. OpenGL ES), two-dimensional (2D) graphics engine (e.g. SGL), etc. The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The three-dimensional graphics processing library is used to implement three-dimensional graphics drawing, image rendering, synthesis and layer processing, etc. The 2D graphics engine is a drawing engine for 2D drawing.

[0092] In the system library, Android Runtime includes the core library and the virtual machine. Android Runtime is responsible for the scheduling and management of the Android system. The core library consists of two parts: one is the function that the Java language needs to call, and the other is the core library of Android. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection and other functions.

[0093] The kernel layer is the layer between hardware and software and belongs to the bottom layer of the Android system. The kernel layer can include various driver interfaces, such as GPU driver, NPU driver, CPU driver, and display driver. The kernel layer can also include power management, processes / threads, etc.

[0094] It should be noted that although the embodiments of the present application are described using the Android system as an example, its basic principles are also applicable to terminal devices based on operating systems such as iOS or Windows.

[0095] Figure 3 The rendering and display process of the game video frame provided by the embodiment of the present application is shown.

[0096] The game application runs on the operating system of the terminal device. In the game scene, the game application modifies the function pointer of the rendering API to generate the game's rendering instruction stream, such as hooking the game's rendering instruction stream. The game's rendering instruction stream is passed to the GPU through the graphics manager and GPU driver in turn. The GPU obtains the function through the rendering instruction and identifies the game's rendering instruction stream based on the characteristics of the game's rendering instruction.

[0097] The rendering instruction stream is an instruction stream composed of multiple drawing instructions (rendering instruction sequence). Generally, a rendering instruction stream is used to draw a frame of image. Specifically, the GPU can execute each drawing instruction in the rendering instruction stream to realize the drawing of each object in a frame of image. The set of objects generated according to the rendering instruction stream can form a frame of image. A frame of image is a frame of static image displayed by the game application. Each frame of static image can be composed of 3D scene image and UI image, etc., wherein the 3D scene image is also called color image. For example, the 3D scene image may include in-game scenery, game characters, background objects, special effects and skills, etc., and the UI image may include control buttons (such as running buttons and skill buttons, etc.), small maps, floating text, user avatars, energy pack controls and blood bars, etc., and the UI image is located on the upper layer of the 3D scene image. It should be understood that both the game characters in the 3D scene image and the rendering control buttons in the UI image belong to the objects in the game frame. In addition, through the rendering instruction stream, the world matrix and projection matrix of the virtual camera corresponding to a frame of image, as well as general information such as the depth image corresponding to a frame of image can also be obtained.

[0098] The game frames in the embodiment of the present application can be divided into two types:

[0099] One type is a real frame generated based on a rendering instruction stream. In a game scenario, the operating system sends rendering instruction streams to the GPU in sequence. The GPU generates a real frame corresponding to each rendering instruction stream in sequence according to the order in which these rendering instruction streams are received. One rendering instruction stream is used to draw one real frame.

[0100] The other type is a predicted frame generated based on two rendering instruction streams obtained successively. "Two rendering instruction streams obtained successively" means that rendering instruction stream 1 is obtained first, and then rendering instruction stream 2 is obtained. After obtaining rendering instruction stream 1 and before obtaining rendering instruction stream 2, the GPU has not obtained other rendering instruction streams. That is, the real frame N-1 obtained by rendering instruction stream 1 and the real frame N+1 obtained by rendering instruction stream 2 are two adjacent real frames. Specifically, the GPU can obtain two rendering instruction streams successively, and separate the 3D scene image and UI image corresponding to a real frame from each rendering instruction stream (for example, the scene image Color0 and the user interface image UI0 corresponding to the real frame N-1 are separated from the rendering instruction stream 1, and the scene image Color1 and the user interface image UI1 corresponding to the real frame N+1 are separated from the rendering instruction stream 2), and obtain the depth image corresponding to each real frame, the world matrix and the projection matrix of the game virtual camera and other general information. The GPU can then use this information to generate a mask image that marks the moving and static pixels, calculate the motion vector of each pixel based on the mask image, and then perform a pixel-by-pixel offset on the 3D scene image corresponding to a certain real frame based on the motion vector. Afterwards, the GPU can call the NPU's neural network model to perform image repair on the blank area after the pixel-by-pixel offset, and superimpose the repaired 3D scene image with the UI image corresponding to the real frame to generate the final predicted frame N.

[0101] Continue to refer to Figure 3 After the real frame N-1 obtained by the above rendering instruction stream 1, the real frame N+1 obtained by the above rendering instruction stream 2, and the predicted frame N by the rendering instruction stream 1 and the rendering instruction stream 2, the GPU can send these frames to the queue to be displayed in the order of real frame N-1, predicted frame N and real frame N+1. Then, the display screen plays the real frame N-1, predicted frame N and real frame N+1 in sequence.

[0102] Above Figure 3 Only the process of generating a predicted frame using two rendering instruction streams is given. In actual game scenarios, the number of rendering instruction streams may be greater than two, and any two rendering instruction streams obtained successively can refer to the above process to generate predicted frames. It should be understood that since the resource consumption of using inter-frame prediction technology to generate predicted frames is less than the resource consumption of the game application rendering real frames, by reducing the number of real frames generated by the game application and using inter-frame prediction technology to generate predicted frames, not only the total amount of game frames can be guaranteed, but also the power consumption load of the chip can be reduced.

[0103] Combine the following FIG. 4A to FIG. 17 The prediction frame generation method provided in the embodiment of the present application is illustrated by way of example.

[0104] Figure 4A-4CSchematic diagrams showing three prediction frame generation methods. The difference between these three figures is that: Figure 4A The specific process of this method is shown. Figure 4B The data flow relationship between the various images involved in this method is shown. Figure 4C The specific process of the method and the data flow relationship between each image are shown. Figure 4A-4C The execution subject in the predicted frame generation method shown can be a terminal device, a GPU in the terminal device, or other software or hardware functional modules in the terminal device. The following example uses a GPU as the execution subject of the predicted frame generation method, which does not limit the present application.

[0105] For example, in a game scenario, the GPU can continuously obtain rendering instruction streams from the game application. A rendering instruction stream indicates the generation of a real frame. Whenever a rendering instruction stream is obtained, such as obtaining rendering instruction stream 1 or obtaining rendering instruction stream 2, the GPU can perform the following steps 1 and 2 for this rendering instruction stream.

[0106] Step 1: The GPU separates the 3D scene image and UI image corresponding to a real frame from the rendering instruction stream.

[0107] The 3D scene image mentioned above refers to a color image that does not contain UI pixels but contains a 3D scene, that is, an RGB image.

[0108] For example, the above-mentioned rendering instruction stream may be a Hook game rendering instruction stream. The GPU may identify the various instructions contained in the Hook game rendering instruction stream through the rendering instruction acquisition function according to the characteristics of the game's rendering instruction stream. The Hook game rendering instruction stream generally includes a variety of drawing instructions, such as scene drawing instructions, UI drawing instructions, and image display instructions. Among them, the scene drawing instructions can be used to draw 3D scene images such as scenery, characters, special effects, and skills in the game, the UI drawing instructions can be used to draw UI images such as control buttons, small maps, and floating text, and the image display instructions are used to put the drawn image data into the specified location of the operating system (such as framebuffer) to complete the frame display.

[0109] The pixel coordinates of the 3D scene image and the pixel coordinates of the UI image are in a one-to-one correspondence, and the UI image is located on the upper layer of the 3D scene image. If the pixel coordinates of an object in the UI image are the same as those of an object in the 3D scene image, the object in the UI image may cause visual occlusion of the object in the 3D scene image. The object in the 3D scene image may change its pixel coordinates based on two reasons: one reason is that the position of objects such as characters, vehicles or birds in the game scene moves, such as Figure 5AAs shown, in the world coordinate system O-XYZ corresponding to the game scene, the initial position of the virtual character is coordinate position 1. When the virtual character runs, it moves to coordinate position 2. In this case, based on the running of the virtual character, the position of the virtual character in the 3D scene image of the real frame N-1 and the 3D scene image of the real frame N+1 moves; another reason is that the virtual camera in the game scene shakes, that is, the perspective of the virtual character corresponding to the game player changes, such as Figure 5B As shown, in the world coordinate system O-XYZ corresponding to the game scene, the position of the virtual character is always located at coordinate position 1 of the world coordinate system O-XYZ, but the virtual camera moves from position P1 to position P2, and the direction of the virtual camera changes. When the position and direction of the virtual camera change, the perspective of the virtual character corresponding to the game player also changes. In this case, the position based on the virtual character in the 3D scene image of the real frame N-1 and the 3D scene image of the real frame N+1 moves.

[0110] When the pixel coordinates of an object in the UI image are the same as those of an object in the 3D scene image, no matter what the above reasons are for changing the pixel coordinates, if the object in the 3D scene image is moved according to steps 3 to 5 below, the object in the UI image will move with the object in the 3D scene image, but in fact the object in the UI image may not move, or the motion vectors of the object in the UI image and the object in the 3D scene image are different, which will cause serious image quality problems in the final predicted frame. Therefore, the GPU needs to separate the 3D scene image and UI image corresponding to a real frame from a rendering instruction stream.

[0111] Specifically, for each game rendering instruction in a rendering instruction stream, the GPU can perform the following operations: according to the characteristics of a game rendering instruction, identify whether it is a UI drawing instruction, such as a UI drawing instruction can include a resource binding instruction, a state setting instruction, and a drawing instruction; if it is a UI drawing instruction, point the output of these instructions to a newly created image, which can be used to save UI pixels, so the newly created image can be called a UI image; if it is a scene drawing instruction, change the original rendering generation result to a newly created image, which is used to save a 3D scene image that does not contain UI pixels. In this way, the GPU separates the UI image and the 3D scene image from the rendering instruction stream.

[0112] For example, Figure 6As shown, the GPU first obtains the rendering instruction stream 1 for instructing the generation of the real frame N-1, and separates the 3D scene image Color0 and the UI image UI0 of the real frame N-1 from the rendering instruction stream 1. If the UI image UI0 is superimposed on the 3D scene image Color0, a complete real frame N-1 is obtained. Then, the GPU can obtain the rendering instruction stream 2 for instructing the generation of the real frame N+1, and separates the 3D scene image Color1 and the UI image UI1 of the real frame N+1 from the rendering instruction stream 2. If the UI image UI1 is superimposed on the 3D scene image Color1, a complete real frame N+1 is obtained.

[0113] Step 2: The GPU obtains the depth image, the world matrix and the projection matrix of the virtual camera corresponding to a real frame according to the rendering instruction stream.

[0114] The GPU can identify the rendering instruction sequence for model rendering based on the characteristics of the game rendering instructions, and determine the glBindBuffer instruction located before the model rendering drawcall instruction. Among them, the glBindBuffer instruction contains the relevant data of the virtual camera. The GPU can obtain the world matrix and projection matrix of the virtual camera from the buffer corresponding to the glBindBuffer instruction. In addition, the GPU can also obtain the depth image from the rendering instruction stream.

[0115] It should be noted that the 3D scene image obtained by the GPU from step 1 and the depth image obtained from step 2 have the same image content, for example, both images contain people, trees, and mountains. The difference between the two images is that the 3D scene image is a three-channel (RGB) color image; the depth image is a single-channel image, which is similar to a grayscale image. Each pixel value of the depth image can be used to represent the distance between the virtual camera and the object, and the depth image reflects the geometric shape of the visible surface of the object.

[0116] In order to facilitate the understanding of the world matrix and projection matrix of the above virtual camera, the following Figure 7 The pixel coordinate system, image coordinate system, normalized device coordinate system (NDC), camera coordinate system and world coordinate system involved in the embodiments of the present application are explained.

[0117] The pixel coordinate system o-wh and the image coordinate system o-xy are two-dimensional coordinate systems. Usually, images are stored in the form of arrays in terminal devices, and the value of each element (i.e., pixel) in the array represents the brightness of the image point. The origin of the pixel coordinate system o-wh is located in the upper left corner of the image, the w axis is horizontal to the right, and the h axis is vertically downward. Each pixel coordinate (w, h) is the number of columns and rows of a pixel in the array, so (w, h) is the coordinate of the pixel coordinate system o-wh in pixels. The origin of the image coordinate system o-xy is located at the center of the image (called the principal point of the image), the x axis is parallel to the w axis, and the y axis is parallel to the h axis. The coordinates in the pixel coordinate system o-wh and the image coordinate system o-xy can be converted to each other.

[0118] Assume that the pixel coordinates in the pixel coordinate system o-wh are (w0, h0), and the physical sizes of each pixel on the x-axis and y-axis are dx and dy. The relationship between the pixel coordinate system o-wh and the image coordinate system o-xy is as follows:

[0119]

[0120] The standardized device coordinate system o-x'y'z' is a three-dimensional coordinate system. In the standardized device coordinate system o-x'y'z', the values ​​of x', y' and z' are in the interval [-1,1], where z' represents the depth value of the image, and x' and y' are obtained by normalizing the coordinates (x, y) in the image coordinate system o-xy. Through the viewport transform, the standardized device coordinate system o-x'y'z' can be transformed into the image coordinate system o-xy or the pixel coordinate system o-wh. Taking a frame of 512*512 size image as an example, the pixel coordinates of the lower right corner point in the pixel coordinate system o-wh are (512, 512). After the coordinate transformation, the coordinates of this point in the standardized device coordinate system o-x'y'z' are (1, 1, z'), where z' represents the depth value of the coordinate.

[0121] The camera coordinate system O0-X0Y0Z0 is a three-dimensional coordinate system. In the camera coordinate system O0-X0Y0Z0, the virtual camera is a virtual camera set for the game scene, not a real camera. In the game scene, if there are multiple players roaming in the game world, since each player has a different position and sight direction in the game world, the frame displayed by the terminal device used by each player is also different. Suppose a coordinate system is created with a certain player as the origin, this coordinate system can be called the camera coordinate system or the observation space coordinate system. Figure 7, the origin O0 of the camera coordinate system is the optical center (projection center) of the virtual camera, the X0 axis and the Y0 axis are parallel to the x-axis and y-axis of the camera plane, the Z0 axis is the optical axis of the virtual camera, the Z0 axis is perpendicular to the camera plane, and oO0 is the focal length of the virtual camera. It should be understood that each player has a different position and sight direction in the game world, so the camera coordinate system O0-X0Y0Z0 established with the virtual camera corresponding to each player as the center will also be different.

[0122] The world coordinate system O-XYZ is a three-dimensional coordinate system introduced to describe the position of each virtual camera. In the game scene, virtual objects such as characters, tables, rooms, animals and vehicles have a specific position in the game. The position of the virtual object in the game world is defined as the coordinate in the world coordinate system, which is called the world space coordinate. For example, the world coordinate system O-XYZ can be a three-dimensional coordinate system established with the center of the earth as the origin. It should be understood that different players have different positions and sight directions in the world coordinate system O-XYZ, which makes the position and orientation of the virtual camera in the world coordinate system O-XYZ also different. The coordinates in the camera coordinate system O0-X0Y0Z0 can be converted into coordinates in the world coordinate system O-XYZ through translation and rotation.

[0123] The world matrix, also called the external parameter matrix, is used to convert the object coordinates in the world coordinate system O-XYZ into the camera coordinate system O0-X0Y0Z0. That is, when the perspective of the virtual camera is used as the origin of the game scene, the world matrix transforms all world space coordinates into observation coordinates relative to the position and direction of the virtual camera.

[0124] For example, the world matrix can be expressed as:

[0125]

[0126] Among them, (Rx, Ry, Rz) can represent the X-axis direction vector of the virtual camera, (Ux, Uy, Uz) can represent the Y-axis direction vector of the virtual camera, (Dx, Dy, Dz) represents the line of sight direction and Z-axis of the virtual camera. (Px, Py, Pz) can represent the world space coordinates of the virtual camera.

[0127] The projection matrix, also known as the intrinsic parameter matrix, is used to convert the object coordinates in the camera coordinate system O0-X0Y0Z0 into descriptions in the standardized device coordinate system o-x'y'z'. When rendering, a three-dimensional scene is usually projected onto the screen (i.e., the camera surface) to become a two-dimensional image. Specifically, all vertex data is first converted from the camera coordinate system O0-X0Y0Z0 to the clipping space, and then the converted clipping coordinates are transformed into the standardized device coordinate system o-x'y'z' by dividing by the w component in the clipping coordinate system. Among them, the camera coordinate system O0-X0Y0Z0 is a right-handed coordinate system, and the standardized device coordinate system o-x'y'z' is a left-handed coordinate system.

[0128] Exemplarily, assuming that the projection matrix is ​​a matrix of a general frustum, in which the coordinates of the four vertices of a conical section are represented as (r, t, -n), (r, b, -n), (f, b, -n), (f, t, -n), then the projection matrix can be expressed as:

[0129]

[0130] Assume that the GPU obtains rendering instruction stream 1 and rendering instruction stream 2 in sequence. Among them, rendering instruction stream 1 can be used to generate real frame N-1, and rendering instruction stream 2 can be used to generate real frame N+1. According to the above steps 1 and 2, the GPU can obtain the 3D scene image Color0, UI image UI0, depth image Depth0, virtual camera's world matrix ViewMatrix0 and virtual camera's projection matrix ProjectionMatrix0 corresponding to the real frame N-1, and the 3D scene image Color1, UI image UI1, depth image Depth1, virtual camera's world matrix ViewMatrix1 and virtual camera's projection matrix ProjectionMatrix1 corresponding to the real frame N+1. Then, the GPU can generate the predicted frame N according to these parameters according to the following steps 3 to 7.

[0131] Step 3: The GPU generates a mask image marking dynamic pixels and static pixels based on the depth image of the real frame N-1, the world matrix and projection matrix of the virtual camera, and the depth image of the real frame N+1, the world matrix and projection matrix of the virtual camera.

[0132] Exemplarily, the GPU may generate a mask image through the following steps (1) to (3).

[0133] (1) Based on the depth image Depth0 of the real frame N-1, the world matrix ViewMatrix0 and the projection matrix ProjectionMatrix0 of the virtual camera of the real frame N-1, calculate the world space coordinates (x0, y0, z0) corresponding to each pixel coordinate (w0, h0) of the 3D scene image Color0.

[0134] Specifically, assuming that the coordinates (w0, h0) of any pixel in the 3D scene image Color0 of the real frame N-1 and the depth image Depth0 of the real frame N-1 are known, the GPU can first convert the coordinates (w0, h0) of any pixel in the 3D scene image Color0 into the image coordinates (x0, y0) of the image coordinate system o0-x0y0, and then calculate the image coordinates (x0, y0) based on the depth image Depth0 of the real frame N-1. The coordinates (x', y', z') in the standardized device coordinate system o0-x'y'z' after normalization are obtained, and then the inverse matrix of the projection matrix ProjectionMatrix0 is used to obtain the camera coordinates in the camera coordinate system O0-X0Y0Z0, and then the inverse matrix of the world matrix ViewMatrix0 is used to obtain the world space coordinates (x0, y0, z0) in the world coordinate system O-XYZ. Fig. 8A As shown, for any pixel coordinate in the 3D scene image Color0, the world space coordinate (x0, y0, z0) corresponding to a pixel coordinate (w0, h0) can be obtained by referring to the above steps.

[0135] As an example, the conversion relationship from the coordinates (x', y', z') of the normalized device coordinate system o0-x'y'z' to the world space coordinates (x0, y0, z0) of the world coordinate system O-XYZ is as follows:

[0136]

[0137] Wherein, ViewMatrix0 represents the world matrix of the virtual camera of the real frame N-1, and ProjectionMatrix0 represents the projection matrix of the virtual camera of the real frame N-1. For the specific implementation of the world matrix ViewMatrix0 and the projection matrix ProjectionMatrix0, refer to the specific description of step 2 above, which will not be repeated here.

[0138] (2) For the world space coordinates (x0, y0, z0) corresponding to each pixel in the 3D scene image Color0 of the real frame N-1 calculated in step (1) above, the world matrix ViewMatrix1 and projection matrix ProjectionMatrix1 of the virtual camera of the real frame N+1 are further used to reproject them to the pixel coordinate system, thereby obtaining the reprojected pixel coordinates (w1, h1).

[0139] Specifically, after calculating the world space coordinates (x0, y0, z0) corresponding to each pixel in the 3D scene image Color0 of the real frame N-1 through step (1) above, the GPU can first use the virtual camera's world matrix ViewMatrix1 to calculate the coordinates in the camera coordinate system O1-X1Y1Z1, and then use the virtual camera's projection matrix ProjectionMatrix1 to calculate the coordinates (x', y', z') in the standardized device coordinate system o1-x'y'z', and then convert the coordinates (x', y', z') in the standardized device coordinate system o1-x'y'z' to the image coordinate system o1-x1y1, and then convert them to pixel coordinates (w1, h1) in the pixel coordinate system. Figure 8B As shown, after reprojecting the world space coordinates (x0, y0, z0) corresponding to each pixel calculated in step (1), the reprojected pixel coordinates (w1, h1) can be obtained.

[0140] As an example, the conversion relationship from the world space coordinates (x0, y0, z0) of the world coordinate system O-XYZ to the coordinates (x', y', z') of the standardized device coordinate system o-x'y'z' is as follows:

[0141]

[0142] Among them, ProjectionMatrix1 represents the projection matrix of the virtual camera of the real frame N+1, and ViewMatrix1 represents the world matrix of the virtual camera of the real frame N+1. For the specific implementation of the world matrix ViewMatrix1 and the projection matrix ProjectionMatrix1, refer to the specific description of step 2 above, which will not be repeated here.

[0143] (3) For each pixel coordinate (w1, h1) after reprojection calculated in step (2) above, the world space coordinate (x1, y1, z1) corresponding to each pixel coordinate (w1, h1) is further calculated based on the depth image Depth1 of the real frame N+1, the world matrix ViewMatrix1 of the virtual camera of the real frame N+1, and the projection matrix ProjectionMatrix1.

[0144] Specifically, after calculating the pixel coordinates (w1, h1) of each pixel after reprojection through step (2) above, the GPU can first convert the pixel coordinates (w1, h1) into the coordinates (x1, y1) of the image coordinate system o1-x1y1, and then calculate the image coordinates (x1, y1) based on the depth image Depth1 of the real frame N+1. After normalization, the coordinates (x', y', z') in the standardized device coordinate system o1-x'y'z' can be obtained. Then, the inverse matrix of the projection matrix ProjectionMatrix1 can be used to obtain the coordinates in the camera coordinate system O1-X1Y1Z1, and then the inverse matrix of the world matrix ViewMatrix1 can be used to obtain the world space coordinates (x1, y1, z1) in the world coordinate system O-XYZ. In this way, Figure 8C As shown, the world space coordinates (x1, y1, z1) corresponding to the reprojected pixel coordinates (w1, h1) are obtained.

[0145] As an example, the conversion relationship from the coordinates (x', y', z') of the standardized device coordinate system o1-x'y'z' to the world space coordinates (x1, y1, z1) of the world coordinate system O-XYZ is as follows:

[0146]

[0147] Wherein, ViewMatrix1 represents the world matrix of the virtual camera of the real frame N+1, and ProjectionMatrix1 represents the projection matrix of the virtual camera of the real frame N+1. For the specific implementation of the world matrix ViewMatrix1 and the projection matrix ProjectionMatrix1, refer to the specific description of step 2 above, which will not be repeated here.

[0148] It should be understood that for the same pixel: if the depth in the depth image Depth0 is different from the depth in the depth image Depth1, then Figure 8C As shown, after the coordinate transformation is performed according to the above steps (1) to (3), the pixel point will correspond to two different positions in the world coordinate system O-XYZ, that is, the world space coordinates (x0, y0, z0) and the world space coordinates (x1, y1, z1) are different. Alternatively, if the depth in the depth image Depth0 is the same as the depth in the depth image Depth1, then after the coordinate transformation is performed according to the above steps (1) to (3), the pixel point will correspond to the same position in the world coordinate system O-XYZ, that is, the world space coordinates (x0, y0, z0) and the world space coordinates (x1, y1, z1) are the same.

[0149] If the following three conditions x0=x1, y0=y1, z0=z1 are met at the same time, then the position of the pixel at the pixel coordinate (w0, h0) on the 3D scene image Color0 in the world coordinate system has not changed, and the pixel is a pixel of a static object (such as a stationary house), and the pixel is marked as Mask=0.

[0150] If any of the above three conditions is not met, such as x0≠x1, y0≠y1 or z0≠z1, then the position of the pixel at the pixel coordinate (w0,h0) on the 3D scene image Color0 changes in the world coordinate system, and the pixel is a pixel of a dynamic object (such as a running person), so the pixel is marked as Mask=1.

[0151] For example, Fig. 9 A schematic diagram of a mask image generated based on the 3D scene image Color0 is shown. It is assumed that the running person is a dynamic object, and other objects such as trees and mountains are static objects. A small square represents a pixel, and each object is composed of multiple pixels. For each pixel in the 3D scene image Color0, the GPU can determine whether a pixel is a pixel of a dynamic object or a pixel of a static object according to steps (1) to (3) above, and then mark the pixel of the dynamic object as Mask=1 and the pixel of the static object as Mask=0. After completing the marking of all pixels in the 3D scene image Color0, the GPU can output the Mask of all pixels to a mask image, thereby generating a mask image for marking dynamic and static pixels. It should be understood that the pixel coordinates of the mask image are in a one-to-one correspondence with the pixel coordinates of the 3D scene image Color0.

[0152] Step 4: The GPU calculates the motion vector of each pixel in the 3D scene image Color0 of the real frame N-1 obtained in step 1 based on the pixel labels of the mask image obtained in step 3.

[0153] For example, for each pixel in the mask image, the GPU may specifically perform the following operations:

[0154] If the pixel mark of a pixel in the mask image is 0, it means that the pixel in the 3D scene image Color0 is a pixel of a static object (referred to as a static pixel for short), and the GPU can calculate the motion vector of the static pixel by reprojecting the coordinates.

[0155] As an example, when referring to (1) to (2) in the above step 3 and obtaining the reprojection coordinates corresponding to each pixel coordinate, for the static pixels marked Mask=0 in the mask image, the GPU can filter out the reprojection coordinates (w1, h1) corresponding to each static pixel coordinate (w0, h0) from the reprojection coordinates of all pixels, and then use [(w1-w0)*0.5, (h1-h0)*0.5] to calculate the motion vector of the static pixel.

[0156] As another example, the GPU may not use the reprojection coordinates obtained from (1) to (2) in step 3 above, but may perform the reprojection process again for the static pixels marked Mask = 0 in the mask image. For example, the GPU may calculate the world space coordinates (x0, y0, z0) corresponding to each static pixel coordinate (w0, h0) based on the depth image Depth0 of the real frame N-1, the world matrix ViewMatrix0 of the virtual camera of the real frame N-1, and the projection matrix ProjectionMatrix0; then, the world space coordinates (x0, y0, z0) of the static pixel are reprojected to the pixel coordinate system using the world matrix ViewMatrix1 and the projection matrix ProjectionMatrix1 of the virtual camera of the real frame N+1, thereby obtaining a new pixel coordinate (w1, h1). After calculating the reprojection coordinates (w1, h1) corresponding to each static pixel coordinate (w0, h0), the GPU may use [(w1-w0)*0.5, (h1-h0)*0.5] to calculate the motion vector of the static pixel.

[0157] For example, assuming that the two-dimensional coordinates of a static pixel in the 3D scene image Color0 are (w0, h0) = (255, 255), and the two-dimensional coordinates of the static pixel after reprojection are (w1, h1) = (255, 269), then the motion vector of the static pixel is: [(255-255)*0.5, (269-255)*0.5] = [0, 7].

[0158] If the pixel mark of a pixel in the mask image is 1, it means that the pixel in the 3D scene image Color0 is a pixel of a dynamic object (referred to as a dynamic pixel for short), and the GPU can calculate the motion vector of the dynamic pixel by searching for the best matching pixel.

[0159] Specifically, the GPU can obtain the brightness Intensity0 corresponding to the coordinates (w0, h0) of a dynamic pixel in the 3D scene image Color0, and the feature vector Special0 corresponding to the dynamic pixel. Then, the GPU can find the best matching pixel corresponding to the dynamic pixel in the 3D scene image Color1 according to the following method: calculate the brightness Intensity1 of each pixel in the 3D scene image Color1 and the feature vector Special1 corresponding to the pixel, and traverse each pixel in the 3D scene image Color1 in turn through an iterative search method to find a pixel (w1, h1) with the smallest absolute value of (Intensity0–Intensity1) and the smallest feature distance between the feature vector Speical0 and the feature vector Special1. The pixel (w1, h1) is the best matching pixel. Then, the GPU can use [(w1-w0)*0.5, (h1-h0)*0.5] to calculate the motion vector of the dynamic pixel. Among them, the eigenvector Special0 and the eigenvector Special1 are used to characterize the brightness and geometric features of the pixel in the image. They are pixels with obvious unique features in the image, such as table corners, wall corners, and clothing patterns. Usually, the main parameters of the eigenvector are the color and depth of the current pixel and the surrounding adjacent pixels within a certain range. For example, assuming that the pixel coordinates of a dynamic pixel in the 3D scene image Color0 are (w0, h0) = (255, 257), and the pixel coordinates of the best matching pixel in the 3D scene image Color1 are (w1, h1) = (259, 239), then the motion vector of the dynamic pixel is: [(259-255)*0.5, (239-257)*0.5] = [2, -9].

[0160] After calculating the motion vector of each pixel in the 3D scene image Color0 according to the above steps, the GPU can output these motion vectors to a motion vector image mv. The motion vector image mv can be used for pixel-by-pixel offset in the following step 5.

[0161] Step 5: Based on the motion vector in the motion vector image mv obtained in step 4, the GPU performs a pixel-by-pixel offset on each pixel in the 3D scene image Color0 of the real frame N-1 obtained in step 1 to generate an intermediate image Color_NewFrame containing a blank area.

[0162] For example, Fig.10 A schematic diagram is shown of offsetting each pixel in the 3D scene image Color0 to the intermediate image Color_NewFrame pixel by pixel.

[0163] First, the GPU can create a new four-channel color image Color_NewFrame (called an intermediate image), and initialize the pixel value Color_NewFrame.rgba of each pixel in the intermediate image Color_NewFrame to (0.0, 0.0, 0.0, 0.5). The intermediate image Color_NewFrame can be used to save the pixel values ​​after pixel-by-pixel offset. Among them, Color_NewFrame.rgb=(0.0, 0.0, 0.0), indicating that the values ​​of the RGB three channels of the intermediate image Color_NewFrame are all 0.0, that is, black; Color_NewFrame.a=0.5, indicating that the value of the A channel of the intermediate image Color_NewFrame is 0.5, and the pixel value needs to be predicted later, that is, the pixel needs to be repaired.

[0164] It should be noted that in the embodiment of the present application, the RGB three channels refer to the R channel, the G channel, and the B channel, the R channel represents red, the G channel represents green, and the B channel represents blue. The intensity value of the RGB three channels can be 0 to 255, or the intensity value of the RGB three channels can be normalized to 0.0 to 1.0. The A channel refers to the alpha channel, which refers to the transparency and translucency of a picture. Usually, the value range of the A channel is 0.0 to 1.0.

[0165] Then, the GPU can perform pixel-by-pixel offset on each pixel Color0.rgba in the 3D scene image Color0 based on the motion vector calculated in step 4 above. As an example, Fig.10 As shown, the GPU can first move the static pixels one by one to the intermediate image Color_NewFrame based on the motion vector of the static pixels calculated in step 4, and then move the dynamic pixels one by one to the intermediate image Color_NewFrame based on the motion vector of the dynamic pixels calculated in step 4, thereby obtaining the intermediate image Color_NewFrame that stores the offset pixel values. As another example, the GPU can also start from the upper left corner of the 3D scene image Color0 and move each pixel row by row and column by column based on the motion vector calculated in step 4, that is, move the static pixels and the dynamic pixels at the same time.

[0166] For each pixel Color0.rgba in the 3D scene image Color0, assuming that the two-dimensional coordinates of a pixel are (w0, h0), the motion vector of the pixel at (w0, h0) in the motion vector image mv is mv0, and the GPU can move the pixel coordinates (w0, h0) according to the motion vector mv0 to obtain the new two-dimensional coordinates (w1, h1). Then, the GPU can reassign the pixel value Color_NewFrame.rgba at the coordinates (w1, h1) of the intermediate image Color_NewFrame to:

[0167] Color_NewFrame.rgb=Color0.rgb;

[0168] Color_NewFrame.a=1.0.

[0169] Among them, Color_NewFrame.rgb = Color0.rgb, which means that the RGB three-channel color at the coordinate (w1, h1) of the intermediate image Color_NewFrame is assigned to the RGB three-channel color at the pixel coordinate (w0, h0) of the 3D scene image Color0. Color_NewFrame.a = 1.0, which means that the value of the A channel at the coordinate (w1, h1) of the intermediate image Color_NewFrame is 1, and the pixel value does not need to be predicted later, that is, the pixel does not need to be repaired.

[0170] It should be noted that the embodiment of the present application is described by taking the value of the A channel of a pixel coordinate as 0.5 to indicate that the coordinate needs to be repaired, and the value of the A channel of a pixel coordinate as 1 to indicate that the coordinate does not need to be repaired, which does not limit the present application. In actual implementation, the value of the A channel of a pixel coordinate can also be 1 to indicate that the pixel of the coordinate needs to be repaired, and the value of the A channel of a coordinate can be 0.5 to indicate that the pixel of the coordinate does not need to be repaired.

[0171] In accordance with Fig.10Take the pixel-by-pixel offset as an example. When an object (such as a tree) in the 3D scene image Color0 is a static object, even if the virtual camera is displaced and rotated, the coordinates of each static pixel of the static object in the world coordinate system remain unchanged. Therefore, after each static pixel is moved one by one according to the motion vector, it is not easy to have pixel occlusion caused by multiple static pixels moving to the same position, nor is it easy to have blank areas caused by no pixels moving. However, when an object (such as a running person) in the 3D scene image Color0 is a dynamic object, the motion vectors corresponding to each dynamic pixel of the dynamic object may be different, so that multiple dynamic pixels may move to the same position (w1, h1) of the intermediate image Color_NewFrame, resulting in pixel occlusion at the position (w1, h1), and some positions may not have pixels moved, resulting in the pixel values ​​of these positions still retaining the initialization value (0.0, 0.0, 0.0, 0.5). In combination with the description of the above embodiment, when the A channel value of certain areas in the intermediate image Color_NewFrame is 0.5, it means that pixel value prediction needs to be performed on these areas later, and these areas are blank areas that need to be filled with pixels later.

[0172] Due to edge characteristics and algorithmic reasons, the motion vectors in the edge area of ​​a dynamic object and the motion vectors in the static area adjacent to the edge area may deviate greatly. When the motion vectors corresponding to the dynamic pixels of the dynamic object are different, multiple dynamic pixels in the edge area may move to the same position, making it more likely that blank areas will appear in the edge area of ​​the dynamic object. Of course, in some cases, blank areas may also appear in non-edge areas of dynamic objects, such as joints and facial areas, which are not limited in the embodiments of the present application.

[0173] It should be noted that the above-mentioned "blank area" refers to certain areas in the intermediate image Color_NewFrame, where no pixel values ​​are moved from the 3D scene image Color0 and the initialization values ​​(0.0, 0.0, 0.0, 0.5) are retained. Pixel filling is required later, but it does not mean that the area is white.

[0174] The following describes how to solve the problem of pixel occlusion that occurs when multiple pixels in the 3D scene image Color0 are moved to the same position in the intermediate image Color_NewFrame.

[0175] The first possible implementation

[0176] like Fig.11As shown, at the initial moment, the coordinates of the four adjacent pixels in the intermediate image Color_NewFrame are (w1, h1), (w2, h1), (w1, h2) and (w2, h2), and the Color_NewFrame.rgba of these four pixels are initialized to (0.0, 0.0, 0.0, 0.5). The coordinates of the four adjacent pixels in the 3D scene image Color0 are (w1, h1), (w2, h1), (w1, h2) and (w2, h2), and the Color_NewFrame.rgba of these four pixels are (r1, g1, b1, a1), (r2, g2, b2, a2), (r3, g3, b3, a3) and (r4, g4, b4, a4). Move pixel 1 of pixel coordinates (w1, h1) based on motion vector mv(1, 1), move pixel 2 of pixel coordinates (w2, h1) based on motion vector mv(0, 1), and move pixel 3 of pixel coordinates (w1, h2) based on motion vector mv(1, 0). Finally, all three pixels are moved to pixel coordinates (w2, h2) of the intermediate image Color_NewFrame, resulting in pixel occlusion at pixel coordinates (w2, h2). In addition, if no pixels (w1, h1), (w2, h1), and (w1, h2) in the intermediate image Color_NewFrame are moved, the pixels at these three positions still retain the initialization values ​​(0.0, 0.0, 0.0, 0.5), which are blank areas.

[0177] Assume that Fig.11 The pixel 1, pixel 2, and pixel 3 shown are moved to the same position P(w0,h0) in the intermediate image Color_NewFrame, and the depth corresponding to pixel 1 is l1, the depth corresponding to pixel 2 is l2, and the depth corresponding to pixel 3 is l3. Fig.12As shown, the origin O0 of the camera coordinate system O0-X0Y0Z0 is the optical center of the virtual camera. The GPU can, with reference to the description of step 3 above, successively convert the coordinate (w2, h2) to the camera coordinate system O0-X0Y0Z0 according to depth l1, depth l2, or depth l3, so as to obtain three positions P1, P2, and P3 in the camera coordinate system O0-X0Y0Z0. Then, the GPU can calculate the distance L1 from position P1 to the origin O0 of the camera coordinate system O0-X0Y0Z0, the distance L2 from position P2 to the origin O0 of the camera coordinate system O0-X0Y0Z0, and the distance L3 from position P3 to the origin O0 of the camera coordinate system O0-X0Y0Z0. Since L1 < L2 < L3 and position P1 is the closest to the origin O0, the GPU can use the pixel value Color0.rgba1 of pixel 1 corresponding to position P1 as the pixel value of the coordinate (w2, h2) finally displayed on the intermediate image Color_NewFrame.

[0178] Among them, the above distances L1, L2, and L3 can be obtained through the following relational expressions:

[0179] L1 = sqrt(x1 * x1 + y1 * y1 + z1 * z1);

[0180] L2 = sqrt(x2 * x2 + y2 * y2 + z2 * z2);

[0181] L3 = sqrt(x3 * x3 + y3 * y3 + z3 * z3).

[0182] Among them, sqrt() is the square root function, which is used to calculate the modulus length of a position in the camera coordinate system O0-X0Y0Z0, that is, the distance from a position in the camera coordinate system O0-X0Y0Z0 to the origin O0.

[0183] It should be noted that the above first possible implementation manner takes calculating the distances from each pixel point to the origin O0 of the camera coordinate system O0-X0Y0Z0 using the camera coordinate system as an example for illustration, and it does not limit this application.

[0185] The coordinates of the coordinate systems can be mutually converted. Therefore, the GPU can also calculate the distances from each pixel point to the origin O0 of the world coordinate system O0-X0Y0Z0 using the world coordinate system, and use the pixel value Color0.rgba1 of the pixel point 1 closest to the origin of the world coordinate system O0-X0Y0Z0 as the pixel value of the coordinate (w2, h2) finally displayed on the intermediate image Color_NewFrame. It should be understood that whether the world coordinate system or the camera coordinate system is adopted, the final calculation result is the same pixel point.

[0186] The second possible implementation

[0187] The GPU calculates the world space coordinates (x0, y0, z0) corresponding to each pixel coordinate (w0, h0) of the 3D scene image Color0 based on the depth image Depth0 of the real frame N-1, the world matrix ViewMatrix0 of the virtual camera of the real frame N-1, and the projection matrix ProjectionMatrix0. Then, using the world space coordinates (x0, y0, z0), the modulus length distance corresponding to each pixel coordinate is calculated = sqrt(x0*x0+y0*y0+z0*z0).

[0188] After completing the offset of all pixels in the 3D scene image Color0, if multiple pixels in the 3D scene image Color0 are moved to the same position (w1, h1) of the intermediate image Color_NewFrame, the GPU can compare the modulus distances of these multiple pixels and use the pixel value of the pixel coordinate corresponding to the minimum modulus distance as the pixel value finally displayed in the intermediate image Color_NewFrame.

[0189] It should be noted that, no matter whether the first possible implementation method or the second possible implementation method is adopted, the final calculation result is the same pixel point.

[0190] In addition, the above two possible implementations are illustrated by moving three pixels to the same position P(w0, h0) in the intermediate image Color_NewFrame, and selecting one pixel from the three pixels as the pixel finally displayed at the position P(w0, h0) in the intermediate image Color_NewFrame, which does not limit the present application. In actual implementation, the above method can also be referred to to select one pixel from two pixels, four pixels or more pixels moved to the same position as the pixel finally displayed at the position in the intermediate image Color_NewFrame.

[0191] Step 6. After the GPU generates the edge image edge_map based on the 3D scene image Color0 and the 3D scene image Color1 obtained in step 1, the edge image edge_map and the intermediate image Color_NewFrame obtained in step 5 are input into the AI ​​model to obtain the 3D scene prediction image Pixel_Color.

[0192] Exemplarily, the GPU may generate the 3D scene prediction image Pixel_Color through the following (a) to (e).

[0193] (a) Use the edge detection algorithm to extract the edge image edge0 of the 3D scene image Color0 and the edge image edge1 of the 3D scene image Color1.

[0194] Among them, the edge image edge0 and the edge image edge1 are both single-channel images, that is, grayscale images.

[0195] The edge image is an image obtained by extracting the edge of the original image using an edge detection algorithm. For example, the edge detection algorithm can be a canny algorithm, a differential operator method, a Laplace Gaussian operator method, a fitting method, a neural network method, a wavelet transform method, a wavelet packet decomposition method or a morphological method.

[0196] Take the canny algorithm as an example. The GPU can first use a Gaussian filter to smooth the 3D scene image Color0 / Color1 to filter out noise, then calculate the gradient strength and direction of each pixel in the image, and then use the non-maximum suppression method to eliminate the stray response caused by edge detection, and then use the double-threshold detection method to determine the real edge and potential edge, and finally suppress the isolated weak edges and retain the real edges, so as to obtain the edge image edge0 / edge1.

[0197] (b) Create two new edge images edge0_warp and edge1_warp.

[0198] Among them, the edge image edge0_warp and the edge image edge1_warp are both single-channel images, that is, grayscale images.

[0199] The edge image edge0_warp may be used to save the warping result of the edge image edge0.

[0200] The edge image edge1_warp can be used to save the warping result of the edge image edge1.

[0201] (c) performing image warping on edge image edge0 in units of pixel blocks, and outputting the warped result of edge image edge0 to edge image edge0_warp; and performing image warping on edge image edge1 in units of pixel blocks, and outputting the warped result of edge image edge1 to edge image edge1_warp.

[0202] The above pixel block may be composed of multiple pixel points.

[0203] As an example, a pixel block may be composed of N*N pixels. Where N is an integer greater than or equal to 2. For example, N=8. According to the description of step 5 above, when the real image is offset pixel by pixel in units of pixels, multiple pixels may move to the same position of the intermediate image Color_NewFrame, resulting in pixel occlusion, resulting in blank areas. It should be understood that the above (c) warping the edge image in units of pixel blocks can solve the problem of pixel occlusion, thereby facilitating image repair of blank areas in subsequent steps.

[0204] The present application provides a 2D grid model. The 2D grid model may be an offline grid model manually constructed.

[0205] As an example, Fig.13 A 2D grid model is shown. The 2D grid model consists of p*q rectangular grids, where p represents the number of rows of rectangular grids in the 2D grid model, and q represents the number of columns of rectangular grids in the 2D grid model. For each rectangular grid, a rectangular grid represents a pixel block, for example, composed of 8*8 pixels. In other words, the 2D grid model consists of 8m*8n pixels. It should be noted that the pixel coordinates of the 2D grid model correspond one-to-one to the pixel coordinates in the 3D scene image Color0, the 3D scene image Color1, the edge image edge0, the edge image edge1, the edge image edge0_warp, and the edge image edge1_warp. Assume that the pixel coordinates of the four vertices of a rectangular grid are (0,0), (7,0), (0,7), and (7,7), respectively, and the motion vectors of the pixel coordinates of the four vertices determined by the above step 4 are mv1=(0,2), mv2=(2,1), mv3=(1,1), and mv4=(1,2), respectively. The GPU can move the pixel coordinates of the four vertices according to the motion vectors corresponding to the pixel coordinates of each vertex, thereby obtaining the pixel coordinates of the four new vertices, which are (0,2), (9,1), (1,8), and (8,9) in sequence. The GPU can connect the pixel coordinates of the four new vertices in sequence to obtain the following: Fig.13 The new mesh shown in the dashed box is then rasterized and the linear interpolation method is used to calculate the coordinates of each pixel in the new mesh. Since the texture coordinates (ux,uy) of each vertex before and after the move remain unchanged, the pixel data before the move can be assigned to the pixel coordinates after the move according to the texture coordinates (ux,uy).

[0206] It should be noted that Fig.13Only a schematic diagram of image distortion of a rectangular grid composed of 8*8 pixels in a 2D grid model is shown. It should be understood that for any rectangular grid of the 2D grid model, the pixel coordinates of the four vertices of the rectangular grid can be moved to perform image distortion, and a new edge image is obtained after the distortion of all rectangular grids is completed.

[0207] Combine the following Fig.14A and Fig. 14B , a specific process of rendering edge image edge0_warp and edge image edge1_warp using a 2D mesh model is illustrated by way of example.

[0208] Fig.14A A schematic diagram showing the process of warping the edge image edge0 to obtain the edge image edge0_warp is shown.

[0209] like Fig.14A As shown, the GPU can first extract the edge image edge0 from the 3D scene image Color0, and the edge image edge0 is a single-channel image. Then, the GPU can move each vertex according to the motion vector mv0 corresponding to each vertex vertex0 in the 2D mesh model to form a distorted new mesh, and then rasterize each new mesh, and linearly interpolate the pixel coordinates of each pixel in the new mesh. Each pixel coordinate is used to determine the position of a pixel in the edge image edge0_warp. Assume that the pixel coordinates of a pixel before moving are (vx, vy), and the pixel coordinates after moving according to the motion vector (mv.x, mv.y) are (vx_n, vy_n) = (vx+mv.x, vy+mv.y). Since the texture coordinates (ux,uy) of a pixel point remain unchanged before and after the movement, that is, (ux_n,uy_n)=(ux,uy), the GPU can collect pixel data from the pixel coordinates (vx,vy) of the edge image edge0 according to the texture coordinates (ux,uy) corresponding to the pixel coordinates (vx_n,vy_n) after the movement, and output the collected pixel data to the pixel coordinates (vx_n,vy_n) of the edge image edge0_warp, and finally obtain the edge image edge0_warp that saves the distortion result of the edge image edge0.

[0210] Fig. 14B A schematic diagram showing how the edge image edge1 is warped to obtain the edge image edge1_warp.

[0211] like Fig. 14BAs shown, the GPU can first extract the edge image edge1 from the 3D scene image Color1, and the edge image edge1 is a single-channel image. Then, the GPU can move each vertex according to the motion vector mv0*(-1.0) corresponding to each vertex vertex0 in the 2D mesh model to form a distorted new mesh, and then rasterize each new mesh, and linearly differencing the pixel coordinates of each pixel in the new mesh. The pixel coordinates of each pixel are used to determine the position of a pixel in the edge image edge1_warp. Assume that the pixel coordinates of a pixel before moving are (vx, vy), and the pixel coordinates after moving according to the motion vector (-mv.x, -mv.y) are (vx_n, vy_n) = (vx-mv.x, vy-mv.y). Since the texture coordinates (ux,uy) of a pixel point remain unchanged before and after the movement, that is, (ux_n,uy_n)=(ux,uy), the GPU can collect pixel data from the pixel coordinates (vx,vy) of the edge image edge1 according to the texture coordinates (ux,uy) corresponding to the pixel coordinates (vx_n,vy_n) after the movement, and output the collected pixel data to the pixel coordinates (vx_n,vy_n) of the edge image edge1_warp, and finally obtain the edge image edge1_warp that saves the distortion result of the edge image edge1.

[0212] In the above embodiment, the GPU performs image distortion on the edge image edge0 and the edge image edge1 in units of pixel blocks, which can effectively prevent multiple pixels from moving to the same position, thereby ensuring the integrity of the edge image.

[0213] (d) Based on the A channel label of the intermediate image Color_NewFrame, the edge image edge0_warp and the edge image edge1_warp are output to the final edge image edge_map.

[0214] Exemplarily, the GPU may create an edge image edge_map, wherein the pixel coordinates of the intermediate image Color_NewFrame, the edge image edge0_warp, the edge image edge1_warp, and the edge image edge_map are in a one-to-one correspondence. The GPU may determine the fill value of the corresponding pixel coordinate in the edge image edge_map based on the specific value of the A channel of each pixel coordinate in the intermediate image Color_NewFrame.

[0215] If A=0.5 of a pixel coordinate in the intermediate image Color_NewFrame, the pixel data of the edge image edge1_warp is used to fill the corresponding pixel coordinate in the edge image edge_map, that is, the edge image edge1_warp is used to fill the blank area.

[0216] If A≠0.5 of a pixel coordinate in the intermediate image Color_NewFrame, the pixel data of the edge image edge0_warp is used to fill the corresponding pixel coordinate of the edge image edge_map. It should be understood that filling with the pixel data of the edge image edge0_warp can generate a complete edge image, which is convenient for the AI ​​model to recognize the edge.

[0217] For example, by Fig.14A Get the edge image edge0_warp, through Fig. 14B After obtaining the edge image edge1_warp, Fig. 14C As shown, in the final edge image edge_map, the area corresponding to the area of ​​A≠0.5 of the intermediate image Color_NewFrame is filled with the data of the edge image edge0_warp, and the area corresponding to the area of ​​A=0.5 of the intermediate image Color_NewFrame is filled with the data of the edge image edge1_warp.

[0218] After the edge image edge0_warp and the edge image edge1_warp are output to the final edge image edge_map, the edge image edge_map is a grayscale image. In some embodiments, the GPU can directly input the single-channel edge image edge_map to the AI ​​model, and then the AI ​​model converts the single-channel edge image edge_map into an RGB image. In other embodiments, the GPU can first convert the single-channel edge image edge_map into an RGB image, and then input the converted RGB image into the AI ​​model.

[0219] (e) The edge image edge_map obtained by (d) and the intermediate image Color_NewFrame obtained in step 5 are input into the AI ​​model of the NPU. After the AI ​​model fills the blank areas of the intermediate image Color_NewFrame with pixels, the GPU obtains the 3D scene prediction image Pixel_Color from the NPU.

[0220] Among them, the above-mentioned AI model can be a neural network model, such as an AI image restoration (AI inpainting) model.

[0221] For example, the terminal device may pre-train a neural network model and load the neural network model onto an NPU for executing a neural network algorithm. Fig.15 As shown, the GPU can input the edge image edge_map obtained by the above (d) and the intermediate image Color_NewFrame obtained by the above step 5 into the AI ​​model. The AI ​​model performs convolution and deconvolution calculations on the edge image edge_map and the intermediate image Color_NewFrame to achieve image restoration of the to-be-filled area in the intermediate image Color_NewFrame, thereby generating a complete 3D scene prediction image Pixel_Color. Then, the NPU returns the 3D scene prediction image Pixel_Color output by the AI ​​model to the GPU.

[0222] Combined with the description of the above embodiments, the traditional AI model is usually loaded on a device with high computing power such as a PC. A complete predicted frame can be obtained by inputting the real frame into the AI ​​model. However, the algorithm of the traditional AI model is complex, and the requirements for computing power and bandwidth are high. It cannot be run in real time on terminal devices with limited computing power such as mobile phones. Different from the traditional AI model, the AI ​​model provided in this application is only used to repair the blank areas of the image, and has low requirements for computing power. It can be run in real time on terminal devices with limited computing power such as mobile phones. In addition, by predicting the data missing area, the generated 3D scene prediction image Pixel_Color is closer to the result of the original image.

[0223] Step 7: GPU superimposes the UI image UI0 of the real frame N-1 obtained in step 1 onto the 3D scene prediction image Pixel_Color obtained in step 6 to generate the final prediction frame image Pixel_Final, i.e., prediction frame N.

[0224] For example, Fig.16 As shown, the GPU can perform alpha blending (ie, A channel blending) of the UI image UI0 of the real frame N-1 generated by the above step 1 and the 3D scene prediction image Pixel_Color generated by the above step 6 to generate the final prediction frame image Pixel_Final, that is, the prediction frame N.

[0225] Among them, the rules of alpha blending can be:

[0226] Pixel_Final.rgb=Pixel_Color.rgb*(1.0–Pixel_UI.a)+Pixel_UI.rgb*Pixel_UI.a;

[0227] Pixel_Final.a=1.0.

[0228] Among them, Pixel_Final.rgb is the RGB three-channel pixel of the predicted frame image Pixel_Final, Pixel_Final.a is the A channel pixel of the predicted frame image Pixel_Final, that is, Pixel_Final.rgba is the RGBA four-channel pixel of the predicted frame image Pixel_Final.

[0229] Pixel_Color.rgb is the RGB three-channel pixel of the 3D scene prediction image Pixel_Color, Pixel_Final.a is the A channel pixel of the 3D scene prediction image Pixel_Color, that is, Pixel_Color.rgba is the RGBA four-channel pixel of the 3D scene prediction image Pixel_Color.

[0230] Pixel_UI.rgb is the RGB three-channel pixel of UI image UI0 of real frame N-1, and Pixel_UI.a is the A channel pixel of UI image UI0 of real frame N-1, that is, Pixel_UI.rgba is the RGBA four-channel pixel of UI image UI0 of real frame N-1. It should be noted that Pixel_UI.a can be obtained from the rendering instruction stream of real frame N-1.

[0231] It should be understood that the transparency effect of the UI interface can be achieved through alpha blending. The degree of transparency can be controlled by Pixel_UI.a. When Pixel_UI.a=0 of a pixel point of the UI interface, it means that the pixel point is completely transparent and will not block the 3D scene prediction image Pixel_Color located at the lower layer of the UI interface; and when Pixel_UI.a=1 of a pixel point of the UI interface, it means that the pixel point is completely opaque and will block the 3D scene prediction image Pixel_Color located at the lower layer of the UI interface.

[0232] Combined with the description of the above embodiment, the GPU can generate the real frame N-1 according to the rendering instruction stream 1 obtained first, generate the real frame N+1 according to the rendering instruction stream 2 obtained later, and generate the predicted frame N according to the rendering instruction stream 1 and the rendering instruction stream 2. The GPU can send these frames to the queue to be displayed in the order of the real frame N-1, the predicted frame N and the real frame N+1. Then, as Fig.17 As shown, the display screen of the terminal device plays the real frame N-1, the predicted frame N and the real frame N+1 in sequence.

[0233] In the above-mentioned predicted frame generation method provided in the present application, the GPU can obtain the three-dimensional scene image and UI image corresponding to each real frame from two game rendering instructions, as well as general information such as the depth image, the world matrix and projection matrix of the game virtual camera, and then use this information to perform pixel-by-pixel offset on the 3D scene image, thereby solving the physical distortion problem existing in the block-by-block offset scheme and the ghosting problem existing in the image mixing scheme; the GPU can then use AI image restoration technology to fill in the blank area after the pixel-by-pixel offset, so that the finally generated predicted frame image is closer to the native picture quality of the game.

[0234] An embodiment of the present application also provides a terminal device, including a processor, the processor is coupled to a memory, and the processor is used to execute a computer program or instruction stored in the memory, so that the terminal device implements the methods in the above embodiments.

[0235] The embodiment of the present application also provides a computer-readable storage medium, in which a computer instruction is stored. When the computer-readable storage medium is run on a computer, the computer is caused to execute the method shown above. The computer instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instruction can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that can be integrated with one or more media. Available media can be magnetic media (e.g., floppy disk, hard disk or tape), optical media or semiconductor media (e.g., solid state drive (SSD)), etc.

[0236] An embodiment of the present application further provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer executes the methods in the above embodiments.

[0237] The embodiment of the present application also provides a chip, which is coupled to a memory, and the chip is used to read and execute a computer program or instruction stored in the memory to perform the methods in the above embodiments. The chip can be a general-purpose processor or a dedicated processor. It should be noted that the chip can be implemented using the following circuits or devices: one or more field programmable gate arrays (field programmable gate array, FPGA), programmable logic devices (programmable logic device, PLD), controllers, state machines, gate logic, discrete hardware components, any other suitable circuits, or any combination of circuits capable of performing the various functions described throughout this application.

[0238] The terminal device, computer-readable storage medium, computer program product and chip provided in the above-mentioned embodiments of the present application are all used to execute the method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects corresponding to the method provided above, and will not be repeated here.

[0239] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0240] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0241] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0242] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0243] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A prediction frame generation method, characterized in that: The method comprises: Obtaining a user interface image, a first scene image, a first depth image, a first projection matrix, and a first world matrix from a first rendering instruction stream; Obtain a second scene image, a second depth image, a second projection matrix, and a second world matrix from a second rendering instruction stream; determining static pixels and dynamic pixels in the first scene image based on the first depth image, the first projection matrix, the first world matrix, the second depth image, the second projection matrix, and the second world matrix; Determine the motion vector of the static pixel according to the pixel coordinates of the static pixel and the reprojected coordinates after reprojecting the static pixel; and determine the matching pixel of the dynamic pixel in the second scene image, and determine the motion vector of the dynamic pixel according to the pixel coordinates of the dynamic pixel and the pixel coordinates of the matching pixel; Move the static pixel based on the motion vector of the static pixel, and move the dynamic pixel based on the motion vector of the dynamic pixel to obtain an intermediate image; Filling pixels in the blank area of ​​the intermediate image based on the AI ​​model to obtain a scene prediction image; The user interface image is superimposed on the scene prediction image to obtain a prediction frame.

2. The method according to claim 1, characterized in that The determining, based on the first depth image, the first projection matrix, the first world matrix, the second depth image, the second projection matrix, and the second world matrix, static pixels and dynamic pixels in the first scene image comprises: Perform the following steps for each pixel in the first scene image: Determine, based on the first depth image, the first projection matrix, and the first world matrix, a first world space coordinate corresponding to a first pixel coordinate of the first scene image; reproject the first world space coordinate to a pixel coordinate system based on the second projection matrix and the second world matrix to obtain a second pixel coordinate; determine, based on the second depth image, the second projection matrix, and the second world matrix, a second world space coordinate corresponding to the second pixel coordinate; For any pixel in the first scene image: If the first world space coordinate and the second world space coordinate of a pixel are the same, the pixel is marked as a static pixel; or if the first world space coordinate and the second world space coordinate of the pixel are different, the pixel is marked as a dynamic pixel.

3. The method according to claim 2, characterized in that The method further comprises: Output the labeling result of each pixel to the mask image.

4. The method according to any one of claims 1 to 3, characterized in that The determining the motion vector of the static pixel according to the pixel coordinates of the static pixel and the reprojected coordinates after reprojecting the static pixel comprises: Determining world space coordinates corresponding to static pixel coordinates (w0, h0) of the first scene image based on the first depth image, the first projection matrix, and the first world matrix; Based on the second projection matrix and the second world matrix, reproject the world space coordinates corresponding to the static pixel coordinates (w0, h0) to a pixel coordinate system to obtain the reprojected coordinates (w1, h1); [(w1-w0)*0.5, (h1-h0)*0.5] is determined as the motion vector of the static pixel.

5. The method according to any one of claims 1 to 3, characterized in that The step of determining a matching pixel of the dynamic pixel in the second scene image, and determining a motion vector of the dynamic pixel according to the pixel coordinates of the dynamic pixel and the pixel coordinates of the matching pixel, comprises: Acquire brightness and feature vectors corresponding to the dynamic pixel coordinates (w0', h0') of the first scene image; Searching for the matching pixel (w1', h1') in the second scene image according to the brightness and the feature vector corresponding to the dynamic pixel coordinate (w0', h0'); [(w1'-w0')*0.5, (h1'-h0')*0.5] is determined as the motion vector of the dynamic pixel.

6. The method according to claim 5, characterized in that The matching pixels satisfy the following conditions: The absolute value of the first difference is smaller than the absolute value of the second difference, the first difference is the difference between the brightness corresponding to the dynamic pixel coordinate (w0', h0') and the brightness corresponding to the matching pixel (w1', h1'), and the second difference is the brightness corresponding to the dynamic pixel coordinate (w0', h0') and the brightness corresponding to other pixel coordinates of the second scene image; and The first feature distance is smaller than the second feature distance. The first feature distance is the feature distance between the feature vector corresponding to the dynamic pixel coordinate (w0', h0') and the feature vector corresponding to the matching pixel (w1', h1'). The second feature distance is the feature distance between the feature vector corresponding to the dynamic pixel coordinate (w0', h0') and the feature vector corresponding to other pixel coordinates of the second scene image.

7. The method according to any one of claims 1 to 6, characterized in that The step of moving the static pixel based on the motion vector of the static pixel and moving the dynamic pixel based on the motion vector of the dynamic pixel to obtain an intermediate image comprises: Creating an intermediate image; Based on the motion vector of each static pixel in the first scene image, move each static pixel one by one to the intermediate image; Based on the motion vector of each dynamic pixel in the first scene image, each dynamic pixel is moved one by one to the intermediate image.

8. The method according to claim 7, characterized in that After creating the intermediate image, the method further comprises: Assign the RGB channels of the intermediate image to 0.0, and assign the A channel of the intermediate image to 0.5; The step of moving each static pixel one by one to the intermediate image based on the motion vector of each static pixel in the first scene image comprises: Based on the motion vector of each static pixel in the first scene image, determine the pixel coordinates corresponding to each static pixel in the intermediate image; assign the RGB channels of the pixel coordinates corresponding to each static pixel in the intermediate image to a first value, and assign the A channel of the pixel coordinates corresponding to each static pixel in the intermediate image to 1.0; wherein the first value is the RGB channel pixel value of each static pixel in the first scene image; The step of moving each dynamic pixel one by one to the intermediate image based on the motion vector of each dynamic pixel in the first scene image comprises: Based on the motion vector of each dynamic pixel in the first scene image, the pixel coordinates corresponding to each dynamic pixel are determined in the intermediate image; the RGB channels of the pixel coordinates corresponding to each dynamic pixel in the intermediate image are assigned a second value, and the A channel of the pixel coordinates corresponding to each dynamic pixel in the intermediate image is assigned 1.0; wherein the second value is the RGB channel pixel value of each dynamic pixel in the first scene image.

9. The method according to claim 8, characterized in that The method further comprises: If a plurality of pixels are moved to the same pixel coordinate of the intermediate image, converting the same pixel coordinate to a camera coordinate system according to the depth of each pixel of the plurality of pixels in the first depth image to obtain the camera coordinate of each pixel of the plurality of pixels; Calculate the distance from the camera coordinate of each pixel in the plurality of pixels to the camera origin according to the camera coordinate of each pixel in the plurality of pixels; The RGB channels of the same pixel coordinates are assigned values ​​according to the RGB channel pixel value of a pixel corresponding to the minimum distance among the multiple distances.

10. The method according to any one of claims 1 to 9, characterized in that Filling pixels in the blank area of ​​the intermediate image based on the AI ​​model to obtain a scene prediction image includes: generating a target edge image according to the first scene image and the second scene image; The target edge image and the intermediate image are input into the AI ​​model to generate the scene prediction image; wherein the scene prediction image is an image obtained by the AI ​​model using the target edge image to fill pixels in the blank area of ​​the intermediate image.

11. The method according to claim 10, characterized in that The step of generating a target edge image according to the first scene image and the second scene image includes: Extracting a first edge image from the first scene image, and extracting a second edge image from the second scene image; Warping the first edge image in units of pixel blocks, and outputting the warping result of the first edge image to a first warped edge image; warping the second edge image in units of pixel blocks, and outputting the warping result of the second edge image to a second warped edge image; The target edge image is generated according to the first warped edge image and the second warped edge image based on the A channel label of the intermediate image.

12. The method according to claim 11, characterized in that The step of distorting the first edge image in units of pixel blocks and outputting the distortion result of the first edge image to a first distorted edge image includes: Determining, in the first warped edge image, pixel coordinates corresponding to pixels in the first edge image according to motion vectors corresponding to respective vertices of each mesh in the 2D mesh model; According to the texture coordinates corresponding to the pixel coordinates in the first warped edge image, pixel values ​​are collected from the first edge image to obtain the first warped edge image which stores the warping result of the first edge image.

13. The method according to claim 11, characterized in that The step of distorting the second edge image in units of pixel blocks and outputting the distortion result of the second edge image to a second distorted edge image includes: Determining, in the second warped edge image, pixel coordinates corresponding to pixels in the second edge image according to the inverse vector of the motion vector corresponding to each vertex of each mesh in the 2D mesh model; According to the texture coordinates corresponding to the pixel coordinates in the second distorted edge image, pixel values ​​are collected from the second edge image to obtain the second distorted edge image which stores the distortion result of the second edge image.

14. The method according to claim 11, characterized in that The first edge image, the second edge image, the first warped edge image, and the second warped edge image are single-channel images.

15. The method according to any one of claims 11 to 14, characterized in that The step of generating the target edge image based on the A channel mark of the intermediate image and the first warped edge image and the second warped edge image comprises: If the A channel mark of a pixel coordinate in the intermediate image is 0.5, the corresponding pixel coordinate in the target edge image is filled with the pixel data of the second edge image; or If the A channel mark of a pixel coordinate in the intermediate image is 1, the corresponding pixel coordinate in the target edge image is filled with the pixel data of the first edge image.

16. The method according to any one of claims 1 to 15, characterized in that The first rendering instruction stream is obtained before the second rendering instruction stream; After superimposing the user interface image on the scene prediction image to obtain a prediction frame, the method further includes: Displaying the first frame, the predicted frame and the second frame in sequence; The first frame is a real image frame generated according to the first rendering instruction stream, and the second frame is a real image frame generated according to the second rendering instruction stream.

17. A terminal device, characterized in that: It comprises a processor and a memory coupled to the processor; wherein the memory stores instructions, and the processor calls the instructions so that the terminal device executes the prediction frame generation method as described in any one of claims 1 to 16.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on a terminal device, the terminal device executes the prediction frame generation method according to any one of claims 1 to 16.

Images