Prediction frame generation method, terminal equipment and storage medium

Abstract

The embodiment of the invention provides a prediction frame generation method, terminal equipment and a storage medium, and relates to the technical field of terminals. In a game scene, terminal equipment can acquire a 3D scene image and a UI image corresponding to each real frame from two game rendering instructions, and acquire a depth image and a projection matrix of a game virtual camera according to the game rendering instructions; dividing long-shot pixels and close-shot pixels in the 3D scene image by using the information; then, calculating a motion vector of each long-shot pixel based on a stereoscopic vision principle, and calculating a motion vector of each close-shot pixel based on a spatial search method; then, according to the motion vector of each long-shot pixel and the motion vector of each close-shot pixel, warping the whole 3D scene image by taking a pixel block as a unit; and finally, superposing the UI image on the distorted 3D scene image to generate a final prediction frame. According to the scheme, the image quality of the prediction frame is improved, and the method only depends on universal resources of games and is suitable for most games.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a prediction frame generation method, a terminal device and a storage medium. Background Art

[0002] At present, game scene rendering has increasingly higher requirements for picture smoothness and picture clarity, which relies on higher screen refresh rates and graphics rendering resolutions, posing severe challenges to terminal devices in terms of chip power consumption load.

[0003] In the rendering process of game scenes, there are many similarities between the two frames of images rendered successively. The related technology proposes to obtain the motion vector of the pixel between the two frames of images based on the similarity between the pixels, and then predict the position of the pixel in the predicted frame based on the motion vector. Since the computing resource requirements for generating the predicted frame are less than the computing resource requirements for the game application to render the real frame, the resource consumption of the terminal device is reduced.

[0004] However, for game scenes with different environments and different screen styles, each game frame has different image information distribution characteristics. How to generate prediction frames in different game scenes has become an urgent problem to be solved. Summary of the invention

[0005] The present application provides a prediction frame generation method, a terminal device and a storage medium, which solve the problem of how to generate prediction frames based on motion vectors in different game scenarios.

[0006] In order to achieve the above objectives, this application adopts the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a method for generating a prediction frame. The method may include:

[0008] A user interface image, a first scene image, a first depth image and a first projection matrix are obtained from a first rendering instruction stream; a second scene image, a second depth image and a second projection matrix are obtained from a second rendering instruction stream; distant pixels and near pixels in the first scene image are determined according to the first depth image; a motion vector of each distant pixel in the first scene image is determined according to the first depth image, the first projection matrix, the second scene image, the second depth image and the second projection matrix; a motion vector of each near pixel in the first scene image is determined according to the second scene image; based on the motion vectors of the distant pixels and the motion vectors of the near pixels, the first scene image is distorted in units of pixel blocks to obtain a scene prediction image; and a user interface image is superimposed on the scene prediction image to obtain a prediction frame.

[0009] In the above scheme, the depth image and the projection matrix of the game virtual camera are included in the game rendering instructions, and are both common resources that are easy to obtain, so they can be adapted to most games, improving the versatility of the prediction frame generation method. In addition, since distant pixels and near pixels are distinguished, and the motion vectors of distant pixels and near pixels are calculated respectively based on the characteristics of distant pixels and near pixels, after the image is distorted based on the motion vectors of distant pixels and near pixels, a prediction frame with better image quality is obtained, that is, the image quality of the prediction frame is improved.

[0010] In a possible implementation, the first rendering instruction stream and the second rendering instruction stream are game rendering instruction streams. The game rendering instruction stream may include a variety of drawing instructions, such as scene drawing instructions, UI drawing instructions, and image display instructions. Among them, the scene drawing instructions can be used to draw 3D scene images such as scenery, characters, special effects and skills in the game, the UI drawing instructions can be used to draw UI images such as control buttons, small maps and floating text, and the image display instructions are used to put the drawn image data into the specified location of the operating system to complete the frame display.

[0011] In a possible implementation, a real game frame may be composed of a 3D scene image and a UI image, etc. Among them, the 3D scene image is also called a color image. For example, the 3D scene image may include in-game scenery, game characters, background objects, special effects and skills, etc., and the UI image may include control buttons, a small map, floating text, a user avatar, an energy pack control and a health bar, etc. The UI image is located on the upper layer of the 3D scene image. It should be understood that both the game characters in the 3D scene image and the rendering control buttons in the UI image are objects in the game frame.

[0012] In a possible implementation, the pixel coordinates of the 3D scene image and the pixel coordinates of the UI image are in a one-to-one correspondence, and the UI image is located on the upper layer of the 3D scene image. If the pixel coordinates of an object in the UI image are the same as those of an object in the 3D scene image, the object in the UI image may cause visual occlusion to the object in the 3D scene image. It should be understood that by separating the 3D scene image and the UI image from the rendering instruction stream, it is convenient to perform scene prediction on the 3D scene image later.

[0013] In a possible implementation manner, the method may further include: determining distant view pixels and near view pixels in the second scene image according to the second depth image.

[0014] In the above scheme, the distant pixels and near pixels of two consecutive frames of images are marked respectively. This facilitates the subsequent calculation of the rotation matrix and translation matrix of the virtual camera based on the similarity of the distant pixels of the two frames of images, and then calculates the motion vector of the distant pixels; and facilitates the subsequent calculation of the best matching pixels based on the similarity of the near pixels of the two frames of images, and then calculates the motion vector of the near pixels.

[0015] In a possible implementation, determining distant view pixels and near view pixels in the first scene image according to the first depth image includes: determining whether a depth value of a pixel coordinate in the first depth image is greater than or equal to a depth threshold; if the depth value of a pixel coordinate in the first depth image is greater than or equal to the depth threshold, marking a pixel coordinate in the mask image as a distant view pixel; or, if the depth value of a pixel coordinate in the first depth image is less than the depth threshold, marking a pixel coordinate in the mask image as a near view pixel.

[0016] In one possible implementation, determining distant view pixels and near view pixels in the second scene image according to the second depth image includes: determining whether a depth value of a pixel coordinate in the second depth image is greater than or equal to a depth threshold; if the depth value of a pixel coordinate in the second depth image is greater than or equal to the depth threshold, marking a pixel coordinate in the mask image as a distant view pixel; or, if the depth value of a pixel coordinate in the second depth image is less than the depth threshold, marking a pixel coordinate in the mask image as a near view pixel.

[0017] In the above scheme, the first depth image and the second depth image are similar to grayscale images. Each depth value of the first depth image and the second depth image can be used to characterize the distance between the virtual camera and the object. The first depth image and the second depth image reflect the geometric shape of the visible surface of the object. It should be understood that when the depth value of a pixel in the scene image is large, the pixel is far away from the virtual camera and belongs to the pixel of the distant object, that is, the distant pixel; when the depth value of a pixel in the scene image is small, the pixel is close to the virtual camera and belongs to the pixel of the near object, that is, the near pixel.

[0018] In a possible implementation, determining a motion vector of each distant view pixel in the first scene image according to the first depth image, the first projection matrix, the second scene image, the second depth image, and the second projection matrix includes: obtaining a feature point pair set E[(x0, y0), (x1, y1)], where the feature point pair set consists of multiple feature point pairs, and each feature point pair consists of a distant view pixel (x0, y0) in the first scene image and a distant view pixel (x1, y1) in the second scene image; determining a three-dimensional coordinate pair set E[(x0, y0, z0), (x1, y1, z1)] in a camera coordinate system corresponding to the feature point pair set according to the first depth image, the first projection matrix, the second depth image, and the second projection matrix; obtaining a rotation matrix and a translation matrix of a virtual camera using the three-dimensional coordinate pair set E[(x0, y0, z0), (x1, y1, z1)]; and determining the motion vector of each distant view pixel in the first scene image according to the rotation matrix and the translation matrix.

[0019] In the above scheme, the distant pixels are usually static pixels, and the motion vectors of the static pixels are generated by the motion of the virtual camera. When the motion vector of a pixel is generated by the motion of the virtual camera, the position of the pixel in the world space remains unchanged. In this case, the motion information of the virtual camera (such as the rotation matrix and translation matrix of the virtual camera) can be estimated by using the principle of stereo vision, which makes it easier to calculate the motion vector of each distant pixel in the first scene image Color0 based on the motion information of the virtual camera.

[0020] In a possible implementation, obtaining a set of feature point pairs includes: determining a plurality of first corner points in distant pixels of a first scene image, and selecting N first feature points from the plurality of first corner points; determining a plurality of second corner points in distant pixels of a second scene image, and selecting N second feature points from the plurality of second corner points; for each first feature point among the N first feature points, selecting a second feature point from the N second feature points as a best matching point of a first feature point, to obtain a set of feature point pairs.

[0021] In the above scheme, feature points can be used to characterize the geometric features of pixels at pixel coordinates in the 3D scene image, which are pixels with obvious features in the 3D scene image, such as table corners, wall corners, and clothing patterns, etc. By selecting feature points with obvious features from two adjacent images, feature point matching is facilitated.

[0022] In a possible implementation, the three-dimensional coordinate pair set is composed of N three-dimensional coordinate pairs. According to the first depth image, the first projection matrix, the second depth image, and the second projection matrix, the three-dimensional coordinate pair set in the camera coordinate system corresponding to the feature point pair set is determined, including: based on the first depth image and the first projection matrix, converting the pixel coordinates of the N first feature points into three-dimensional coordinates in the camera coordinate system; based on the second depth image and the second projection matrix, converting the pixel coordinates of the N second feature points into three-dimensional coordinates in the camera coordinate system.

[0023] In the above scheme, the coordinates in the 3D scene image can be converted into NDC coordinates by using the depth image, and then the NDC coordinates can be converted into three-dimensional coordinates in the camera coordinate system by using the projection matrix. These coordinate conversions facilitate the subsequent construction of the space transformation matrix.

[0024] In a possible implementation, the rotation matrix and translation matrix of the virtual camera are obtained using the three-dimensional coordinate pair set, including: constructing a space transformation matrix using the three-dimensional coordinate pair set; and decomposing the rotation matrix and translation matrix from the space transformation matrix.

[0025] In the above scheme, the singular value decomposition method can be used to decompose the rotation matrix Rotation and translation matrix Translation of the virtual camera from the space transformation matrix ViewMatrix. The rotation matrix Rotation and translation matrix Translation are the data volume of the virtual camera transforming from the first scene image Color0 to the second scene image Color1, that is, the association between the two camera coordinate systems is established.

[0026] In a possible implementation, the motion vector of each distant pixel in the first scene image is determined according to the rotation matrix and the translation matrix, including: determining the three-dimensional coordinates in the camera coordinate system corresponding to each distant pixel of the first scene image according to the first depth image and the first projection matrix; reprojecting the three-dimensional coordinates in the camera coordinate system corresponding to each distant pixel of the first scene image to the pixel coordinate system based on the rotation matrix, the translation matrix and the second projection matrix to obtain the reprojected coordinates; determining the motion vector of each distant pixel according to the coordinates (x0, y0) of each distant pixel in the first scene image and the reprojected coordinates (x0, y0) corresponding to each distant pixel coordinate (x0, y0). Exemplarily, the terminal device can determine the motion vector of each distant pixel according to the relationship [(x1-x0)*0.5, (y1-y0)*0.5].

[0027] In the above scheme, each distant pixel of the first scene image is first converted into a three-dimensional coordinate in a camera coordinate system, and then converted to another camera coordinate system through rotation and translation; and then reprojected to the pixel coordinate system, so that the position of a distant pixel in the second scene image is obtained.

[0028] In a possible implementation, determining the motion vector of each near-view pixel in the first scene image according to the second scene image includes: obtaining the brightness and feature vector of each near-view pixel in the first scene image; for each near-view pixel in the first scene image, searching for a matching pixel in the second scene image according to the brightness and feature vector of a near-view pixel; determining the motion vector of each far-view pixel (x0', y0') according to each far-view pixel (x0', y0') of the first scene image and the matching pixel (x1', y1') corresponding to each far-view pixel (x0', y0'). Exemplarily, the terminal device may determine the motion vector of each near-view pixel according to the relationship [(x1'-x0')*0.5, (y1'-y0')*0.5].

[0029] Among them, the matching pixel of a near-view pixel satisfies: Condition 1, the absolute value of the first difference is less than the absolute value of the second difference, the first difference is the difference between the brightness of a near-view pixel and the brightness of the matching pixel, and the second difference is the difference between the brightness of a near-view pixel and the brightness of other pixels in the second scene image. Condition 2, the first feature distance is less than the second feature distance, the first feature distance is the feature distance between the feature vector of the near-view pixel and the feature vector of the matching pixel, and the second feature distance is the feature distance between the feature vector of the near-view pixel and the feature vector of other pixel coordinates in the second scene image.

[0030] In the above solution, when a pixel is a near-view pixel, the near-view pixel may be a dynamic pixel, and the motion vector of the near-view pixel can be calculated by searching for the best matching pixel.

[0031] In a possible implementation, based on motion vectors of distant pixels and motion vectors of near pixels, the first scene image is warped in units of pixel blocks to obtain a scene prediction image, including: determining pixel coordinates of vertices of each pixel block of a 2D grid model in the scene prediction image after movement based on the motion vectors of distant pixels and motion vectors of near pixels; determining each pixel coordinate in the scene prediction image and texture coordinates corresponding to each pixel coordinate based on linear differences of a grid area composed of pixel coordinates of vertices of each pixel block after movement; and collecting pixel data from the first scene image according to the texture coordinates corresponding to each pixel coordinate, and outputting the collected pixel data to the scene prediction image.

[0032] Exemplarily, a pixel block may be composed of M*M pixels, where M is an integer greater than or equal to 2.

[0033] In the traditional prediction frame generation method based on AI models, AI models are usually loaded on devices with high computing power such as PCs. A complete prediction frame can be obtained by inputting the real frame into the AI ​​model. However, the AI ​​model distorts in units of pixels, and the algorithm is relatively complex, with high requirements for computing and bandwidth, and cannot run in real time on terminal devices with limited computing power such as mobile phones. Different from the traditional AI model, in the above scheme, image distortion is performed in units of pixel blocks, and the depth image and the projection matrix of the game virtual camera are both universal resources that are easy to obtain. Therefore, it can be adapted to most games and terminal devices with limited computing power, which improves the versatility of the prediction frame generation method.

[0034] In a possible implementation, the first rendering instruction stream is obtained before the second rendering instruction stream. After superimposing the user interface image UI0 on the scene prediction image Pixel_Color to obtain the predicted frame, the method may further include: displaying the first frame, the predicted frame, and the second frame in sequence. The first frame is a real image frame generated according to the first rendering instruction stream, and the second frame is a real image frame generated according to the second rendering instruction stream.

[0035] It should be understood that according to the first rendering instruction stream and the second rendering instruction, not only two real frames can be generated, but also a predicted frame can be inserted between the two real frames. Since the computing resource requirement for generating the predicted frame is less than the resource requirement for rendering the real frame by the game application, the resource consumption of the terminal device is reduced.

[0036] In a second aspect, the present application provides a device, the device comprising a unit for executing the method in the first aspect. The device may correspond to executing the prediction frame generation method provided in any one of the first aspects. For the relevant description of the units in the device, please refer to the description of the first aspect above, and for the sake of brevity, it will not be repeated here.

[0037] In a third aspect, the present application provides a terminal device, the terminal device comprising a memory and one or more processors. The memory is used to store computer program code, and the computer program code comprises computer instructions. When the computer instructions are called by the processor, the terminal device executes the prediction frame generation method provided in any one of the first aspects.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium. The computer-readable storage medium includes computer instructions. When the computer instructions are executed on a terminal device, the terminal device executes the prediction frame generation method provided in any one of the first aspects.

[0039] In a fifth aspect, the present application provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the prediction frame generation method provided in any one of the first aspects.

[0040] In a sixth aspect, the present application provides a chip system. The chip system includes one or more interface circuits and one or more processors. The interface circuit and the processor are interconnected by a line. The chip system can be applied to a terminal device including a communication module and a memory. The interface circuit is used to receive a signal from the memory of the terminal device and send the received signal to the processor, the signal including a computer instruction stored in the memory. When the processor calls the computer instruction, the terminal device can execute the prediction frame generation method provided in any one of the first aspects.

[0041] It can be understood that the beneficial effects that can be achieved by the above-mentioned apparatus of the second aspect, the terminal device of the third aspect, the computer-readable storage medium of the fourth aspect, the computer program product of the fifth aspect, and the chip system of the sixth aspect can be referred to as the beneficial effects in the first aspect and any possible implementation method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of the hardware structure of a mobile phone provided in an embodiment of the present application;

[0043] Figure 2 A schematic diagram of the architecture of a terminal device provided in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of a rendering and display process of a game frame provided in an embodiment of the present application;

[0045] Figure 4A A flowchart of a method for generating a prediction frame provided in an embodiment of the present application;

[0046] Figure 4B A schematic diagram of a flow chart of another prediction frame generation method provided in an embodiment of the present application;

[0047] Figure 4C A flowchart of another method for generating a prediction frame provided in an embodiment of the present application;

[0048] Figure 5A A schematic diagram of changing pixel coordinates based on the movement of a virtual object in a game scene provided by an embodiment of the present application;

[0049] Figure 5B A schematic diagram of changing pixel coordinates based on the movement of a virtual camera in a game scene provided by an embodiment of the present application;

[0050] Figure 6A schematic diagram of separating a 3D scene image and a UI image from a rendering instruction stream provided in an embodiment of the present application;

[0051] Figure 7 Schematic diagrams of various coordinate systems provided in embodiments of the present application;

[0052] Figure 8 A schematic diagram of marking distant view pixels and near view pixels provided in an embodiment of the present application;

[0053] Fig. 9 A schematic diagram of a process for calculating a motion vector of a distant pixel provided in an embodiment of the present application;

[0054] Fig.10 A schematic diagram of extracting feature points from a 3D scene image provided by an embodiment of the present application;

[0055] Fig.11 A schematic diagram of obtaining feature point pairs provided in an embodiment of the present application;

[0056] Fig.12 A schematic diagram of obtaining a motion vector of a distant pixel by reprojection provided in an embodiment of the present application;

[0057] Fig.13 A schematic diagram of a process for calculating a motion vector of a near-view pixel provided in an embodiment of the present application;

[0058] Fig.14 A schematic diagram of a 2D grid model provided in an embodiment of the present application;

[0059] Fig.15 A schematic diagram of distorting an image in units of pixel blocks based on a motion vector provided in an embodiment of the present application;

[0060] Fig.16 A schematic diagram of superimposing a UI image onto a 3D scene prediction image provided in an embodiment of the present application;

[0061] Fig.17 A schematic diagram of a terminal device playing a game screen frame by frame provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0063] In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B. In the description of this application, "and / or" is only a way to describe the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0064] The terms "first" and "second" in the specification and claims of this application are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects. For example, the first frame and the first frame are used to distinguish different video frames, rather than to describe a specific order of video frames. In the embodiments of this application, "multiple" refers to two or more.

[0065] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the phrases "in some embodiments" etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0066] To facilitate understanding of this application, some of the terms used in this application are explained below.

[0067] 1. Frame rate refers to the number of frames to be displayed generated per second by the application when it is running. The terminal device usually needs to go through the processes of drawing, rendering and synthesis to generate the frames to be displayed. The unit of frame rate is frames per second (fps). Among them, frame drawing refers to the drawing of the picture of the display interface, frame rendering is to color the drawn picture or add 3D effects, etc., and frame synthesis is the process of synthesizing at least one rendered view into the interface to be displayed.

[0068] 2. Inter-frame prediction technology: It refers to using the correlation of the frame time domain to use the pixels of the adjacent real frames to predict the pixels of the intermediate frame, so as to effectively remove the temporal redundancy. Generally, in the field of video playback, inter-frame prediction technology is relatively mature, for example, weighted mixing of two video frames, or using optical flow to calculate motion vectors for pixel blurring.

[0069] The present application mainly relates to inter-frame prediction in the field of games. Specifically, when a terminal device runs a game application, the frame rate of the game application can reach 60fps or even 100fps, which occupies a large amount of system resources, causing the terminal device to have problems such as fast power consumption, battery heating or system freeze. Studies have shown that the resource consumption of using inter-frame prediction technology to generate predicted frames is less than the resource consumption of game applications to render real frames. Therefore, by reducing the number of real frames generated by game applications and using inter-frame prediction technology to generate predicted frames, the total amount of game frames can be guaranteed, and the power consumption load of the chip can also be reduced.

[0070] However, the clarity and sharpness of images rendered by games are significantly higher than those of videos. In game scenarios, the predicted frames must not only approach or reach the clarity and sharpness of the real frames, but also ensure the correctness of the picture content and prevent deformation of objects in the image. Therefore, the traditional video inter-frame prediction technology used in the field of video playback cannot meet the needs of generating game prediction frames.

[0071] Generally, there are several main categories of frame prediction technologies for game scenes: one category of frame prediction technology is to perform semi-transparent blending of two consecutive frames of images using motion vectors (mv) as weights, but this technology does not analyze the image information distribution characteristics of the game frames, resulting in the final blended predicted frames being prone to image quality problems such as ghosting; another category of frame prediction technology is to use pixels as units, using motion vectors as the amount of motion of pixels, and offsetting each pixel of the image. This technology relies on more game resources, has a high demand for computing power, and cannot be universally adapted to different games.

[0072] In view of the above problems, the present application provides a prediction frame generation method. In this method, the terminal device can obtain the 3D scene image and UI image corresponding to each real frame from two game rendering instructions, and obtain general information such as the depth image and the projection matrix of the game virtual camera according to the game rendering instructions; then use this information to divide the distant pixels and the near pixels in the 3D scene image; then calculate the motion vector of each distant pixel based on the principle of stereoscopic vision, and calculate the motion vector of each near pixel based on the spatial search method; then, according to the motion vectors of each distant pixel and the motion vectors of the near pixel, distort the entire 3D scene image in units of pixel blocks; finally, superimpose the UI image on the distorted 3D scene image to generate the final prediction frame.

[0073] On the one hand, the depth image and the projection matrix of the game virtual camera are included in the game rendering instructions and are relatively easy-to-obtain general resources, so they can be adapted to most games, improving the versatility of the prediction frame generation method. On the other hand, since distant pixels and near pixels are distinguished, and the motion vectors of distant pixels and near pixels are calculated based on their characteristics, after image distortion based on the motion vectors of distant pixels and near pixels, a prediction frame with better image quality can be obtained, that is, the image quality of the prediction frame is improved.

[0074] In some embodiments, the terminal device is also referred to as a terminal or user equipment (UE). For example, the terminal device may be a mobile phone, a smart screen, a smart TV, a tablet computer (Pad), a wearable device, a virtual reality (VR) device, an augmented reality (AR) device, and other terminal devices with limited computing power. It should be understood that the prediction frame generation method provided in the present application can also be applied to terminal devices with sufficient computing power such as PCs. The embodiments of the present application do not impose any restrictions on the specific type of terminal device. Regardless of the type of terminal device to which the prediction frame generation method provided in the present application is applied, the computing resource requirement for generating the prediction frame is less than the resource requirement for the game application to render the real frame, thereby reducing the resource consumption of the terminal device.

[0075] In order to better understand the embodiments of the present application, the structure of the terminal device of the embodiments of the present application is introduced below.

[0076] Take the terminal device as an example, which is a mobile phone. Figure 1 A schematic diagram of the hardware structure of a mobile phone 100 provided in an embodiment of the present application.

[0077] like Figure 1As shown, the mobile phone 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a SIM card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, and a bone conduction sensor 180M, etc.

[0078] The processor 110 may include one or more processing units. For example, the processor 110 may include a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), an image signal processor (ISP), a neural-network processing unit (NPU), a video codec, a modem, and / or a baseband processor.

[0079] The CPU is the final execution unit for information processing and program execution. Its main tasks include processing instructions, executing operations, controlling time, and processing data. The CPU may include a controller, an arithmetic unit, a cache memory, and a bus for connecting these components. The controller is the nerve center and command center of the mobile phone 100. The controller can generate an operation control signal based on the instruction opcode and timing signal to complete the control of fetching and executing instructions. The arithmetic unit refers to a component that performs various arithmetic and logical operations. The cache memory can store instructions or data that the processor 110 has just used or circulated.

[0080] GPU, also known as display core, visual processor or display chip, is a microprocessor for graphics and image processing. GPU connects display screen 194, AP and CPU. GPU can be used to perform complex mathematical and geometric calculations, floating-point operations, parallel calculations, and can also be used for graphics rendering. GPU reduces the graphics card's dependence on the CPU, especially when processing 3D graphics. GPU uses core technologies such as cubic environment material mapping and vertex blending, so GPU performs some of the work originally done by the CPU.

[0081] NPU is a neural network (NN) computing processor, which can quickly process input information and continuously self-learn by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain. NPU can realize applications such as intelligent cognition of the mobile phone 100.

[0082] In the embodiment of the present application, the prediction frame generation method is mainly performed by the GPU. For example, the GPU can obtain the 3D scene image and UI image corresponding to each real frame from two game rendering instructions, as well as general information such as the depth image and the projection matrix of the game virtual camera, and use this information to distinguish between distant pixels and near pixels; then calculate the motion vector of each distant pixel based on the principle of stereoscopic vision, and calculate the motion vector of each near pixel based on the spatial search method; then, according to the motion vector of each pixel, distort the 3D scene image in units of pixel blocks; finally, superimpose the UI image on the distorted 3D scene image to generate the final prediction frame. Please refer to the specific description of the following embodiment, which will not be repeated here.

[0083] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the mobile phone 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application (application, APP) required for at least one function, such as a game application, etc. The data storage area can store configuration files of each APP, as well as data created during the use of the mobile phone 100.

[0084] The mobile phone 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the AP, etc. For example, the background sound of the game, the real person conversation, etc. can be played.

[0085] The display screen 194 includes a signal processing module and a display panel, and is used to display images, videos, etc., such as displaying a game interface.

[0086] It is understandable that the hardware structure illustrated in the embodiment of the present application does not constitute a specific limitation on the mobile phone 100. In other embodiments, the mobile phone 100 may include more or fewer components than shown, or combine some components, or separate some components, or arrange the components differently. Figure 1 The components shown may be implemented in hardware, software or a combination of software and hardware.

[0087] The software system of the terminal device can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, or a micro-service architecture. The embodiment of the present invention takes the Android system of a layered architecture as an example to exemplify the software architecture of the terminal device.

[0088] Figure 2 The schematic diagram of the architecture of the terminal device provided in the embodiment of the present application is shown. Figure 2 As shown, the terminal device can adopt a layered architecture to divide the software into several layers, each with clear roles and division of labor. The layers communicate through software interfaces. In some embodiments, the software layers of the software structure are divided from top to bottom into: application (APP) layer, application framework (FW) layer, system library (FWK LIB) and kernel layer. The above software architecture runs on the hardware layer, which may include GPU, NPU and display screen, etc.

[0089] The application layer, also called the application layer, can include a series of application packages. For example, the application layer can include game applications, camera applications, calendar applications, phone applications, and settings applications. When these application packages are run, they can access various service modules provided by the application framework layer through the application programming interface (API) and execute corresponding intelligent services.

[0090] The application framework layer provides API and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 2 As shown, the application framework layer may include a graphics manager, a video codec, a window manager, a resource manager, and a notification manager. Among them, the graphics manager is responsible for calling the interface in the graphics library to complete drawing, rendering, and synthesizing the interface, the video codec is responsible for video encoding and decoding, the window manager manages all windows in the system, the resource manager is responsible for system resources, and the notification manager is responsible for managing the prompt messages in the status bar at the top of the mobile phone.

[0091] The system library can include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (e.g. OpenGL ES), two-dimensional (2D) graphics engine (e.g. SGL), etc. The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The three-dimensional graphics processing library is used to implement three-dimensional graphics drawing, image rendering, synthesis and layer processing, etc. The 2D graphics engine is a drawing engine for 2D drawing.

[0092] In the system library, Android Runtime includes the core library and the virtual machine. Android Runtime is responsible for the scheduling and management of the Android system. The core library consists of two parts: one is the function that the Java language needs to call, and the other is the core library of Android. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection and other functions.

[0093] The kernel layer is the layer between hardware and software and belongs to the bottom layer of the Android system. The kernel layer can include various driver interfaces, such as GPU driver, NPU driver, CPU driver, and display driver. The kernel layer can also include power management, processes / threads, etc.

[0094] It should be noted that although the embodiments of the present application are described using the Android system as an example, its basic principles are also applicable to terminal devices based on operating systems such as iOS or Windows.

[0095] Figure 3 The rendering and display process of the game frame provided by the embodiment of the present application is shown.

[0096] The game application runs on the operating system of the terminal device. Figure 3 As shown, in the game scenario, the game application modifies the function pointer of the rendering API and generates the rendering instruction stream of the game, such as Hooking the game rendering instruction stream; the rendering instruction stream of the game is successively transmitted to the GPU through the graphics manager and GPU driver, etc.; the GPU obtains the function through the rendering instruction and identifies the rendering instruction stream of the game according to the characteristics of the game rendering instruction.

[0097] The rendering instruction stream is an instruction stream composed of multiple drawing instructions (rendering instruction sequence). Generally, a rendering instruction stream is used to draw a frame of image. Specifically, the GPU can execute each drawing instruction in the rendering instruction stream to realize the drawing of each object in a frame of image. The set of objects generated according to the rendering instruction stream can form a frame of image. A frame of image is a frame of static image displayed by the game application. Each frame of static image can be composed of 3D scene image and UI image, etc., wherein the 3D scene image is also called color image. For example, the 3D scene image may include in-game scenery, game characters, background objects, special effects and skills, etc., and the UI image may include control buttons (such as running buttons and skill buttons, etc.), small maps, floating text, user avatars, energy pack controls and blood bars, etc., and the UI image is located on the upper layer of the 3D scene image. It should be understood that both the game characters in the 3D scene image and the rendering control buttons in the UI image belong to the objects in the game frame. In addition, through the rendering instruction stream, the projection matrix of the virtual camera corresponding to a frame of image and the general information such as the depth image corresponding to a frame of image can also be obtained.

[0098] The game frames in the embodiment of the present application can be divided into two types:

[0099] One type is a real frame generated based on a rendering instruction stream. In a game scenario, the operating system sends rendering instruction streams to the GPU in sequence. The GPU generates a real frame corresponding to each rendering instruction stream in sequence according to the order in which these rendering instruction streams are received. One rendering instruction stream is used to draw one real frame.

[0100] The other type is a predicted frame generated based on two rendering instruction streams obtained successively. "Two rendering instruction streams obtained successively" means that rendering instruction stream 1 is obtained first, and then rendering instruction stream 2 is obtained. After obtaining rendering instruction stream 1 and before obtaining rendering instruction stream 2, the GPU has not obtained other rendering instruction streams. That is, the real frame N-1 obtained by rendering instruction stream 1 and the real frame N+1 obtained by rendering instruction stream 2 are two adjacent real frames. Specifically, the GPU can obtain two rendering instruction streams in sequence, and separate the 3D scene image and UI image corresponding to a real frame from each rendering instruction stream (for example, the scene image Color0 and the user interface image UI0 corresponding to the real frame N-1 are separated from the rendering instruction stream 1, and the scene image Color1 and the user interface image UI1 corresponding to the real frame N+1 are separated from the rendering instruction stream 2), as well as obtain the depth image corresponding to each real frame, the projection matrix of the game virtual camera and other general information, and then use this information to distinguish between distant pixels and near pixels; then calculate the motion vector of each distant pixel based on the principle of stereoscopic vision, and calculate the motion vector of each near pixel based on the spatial search method; then, according to the motion vector of each pixel, the 3D scene image is distorted in units of pixel blocks; finally, the UI image is superimposed on the distorted 3D scene image to generate the final predicted frame N.

[0101] Continue to refer to Figure 3 After the real frame N-1 obtained by the above rendering instruction stream 1, the real frame N+1 obtained by the above rendering instruction stream 2, and the predicted frame N by the rendering instruction stream 1 and the rendering instruction stream 2, the GPU can send these frames to the queue to be displayed in the order of real frame N-1, predicted frame N and real frame N+1. Then, the display screen plays the real frame N-1, predicted frame N and real frame N+1 in sequence.

[0102] Above Figure 3 Only the process of generating a predicted frame using two rendering instruction streams is given. In actual game scenarios, the number of rendering instruction streams may be greater than two, and any two rendering instruction streams obtained successively can refer to the predicted frames generated by the above process. It should be understood that since the resource consumption of using inter-frame prediction technology to generate predicted frames is less than the resource consumption of the game application rendering real frames, by reducing the number of real frames generated by the game application and using inter-frame prediction technology to generate predicted frames, not only the total amount of game frames can be guaranteed, but also the power consumption load of the chip can be reduced.

[0103] Combine the following FIG. 4A to FIG. 17 The prediction frame generation method provided in the embodiment of the present application is illustrated by way of example.

[0104] Figure 4A-4CSchematic diagrams showing three prediction frame generation methods. The difference between these three figures is that: Figure 4A The specific process of this method is shown. Figure 4B The data flow relationship between the various images involved in this method is shown. Figure 4C The specific process of the method and the data flow relationship between each image are shown. Figure 4A-4C The execution subject in the predicted frame generation method shown may be a terminal device, a GPU in the terminal device, or other software or hardware functional modules in the terminal device.

[0105] For example, in a game scenario, the GPU can continuously obtain rendering instruction streams from the game application. A rendering instruction stream indicates the generation of a real frame. Whenever a rendering instruction stream is obtained, such as obtaining rendering instruction stream 1 or obtaining rendering instruction stream 2, the GPU can perform the following steps 1 and 2 for this rendering instruction stream.

[0106] Step 1: The GPU separates the 3D scene image and UI image corresponding to a real frame from the rendering instruction stream.

[0107] The 3D scene image mentioned above refers to a color image that does not contain UI pixels but contains a 3D scene, such as an RGB image.

[0108] For example, the above-mentioned rendering instruction stream may be a Hook game rendering instruction stream. The GPU may identify the various instructions contained in the Hook game rendering instruction stream through the rendering instruction acquisition function according to the characteristics of the game's rendering instruction stream. The Hook game rendering instruction stream generally includes a variety of drawing instructions, such as scene drawing instructions, UI drawing instructions, and image display instructions. Among them, the scene drawing instructions may be used to draw 3D scene images such as scenery, characters, special effects, and skills in the game, the UI drawing instructions may be used to draw UI images such as control buttons, small maps, and floating text, and the image display instructions are used to put the drawn image data into a specified location of the operating system (such as framebuffer0) to complete the frame display.

[0109] The pixel coordinates of the 3D scene image and the pixel coordinates of the UI image are in a one-to-one correspondence, and the UI image is located on the upper layer of the 3D scene image. If the pixel coordinates of an object in the UI image are the same as those of an object in the 3D scene image, the object in the UI image may cause visual occlusion of the object in the 3D scene image. The object in the 3D scene image may change its pixel coordinates based on two reasons: one reason is that the position of objects such as characters, vehicles or birds in the game scene moves, such as Figure 5AAs shown, in the world coordinate system O-XYZ corresponding to the game scene, the initial position of the virtual character is coordinate position 1. When the virtual character runs, it moves to coordinate position 2. In this case, based on the running of the virtual character, the position of the virtual character in the 3D scene image of the real frame N-1 and the 3D scene image of the real frame N+1 moves; another reason is that the virtual camera in the game scene shakes, that is, the perspective of the virtual character corresponding to the game player changes, such as Figure 5B As shown, in the world coordinate system O-XYZ corresponding to the game scene, the position of the virtual character is always located at coordinate position 1 of the world coordinate system O-XYZ, but the virtual camera moves from position P1 to position P2, and the direction of the virtual camera changes. When the position and direction of the virtual camera change, the perspective of the virtual character corresponding to the game player also changes. In this case, the position based on the virtual character in the 3D scene image of the real frame N-1 and the 3D scene image of the real frame N+1 moves.

[0110] When the pixel coordinates of an object in the UI image are the same as those of an object in the 3D scene image, no matter what the above reasons are for changing the pixel coordinates, if the object in the 3D scene image is moved according to steps 3 to 5 below, the object in the UI image will move with the object in the 3D scene image, but in fact the object in the UI image may not move, or the motion vectors of the object in the UI image and the object in the 3D scene image are different, which will result in serious image quality problems in the final predicted frame. Therefore, the GPU needs to separate the 3D scene image and the UI image from the rendering instruction stream.

[0111] Specifically, the GPU can identify whether it is a UI drawing instruction based on the characteristics of the game rendering instruction. For example, the UI drawing instruction may include resource binding instructions, state setting instructions, and drawing instructions, and point the output of these instructions to a newly created image. The newly created image can be used to save UI pixels, so the newly created image can be called a UI image. In addition, the GPU can change the original rendering generation result to a newly created image for saving a 3D scene image that does not contain UI pixels. In this way, the GPU separates the UI image and the 3D scene image corresponding to a real frame from a rendering instruction stream.

[0112] For example, Figure 6As shown, the GPU first obtains the rendering instruction stream 1 for instructing the generation of the real frame N-1, and separates the 3D scene image Color0 and the UI image UI0 of the real frame N-1 from the rendering instruction stream 1. If the UI image UI0 is superimposed on the 3D scene image Color0, a complete real frame N-1 is obtained. Then, the GPU can obtain the rendering instruction stream 2 for instructing the generation of the real frame N+1, and separates the 3D scene image Color1 and the UI image UI1 of the real frame N+1 from the rendering instruction stream 2. If the UI image UI1 is superimposed on the 3D scene image Color1, a complete real frame N+1 is obtained.

[0113] Step 2: The GPU obtains the depth image and the projection matrix of the virtual camera according to the rendering instruction stream.

[0114] The GPU can identify the rendering instruction sequence for model rendering based on the characteristics of the game rendering instructions, and determine the glBindBuffer instruction located before the model rendering drawcall instruction. Among them, the glBindBuffer instruction contains the relevant data of the virtual camera. The GPU can obtain the projection matrix of the virtual camera from the buffer corresponding to the glBindBuffer instruction. In addition, the GPU can also obtain the depth image from the rendering instruction stream.

[0115] It should be noted that the image content of the 3D scene image obtained by the GPU from step 1 and the depth image obtained from step 2 is the same, for example, both images contain people, trees, mountains, etc. The difference between the two images is that the 3D scene image is a multi-channel (such as RGB channel) color image; the depth image is a single-channel image, and the depth image is similar to a grayscale image. Each depth value of the depth image can be used to characterize the distance between the virtual camera and the object, and the depth image reflects the geometric shape of the visible surface of the object. Among them, the RGB channel refers to the R channel, the G channel and the B channel, the R channel represents red, the G channel represents green, and the B channel represents blue. As an example, the intensity value range of the RGB three channels of the 3D scene image and the single channel of the depth image can be 0 to 255. As another example, the intensity value range of the RGB three channels of the 3D scene image and the single channel of the depth image can be normalized to 0.0 to 1.0.

[0116] In order to facilitate the understanding of the projection matrix of the virtual camera, the following Figure 7 This article provides examples of several basic coordinate systems involved in game rendering scenes.

[0117] The pixel coordinate system o-xy is a two-dimensional coordinate system. Usually, images are stored in the form of arrays in terminal devices, and the value of each element (i.e., pixel) in the array represents the brightness of the image point. The origin of the pixel coordinate system o-xy is located in the lower left corner of the image, the x-axis is horizontal to the right, and the y-axis is vertically upward. Each pixel coordinate (x, y) is the number of columns and rows of a pixel in the array, so the pixel coordinate (x, y) is the coordinate in the pixel coordinate system o-xy in pixels. A pixel coordinate (x, y) is used to represent the specific position of a pixel in the image.

[0118] It should be noted that the present application does not specifically limit the establishment method of the pixel coordinate system o-xy, which can be determined according to actual use requirements. For example, the pixel coordinate system o-xy can also be a two-dimensional coordinate system with the center point of the image as the origin, the x-axis horizontally to the right, and the y-axis vertically upward; or, the pixel coordinate system o-xy can also be a two-dimensional coordinate system with the upper left corner of the image as the origin, the x-axis horizontally to the right, and the y-axis vertically downward.

[0119] The normalized device coordinate system (NDC) o-x'y'z' is a three-dimensional coordinate system. In the normalized device coordinate system o-x'y'z', the values ​​of x', y' and z' are in the interval [-1,1], where x' and y' are obtained by normalizing the coordinates (x, y) in the pixel coordinate system o-xy, and z' represents the depth value of the image. Through the viewport transform, the normalized device coordinate system o-x'y'z' can be transformed into the pixel coordinate system o-xy.

[0120] For example, Figure 7 The two-dimensional coordinates (x, y) in the pixel coordinate system o-xy shown and the three-dimensional coordinates x', y', z' in the NDC coordinate system can be converted by the following relationship:

[0121] x',y',z'=[x*2-1,y*2-1,Depth.x];

[0122] x,y=[(x',y')+1.0]*0.5.

[0123] Among them, Depth.x represents the depth value of the corresponding position of the two-dimensional coordinate (x, y) in the depth image.

[0124] Take a frame of 512*512 size image as an example. The pixel coordinates of the lower left corner point in the pixel coordinate system o-xy are (0, 0). After the coordinate transformation, the coordinates in the standardized device coordinate system o-x'y'z' are (-1, -1, z'). The pixel coordinates of the center point in the pixel coordinate system o-xy are (255, 255) (i.e. (0.5, 0.5)). After the coordinate transformation, the coordinates in the standardized device coordinate system o-x'y'z' are (0, 0, z'). The pixel coordinates of the upper right corner point in the pixel coordinate system o-xy are (512, 512) (i.e. (1.0, 1.0)). After the coordinate transformation, the coordinates in the standardized device coordinate system o-x'y'z' are (1, 1, z'). Among them, z' represents the depth value of a pixel point, which can be obtained from the depth image.

[0125] The camera coordinate system O-XYZ is a three-dimensional coordinate system. In the camera coordinate system O-XYZ, the virtual camera is a virtual camera set for the game scene, not a real camera. In the game scene, if there are multiple players roaming in the game world, since each player has a different position and sight direction in the game world, the frame displayed by the terminal device used by each player is also different. Suppose a coordinate system is created with the virtual camera corresponding to a certain player as the origin. This coordinate system can be called the camera coordinate system or the observation space coordinate system. Figure 7 , the origin O of the camera coordinate system is the optical center (projection center) of the virtual camera, the X-axis and Y-axis are parallel to the x-axis and y-axis of the camera plane, the Z-axis is the optical axis of the virtual camera, the Z-axis is perpendicular to the camera plane, and oO is the focal length of the virtual camera. It should be understood that since each player has a different position and sight direction in the game world, the camera coordinate system O-XYZ established with the virtual camera corresponding to each player as the center will also be different.

[0126] The projection matrix, also known as the intrinsic parameter matrix, is used to convert the object coordinates in the camera coordinate system O-XYZ into descriptions in the standardized device coordinate system o-x'y'z'. When rendering, a three-dimensional scene is usually projected onto the camera plane to become a two-dimensional image. Specifically, all vertex data is first converted from the camera coordinate system O-XYZ to the clipping space, and then the converted clipping coordinates are transformed into the standardized device coordinate system o-x'y'z' by dividing by the w component in the clipping coordinate system. Among them, the camera coordinate system O-XYZ is a right-handed coordinate system, and the standardized device coordinate system o-x'y'z' is a left-handed coordinate system.

[0127] Exemplarily, assuming that the projection matrix is ​​a matrix of a general frustum, in which the coordinates of the four vertices of a conical section are represented as (r, t, -n), (r, b, -n), (f, b, -n), (f, t, -n), then the projection matrix can be expressed as:

[0128]

[0129] Assume that the GPU obtains rendering instruction stream 1 and rendering instruction stream 2 in sequence. Among them, rendering instruction stream 1 can be used to generate real frame N-1, and rendering instruction stream 2 can be used to generate real frame N+1. According to the above steps 1 and 2, the GPU can obtain the 3D scene image Color0, UI image UI0, depth image Depth0, and projection matrix ProjectionMatrix0 of the virtual camera corresponding to the real frame N-1, and the 3D scene image Color1, UI image UI1, depth image Depth1, and projection matrix ProjectionMatrix1 of the virtual camera corresponding to the real frame N+1. Then, the GPU can generate the predicted frame N according to these parameters according to the following steps 3 to 7.

[0130] Step 3: The GPU generates a mask image that marks distant pixels and near pixels based on the depth image.

[0131] Exemplarily, the GPU can create two Rlpha8 (R8) single-channel color images, respectively as mask image Mask0 and mask image Mask1. Each pixel in the R8 single channel occupies 8 bits, has only transparency but no color. Mask image Mask0 can be used to save the marking results of distant pixels and near pixels of 3D scene image Color0 corresponding to real frame N-1, and mask image Mask1 can be used to save the marking results of distant pixels and near pixels of 3D scene image Color1 corresponding to real frame N+1.

[0132] It should be noted that the pixel coordinates of the 3D scene image Color0, the 3D scene image Color1, the depth image Depth0, the depth image Depth1, the mask image Mask0 and the mask image Mask1 are in a one-to-one correspondence.

[0133] For each pixel in the depth image Depth0, the GPU can make the following judgments:

[0134] Determine whether the depth value of a pixel coordinate (x, y) in the depth image Depth0 is greater than or equal to the depth threshold. If the depth value of a pixel coordinate (x, y) in the depth image Depth0 is greater than or equal to the depth threshold, then the pixel coordinate (x, y) in the mask image Mask0 is marked as mask = 1.0, which is used to indicate that the pixel at the pixel coordinate (x, y) in the 3D scene image Color0 is a distant pixel.

[0135] If the depth value of a pixel coordinate (x, y) in the depth image Depth0 is less than the depth threshold, the pixel coordinate (x, y) in the mask image Mask0 is marked as mask = 0.0, indicating that the pixel at the pixel coordinate (x, y) in the 3D scene image Color0 is a near-field pixel.

[0136] For each pixel in the depth image Depth1, the GPU can make the following judgments:

[0137] Determine whether the depth value of a pixel coordinate (x, y) in the depth image Depth1 is greater than or equal to the depth threshold. If the depth value of a pixel coordinate (x, y) in the depth image Depth1 is greater than or equal to the depth threshold, the GPU can mark the pixel coordinate (x, y) in the mask image Mask1 as mask = 1.0, indicating that the pixel at the pixel coordinate (x, y) in the 3D scene image Color1 is a distant pixel.

[0138] If the depth value of a pixel coordinate (x, y) in the depth image Depth1 is less than the depth threshold, the GPU can mark the pixel coordinate (x, y) in the mask image Mask1 as mask = 0.0, indicating that the pixel at the pixel coordinate (x, y) in the 3D scene image Color1 is a foreground pixel.

[0139] The above-mentioned depth threshold can be a pre-set empirical value. Taking the intensity value range of the depth image as 0 to 255 as an example, the empirical value can be around 128, for example, the depth threshold is 120, 128, 130 or 136. Taking the intensity value range of the depth image as 0.0 to 1.0 as an example, the empirical value can be around 0.5, for example, the depth threshold is 0.4, 0.5, or 0.6. It should be understood that the depth threshold may also be other values, and this application does not make specific limitations.

[0140] For example, Figure 8 Schematic diagram of marking distant pixels and near pixels is shown. Assume that the intensity value of the depth image ranges from 0 to 255, and the depth threshold is 130. Figure 8As shown, the pixel coordinates in the depth image and the mask image are in a one-to-one correspondence. As an example, the GPU can start from the upper left corner or the lower left corner of the depth image, traverse each pixel coordinate row by row and column by column, and determine whether the depth value of the pixel coordinate is greater than or equal to 130. As another example, the GPU can determine in parallel whether the depth value of each pixel coordinate is greater than or equal to 130. If the depth value of a pixel coordinate is greater than or equal to 130, then the corresponding pixel coordinate in the mask image is marked as mask=1.0. If the depth value of a pixel coordinate is less than 130, then the corresponding pixel coordinate in the mask image is marked as mask=0.0.

[0141] After generating the mask images Mask0 and Mask1 for marking distant pixels and near pixels, the GPU can determine whether the pixel at the corresponding pixel coordinate in the 3D scene image Color0 is a distant pixel or a near pixel according to the mask of each pixel coordinate. If it is a distant pixel, the motion vector of the distant pixel is calculated, that is, the following step 4 is executed. If it is a near pixel, the motion vector of the near pixel is calculated, that is, the following step 5 is executed.

[0142] It should be noted that the embodiment of the present application is described by taking the generation of mask images Mask0 and Mask1 for marking distant pixels and near pixels as an example, which does not limit the present application. As another example, the GPU may also only generate mask image Mask0 for marking distant pixels and near pixels, but not generate mask image Mask1 for marking distant pixels and near pixels. In this case, if the following steps 4 and 5 are executed, the GPU needs to compare each distant pixel / each near pixel in the 3D scene image Color0 with all pixels of the 3D scene image Color1 to calculate the motion vector of the distant pixel / near pixel. The description of the following embodiment can be referred to, and will not be repeated here.

[0143] The embodiment of the present application does not specifically limit the execution of step 4 and step 5. For example, step 4 and step 5 can be executed simultaneously; or, step 4 can be executed first, and then step 5; or, step 5 can be executed first, and then step 4.

[0144] Step 4: For the distant pixels marked by the mask image, the GPU can calculate the motion vector of each distant pixel in the 3D scene image Color0 based on the principle of stereoscopic vision.

[0145] In general, the pixels in the distant area (i.e., distant pixels) are the pixels of static objects (i.e., static pixels), and the motion vectors of static pixels are generated by the movement of the virtual camera. When the motion vector of a pixel is generated by the movement of the virtual camera, the position of the pixel in the world space remains unchanged. In this case, the GPU can estimate the motion information of the virtual camera (such as the rotation matrix and translation matrix of the virtual camera) through the principle of stereo vision, and then calculate the motion vector of each distant pixel in the 3D scene image Color0 based on the motion information of the virtual camera.

[0146] like Fig. 9 As shown, the GPU can perform the following (1) to (10) to calculate the motion vector of the distant pixel.

[0147] (1) Calculate the corner point corner0 in the 3D scene image Color0 and the corner point corner1 in the 3D scene image Color1.

[0148] As an example, for any distant pixel coordinate (x, y) in the 3D scene image Color0 / Color1, such as Fig.10 As shown, the GPU can determine the four pixel coordinates (x-2,y), (x+2,y), (x,y-2), and (x,y+2) located around the distant pixel coordinate (x,y). Then, the brightness Intensity(x,y) of the pixel coordinate (x,y), the brightness Intensity(x-2,y) of the pixel coordinate (x-2,y), the brightness Intensity(x+2,y) of the pixel coordinate (x+2,y), the brightness Intensity(x,y-2) of the pixel coordinate (x,y+2), and the brightness Intensity(x,y+2) of the pixel coordinate (x,y+2) are calculated. Then, calculate whether the absolute value of the difference between the brightness Intensity (x, y) and the brightness Intensity (x-2, y) is greater than or equal to the brightness threshold, whether the absolute value of the difference between the brightness Intensity (x, y) and the brightness Intensity (x+2, y) is greater than or equal to the brightness threshold, whether the absolute value of the difference between the brightness Intensity (x, y) and the brightness Intensity (x, y-2) is greater than or equal to the brightness threshold, and whether the absolute value of the difference between the brightness Intensity (x, y) and the brightness Intensity (x, y+2) is greater than or equal to the brightness threshold. If at least three of these four absolute values ​​are greater than or equal to the brightness threshold, it can be determined that the pixel at the distant pixel coordinate (x, y) is a corner point.

[0149] The brightness threshold may be a preset empirical value.

[0150] For example, the brightness Intensity of a pixel coordinate can be determined using the following relationship:

[0151] Intensity=Color.r*0.299+Color.g*0.587+Color.b*0.114.

[0152] Among them, Color.r is used to represent the color value of the R channel, Color.g is used to represent the color value of the G channel, and Color.b is used to represent the color value of the B channel.

[0153] (2) Select N feature points Special0 from all corner points corner0 of the 3D scene image Color0, and select N feature points Special1 from all corner points corner1 of the 3D scene image Color1.

[0154] Wherein, N can be an integer greater than or equal to 8.

[0155] As an example, Fig.10 As shown, for any corner point corner(x,y) in the 3D scene image Color0 / Color1, the GPU can first determine the four pixel coordinates (x-1,y), (x+1,y), (x,y-1), and (x,y+1) around the corner point (x,y), and then calculate the brightness Intensity(x-1,y), Intensity(x+1,y), Intensity(x,y-1), and Intensity(x,y+1) corresponding to these four pixel coordinates respectively.

[0156] Then, the GPU can calculate the brightness difference Ix in the x-axis direction and the brightness difference Iy in the y-axis direction respectively:

[0157] Ix=Intensity(x+1,y)-Intensity(x-1,y);

[0158] Iy=Intensity(x,y+1)-Intensity(x,y-1).

[0159] The GPU can then calculate the depth difference dx in the x-axis direction and the depth difference dy in the y-axis direction respectively:

[0160] dx=Depth_mip(x+1,y)-Depth_mip(x-1,y);

[0161] dy=Depth_mip(x,y+1)-Depth_mip(x,y-1).

[0162] Then, the GPU can calculate the response function F(corner) using an exponential function based on the brightness difference Ix in the x-axis direction, the brightness difference Iy in the y-axis direction, the depth difference dx in the x-axis direction, and the depth difference dy in the y-axis direction:

[0163] F(corner)=exp(Ix,Iy,dx,dy).

[0164] Among them, exp() is the exponential function.

[0165] After obtaining the response function F0(corner) of each corner point corner0 in the 3D scene image Color0, all response functions F0(corner) are sorted from large to small according to the response function value, and N corner points corner0 corresponding to the largest N response function values ​​are selected as feature points Special0, that is, N feature points Special0 are selected.

[0166] After obtaining the response function F1(corner) of each corner point corner1 in the 3D scene image Color1, all response functions F1(corner) are sorted from large to small according to the response function value, and N corner points corner1 corresponding to the largest N response function values ​​are selected as feature points Special1, that is, N feature points Special1 are selected.

[0167] It should be noted that the above embodiment is described by taking the response function F(corner) of all corner points in a 3D scene image in order from large to small according to the response function value, and selecting N corner points corresponding to the largest N response function values ​​as feature points Special as an example, which does not limit the present application. As another example, the GPU may also determine whether the response function value of the response function F(corner) of a corner point is greater than or equal to the response threshold, and if it is greater than the response threshold, the corner point is taken as a feature point Special.

[0168] (3) Feature point matching is performed on the N feature points Special0 and the N feature points Special1 to obtain a feature point pair set E[(x0, y0), (x1, y1)].

[0169] The above feature point pair set E[(x0,y0),(x1,y1)] consists of multiple feature point pairs. For each feature point pair in the multiple feature point pairs, a feature point pair consists of a pixel coordinate (x0,y0) and a pixel coordinate (x1,y1). Among them, the pixel coordinate (x0,y0) is the pixel coordinate of the feature point in the 3D scene image Color0, and the pixel coordinate (x1,y1) is the pixel coordinate of the feature point in the 3D scene image Color1 that matches the pixel coordinate (x1,y1).

[0170] For example, Fig.11 As shown, the 3D scene image Color0 and the 3D scene image Color1 each include 11 feature points. The feature points of the 3D scene image Color0 and the feature points of the 3D scene image Color1 are in a one-to-one correspondence. For example, feature point 1 and feature point 2 are a set of feature point pairs.

[0171] As an example, for any feature point Special0 in the 3D scene image Color0, the GPU can determine the feature point Special1 corresponding to the feature point Special0 in the following manner: calculate a feature vector SpecialVector0 corresponding to a feature point Special0; calculate a feature vector SpecialVector1 corresponding to each feature point Special1 in the 3D scene image Color1; calculate a feature distance (such as Hamming distance) between a feature vector SpecialVector0 and each feature vector SpecialVector1; use a feature point Special1 corresponding to the minimum feature distance as the feature point Special1 corresponding to the feature point Special0, thereby obtaining a set of pairwise corresponding feature point pairs [(x0, y0), (x1, y1)]. Among them, the pixel coordinates (x0, y0) are the pixel coordinates of the feature point in the 3D scene image Color0, and the pixel coordinates (x1, y1) are the pixel coordinates of the feature point in the 3D scene image Color1 that matches the pixel coordinates (x0, y0).

[0172] (4) Based on the depth image Depth0, the depth image Depth1, the projection matrix ProjectionMatrix0 of the virtual camera, and the projection matrix ProjectionMatrix1 of the virtual camera, calculate the three-dimensional coordinate pair set E[(x0,y0,z0),(x1,y1,z1)] in the camera coordinate system corresponding to the feature point pair set E[(x0,y0),(x1,y1)].

[0173] Exemplarily, the GPU may calculate the three-dimensional coordinates (x0, y0, z0) in the camera coordinate system corresponding to the feature point (x0, y0) using the following relationship:

[0174] (x t0 ,y t0 ,z t0 ,w t0 )=[x0*2-1,y0*2-1,Depth0.x,1.0]*ProjectionMatrix0 -1 ;

[0175]

[0176] Among them, Depth0.x represents the depth value corresponding to the feature point (x0, y0), w t0 is a homogeneous coordinate, and ProjectionMatrix0 is the projection matrix of the virtual camera corresponding to the 3D scene image Color0.

[0177] Exemplarily, the GPU may calculate the three-dimensional coordinates (x1, y1, z1) in the camera coordinate system corresponding to the feature point (x1, y1) using the following relationship:

[0178] (x t1 ,y t1 ,z t1 ,w t1 )=[x1*2-1,y1*2-1,Depth1.x,1.0]*ProjectionMatrix1 -1 ;

[0179]

[0180] Among them, Depth1.x represents the depth value corresponding to the feature point (x1, y1), w t1 is a homogeneous coordinate, and ProjectionMatrix1 is the projection matrix of the virtual camera corresponding to the 3D scene image Color1.

[0181] (5) Use the eight three-dimensional coordinate pairs in the three-dimensional coordinate pair set E[(x0, y0, z0), (x1, y1, z1)] to construct the spatial transformation matrix ViewMatrix.

[0182] The number of three-dimensional coordinate pairs is 8, which is determined based on experience.

[0183] It should be noted that the above embodiment is described by taking 8 three-dimensional coordinate pairs to construct the space transformation matrix ViewMatrix as an example, which does not limit the present application. As an example, the GPU can also construct the space transformation matrix ViewMatrix with more than 8 three-dimensional coordinate pairs. It should be understood that the fewer the number of three-dimensional coordinate pairs used, the lower the GPU calculation amount, and the lower the accuracy and stability of constructing the space transformation matrix ViewMatrix; the more the number of three-dimensional coordinate pairs used, the higher the GPU calculation amount, and the higher the accuracy and stability of constructing the space transformation matrix ViewMatrix.

[0184] (6) Decompose the virtual camera’s rotation matrix Rotation and translation matrix Translation from the spatial transformation matrix ViewMatrix.

[0185] For example, the GPU can use the singular value decomposition method to decompose the rotation matrix Rotation and translation matrix Translation of the virtual camera from the space transformation matrix ViewMatrix. The rotation matrix Rotation and translation matrix Translation are the data volume of the virtual camera transforming from the 3D scene image Color0 to the 3D scene image Color1.

[0186] Further integration Fig.12 For any distant pixel (x0, y0) in the 3D scene image Color0, the GPU can calculate the motion vector of each distant pixel (x0, y0) in parallel through the following (7) to (10).

[0187] (7) Based on the depth image Depth0 and the projection matrix ProjectionMatrix0 of the virtual camera, the three-dimensional coordinates (x0, y0, z0) in the camera coordinate system corresponding to each distant pixel (x0, y0) are calculated.

[0188] For example, the GPU may calculate the three-dimensional coordinates (x0, y0, z0) in the camera coordinate system corresponding to each distant view pixel (x0, y0) using the following relationship:

[0189] (x t0 ,y t0 ,z t0 ,w t0 )=[x0*2-1,y0*2-1,Depth0.x,1.0]*ProjectionMatrix0 -1 ;

[0190]

[0191] Among them, Depth0.x represents the depth value corresponding to the distant pixel (x0, y0), w t0 is a homogeneous coordinate, and ProjectionMatrix0 is the projection matrix of the virtual camera corresponding to the 3D scene image Color0.

[0192] (8) Based on the rotation matrix Rotation, the translation matrix Translation and the projection matrix ProjectionMatrix1, the three-dimensional coordinates (x0, y0, z0) in the camera coordinate system are reprojected to the pixel coordinate system to obtain the reprojected pixel coordinates (x1, y1).

[0193] It should be noted that in the game scene, the projection matrix is ​​the intrinsic matrix of the virtual camera and is generally fixed. Therefore, the projection matrix ProjectionMatrix0 and the projection matrix ProjectionMatrix1 are the same.

[0194] Exemplarily, the GPU may calculate the reprojected coordinates (x1, y1) corresponding to the three-dimensional coordinates (x0, y0, z0) in the camera coordinate system using the following relationship:

[0195] (mv t.x ,mv t.y ,mv t.z ,mv t.w )=[(x0,y0,z0)*Rotation+Translation]*ProjectionMatrix1;

[0196]

[0197] Among them, mv t.w is a homogeneous coordinate, and ProjectionMatrix1 is the projection matrix of the virtual camera corresponding to the 3D scene image Color1.

[0198] (9) Based on the distant pixel (x0, y0) and the reprojected coordinates (x1, y1), calculate the motion vector (mv.x, mv.y) of the distant pixel (x0, y0).

[0199] Exemplarily, the GPU may calculate the motion vector (mv.x, mv.y) using the following relationship:

[0200] (mv.x,mv.y)=[(x1-x0)*0.5, (y1-y0)*0.5].

[0201] For example, assuming that the pixel coordinates of a distant pixel in the 3D scene image Color0 are (x0, y0) = (255, 255), and the pixel coordinates of the best matching pixel in the 3D scene image Color1 are (x1, y1) = (261, 263), then the motion vector of the distant pixel is: [(261-255)*0.5, (263-255)*0.5] = [3, 4].

[0202] (10) Output the motion vector (mv.x, mv.y) to the pixel coordinates (x0, y0) of the image mv0, where the image mv0 is used to store the motion vector corresponding to each distant pixel coordinate.

[0203] After the motion vectors corresponding to the coordinates of each distant pixel are obtained by parallel calculation, the image mv0 has stored the motion vectors of all distant pixels in the 3D scene image Color0.

[0204] Step 5: For the near-field pixels marked by the mask image, the GPU can calculate the motion vector of each near-field pixel in the 3D scene image Color0 according to the spatial search algorithm.

[0205] If the pixel mark of a pixel (x, y) in the mask image Mask0 is 0.0, it means that the pixel at the pixel coordinate (x, y) in the 3D scene image Color0 is a foreground pixel, and the GPU can calculate the motion vector of the foreground pixel by searching for the best matching pixel (ie, iterative method).

[0206] like Fig.13 As shown, the GPU can execute the following ① to ⑨.

[0207] ① Calculate the brightness Intensity0 corresponding to the pixel coordinates (x0, y0) of the foreground pixel in the 3D scene image Color0, and the feature vector Special0 corresponding to the foreground pixel. The feature vector Special0 is used to characterize the geometric features of the pixel at the pixel coordinates (x0, y0) in the 3D scene image Color0.

[0208] ② Calculate the brightness Intensity1 corresponding to the pixel coordinates (x1, y1) of the foreground pixel in the 3D scene image Color1, and the feature vector Special1 corresponding to the foreground pixel. The feature vector Special1 is used to characterize the geometric features of the pixel at the pixel coordinates (x1, y1) in the 3D scene image Color1.

[0209] It should be noted that the feature vectors Special0 and Special1 can be used to characterize the geometric features of the pixel at the pixel coordinates in the 3D scene image, which are pixels with obvious features in the 3D scene image, such as table corners, wall corners, and clothing patterns. Usually, the main parameters of the feature vector are the color and depth of the current pixel and the surrounding adjacent pixels within a certain range.

[0210] After obtaining the brightness Intensity0 and the feature vector Special0 corresponding to the pixel coordinates (x0, y0) of the foreground pixel in the 3D scene image Color0, and the brightness Intensity1 and the feature vector Special1 corresponding to the pixel coordinates (x1, y1) of the foreground pixel in the 3D scene image Color1, the GPU can calculate in parallel the motion vector of each foreground pixel in the 3D scene image Color0 according to the following ③ to ⑨.

[0211] The following is an example of how the motion vector of a near-view pixel (x0, y0) in the 3D scene image Color0 is calculated through the following ③ to ⑨.

[0212] ③ Obtain the first absolute value, and determine whether the first absolute value is less than or equal to the second absolute value.

[0213] The first absolute value is the absolute value of the difference between the brightness Intensity0 and Intensity1 in the current round of judgment process, and the second absolute value is the absolute value of the difference between the brightness Intensity0 and Intensity1 in each round of judgment process before the current round of judgment process.

[0214] If the first absolute value is less than or equal to the second absolute value, then execute the following ④.

[0215] If the first absolute value is greater than the second absolute value, then execute the following ⑥.

[0216] ④ Obtain the first characteristic distance, and determine whether the first characteristic distance is less than or equal to the second characteristic distance.

[0217] The first characteristic distance is the characteristic distance between the characteristic vector Special0 and the characteristic vector Special1 in the current round of judgment process, and the second characteristic distance is the characteristic distance between the characteristic vector Special0 and the characteristic vector Special1 in each round of judgment process before the current round of judgment process.

[0218] If the first characteristic distance is less than or equal to the second characteristic distance, then execute the following ⑤.

[0219] If the first characteristic distance is greater than the second characteristic distance, then execute the following ⑥.

[0220] ⑤ Mark the pixel coordinates (x1, y1) in this round of judgment process. It should be understood that the pixel coordinates (x1, y1) marked in this round of judgment process are the best matching pixels obtained so far.

[0221] ⑥ Detect whether there are any undetermined near-field pixels in the 3D scene image Color1.

[0222] If there are still undetermined near-view pixels in the 3D scene image Color1, then return to the above ③.

[0223] If there are no undetermined near-view pixels in the 3D scene image Color1, then execute the following ⑦.

[0224] ⑦ Take the pixel at the last marked pixel coordinate (x1, y1) as the best matching pixel.

[0225] It should be understood that by iteratively searching through each pixel in the 3D scene image Color1, a pixel coordinate (x1, y1) with the smallest absolute value of (Intensity0–Intensity1) and the smallest characteristic distance between the feature vector Special0 and the feature vector Special1 can be found. The pixel coordinate (x1, y1) has the highest matching degree with the pixel coordinate (x0, y0), and the pixel at the pixel coordinate (x1, y1) is the best matching pixel of the pixel coordinate (x0, y0).

[0226] ⑧ Use [(x1-x0)*0.5,(y1-y0)*0.5] to calculate the motion vector corresponding to the pixel coordinates (x0,y0).

[0227] For example, assuming that the pixel coordinates of a foreground pixel in the 3D scene image Color0 are (x0, y0) = (255, 257), and the pixel coordinates of the best matching pixel in the 3D scene image Color1 are (x1, y1) = (259, 239), then the motion vector of the foreground pixel is: [(259-255)*0.5, (239-257)*0.5] = [2, -9].

[0228] ⑨ Output the motion vector to the pixel coordinates (x0, y0) of the image mv1, where the image mv1 is used to store the motion vector corresponding to each near-view pixel coordinate.

[0229] After the motion vectors corresponding to the coordinates of each near-view pixel are obtained by parallel calculation, the image mv1 has stored the motion vectors of all near-view pixels in the 3D scene image Color0.

[0230] As an example, after obtaining image mv0 according to step 4 and image mv1 according to step 5, the GPU may also merge image mv0 and image mv1 into a motion vector image mv. The motion vector image mv may be used for image distortion in step 6 below.

[0231] Step 6: The GPU warps the 3D scene image Color0 in units of pixel blocks according to the motion vectors of each pixel, and outputs the warped result to the 3D scene prediction image Pixel_Color.

[0232] The above “motion vector of each pixel” includes the motion vector of each distant pixel in the 3D scene image Color0 obtained in step 4 , and the motion vector of each near pixel in the 3D scene image Color0 obtained in step 5 .

[0233] For example, the GPU may create a new 3D scene prediction image Pixel_Color. The 3D scene prediction image Pixel_Color may be an RGBA four-channel image. Then, the GPU may perform image distortion on the 3D scene image Color0 in units of pixel blocks, and output the distortion result of the 3D scene image Color0 to the 3D scene prediction image Pixel_Color.

[0234] The above pixel block may be composed of multiple pixel points.

[0235] For example, a pixel block may be composed of M*M pixels, where M is an integer greater than or equal to 2. For example, M=8.

[0236] The present application provides a 2D grid model. The 2D grid model may be an offline grid model manually constructed.

[0237] As an example, Fig.14A 2D grid model is shown. The 2D grid model consists of p*q rectangular grids, where p represents the number of rows of the rectangular grid in the 2D grid model, and q represents the number of columns of the rectangular grid in the 2D grid model. For each rectangular grid, a rectangular grid represents a pixel block, for example, composed of 8*8 pixels. In other words, the 2D grid model consists of 8p*8q pixels. It should be noted that the pixel coordinates of the 2D grid model, the pixel coordinates of the 3D scene image Color0, and the pixel coordinates in the image mv are in a one-to-one correspondence. Assume that the pixel coordinates of the four vertices of a rectangular grid are (0,0), (7,0), (0,7), (7,7), respectively, and the motion vectors of the pixel coordinates of the four vertices determined by the above step 4 or step 5 are mv1=(1,1), mv2=(1,01), mv3=(0,1), mv4=(2,1), respectively. The GPU can move the pixel coordinates of the four vertices according to the motion vectors corresponding to the pixel coordinates of each vertex, thereby obtaining the pixel coordinates of the four new vertices, which are (1,1), (8,0), (0,6), and (9,8) in sequence. The GPU can connect the pixel coordinates of the four new vertices in sequence to obtain the following: Fig.14 The new grid shown in the dotted box is then rasterized, and the linear difference is used to obtain the texture coordinates corresponding to each pixel coordinate in the new grid. Since the texture coordinates (ux,uy) of each vertex before and after the movement remain unchanged, the pixel data before the movement can be assigned to the pixel coordinates after the movement according to the texture coordinates (ux,uy).

[0238] It should be noted that Fig.14 Only a schematic diagram of image distortion of a rectangular grid composed of 8*8 pixels in a 2D grid model is shown. It should be understood that for any rectangular grid of the 2D grid model, the pixel coordinates of the four vertices of the rectangular grid can be moved to perform image distortion, referring to the description in the previous paragraph. After the distortion of all rectangular grids is completed, a new edge image is obtained.

[0239] Combine the following Fig.15 The specific process of performing image distortion on the 3D scene image Color0 in units of pixel blocks and outputting the distortion result to the 3D scene prediction image Pixel_Color is described by way of example.

[0240] Fig.15 A schematic diagram showing how to warp an image in units of pixel blocks based on motion vectors.

[0241] like Fig.15As shown, the GPU can move each vertex according to the motion vector mv corresponding to each vertex vertex in the 2D mesh model to form a distorted new mesh. Afterwards, the GPU can rasterize each new mesh and linearly interpolate the pixel coordinates of each pixel in the new mesh. Each pixel coordinate is used to determine the position of a pixel in the 3D scene prediction image Pixel_Color. Assuming that the pixel coordinates of a pixel before moving are (vx, vy), and the pixel coordinates after moving according to the motion vector (mv.x, mv.y) are (vx_n, vy_n), the following relationship exists:

[0242] (vx_n,vy_n)=(vx+mv.x,vy+mv.y).

[0243] Since the texture coordinates (ux,uy) of a pixel point remain unchanged before and after the movement, that is, (ux_n,uy_n) = (ux,uy), the GPU can collect pixel data from the pixel coordinates (vx,vy) of the 3D scene image Color0 according to the texture coordinates (ux,uy) corresponding to the pixel coordinates (vx_n,vy_n) after the movement, and output the collected pixel data to the pixel coordinates (vx_n,vy_n) of the 3D scene prediction image Pixel_Color, and finally obtain the 3D scene prediction image Pixel_Color that saves the distortion result of the 3D scene image Color0.

[0244] In the traditional prediction frame generation method based on artificial intelligence (AI) model, the AI ​​model is usually loaded on a device with high computing power such as PC. A complete prediction frame can be obtained by inputting the real frame into the AI ​​model. However, the AI ​​model is distorted in units of pixels, and the algorithm is relatively complex, with high requirements for computing and bandwidth, and cannot be run in real time on terminal devices with limited computing power such as mobile phones. Different from the traditional AI model, the prediction frame generation method provided in this application performs image distortion in units of pixel blocks, and the depth image and the projection matrix of the game virtual camera are relatively easy to obtain universal resources, so it can be adapted to most games and terminal devices with limited computing power, which improves the versatility of the prediction frame generation method.

[0245] Step 7: GPU superimposes the UI image UI0 of the real frame N-1 obtained in step 1 onto the 3D scene prediction image Pixel_Color obtained in step 6 to generate the final prediction frame image Pixel_Final, i.e., prediction frame N.

[0246] For example, Fig.16As shown, the GPU can perform alpha blending (ie, A channel blending) of the UI image UI0 of the real frame N-1 generated by the above step 1 and the 3D scene prediction image Pixel_Color generated by the above step 6 to generate the final prediction frame image Pixel_Final, that is, the prediction frame N.

[0247] Among them, the rules of alpha blending can be:

[0248] Pixel_Final.rgb=Pixel_Color.rgb*(1.0–Pixel_UI.a)+Pixel_UI.rgb*Pixel_UI.a;

[0249] Pixel_Final.a=1.0.

[0250] Among them, Pixel_Final.rgb is the RGB three-channel pixel of the predicted frame image Pixel_Final, Pixel_Final.a is the A channel pixel of the predicted frame image Pixel_Final, that is, Pixel_Final.rgba is the RGBA four-channel pixel of the predicted frame image Pixel_Final.

[0251] Pixel_Color.rgb is the RGB three-channel pixel of the 3D scene prediction image Pixel_Color, Pixel_Final.a is the A channel pixel of the 3D scene prediction image Pixel_Color, that is, Pixel_Color.rgba is the RGBA four-channel pixel of the 3D scene prediction image Pixel_Color.

[0252] Pixel_UI.rgb is the RGB three-channel pixel of UI image UI0 of real frame N-1, and Pixel_UI.a is the A channel pixel of UI image UI0 of real frame N-1, that is, Pixel_UI.rgba is the RGBA four-channel pixel of UI image UI0 of real frame N-1. It should be noted that Pixel_UI.a can be obtained from the rendering instruction stream of real frame N-1.

[0253] It should be understood that the transparency effect of the UI interface can be achieved through alpha blending. The degree of transparency can be controlled by Pixel_UI.a. When Pixel_UI.a=0 of a pixel point of the UI interface, it means that the pixel point is completely transparent and will not block the 3D scene prediction image Pixel_Color located at the lower layer of the UI interface; and when Pixel_UI.a=1 of a pixel point of the UI interface, it means that the pixel point is completely opaque and will block the 3D scene prediction image Pixel_Color located at the lower layer of the UI interface.

[0254] Combined with the description of the above embodiment, the GPU can generate the real frame N-1 according to the rendering instruction stream 1 obtained first, generate the real frame N+1 according to the rendering instruction stream 2 obtained later, and generate the predicted frame N according to the rendering instruction stream 1 and the rendering instruction stream 2. The GPU can send these frames to the queue to be displayed in the order of the real frame N-1, the predicted frame N and the real frame N+1. Then, as Fig.17 As shown, the display screen of the terminal device plays the real frame N-1, the predicted frame N and the real frame N+1 in sequence.

[0255] In the above-mentioned prediction frame generation method provided in the present application, on the one hand, the depth image and the projection matrix of the game virtual camera are included in the game rendering instructions, which are universal resources that are relatively easy to obtain and can be adapted to most games, thereby improving the versatility of the prediction frame generation method; on the other hand, since distant pixels and near-field pixels are distinguished, and the motion vectors of distant pixels and near-field pixels are calculated respectively based on the characteristics of distant pixels and near-field pixels, after image distortion is performed based on the motion vectors of distant pixels and near-field pixels, a prediction frame with better image quality is obtained, that is, the image quality of the prediction frame is improved.

[0256] An embodiment of the present application also provides a terminal device, including a processor, the processor is coupled to a memory, and the processor is used to execute a computer program or instruction stored in the memory, so that the terminal device implements the methods in the above embodiments.

[0257] The embodiment of the present application also provides a computer-readable storage medium, in which a computer instruction is stored. When the computer-readable storage medium is run on a computer, the computer is caused to execute the method shown above. The computer instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instruction can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that can be integrated with one or more media. Available media can be magnetic media (e.g., floppy disk, hard disk or tape), optical media or semiconductor media (e.g., solid state drive (SSD)), etc.

[0258] An embodiment of the present application further provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer executes the methods in the above embodiments.

[0259] The embodiment of the present application also provides a chip, which is coupled to a memory, and the chip is used to read and execute a computer program or instruction stored in the memory to perform the methods in the above embodiments. The chip can be a general-purpose processor or a dedicated processor. It should be noted that the chip can be implemented using the following circuits or devices: one or more field programmable gate arrays (field programmable gate array, FPGA), programmable logic devices (programmable logic device, PLD), controllers, state machines, gate logic, discrete hardware components, any other suitable circuits, or any combination of circuits capable of performing the various functions described throughout this application.

[0260] The terminal device, computer-readable storage medium, computer program product and chip provided in the above-mentioned embodiments of the present application are all used to execute the method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects corresponding to the method provided above, and will not be repeated here.

[0261] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0262] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0263] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A prediction frame generation method, characterized in that: The method comprises: Obtaining a user interface image, a first scene image, a first depth image, and a first projection matrix from a first rendering instruction stream; Acquire a second scene image, a second depth image, and a second projection matrix from a second rendering instruction stream; Determining distant view pixels and near view pixels in the first scene image according to the first depth image; Determine a motion vector of each distant view pixel in the first scene image according to the first depth image, the first projection matrix, the second scene image, the second depth image, and the second projection matrix; determining, according to the second scene image, a motion vector of each near-view pixel in the first scene image; Based on the motion vectors of the distant view pixels and the motion vectors of the near view pixels, warping the first scene image in units of pixel blocks to obtain a scene prediction image; The user interface image is superimposed on the scene prediction image to obtain a prediction frame.

2. The method according to claim 1, characterized in that The determining, according to the first depth image, distant view pixels and near view pixels in the first scene image includes: Determine whether a depth value of a pixel coordinate in the first depth image is greater than or equal to a depth threshold; If the depth value of a pixel coordinate in the first depth image is greater than or equal to the depth threshold, marking the pixel coordinate in the mask image as a distant view pixel; or, If the depth value of a pixel coordinate in the first depth image is less than the depth threshold, the pixel coordinate in the mask image is marked as a near-view pixel.

3. The method according to claim 1, characterized in that The method further comprises: According to the second depth image, distant view pixels and near view pixels in the second scene image are determined.

4. The method according to claim 3, characterized in that The determining, according to the second depth image, distant view pixels and near view pixels in the second scene image comprises: Determine whether a depth value of a pixel coordinate in the second depth image is greater than or equal to a depth threshold; If the depth value of a pixel coordinate in the second depth image is greater than or equal to the depth threshold, marking the pixel coordinate in the mask image as a distant view pixel; or, If the depth value of a pixel coordinate in the second depth image is less than the depth threshold, the pixel coordinate in the mask image is marked as a near-view pixel.

5. The method according to any one of claims 1 to 4, characterized in that The determining, according to the first depth image, the first projection matrix, the second scene image, the second depth image, and the second projection matrix, a motion vector of each distant view pixel in the first scene image comprises: Acquire a feature point pair set, the feature point pair set consisting of a plurality of feature point pairs, each feature point pair consisting of a distant view pixel in the first scene image and a distant view pixel in the second scene image; Determine a set of three-dimensional coordinate pairs in a camera coordinate system corresponding to the set of feature point pairs according to the first depth image, the first projection matrix, the second depth image, and the second projection matrix; Using the three-dimensional coordinate pair set, obtaining a rotation matrix and a translation matrix of a virtual camera; A motion vector of each distant view pixel in the first scene image is determined according to the rotation matrix and the translation matrix.

6. The method according to claim 5, characterized in that The step of obtaining a feature point pair set includes: Determine a plurality of first corner points in distant view pixels of the first scene image, and select N first feature points from the plurality of first corner points; Determine a plurality of second corner points in distant pixels of the second scene image, and select N second feature points from the plurality of second corner points; For each first feature point among the N first feature points, a second feature point is selected from the N second feature points as a best matching point of a first feature point to obtain the feature point pair set.

7. The method according to claim 6, characterized in that The three-dimensional coordinate pair set consists of N three-dimensional coordinate pairs; the determining, according to the first depth image, the first projection matrix, the second depth image and the second projection matrix, a three-dimensional coordinate pair set in a camera coordinate system corresponding to the feature point pair set includes: Based on the first depth image and the first projection matrix, convert the pixel coordinates of the N first feature points into three-dimensional coordinates in the camera coordinate system; Based on the second depth image and the second projection matrix, pixel coordinates of the N second feature points are converted into three-dimensional coordinates in the camera coordinate system.

8. The method according to any one of claims 5 to 7, characterized in that The step of obtaining the rotation matrix and translation matrix of the virtual camera by using the set of three-dimensional coordinate pairs includes: Constructing a spatial transformation matrix using the three-dimensional coordinate pair set; The rotation matrix and the translation matrix are decomposed from the spatial transformation matrix.

9. The method according to any one of claims 5 to 7, characterized in that The step of determining the motion vector of each distant view pixel in the first scene image according to the rotation matrix and the translation matrix includes: Determine, according to the first depth image and the first projection matrix, a three-dimensional coordinate in the camera coordinate system corresponding to each distant view pixel of the first scene image; Based on the rotation matrix, the translation matrix and the second projection matrix, reprojecting the three-dimensional coordinates in the camera coordinate system corresponding to each distant pixel of the first scene image to a pixel coordinate system to obtain reprojected coordinates; A motion vector of each distant view pixel is determined according to coordinates of each distant view pixel of the first scene image and reprojection coordinates corresponding to the coordinates of each distant view pixel.

10. The method according to claim 9, characterized in that The step of determining the motion vector of each distant view pixel according to the coordinates of each distant view pixel of the first scene image and the reprojection coordinates corresponding to the coordinates of each distant view pixel comprises: Determine the motion vector of each distant view pixel according to the relationship [(x1-x0)*0.5, (y1-y0)*0.5]; Wherein, (x0, y0) represents a distant pixel coordinate of the first scene image, and (x1, y1) represents the reprojection coordinate corresponding to the distant pixel coordinate.

11. The method according to any one of claims 1 to 9, characterized in that The step of determining the motion vector of each near-view pixel in the first scene image according to the second scene image includes: Obtaining the brightness and feature vector of each near-view pixel in the first scene image; For each near-view pixel in the first scene image, searching for a matching pixel in the second scene image according to the brightness and feature vector of a near-view pixel; According to each distant view pixel of the first scene image and the matching pixels corresponding to each distant view pixel, a motion vector of each distant view pixel is determined.

12. The method according to claim 11, characterized in that The step of determining the motion vector of each distant view pixel according to each distant view pixel of the first scene image and the matching pixels corresponding to each distant view pixel comprises: Determine the motion vector of each near-view pixel according to the relationship [(x1'-x0')*0.5, (y1'-y0')*0.5]; Wherein, (x0', y0') represents a near-view pixel coordinate of the first scene image, and (x1', y1') represents a matching pixel coordinate corresponding to the near-view pixel coordinate.

13. The method according to claim 11 or 12, characterized in that: The matching pixels satisfy the following conditions: The absolute value of the first difference is smaller than the absolute value of the second difference, the first difference is the difference between the brightness of a foreground pixel and the brightness of the matching pixel, and the second difference is the difference between the brightness of the foreground pixel and the brightness of other pixels in the second scene image; and The first feature distance is smaller than the second feature distance, the first feature distance being the feature distance between the feature vector of the foreground pixel and the feature vector of the matching pixel, and the second feature distance being the feature distance between the feature vector of the foreground pixel and the feature vectors of other pixel coordinates in the second scene image.

14. The method according to any one of claims 1 to 13, characterized in that The step of distorting the first scene image in units of pixel blocks based on the motion vectors of the distant view pixels and the motion vectors of the near view pixels to obtain a scene prediction image includes: Determining pixel coordinates of vertices of each pixel block of the 2D mesh model after movement in the scene prediction image based on the motion vector of the distant pixel and the motion vector of the near pixel; Performing linear interpolation on a grid area composed of pixel coordinates after the vertices of each pixel block are moved, to determine each pixel coordinate in the scene prediction image and a texture coordinate corresponding to each pixel coordinate; According to the texture coordinates corresponding to the respective pixel coordinates, pixel data is collected from the first scene image, and the collected pixel data is output to the scene prediction image.

15. The method according to claim 14, characterized in that The pixel block consists of M*M pixels, where M is an integer greater than or equal to 2.

16. The method according to any one of claims 1 to 15, characterized in that The first rendering instruction stream is obtained before the second rendering instruction stream; After superimposing the user interface image on the scene prediction image to obtain a prediction frame, the method further includes: Displaying the first frame, the predicted frame and the second frame in sequence; The first frame is a real image frame generated according to the first rendering instruction stream, and the second frame is a real image frame generated according to the second rendering instruction stream.

17. A terminal device, characterized in that: It comprises a processor and a memory coupled to the processor; wherein the memory stores instructions, and the processor calls the instructions so that the terminal device executes the prediction frame generation method as described in any one of claims 1 to 16.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on a terminal device, the terminal device executes the prediction frame generation method according to any one of claims 1 to 16.

Images