Image display method and device for virtual scene, medium and program product

Abstract

The invention discloses an image display method and device for a virtual scene, a medium and a program product, and belongs to the technical field of virtual world. The method comprises the following steps: acquiring a first scene depth texture map and a first scene color texture map of a first image frame; based on the first scene depth texture map, obtaining a first spatial position of a vertex of the target triangular surface in a first clipping space; mapping the first spatial position to a second spatial position in a second clipping space based on the first camera parameter and a current second camera parameter; and generating a second scene depth texture map and a second scene color texture map of the second image frame based on the second spatial position and the first scene color texture map. According to the scheme, the display frame rate of the virtual scene can be improved, so that the display effect of the virtual scene is improved.

Description

Technical Field

[0001] The present application relates to the field of virtual world technology, and in particular to an image display method, device, medium and program product for a virtual scene. Background Art

[0002] In an application program including a virtual scene, elements or objects in the virtual scene are usually rendered into a two-dimensional image, and the rendered two-dimensional image is displayed on a screen.

[0003] In the related art, the display effect of the virtual scene is related to the frame rate of image rendering. The higher the frame rate, the better the display effect of the virtual scene. Conversely, the lower the frame rate, the worse the display effect of the virtual scene.

[0004] However, when the display frame rate of the virtual scene is high, a higher terminal processing capacity is required. Due to the limitation of the terminal's processing performance, many terminals cannot support high frame rate display, resulting in limited display effect of the virtual scene. Summary of the invention

[0005] Each embodiment of the present application provides an image display method, device, medium and program product for a virtual scene, and the technical solution is as follows:

[0006] According to one aspect of the present application, there is provided an image display method for a virtual scene, the method comprising:

[0007] Acquire a first scene depth texture map and a first scene color texture map of a first image frame; the first image frame is an image frame generated by the most recent rendering of the virtual scene;

[0008] Based on the first scene depth texture map, obtaining a first spatial position of a vertex of a target triangular face in a first clipping space; the target triangular face is a triangular face of a target scene object in the first image frame; the first clipping space is a clipping space corresponding to a first camera parameter of the first image frame;

[0009] Based on the first camera parameter and the current second camera parameter, mapping the first spatial position to a second spatial position in a second cropping space; the second cropping space is a cropping space corresponding to the second camera parameter;

[0010] Based on the second spatial position and the first scene color texture map, generating a second scene depth texture map and a second scene color texture map of a second image frame;

[0011] The second image frame is displayed based on the second scene depth texture map and the second scene color texture map.

[0012] According to another aspect of the present application, there is provided an image display device for a virtual scene, the device comprising:

[0013] A first texture acquisition module, used to acquire a first scene depth texture map and a first scene color texture map of a first image frame; the first image frame is an image frame generated by the most recent rendering of the virtual scene;

[0014] A first position acquisition module, configured to acquire a first spatial position of a vertex of a target triangular face in a first clipping space based on the first scene depth texture map; the target triangular face is a triangular face of a target scene object in the first image frame; the first clipping space is a clipping space corresponding to a first camera parameter of the first image frame;

[0015] a second position acquisition module, configured to map the first spatial position to a second spatial position in a second cropping space based on the first camera parameter and a current second camera parameter; the second cropping space is a cropping space corresponding to the second camera parameter;

[0016] A second texture acquisition module, configured to generate a second scene depth texture map and a second scene color texture map of a second image frame based on the second spatial position and the first scene color texture map;

[0017] A display module is used to display the second image frame based on the second scene depth texture map and the second scene color texture map.

[0018] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the above-mentioned image display method for a virtual scene.

[0019] On the other hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, and the computer program is loaded and executed by a processor to implement the above-mentioned image display method for a virtual scene.

[0020] In another aspect, a computer program product is provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the image display method for a virtual scene provided in the above-mentioned various optional implementations.

[0021] The beneficial effects of the technical solution provided by this application include at least:

[0022] After drawing the first image frame of the virtual scene, the first scene depth texture map and the first scene color texture map of the first image frame can be obtained, and then based on the first scene depth texture map, the first spatial position of the vertices of the triangular surface of the target scene object in the first image frame in the first clipping space is obtained, and based on the first spatial position, the first camera parameter corresponding to the first image frame, and the current second camera parameter, the first spatial position is mapped to the second spatial position, and then based on the second spatial position and the first scene color texture map, the second scene depth texture map and the second scene color texture map of the second image frame are generated to display the second image frame; in the above scheme, after drawing the first image frame of the virtual scene, for the target scene object in the first image frame, the pixel position, depth and color corresponding to the target scene object in the next frame are obtained by spatial mapping and camera parameter prediction, and the target scene object does not need to be redrawn and rendered, thereby greatly reducing the workload of rendering the next image frame, thereby reducing the processing resource occupation of image display, and improving the display frame rate of the virtual scene, thereby improving the display effect of the virtual scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a structural block diagram of a computer system provided by an exemplary embodiment of the present application;

[0024] Figure 2 is a flow chart of an image display method for a virtual scene provided by the present application;

[0025] Figure 3 It is a framework diagram of an image prediction process involved in this application;

[0026] Figure 4 is a flow chart of an image display method for a virtual scene provided by the present application;

[0027] Figure 5 is a schematic diagram of a screen space aggregate mesh in scene depth involved in this application;

[0028] Figure 6 is a schematic diagram of a screen space aggregate grid in scene color involved in this application;

[0029] Figure 7 is a flow chart of an image display method for a virtual scene provided by the present application;

[0030] Figure 8 This is a diagram of the implementation framework of the frame prediction involved in this application;

[0031] Fig. 9 It is a schematic diagram of the process of outputting vertex information;

[0032] Fig.10It is a schematic diagram of the process of outputting the spatial position and UV coordinates of pixels;

[0033] Fig.11 It is the process of outputting depth and color;

[0034] Fig.12 It is the scene color map output by frame prediction;

[0035] Fig.13 It is the timing diagram of the upper screen;

[0036] Fig.14 This is the basic structure diagram of the "paired rendering pipeline";

[0037] Fig.15 This is another basic structural diagram of "paired rendering pipeline";

[0038] Fig.16 This is a basic diagram of the rendering pipeline that interpolates intermediate frames in the rendering thread;

[0039] Fig.17 is a structural block diagram of an image display device for a virtual scene provided by the present application;

[0040] Fig.18 It is a structural block diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0042] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0043] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0044] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the object behaviors such as attack operations involved in this application are all obtained with full authorization.

[0045] It should be understood that although the terms first, second, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first parameter may also be referred to as the second parameter, and similarly, the second parameter may also be referred to as the first parameter. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0046] For ease of understanding, several terms involved in this application are explained below.

[0047] 1) Virtual scene: A virtual scene is a virtual scene displayed (or provided) when an application is running on a terminal. The virtual scene can be a simulated environment scene of the real world, a semi-simulated and semi-fictitious three-dimensional environment scene, or a purely fictitious three-dimensional environment scene. The virtual scene can be any one of a two-dimensional virtual scene, a 2.5-dimensional virtual scene, and a three-dimensional virtual scene. The following embodiments are illustrated by taking the virtual scene as a three-dimensional virtual scene, but are not limited to this. Optionally, the virtual scene can also be used for a virtual scene battle between at least two virtual characters. Optionally, the virtual scene can also be used for a battle between at least two virtual characters using virtual props. Optionally, the virtual scene can also be used for a battle between at least two virtual characters using virtual props within a target area, and the target area will continue to shrink as time passes in the virtual scene.

[0048] The virtual scene is usually generated by an application in a computer device such as a terminal and displayed based on the hardware in the terminal (such as a screen). The terminal can be a mobile terminal such as a smart phone, tablet computer or e-book reader; or, the terminal can also be a personal computer device such as a laptop or a fixed computer.

[0049] 2) Rendering pipeline: refers to a series of processing steps that convert three-dimensional scene data into two-dimensional images in computer graphics and game development.

[0050] 3) Screen: refers to the process in which the computer displays the rendering results on a display device (such as a monitor, projector, etc.) after completing all the rendering of a frame so that it can be viewed by the user.

[0051] 4) Vertical synchronization: After the software calls the screen display command, it still takes a certain amount of time to render the result to the display device. Therefore, vertical synchronization will block the software from calling the screen display command again before the display device completes the rendering, and put the software into a waiting state until the current rendering is completed, to ensure that the next time the screen display command takes effect, the current screen rendering has been completed. Due to the existence of the vertical synchronization mechanism, the interval between each frame will inevitably not be less than the waiting time for the device's vertical synchronization.

[0052] 5) Homogeneous Coordinates: Homogeneous coordinates are a method of representing coordinates used in geometry, computer graphics, and many other fields. In a homogeneous coordinate system, a point in n-dimensional space is represented by a point in n+1-dimensional space. For example, in three-dimensional space, a point is usually represented by three coordinates [x, y, z] T In homogeneous coordinates, it is represented as four coordinates [x, y, z, w] T , where w are homogeneous coordinates. Their main advantage is that they allow us to represent and manipulate affine transformations (such as translation, rotation, scaling) and perspective projections using matrices. In non-homogeneous coordinates, translations cannot be represented by matrix multiplication, which makes the combination and application of transformations more complicated. In homogeneous coordinates, all of these transformations can be represented by matrix multiplication, which makes the combination and application of transformations much simpler. In this article, all coordinates that use transformation matrices to transform the coordinate space are homogeneous coordinates.

[0053] 6) Model matrix (M matrix): This matrix describes the position, direction and size of the 3D model in the world and is responsible for converting coordinates from its local space (also called object space) to world space. In other words, a coordinate value in local space multiplied by the M matrix can obtain the corresponding coordinate in world space.

[0054] 7) View matrix (V matrix): This matrix describes the position and direction of the camera (or observer) in the world and is responsible for converting coordinates from world space to view space (also called camera space). In other words, a coordinate value in world space multiplied by the V matrix can obtain the corresponding coordinate in view space.

[0055] 8) Projection matrix (P matrix): This matrix is ​​responsible for defining how to map the 3D world to the 2D screen, converting coordinates from the view space to the screen's clipping space. In other words, a coordinate value in the view space is multiplied by the P matrix to obtain the corresponding coordinate in the clipping space.

[0056] 9) Scene Color: refers to the color information of each pixel in the frame buffer after the objects in the scene are drawn during the rendering process.

[0057] 10) Scene Depth: refers to the depth information of each pixel in the frame buffer after the objects in the scene are drawn during the rendering process, that is, the distance information from the pixel to the camera.

[0058] 11) Compute Shader Pass: refers to the process of calling the GPU through the compute shader to perform the operations specified in the shader and output the results.

[0059] 12) Mesh Pass: refers to the process in which the CPU draws mesh data into a series of pixels through vertex shading, rasterization, pixel shading, depth testing and other processes.

[0060] 13) UV coordinates: In computer graphics, UV coordinates are a method used to represent two-dimensional texture coordinates. In the UV coordinate system, the U axis represents the horizontal direction and the V axis represents the vertical direction. The range of UV coordinates is usually between 0 and 1, where (0,0) represents the upper left corner of the texture and (1,1) represents the lower right corner of the texture.

[0061] Figure 1 The structure block diagram of a computer system provided by an exemplary embodiment of the present application is shown. The computer system 100 includes: a first terminal 110, a server 120, and a second terminal 130.

[0062] The first terminal 110 is installed and runs a client 111 that supports virtual scenes, and the client 111 may be a multiplayer online battle program. When the first terminal runs the client 111, the user interface of the client 111 is displayed on the screen of the first terminal 110.

[0063] The second terminal 130 is installed and runs a client 131 that supports virtual scenes, and the client 131 may be a multiplayer online battle program. When the second terminal 130 runs the client 131, a user interface of the client 131 is displayed on the screen of the second terminal 130.

[0064] Optionally, the client installed on the first terminal 110 and the second terminal 130 is the same, or the client installed on the two terminals is the same type of client on different operating system platforms (Android or IOS). The first terminal 110 may refer to one of a plurality of terminals, and the second terminal 130 may refer to another of a plurality of terminals. This embodiment is only illustrated by taking the first terminal 110 and the second terminal 130 as examples. The device types of the first terminal 110 and the second terminal 130 are the same or different, and the device type includes at least one of: a smart phone, a tablet computer, an e-book reader, an MP3 player, an MP4 player, a laptop portable computer, and a desktop computer.

[0065] Figure 1 Only two terminals are shown, but in different embodiments, there are multiple other terminals 140 that can access the server 120. Optionally, there are one or more terminals 140 corresponding to the developer, and a development and editing platform for the client supporting the virtual scene is installed on the terminal 140. The developer can edit and update the client on the terminal 140, and transmit the updated client installation package to the server 120 via a wired or wireless network. The first terminal 110 and the second terminal 130 can download the client installation package from the server 120 to update the client.

[0066] The first terminal 110 , the second terminal 130 , and other terminals 140 are connected to the server 120 via a wireless network or a wired network.

[0067] The server 120 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. The server 120 is used to provide background services for clients that support three-dimensional virtual scenes. Optionally, the server 120 undertakes the main computing work, and the terminal undertakes the secondary computing work; or, the server 120 undertakes the secondary computing work, and the terminal undertakes the main computing work; or, the server 120 and the terminal adopt a distributed computing architecture for collaborative computing.

[0068] In an illustrative example, the server 120 includes a processor 122, a user account database 123, a battle service module 124, and a user-oriented input / output interface (I / O interface) 125. The processor 122 is used to load instructions stored in the server 120 and process data in the user account database 123 and the battle service module 124; the user account database 123 is used to store data of user accounts used by the first terminal 110, the second terminal 130, and other terminals 140, such as the user account's avatar, the user account's nickname, the user account's combat power index, and the service area where the user account is located; the battle service module 124 is used to provide multiple battle rooms for users to fight, such as 1V1 battle, 3V3 battle, 5V5 battle, etc.; the user-oriented I / O interface 125 is used to establish communication and exchange data with the first terminal 110 and / or the second terminal 130 through a wireless network or a wired network.

[0069] Figure 2 A flowchart of an image display method for a virtual scene provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device. The computer device can be Figure 1 The first terminal 110 and the second terminal 130 in the system shown in the figure, or the computer device can also be Figure 1 The server 120 in the system shown, or the computer device may also include the above Figure 1 In the system shown, a first terminal 110, a server 120 and a second terminal 130; the method includes:

[0070] Step 210: Acquire a first scene depth texture map and a first scene color texture map of a first image frame; the first image frame is an image frame generated by the most recent rendering of the virtual scene.

[0071] In the embodiment of the present application, the rendering of the virtual scene refers to the process of mapping the scene objects in the virtual scene from a three-dimensional model to a two-dimensional image by rendering. In other words, the first image frame is a two-dimensional image of the virtual scene generated entirely by rendering.

[0072] For example, the first image frame is an image frame generated after the computer device sequentially rasterizes and colors the triangular faces of the model that are not blocked in the viewing cone and outputs the color (Color) and depth (Depth) to the frame buffer (Frame Buffer).

[0073] In an embodiment of the present application, when a computer device is displaying a scene image of a virtual scene, when it is necessary to generate an image frame by prediction, it can use the image frame generated by the most recent rendering (i.e., the first image frame) to generate the next image frame to be displayed by prediction through the image frame; wherein, the above-mentioned generation of the next image frame to be displayed by prediction means that part or all of the image frame is generated by prediction through the previous first image frame. The "prediction" process here can refer to the subsequent steps.

[0074] For the first image frame of the virtual scene, when the image frame is displayed on the screen, the scene depth texture map (SceneDepth) and the scene color texture map (SceneColor) of the image frame are required; wherein the scene depth texture map contains the depth corresponding to each pixel in the image frame of the virtual scene, and the relationship between the depth and the pixel in the scene depth texture map is associated by the UV coordinates of the pixel; for example, the scene depth texture map contains the depth corresponding to the UV coordinates of each pixel.

[0075] Correspondingly, the scene color texture map contains the color corresponding to each pixel in the image frame of the virtual scene. The relationship between the color value and the pixel in the scene color texture map is associated by the UV coordinates of the pixel; for example, the scene color texture map contains the color value corresponding to the UV coordinate of each pixel.

[0076] Step 220: Based on the first scene depth texture map, obtain the first spatial position of the vertices of the target triangle in the first clipping space; the target triangle is the triangle of the target scene object in the first image frame; the first clipping space is the clipping space corresponding to the first camera parameter of the first image frame.

[0077] In a virtual scene, the model of each scene object can be surrounded by a number of triangular faces. The points on the three corners of each triangular face can be called the vertices of the triangular face.

[0078] For ease of understanding, the following introduces several spaces involved in the rendering process:

[0079] 1) Object space: also known as model space / local space / modeling space, it is the coordinate system that defines the triangular faces of the model. Each model has its own coordinate system. For example, the geometric center of the model is the origin of this coordinate system. The position attributes of the vertices of the triangular faces in this model are all relative to the coordinate values ​​of this origin. This coordinate space can be called object space.

[0080] 2) World space: When placing a model into a virtual scene (such as a game world), a model matrix needs to be applied to the model. This matrix moves, rotates, and scales the model. The transformed model is placed in the world space at a specific angle, size, and position. There is no correlation between the object spaces of different objects, but the world space can provide a reference for the positions of different objects in a scene.

[0081] 3) Observation space: also known as perspective space, the coordinate system of this space is based on the camera; for example, with the position of the camera as the origin, the camera faces the positive direction of the Z axis, the right side is the positive direction of the X axis, and the top is the positive direction of the Y axis to establish a coordinate system, the coordinate space corresponding to this coordinate system is the observation space; the rendering pipeline is a series of processing processes that convert three-dimensional scene data into two-dimensional images, and in the world space, in addition to the model, there is also a camera that observes the model. At this time, defining the above observation space can correctly obtain the image observed by the camera, and it can be easily calculated.

[0082] 4) Clipping space: Clipping space is the space where the clipping operation is performed after the model of the world space is projected onto the corresponding projection surface. Since the models of the scene objects in the entire virtual scene are in the observation space, and the camera has a certain visible range, this range can be called a cone. The objects inside the cone are visible to the camera (can be displayed on the screen), while the objects outside the cone are invisible. In order to distinguish which objects are visible and which objects are invisible, a perspective projection (Perspective) matrix is ​​applied to the vertices of the model in the observation space, and the vertices of the model can be transformed into a new space, which is the clipping space. The above perspective matrix calculation process can be performed by the vertex shader. The output of the vertex shader is the coordinate value of the vertex in the clipping space. If a vertex is in the cone, the value range of the xyz component of the vertex is [-w, w]. If the absolute value of any component in xyz exceeds w, the vertex will be clipped (will not appear on the screen). Among them, the value of w is specified as 1 before the transformation, and the value after the transformation corresponds to the depth before the transformation.

[0083] 5) Standardized device space: After the vertex shader performs the matrix transformation, it can perform perspective division, that is, divide the three components of the vertex xyz by the w component, and then convert it to the standardized device space. All coordinates are normalized and fall within the range of [-1, 1].

[0084] 6) Screen space: For the coordinates in the standardized device space, the viewport transformation matrix is ​​applied to translate and scale these coordinates to the screen space. The screen space can be regarded as a coordinate system with (0, 0) as the origin, the upward direction as the positive direction of the y-axis, and the right direction as the positive direction of the x-axis. Its coordinate range is determined by the resolution of the screen.

[0085] In an embodiment of the present application, the coordinates of pixels in the scene depth texture map and the scene color texture map can be obtained by matrix transformation of the coordinates in the clipping space. Correspondingly, for the coordinates of the vertices of the triangular surface in the first scene depth texture map, the spatial coordinates of the vertex in the first clipping space corresponding to the first camera parameters (i.e., the above-mentioned first spatial coordinates) can be reversely derived.

[0086] The first camera parameters corresponding to the first image frame may also be referred to as camera parameters at the moment of rendering and displaying the first image frame.

[0087] Step 230: Based on the first camera parameter and the current second camera parameter, map the first spatial position to a second spatial position in a second cropping space; the second cropping space is a cropping space corresponding to the second camera parameter.

[0088] For two different moments, the parameters of the camera observing the virtual scene will usually change (move, rotate, etc.). Assuming that the position of the target scene object in the virtual scene remains unchanged, when the spatial coordinates of the vertices of the triangular surface of the target scene object at the previous moment in the clipping space at the previous moment are known, the spatial coordinates of the vertices of the triangular surface in the clipping space at the current moment can be derived (also called mapped) by combining the camera parameters of the previous moment and the current moment.

[0089] Step 240: Generate a second scene depth texture map and a second scene color texture map of a second image frame based on the second spatial position and the first scene color texture map.

[0090] When the spatial coordinates of the vertices of the triangular surface on the target scene object in the virtual scene at the current moment in the clipping space are derived, the depth of each pixel on the triangular surface at the current moment can be determined through the spatial coordinates of the vertices of the triangular surface on the target scene object in the clipping space at the current moment, and the color value of the pixel at the previous moment is assigned to the current moment, so as to obtain the second scene depth texture map and the second scene color texture map at the current moment.

[0091] Step 250: Display a second image frame based on the second scene depth texture map and the second scene color texture map.

[0092] After obtaining the second scene depth texture map and the second scene color texture map at the current moment, the second image frame at the current moment can be displayed on the screen.

[0093] Since the second image frame is an image frame predicted based on the most recent rendering frame, the second image frame can also be called a predicted frame. That is, part or all of the information in the second image frame is obtained by predicting the previous rendering frame, rather than directly obtained from the model mapping of the scene object in the virtual scene.

[0094] In summary, the scheme shown in the embodiment of the present application, after drawing the first image frame of the virtual scene, the first scene depth texture map and the first scene color texture map of the first image frame can be obtained, and then based on the first scene depth texture map, the first spatial position of the vertex of the triangle surface of the target scene object in the first image frame in the first clipping space is obtained, and based on the first spatial position, the first camera parameter corresponding to the first image frame, and the current second camera parameter, the first spatial position is mapped to the second spatial position, and then based on the second spatial position and the first scene color texture map, the second scene depth texture map and the second scene color texture map of the second image frame are generated to display the second image frame; in the above scheme, after drawing the first image frame of the virtual scene, for the target scene object in the first image frame, the pixel position, depth and color corresponding to the target scene object in the next frame are obtained by spatial mapping and camera parameter prediction, and the target scene object does not need to be redrawn and rendered, thereby greatly reducing the workload of rendering the next image frame, thereby reducing the processing resource occupancy of the image display, and improving the display frame rate of the virtual scene, thereby improving the display effect of the virtual scene.

[0095] This application Figure 2 The illustrated embodiment provides a scheme for obtaining a predicted frame by predicting a rendered frame, thereby realizing inserting a predicted frame between rendered frames to improve the frame rate. Since all or part of the information in the predicted frame can be directly obtained through the prediction of the previous frame, the complexity of predictive frame generation can be greatly reduced, thereby realizing an improvement in the frame rate while reducing the occupation of processing resources.

[0096] For example, please refer to Figure 3 , which shows a framework diagram of an image prediction process involved in this application. Figure 3As shown, in the process of displaying the scene image of the virtual scene 31, at the first moment, the scene elements 31b and 31c in the virtual scene that are within the field of view of the camera 31a are mapped into a two-dimensional image through the rendering pipeline to obtain a first scene depth texture map 32 and a first scene color texture map 33, and then the first image frame at the first moment is displayed according to the first scene depth texture map 32 and the first scene color texture map 33.

[0097] At a second moment after the first moment, the camera parameters of camera 31a change. At this time, the scene elements in the virtual scene within the field of view of camera 31a are not directly mapped into a new two-dimensional image through the changed camera parameters, but based on the first scene depth texture map 32, the vertices of the triangular faces in the target scene elements (such as scene element 31b and scene element 31c) are predicted to have a first spatial position in the clipping space 34 corresponding to the camera parameters at the first moment, and then, by combining the camera parameters at the first moment and the camera parameters at the second moment, the first spatial position can be reprojected to the second spatial position in the clipping space 35 corresponding to the camera parameters at the second moment through reprojection, thereby obtaining the second scene depth texture map 36 at the second moment; and then, by combining the second spatial position and the first scene color texture map 33, the second scene color texture map 37 at the second moment can be obtained, and then the second image frame at the second moment is displayed based on the second scene depth texture map 36 and the second scene color texture map 37.

[0098] Among them, in the second image frame, at least the information corresponding to the target scene element is predicted by the information in the first image frame that has been rendered before. Compared with the method of directly rendering from the virtual scene, the above scheme can reduce the resource usage in the process of generating the second image frame.

[0099] Based on the above Figure 2 For the examples shown, please refer to Figure 4 , which shows a flowchart of an image display method for a virtual scene provided by an exemplary embodiment of the present application. Figure 4 As shown, the above step 220 can be implemented as step 220a, step 220b and step 220c:

[0100] Step 220a: For each image block corresponding to the target scene object in the first scene depth texture map, determine the pixel point with the largest gradient change in the image block to obtain the pixel point in the first scene depth texture map corresponding to the vertex of the target triangle.

[0101] Since the models of scene objects in the virtual scene are surrounded by triangular surfaces, the depth change between two adjacent points on the same triangular surface will be smoother, while the depth change between two adjacent points belonging to different triangular surfaces will be more abrupt. This situation can be reflected by the gradient change values ​​of adjacent pixels in the scene depth texture map.

[0102] Based on the above principles, in an embodiment of the present application, for an existing first scene depth texture map, the first scene depth texture map can be divided into multiple image blocks, and it is assumed that in the first scene depth texture map, each image block in the area corresponding to the target scene object has a vertex of a triangular surface on the target scene object. At this time, for the pixel point with the largest gradient change in the above image block, it can be considered that the pixel point corresponds to the vertex of the triangular surface on the target scene object.

[0103] Step 220b: based on the pixel points in the first scene depth texture map corresponding to the vertices of the target triangle, obtain the vertex space positions and vertex UV coordinates of the vertices of the target triangle in the first clipping space.

[0104] After determining the pixel points corresponding to the vertices of the triangular faces on the target scene object in each image block in the area corresponding to the target scene object in the first scene depth texture map, since the first scene depth texture map contains the depth and UV coordinates of the pixel points, the vertex space positions and vertex UV coordinates of the vertices of the triangular faces on the target scene object in the first clipping space can be determined.

[0105] In some embodiments, based on the pixel points in the first scene depth texture map corresponding to the vertices of the target triangular surface, obtaining the vertex space positions and vertex UV coordinates corresponding to the vertices of the target triangular surface in the first clipping space includes:

[0106] The spatial positions and UV coordinates of the pixels in the first scene depth texture map corresponding to the vertices of the target triangle are obtained as the spatial positions and UV coordinates of the vertices in the first clipping space corresponding to the vertices of the target triangle.

[0107] In a possible implementation of the present application, a computer device can directly use the pixel points corresponding to the vertices of the triangular faces on the target scene object in each image block in the area corresponding to the target scene object in the first scene depth texture map as the vertices of the triangular faces on the target scene object, and use their spatial positions and UV coordinates as the vertex spatial positions and vertex UV coordinates. This solution has a simple execution process, consumes less resources, and can ensure the efficiency of image prediction.

[0108] In some embodiments, based on the first camera parameter and the current second camera parameter, mapping the first spatial position to the second spatial position in the second cropping space includes:

[0109] Get the foreground pixel points and background pixel points corresponding to the vertices of the target triangle in the first scene depth texture map, where the foreground pixel points are the pixel points with the closest depth among the pixel points surrounding the position of the vertices of the target triangle in the first scene depth texture map, and the background pixel points are the pixel points with the farthest depth among the pixel points surrounding the position of the vertices of the target triangle in the first scene depth texture map;

[0110] Based on the first camera parameter and the second camera parameter, the spatial position of the foreground pixel point in the first cropping space and the spatial position of the background pixel point in the first cropping space are respectively mapped to the second cropping space to obtain the spatial position of the foreground pixel point in the second cropping space and the spatial position of the background pixel point in the second cropping space;

[0111] In response to the distance between the spatial position of the foreground pixel point in the second clipping space and the spatial position of the background pixel point in the second clipping space being greater than a distance threshold, the spatial position of the foreground pixel point in the second clipping space and the UV coordinates of the background pixel point are used as the vertex spatial positions and vertex UV coordinates corresponding to the vertices of the target triangle in the second clipping space;

[0112] In response to the spatial position of the foreground pixel in the second clipping space, the distance between the spatial position of the foreground pixel in the second clipping space and the spatial position of the background pixel in the second clipping space is not greater than a distance threshold, the spatial position and UV coordinate of the foreground pixel in the second clipping space are used as the vertex spatial positions and vertex UV coordinates of the vertices of the target triangle surface corresponding to the vertex spatial positions and vertex UV coordinates in the second clipping space.

[0113] If the vertex is located at the edge of the object and the background, since the movement speed of the foreground object pixels is much faster than that of the background pixels when the perspective camera moves horizontally, stretching and aliasing may be caused if the vertex position coordinates and UV are used directly. In this regard, in an embodiment of the present application, based on the vertices of the target triangle corresponding to the pixel points in the first scene depth texture map, in the process of obtaining the vertex spatial positions and vertex UV coordinates of the vertices of the target triangle corresponding to the vertices in the second clipping space, the two pixel points with the nearest and farthest depths can be selected from the pixel points corresponding to the vertices in the first scene depth texture map and the multiple pixel points around them, respectively as the foreground and background pixel points, and then the two pixel points are projected to the second clipping space by reprojection. Specifically, the spatial positions (spatial coordinates, such as homogeneous coordinates) of the two pixel points in the first clipping space are multiplied by the reprojection transformation matrix to obtain the spatial coordinates of the two pixel points in the second clipping space. Position, wherein the reprojection transformation matrix is ​​determined by the first camera parameters and the second camera parameters; then, the spatial positions of the two pixel points after reprojection are compared. If the spatial positions after reprojection are very different, it is considered that the two foreground and background pixel points belong to different objects. The spatial position of the foreground pixel point in the second clipping space and the UV coordinate of the background pixel point corresponding to the second clipping space are taken as the vertex spatial position and vertex UV coordinate of the corresponding vertex. Otherwise, the spatial position of the foreground pixel point in the second clipping space and the UV coordinate corresponding to the second clipping space are taken as the corresponding vertex spatial position and vertex UV coordinate, thereby suppressing the stretching and aliasing of the edge position of the target scene object and improving the accuracy of image prediction.

[0114] Among them, through the above scheme, the vertices in the image block are located by the pixel points with the largest gradient change in the image block of the scene depth texture map, so as to determine the vertex spatial positions and vertex UV coordinates of the vertices of the triangular surface on the target scene object in the first clipping space, and then determine the spatial positions of each point on the entire triangular surface in the first clipping space, thereby realizing the reverse deduction of the spatial positions of the triangular surfaces of the target scene object in the clipping space through the scene depth texture map.

[0115] In some embodiments, determining the pixel point with the largest gradient change in the image block to obtain the vertex of the target triangle surface corresponding to the pixel point in the first scene depth texture map also includes:

[0116] Obtain the sum of the squares of the gradient differences of each pixel in the image block on the two coordinate components as the gradient change value at each pixel in the image block;

[0117] The two coordinate components are two components in the UV coordinate system of the first scene depth texture map.

[0118] The solution shown in the embodiment of the present application provides a method for accurately representing the change of pixel depth by using the sum of the squares of the gradient differences of the pixel on two coordinate components as the gradient change value at the pixel, thereby ensuring the accuracy of vertex prediction.

[0119] In some embodiments, obtaining the square sum of the gradient difference values ​​of each pixel in the image block on two coordinate components as the gradient change value at each pixel in the image block includes:

[0120] For each pixel in the image block, the sum of squares of the gradient differences of the pixel on two coordinate components is calculated based on the difference between the depth of the pixel to the right of the pixel and the depth of the pixel, and the difference between the depth of the pixel below the pixel and the depth of the pixel.

[0121] In an embodiment of the present application, a computer device may take the square of the difference between the depth of the pixel to the right of the pixel and the depth of the pixel, as well as the square of the difference between the depth of the pixel below the pixel and the depth of the pixel, and take the sum of the two squares as the sum of the squares of the gradient differences of the pixel on the two coordinate components, thereby reducing the consumption of computing resources and ensuring the efficiency of vertex prediction while ensuring the accuracy of the predicted vertices.

[0122] In the solution provided in the embodiment of the present application, the cached scene color (SceneColor) and scene depth (SceneDepth) of the k-1th frame and the camera motion parameters of the kth frame can be used to generate the scene color (SceneColor) and scene depth (SceneDepth) drawing results of the kth frame.

[0123] In the normal process of drawing scene objects, the unobstructed model triangles within the viewing cone will be rasterized, colored, and the color (Color) and depth (Depth) will be output to the frame buffer (Frame Buffer) in sequence; in the frame prediction implemented in this application, the scene color (SceneColor) and scene depth (SceneDepth) after the drawing is completed will be used to reversely restore all the triangles of the static object appearing on the screen. In order to facilitate the connection of vertices into a Mesh and make the vertices more evenly distributed, this application allocates a vertex in each pixel block (Tile) composed of n×n pixels in the scene depth texture map (SceneDepth Texture), and the vertex is located at the pixel where the vertex is most likely to exist in the pixel block (Tile). This application uses a first-order nonlinear operator ——u is the horizontal coordinate of the scene depth texture map (SceneDepthTexture), and v is the vertical coordinate——that is, the sum of the squares of the components of the depth map gradient is used to measure the change in the depth field gradient at each pixel in the SceneDepth depth map: the result of this operator is a scalar, which can be easily compared in size; and the square can make the comparison unaffected by the positive and negative gradients, so that only the rate of change of the gradient is considered. After using the operator to find the pixel with the largest change in the sum of squared gradients in the n×n pixel block (Tile), it can be assumed that the vertex is located at the location of this pixel. After that, the vertices are connected to each other into triangles, and all the triangles that appear on the screen can be restored. This application calls the set of all triangles restored on the screen a "screen space aggregated mesh." Please refer to Figure 5 and Figure 6 , which shows a schematic diagram of the screen space aggregated mesh in scene depth involved in the present application, and a schematic diagram of the screen space aggregated mesh in scene color.

[0124] After restoring the screen space aggregate mesh, according to the VP matrix of the kth frame (that is, the product of the V matrix of the kth frame and the P matrix, which is usually determined by the camera parameters of the game engine, denoted as P k V k ) and the VP matrix of the k-1th frame (denoted as P k-1 V k-1 , whose inverse matrix is ), the following reprojection formula can be used to convert the triangle vertices of the k-1th frame into homogeneous coordinates in the clip space. Reproject to the homogeneous coordinates of the k-th frame spatial position Since the reprojected triangle vertices all come from stationary objects, the product of the M matrix of the object in the k-1th frame and the kth frame is the identity matrix (i.e. ), and the VP matrix is ​​related to the camera, that is to say, the reprojection result of the vertices of the stationary object according to this method is accurate. In the embodiment of the present application, it can be assumed that the target scene object in the virtual scene is a stationary object. It is worth mentioning that the clip space position and its corresponding screen space UV coordinate can be converted to each other through perspective division and viewport transformation. As shown in the following formula:

[0125]

[0126] in, The matrix product is the reprojection transformation matrix, and the clip space position homogeneous coordinates of the k-1th frame are Multiplying by the reprojection transformation matrix gives the homogeneous coordinates of the k-th frame's spatial position ——The process of transforming the coordinates of the k-1th frame to the coordinates of the kth frame through matrix multiplication is called "reprojection"; correspondingly, the process of transforming the coordinates of the kth frame to the coordinates of the k-1th frame by multiplying the inverse matrix of the reprojection transformation matrix is ​​called "inverse reprojection".

[0127] Based on the above Figure 2 or Figure 4 For the examples shown, please refer to Figure 7 , which shows a flowchart of an image display method for a virtual scene provided by an exemplary embodiment of the present application. Figure 7 As shown, the above step 240 can be implemented as step 240a and step 240b:

[0128] Step 240a: Based on the second spatial position, obtain the screen space coordinates of each pixel in the target triangle surface in the first image frame, the screen space coordinates include UV coordinates and depth values, and map the screen space coordinates of each pixel in the target triangle surface in the first image frame to the screen space coordinates of each pixel in the target triangle surface in the second image frame to obtain a second scene depth texture map.

[0129] In an embodiment of the present application, a computer device may draw all triangular faces composed of vertices in a previous second clipping space to a frame buffer, the triangular faces will be rasterized into pixels in a second image frame, and the UV coordinates of the first image frame recorded on the vertices will also be interpolated to each pixel; wherein, a depth value is sampled from the first depth color texture map using the UV coordinates of the first image frame recorded by the vertices interpolated to the pixels, and the UV coordinates are combined with the depth value into screen space coordinates of the first image frame, and the first clipping space coordinates are obtained after scaling, and are mapped to the second image frame to obtain second clipping space coordinates, and the second screen space coordinates are obtained after scaling, and the z component thereof is the depth value of the pixel in the second image frame, and the second depth texture map can be obtained after all second pixel depth values ​​are cached into the texture.

[0130] Step 240b: based on the UV coordinates of each pixel of the target triangle surface corresponding to the first image frame, sample the color value from the first scene color texture map to obtain a second scene color texture map.

[0131] In an embodiment of the present application, after obtaining the UV coordinates of each pixel of the target triangle surface corresponding to the first image frame, the computer device can sample the color values ​​of each pixel of the target triangle surface from the first scene color texture map, thereby obtaining the color values ​​of each pixel of the target triangle surface corresponding to the second scene color texture map.

[0132] In some embodiments, based on the UV coordinates of each pixel of the target triangle corresponding to the first image frame, sampling the color value from the first scene color texture map to obtain the second scene color texture map includes:

[0133] Reversely remap the spatial position of each pixel in the target triangle in the second clipping space back to the first clipping space to obtain the reversely remapped position of each pixel in the target triangle in the first clipping space;

[0134] Based on the reverse remapped position of each pixel in the target triangle in the first clipping space, obtain the reverse remapped UV coordinates and reverse remapped depth of each pixel in the target triangle;

[0135] Based on the inversely remapped UV coordinates and the inversely remapped depth of each pixel in the target triangle, sampling a color value from the first scene color texture map to obtain a second scene color texture map;

[0136] In response to the difference between the depth of the inverse remapped first pixel and the depth sampled from the first depth color texture map by the first pixel being less than a depth difference threshold, a color value is sampled from the first scene color texture map using the inverse remapped UV coordinate of the first pixel as the color value of the first pixel in the second scene color texture map; the first pixel is any one of the pixels in the target triangle surface;

[0137] In response to the difference between the inversely remapped depth of the first pixel and the depth sampled by the first pixel from the first depth color texture map being not less than a depth difference threshold, a color value is sampled from the first scene color texture map using the UV coordinate of the vertex corresponding to the first pixel in the first image frame as the color value of the first pixel in the second scene color texture map.

[0138] In the embodiment of the present application, the UV coordinates and depth values ​​of the pixels in the second image frame are combined into a three-dimensional coordinate, and after scaling, the second position of the pixel in the second clipping space can be obtained; based on the first camera parameters and the second camera parameters, it is remapped back to the first clipping space to obtain the remapped first position of the first clipping space, and the remapped first position is scaled to obtain the remapped first UV coordinates and the remapped first depth (i.e., the inversely remapped UV coordinates and the inversely remapped depth of each pixel in the target triangle); then, the pixel is remapped to the first depth, and compared with the depth value sampled from the first depth color texture map. If the difference is less than a threshold, the remapped first UV coordinate is used to sample the color value from the first scene color texture map; if the difference is greater than a threshold, the UV coordinates of the first image frame recorded by the vertex interpolated to the pixel are directly used to sample the color value from the first scene color texture map. After all the second pixel color values ​​are cached in the texture, the second color texture map can be obtained. Through the above processing, the shearing aliasing in the color sampling process can be suppressed, and the prediction accuracy of the scene color texture map can be improved.

[0139] In some embodiments, the target scene object comprises a stationary object displayed in the first image frame and stationary in the virtual scene;

[0140] Alternatively, the target scene object includes all scene objects displayed in the first image frame.

[0141] In an embodiment of the present application, the computer device may only predict and generate corresponding image texture data for stationary objects in the virtual scene, thereby ensuring the accuracy of the predicted and generated image.

[0142] Alternatively, the computer device may also predict all objects in the virtual scene to generate corresponding image texture data, thereby ensuring the efficiency of predictive image generation.

[0143] In some embodiments, in response to the target scene object including a stationary object, based on the second spatial position, obtaining the screen space coordinates of each pixel in the target triangular surface corresponding to the first image frame, the screen space coordinates including UV coordinates and depth values, mapping the screen space coordinates of each pixel in the target triangular surface corresponding to the first image frame to the screen space coordinates of each pixel in the target triangular surface corresponding to the second image frame, so as to obtain a second scene depth texture map, including:

[0144] Based on the second spatial position, the screen space coordinates of each pixel in the target triangular surface corresponding to the first image frame are obtained, where the screen space coordinates include UV coordinates and depth values, and the screen space coordinates of each pixel in the target triangular surface corresponding to the first image frame are mapped to the screen space coordinates of each pixel in the target triangular surface corresponding to the second image frame, so as to obtain a scene depth texture map corresponding to the stationary object;

[0145] Based on the second camera parameters, the moving object in the virtual scene is rendered to obtain a scene depth texture map corresponding to the moving object;

[0146] Acquire a second scene depth texture map based on the scene depth texture map corresponding to the stationary object and the scene depth texture map corresponding to the moving object;

[0147] Based on the UV coordinates of each pixel of the target triangle surface corresponding to the first image frame, sampling the color value from the first scene color texture map to obtain the second scene color texture map, including:

[0148] Based on the UV coordinates of each pixel of the target triangle surface corresponding to the first image frame, sampling the color value from the first scene color texture map to obtain the scene color texture map corresponding to the stationary object;

[0149] Based on the second camera parameters, the moving object in the virtual scene is rendered to obtain a scene color texture map corresponding to the moving object;

[0150] Based on the scene color texture map corresponding to the stationary object and the scene color texture map corresponding to the moving object, a second scene color texture map is acquired.

[0151] In the embodiment of the present application, when the target scene object includes a stationary object displayed in the first image frame and stationary in the virtual scene, through the above scheme (such as the above Figure 2 , Figure 4 or Figure 7 The various steps in the scheme shown in the figure) can predict the scene depth texture map and scene color texture map of the stationary object. For the moving object in the virtual scene, the computer device can generate the scene depth texture map and scene color texture map corresponding to the moving object by mapping the moving object to a two-dimensional image. That is, according to the current second camera parameters, the moving object in the virtual scene is projected from the three-dimensional model to the two-dimensional plane to obtain the scene depth texture map and scene color texture map corresponding to the moving object. Then, the scene depth texture map and scene color texture map of the stationary object and the scene depth texture map and scene color texture map corresponding to the moving object are merged to obtain a complete second scene depth texture map and second scene color texture map of the second image frame.

[0152] That is to say, in an embodiment of the present application, during the process of rendering the first image frame, the computer device can determine which scene objects in the virtual scene are stationary objects and which are moving objects, and mark or record the UV coordinates of the stationary objects or moving objects in the first image frame. When subsequently predicting and generating the second image frame, the computer device can predict the UV coordinates, depth and color information of the stationary object in the second image frame based on partial data of the UV coordinates corresponding to the stationary object in the first image frame. As for the moving object, the computer device can obtain the UV coordinates, depth and color information of the moving object in the second image frame through the rendering process, and finally combine the two to obtain the second scene depth texture map and the second scene color texture map of the final second image frame.

[0153] Please refer to Figure 8 , which shows the implementation framework diagram of the frame prediction involved in this application.

[0154] like Figure 8 As shown, first, a compute shader pass is used to calculate the position and UV of each vertex of the screen space aggregate mesh according to the scene depth of the k-1 frame. It dispatches a GPU thread for each n×n pixel block of the scene depth texture map, in which all pixels in the pixel block are traversed and sampled to obtain the depth value of each pixel; we can use the difference ((Depth(u+1,v)-Depth(u,v)) 2 +(Depth(u,v+1)-Depth(u,v)) 2 ) to approximate the differential operator mentioned above That is to say, the change in the depth field gradient at each pixel is calculated by adding the square of the difference between the depth of the pixel to the right of the pixel and the depth of the pixel, plus the square of the difference between the depth of the pixel below the pixel and the depth of the pixel; after traversing the pixel block (Tile) and calculating the change in the depth field gradient for each pixel therein (that is, the calculation result of the differential operator), the pixel position with the largest change in the depth field gradient can be obtained, which can be considered as the position of the vertex in the pixel block (Tile).

[0155] In order to avoid deformation caused by stretching, after finding the vertex position, the pixels with the closest and farthest depth values ​​are selected from the four adjacent pixels at that position and to its right, below, and to the lower right (the vertices of the target triangle correspond to the pixels around the position in the first scene depth texture map). If the selected foreground object pixels and background pixels are located on the same object, then their position coordinates are still relatively close after multiplying by the reprojection transformation matrix. At this time, the vertices can use the clip space position and UV coordinates of the foreground pixels; conversely, if the reprojected position distance is too far, then it can be considered that the foreground object pixels and background pixels belong to different objects (such as the bus and red building in the figure): At this time, the vertices will use the clip space position of the foreground pixels and the UV coordinates of the background pixels, thereby ensuring that the pixel color of the foreground object does not appear in the stretched area. After the stretch deformation correction is completed, the reprojected vertex information, including the clip space position and UV coordinates, is output to the texture for use in subsequent steps. The above process of outputting vertex information is as follows: Fig. 9 shown.

[0156] After that, a Mesh Drawing Pass is executed to draw the screen space collection mesh to the screen to generate the scene color (SceneColor) and scene depth (SceneDepth) of the kth frame.

[0157] In the vertex shader, the vertex information output by the previous compute shader is first read to obtain the clip space position and UV coordinates of the vertex. At the same time, in order to prevent undrawn blank spaces at the edge of the screen, the position and UV of the vertex at the edge of the screen will always remain close to the edge. After that, the hardware will interpolate the clip space position and UV coordinates of the pixel through rasterization based on the clip space position and UV coordinates of the vertex, and output it to the pixel shader. The above process of outputting the spatial position and UV coordinates of the pixel is as follows: Fig.10 shown.

[0158] In the pixel shader, first sample the scene depth texture of the k-1 frame according to the UV coordinates interpolated by rasterization to obtain the depth value of the k-1 frame. Then use the depth value as the z component, and map the UV coordinates interpolated by rasterization from the [0,1] interval to [-1,1] as the xy component as the homogeneous coordinates. (The w component is directly set to 1.) After multiplying the homogeneous coordinate by the reprojection transformation matrix, the z component of the coordinate is the depth value of the pixel in the kth frame.

[0159] Similar to stretching deformation, if you directly use the above method of "getting the kth frame value by reprojecting the kth frame value" to reproject the scene color (SceneColor) pixel value of the k-1th frame to get the scene color of the kth frame, shearing will occur. To prevent shearing, the depth value of the kth frame just obtained can be used as the z component, and the UV value of the pixel processed by the pixel shader on the kth frame screen is mapped from the [0,1] interval to [-1,1] as the xy component as the homogeneous coordinate (The w component is also directly taken as 1), multiply it by the inverse matrix of the reprojection transformation matrix for inverse reprojection, and compare the z component of the transformed coordinate with the depth value of the k-1th frame obtained previously: if they are roughly equal, use the xy component of the coordinate obtained after inverse reprojection to map from the [-1,1] interval to [0,1] as the UV coordinate of the k-1th frame to sample the scene color (SceneColor) of the k-1th frame and output it as the color of this pixel in the kth frame; otherwise, directly use the UV coordinate interpolated by rasterization to sample the scene color (SceneColor) of the k-1th frame as the output pixel color. The above process of outputting depth and color is as follows: Fig.11 As shown. At this point, the scene color (SceneColor) and scene depth (SceneDepth) of the predicted frame can be obtained.

[0160] During the game, when the camera moves forward and rotates right, the scene color (SceneColor) output by frame prediction is as follows Fig.12 As shown. It can be seen that when the frame prediction technology based on screen space vertex-by-vertex reprojection proposed in this application is predicted, since the screen space collection mesh is still continuous after reprojection, no pixel loss will occur after rasterization interpolation; and it does not rely on any hardware and application programming interface (Application Programming Interface, API) extension, and can generate predicted frames with very low overhead while maintaining high compatibility.

[0161] The frame prediction technology based on screen space vertex-by-vertex reprojection provided in this application can directly predict and generate the next frame's static object drawing result at the software layer based on the camera parameters and the static object rendering result in the previous frame. With the corresponding pipeline, it can meet the high frame rate requirements on most devices with low power consumption. Please refer to Fig.13 As shown in the screen timing diagram, since the process of generating the static object drawing result by frame prediction no longer depends on the rendering result of the next frame, each rendered frame can be displayed on the screen immediately after the drawing is completed, and no additional waiting time for the screen is added.

[0162] Since frame prediction is only used to predict the drawing results of static objects, in the pipeline proposed in this application, moving objects are drawn separately from static objects. The drawing of moving objects is continued only after the drawing or prediction generation of static objects in each frame is completed.

[0163] Based on the above Figure 2 , Figure 4 or Figure 7 In some embodiments, based on the second spatial position, the screen space coordinates of each pixel in the target triangular surface corresponding to the first image frame are obtained, the screen space coordinates include UV coordinates and depth values, and the screen space coordinates of each pixel in the target triangular surface corresponding to the first image frame are mapped to the screen space coordinates of each pixel in the target triangular surface corresponding to the second image frame, so as to obtain a scene depth texture map corresponding to the stationary object, including:

[0164] Obtaining, through the first rendering pipeline, screen space coordinates of each pixel in the target triangle surface corresponding to the first image frame based on the second spatial position, the screen space coordinates including UV coordinates and depth values, and mapping the screen space coordinates of each pixel in the target triangle surface corresponding to the first image frame to the screen space coordinates of each pixel in the target triangle surface corresponding to the second image frame, so as to obtain a scene depth texture map corresponding to the stationary object;

[0165] Based on the second camera parameters, the moving object in the virtual scene is rendered to obtain a scene depth texture map corresponding to the moving object, including:

[0166] Through the second rendering pipeline, based on the second camera parameters, the moving object in the virtual scene is rendered to obtain a scene depth texture map corresponding to the moving object;

[0167] Based on the UV coordinates of each pixel of the target triangle surface corresponding to the first image frame, sampling the color value from the first scene color texture map to obtain the scene color texture map corresponding to the stationary object, including:

[0168] By using the first rendering pipeline, based on the UV coordinates of each pixel of the target triangle surface corresponding to the first image frame, a color value is sampled from the first scene color texture map to obtain a scene color texture map corresponding to the stationary object;

[0169] Based on the second camera parameters, the moving object in the virtual scene is rendered to obtain a scene color texture map corresponding to the moving object, including:

[0170] Through the second rendering pipeline, based on the second camera parameters, the moving object in the virtual scene is rendered to obtain a scene color texture map corresponding to the moving object.

[0171] In other embodiments, the above process may also be executed through the same rendering pipeline.

[0172] Frame prediction can efficiently generate the drawing results of static objects. This application proposes two rendering pipelines adapted to frame prediction: one is a rendering pipeline with "rendering frame-prediction frame" as a pair (referred to as "paired rendering pipeline"), and the other is a rendering pipeline that directly interpolates intermediate frames in the rendering thread (referred to as "intermediate frame interpolation pipeline").

[0173] Please refer to Fig.14 , which shows the basic structure of the "paired rendering pipeline": the pipeline treats every two frames of the game as a pair (respectively called "rendering frame" and "prediction frame"), and separates the drawing of dynamic and static objects into different passes. In the rendering frame, firstly, the opaque and translucent objects in the scene are drawn in the opaque base scene rendering channel (BasePass Opaque) and the translucent base scene rendering channel (Base Pass Translucent), respectively. After the drawing is completed, the scene color (SceneColor) and scene depth (SceneDepth) at this time are cached in the cache texture (at this time, there are only static objects in the cached texture); then, the opaque and translucent dynamic objects are drawn on the rendering target texture (Render Traget) through the opaque motion object rendering channel (Movable Pass Opaque) and the translucent motion object rendering channel (MovablePass Translucent), and then the rendering target texture (Render Traget) is displayed on the screen after post-processing. In the predicted frame, the drawing results of static objects are directly generated by frame prediction through the cached scene color (SceneColor) and scene depth (SceneDepth) and output to the rendering target texture (Render Traget); thereafter, the opaque and translucent dynamic objects of this frame are continued to be drawn through the opaque moving object rendering channel (Movable Pass Opaque) and the translucent moving object rendering channel (MovablePassTranslucent), and then the rendering target texture (Render Traget) is put on the screen after post-processing. It can be seen that in the "paired rendering pipeline", the ratio of graphics frame to logical frame is 1:1, which can improve performance, reduce battery power consumption, and fully guarantee the player's operation delay and feel at high frame rates.

[0174] It can be seen that the above method uses a paired rendering pipeline. Since the overhead of generating static scenes by frame prediction is much lower than that of drawing static scenes, it will cause uneven loads between rendered frames and predicted frames, which will cause the GPU underlying driver to estimate the load incorrectly and thus reduce the frequency, resulting in a drop in frame rate. Since the GPU driver of the mobile device uses the upper screen call as the boundary to distinguish between two GPU frames, in order to make the GPU load more balanced, this application proposes the following Fig.15 The basic structure of the "paired rendering pipeline" shown in FIG. 1 is as follows: in the prediction frame, the drawing results of the frame prediction and the moving object are output to the rendering target texture No. 1 (RenderTarget 1, i.e., RT1), and are not displayed on the screen temporarily after the prediction frame ends; in the rendering frame, the opaque basic scene rendering channel (Base Pass Opaque) is divided into two parts (Part 1 and Part 2 respectively), and Part 1 is first drawn to the rendering target texture No. 0 (Render Target 0, i.e., RT0) at the beginning, and then the drawing result of the prediction frame, i.e., the rendering target texture No. 1 (Render Target 1, i.e., RT1) is displayed on the screen; after the display is completed, the remaining drawing of the rendering frame is completed on the rendering target texture No. 0 (RenderTarget 0, i.e., RT0) and displayed on the screen. In this way, the GPU load can be balanced by the number of objects drawn in the two parts (Part 1 and Part 2) of the opaque basic scene rendering channel (Base Pass Opaque).

[0175] On the other hand, if the game is not sensitive to the response time of the player's input and there is a need to further reduce the performance overhead, this application also provides the following Fig.16The rendering pipeline for interpolating intermediate frames directly in the rendering thread (referred to as the "intermediate frame interpolation pipeline") shown in FIG. 1 is as follows: when the Render thread starts, the drawing of the first part (Part 1) of the opaque basic scene rendering channel (Base PassOpaque) is first completed, and the camera parameters of the previous frame and the current frame are used to interpolate the camera matrix and other parameters of the "intermediate frame", and then the interpolated VP matrix and the scene color (SceneColor) and scene depth (SceneDepth) textures cached in the previous frame are used to generate the drawing results of all scene objects in the "intermediate frame" through frame prediction; then the motion parameter matrices of all moving objects are interpolated. At this time, two uniform parameter buffers (UniformBuffer) can be used for all moving objects. When drawing the moving object rendering channel 1 (Moveable Pass 1), the uniform parameter buffer 1 (Uniform Buffer 1) containing the interpolation parameters is used, and the uniform parameter buffer 2 (UniformBuffer 2) that directly uses the game logic output parameters is used for the subsequent moving object rendering channel 2 (Moveable Pass 2); when the moving object rendering channel 1 (Moveable Pass 1) is completed using the interpolated intermediate frame parameters, the moving object rendering channel 1 (Moveable Pass 2) is completed using the interpolated intermediate frame parameters. After drawing Pass 1), post-processing is performed and the intermediate frame is displayed on the screen; thereafter, scene objects are drawn and cached in the second part of the opaque basic scene rendering channel (Base Pass Opaque Part2) and the translucent basic scene rendering channel (Base Pass Translucent), and then the moving object rendering channel 2 (Moveable Pass 2) is completed using the uniform parameter buffer 2 (Uniform Buffer 2) and post-processed on the screen. It can be seen that although this high frame rate implementation method cannot shorten the player input response time to improve the operation feel, it can make the camera movement and sliding screen rotation very close to the effect of high frame rate rendering, and there is no need to modify the game logic - for example, if the logic frame rate is set to 60Hz at this time, the picture effect of the camera movement can be very close to the effect of 120Hz, but most of the code logic runs at 60Hz, so the screen frame rate and picture smoothness can be greatly improved with very low overhead.

[0176] After testing, the performance data of the frame prediction technology and the supporting rendering pipeline mentioned in this article on a device equipped with a certain chip, using the original pipeline, paired rendering pipeline and intermediate frame interpolation pipeline are shown in Table 1. It can be seen that thanks to the frame prediction, which reduces the drawing of static scene objects once every two frames, the paired rendering pipeline has a 23.7% increase in frame rate and a 9.38% decrease in power consumption compared to the original pipeline; in the intermediate frame interpolation pipeline, since the game logic frame rate is only half of the screen frame rate at this time, the CPU overhead is further reduced. At this time, compared with the original pipeline, its frame rate is increased by 30.7% and power consumption is reduced by 15.78%.

[0177] Table 1

[0178]

[0179] On another device, the performance data obtained when the test was limited to 90FPS is shown in Table 2 below: It can be seen that the average screen frame rates using the original pipeline, paired rendering pipeline, and intermediate frame interpolation pipeline are roughly the same, the power consumption of the paired rendering pipeline is reduced by about 11% compared to the original pipeline, and the power consumption of the intermediate frame interpolation pipeline is reduced by about 21%.

[0180] Table 2

[0181]

[0182] Fig.17 A block diagram of an image display device for a virtual scene provided by an exemplary embodiment of the present application is shown. The device includes:

[0183] A first texture acquisition module 1701 is used to acquire a first scene depth texture map and a first scene color texture map of a first image frame; the first image frame is an image frame generated by the most recent rendering of the virtual scene;

[0184] A first position acquisition module 1702 is used to acquire a first spatial position of a vertex of a target triangular face in a first clipping space based on the first scene depth texture map; the target triangular face is a triangular face of a target scene object in the first image frame; the first clipping space is a clipping space corresponding to a first camera parameter of the first image frame;

[0185] A second position acquisition module 1703, configured to map the first spatial position to a second spatial position in a second cropping space based on the first camera parameter and a current second camera parameter; the second cropping space is a cropping space corresponding to the second camera parameter;

[0186] A second texture acquisition module 1704 is used to generate a second scene depth texture map and a second scene color texture map of a second image frame based on the second spatial position and the first scene color texture map;

[0187] The display module 1705 is configured to display the second image frame based on the second scene depth texture map and the second scene color texture map.

[0188] In some embodiments, the first location acquisition module is used to:

[0189] For each image block corresponding to the target scene object in the first scene depth texture map, determine the pixel point with the largest gradient change in the image block to obtain the pixel point in the first scene depth texture map corresponding to the vertex of the target triangle surface;

[0190] Based on the pixel points in the first scene depth texture map corresponding to the vertices of the target triangular surface, the vertex space positions and vertex UV coordinates of the vertices of the target triangular surface in the first clipping space are obtained.

[0191] In some embodiments, the first position acquisition module is used to obtain the spatial positions and UV coordinates of the pixels in the first scene depth texture map corresponding to the vertices of the target triangle as the vertex spatial positions and vertex UV coordinates in the first clipping space corresponding to the vertices of the target triangle.

[0192] In some embodiments, the first location acquisition module is used to:

[0193] Obtain foreground pixels and background pixels corresponding to the vertices of the target triangular surface in the first scene depth texture map, wherein the foreground pixels are pixels with the closest depth among the pixels surrounding the positions corresponding to the vertices of the target triangular surface in the first scene depth texture map, and the background pixels are pixels with the farthest depth among the pixels surrounding the positions corresponding to the vertices of the target triangular surface in the first scene depth texture map;

[0194] Based on the first camera parameter and the second camera parameter, respectively map the spatial position of the foreground pixel in the first cropping space and the spatial position of the background pixel in the first cropping space to the second cropping space to obtain the spatial position of the foreground pixel in the second cropping space and the spatial position of the background pixel in the second cropping space;

[0195] In response to the distance between the spatial position of the foreground pixel in the second clipping space and the spatial position of the background pixel in the second clipping space being greater than a distance threshold, using the spatial position of the foreground pixel in the second clipping space and the UV coordinates of the background pixel as the vertex spatial positions and vertex UV coordinates corresponding to the vertices of the target triangle in the second clipping space;

[0196] In response to the distance between the spatial position of the foreground pixel in the second clipping space and the spatial position of the background pixel in the second clipping space being no greater than the distance threshold, the spatial position and UV coordinate of the foreground pixel in the second clipping space are used as the vertex spatial positions and vertex UV coordinates of the vertices of the target triangle in the second clipping space.

[0197] In some embodiments, the first position acquisition module is also used to obtain the sum of the squares of the gradient differences of each pixel in the image block on the two coordinate components as the gradient change value at each pixel in the image block; wherein the two coordinate components are two components in the UV coordinate system of the first scene depth texture map.

[0198] In some embodiments, the first position acquisition module is used to calculate, for each pixel in the image block, the sum of the squares of the gradient differences of the pixel on two coordinate components based on the difference between the depth of the pixel to the right of the pixel and the depth of the pixel, and the difference between the depth of the pixel below the pixel and the depth of the pixel.

[0199] In some embodiments, the second texture acquisition module 1704 is used to:

[0200] Based on the second spatial position, obtaining the screen space coordinates of each pixel in the target triangular surface corresponding to the first image frame, the screen space coordinates including UV coordinates and depth values, and mapping the screen space coordinates of each pixel in the target triangular surface corresponding to the first image frame to the screen space coordinates of each pixel in the target triangular surface corresponding to the second image frame, so as to obtain the second scene depth texture map;

[0201] Based on UV coordinates corresponding to each pixel of the target triangle in the first image frame, color values ​​are sampled from the first scene color texture map to obtain the second scene color texture map.

[0202] In some embodiments, the second texture acquisition module 1704 is used to:

[0203] Reversely remap the spatial position of each pixel in the target triangular surface in the second clipping space back to the first clipping space to obtain the reversely remapped position of each pixel in the target triangular surface in the first clipping space;

[0204] Based on the reverse remapped position of each pixel in the target triangular surface in the first clipping space, obtaining the reverse remapped UV coordinates and reverse remapped depth of each pixel in the target triangular surface;

[0205] Based on the inversely remapped UV coordinates and the inversely remapped depth of each pixel in the target triangle, sampling a color value from the first scene color texture map to obtain the second scene color texture map;

[0206] In response to the difference between the inversely remapped depth of the first pixel and the depth obtained by sampling the first pixel from the first depth color texture map being less than a depth difference threshold, a color value is sampled from the first scene color texture map using the inversely remapped UV coordinate of the first pixel as the color value of the first pixel in the second scene color texture map; the first pixel is any one of the pixels in the target triangle;

[0207] In response to the difference between the inversely remapped depth of the first pixel and the depth sampled by the first pixel from the first depth color texture map being not less than the depth difference threshold, a color value is sampled from the first scene color texture map using the UV coordinates of the vertex corresponding to the first pixel in the first image frame as the color value of the first pixel in the second scene color texture map.

[0208] In some embodiments, the target scene object comprises a stationary object displayed in the first image frame and stationary in the virtual scene;

[0209] Alternatively, the target scene object includes all scene objects displayed in the first image frame.

[0210] In some embodiments, in response to the target scene object including the stationary object, the second texture acquisition module 1704 is configured to:

[0211] Based on the second spatial position, obtaining the screen space coordinates of each pixel in the target triangular surface corresponding to the first image frame, the screen space coordinates including UV coordinates and depth values, and mapping the screen space coordinates of each pixel in the target triangular surface corresponding to the first image frame to the screen space coordinates of each pixel in the target triangular surface corresponding to the second image frame, so as to obtain a scene depth texture map corresponding to the stationary object;

[0212] Based on the second camera parameters, rendering the moving object in the virtual scene to obtain a scene depth texture map corresponding to the moving object;

[0213] Based on the scene depth texture map corresponding to the stationary object and the scene depth texture map corresponding to the moving object, acquiring the second scene depth texture map;

[0214] Based on the UV coordinates of each pixel of the target triangle surface in the first image frame, sampling color values ​​from the first scene color texture map to obtain the scene color texture map corresponding to the stationary object;

[0215] Based on the second camera parameters, rendering the moving object in the virtual scene to obtain a scene color texture map corresponding to the moving object;

[0216] Based on the scene color texture map corresponding to the stationary object and the scene color texture map corresponding to the moving object, the second scene color texture map is acquired.

[0217] In some embodiments, the second texture acquisition module 1704 is used to:

[0218] Obtaining, through the first rendering pipeline, screen space coordinates of each pixel in the target triangular surface corresponding to the first image frame based on the second spatial position, the screen space coordinates including UV coordinates and depth values, and mapping the screen space coordinates of each pixel in the target triangular surface corresponding to the first image frame to the screen space coordinates of each pixel in the target triangular surface corresponding to the second image frame, so as to obtain a scene depth texture map corresponding to the stationary object;

[0219] Through the second rendering pipeline, based on the second camera parameters, the moving object in the virtual scene is rendered to obtain a scene depth texture map corresponding to the moving object;

[0220] By means of the first rendering pipeline, based on UV coordinates of each pixel of the target triangle surface in the first image frame, a color value is sampled from the first scene color texture map to obtain a scene color texture map corresponding to the stationary object;

[0221] Through the second rendering pipeline, based on the second camera parameters, the moving object in the virtual scene is rendered to obtain a scene color texture map corresponding to the moving object.

[0222] One point that needs to be explained is that the device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example to implement its functions. In actual applications, the above-mentioned functions can be assigned to different functional modules according to actual needs, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0223] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method; the technical effects achieved by each module performing operations are the same as those in the embodiment of the method, and will not be elaborated here.

[0224] Fig.18 The structure block diagram of a computer device 1800 provided by an exemplary embodiment of the present application is shown. The computer device 1800 may be a portable mobile terminal, such as a smart phone or a tablet computer. The computer device 1800 may also be called a user device, a portable terminal, or other names.

[0225] Typically, the computer device 1800 includes a processor 1801 and a memory 1802 .

[0226] The processor 1801 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1801 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1801 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1801 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1801 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0227] The memory 1802 may include one or more computer-readable storage media, which may be tangible and non-transitory. The memory 1802 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1802 is used to store at least one instruction, which is used to be executed by the processor 1801 to implement the image display method for a virtual scene provided in the embodiment of the present application.

[0228] In some embodiments, the computer device 1800 may further include: a peripheral device interface 1803 and at least one peripheral device. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1804 , a touch display screen 1805 , a camera 1806 , an audio circuit 1807 and a power supply 1808 .

[0229] In some embodiments, the computer device 1800 further includes one or more sensors 1809 , including but not limited to: an acceleration sensor 1810 , a gyroscope sensor 1811 , a pressure sensor 1812 , an optical sensor 1813 , and a proximity sensor 1814 .

[0230] Those skilled in the art will appreciate that the structure shown above does not constitute a limitation on the computer device 1800 , and may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0231] In an exemplary embodiment, a chip is also provided, which includes a programmable logic circuit and / or program instructions. When the chip is run on a computer device, it is used to implement the image display method for a virtual scene described in the above aspects.

[0232] In an exemplary embodiment, a computer program product is also provided, the computer program product includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor reads and executes the computer instructions from the computer-readable storage medium to implement the image display method for a virtual scene provided by each of the above method embodiments.

[0233] In an exemplary embodiment, a computer-readable storage medium is further provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the image display method for a virtual scene provided by the above-mentioned method embodiments.

[0234] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0235] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0236] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for displaying an image of a virtual scene, characterized in that: The method comprises: Acquire a first scene depth texture map and a first scene color texture map of a first image frame; the first image frame is an image frame generated by the most recent rendering of the virtual scene; Based on the first scene depth texture map, obtaining a first spatial position of a vertex of a target triangular face in a first clipping space; the target triangular face is a triangular face of a target scene object in the first image frame; the first clipping space is a clipping space corresponding to a first camera parameter of the first image frame; Based on the first camera parameter and the current second camera parameter, mapping the first spatial position to a second spatial position in a second cropping space; the second cropping space is a cropping space corresponding to the second camera parameter; Based on the second spatial position and the first scene color texture map, generating a second scene depth texture map and a second scene color texture map of a second image frame; The second image frame is displayed based on the second scene depth texture map and the second scene color texture map.

2. The method according to claim 1, characterized in that The obtaining, based on the first scene depth texture map, a first spatial position of a vertex of a target triangle in a first clipping space comprises: For each image block corresponding to the target scene object in the first scene depth texture map, determine the pixel point with the largest gradient change in the image block to obtain the pixel point in the first scene depth texture map corresponding to the vertex of the target triangle surface; Based on the pixel points in the first scene depth texture map corresponding to the vertices of the target triangular surface, the vertex space positions and vertex UV coordinates of the vertices of the target triangular surface in the first clipping space are obtained.

3. The method according to claim 2, characterized in that The step of obtaining the vertex spatial positions and vertex UV coordinates of the vertices of the target triangular surface in the first clipping space corresponding to the vertices of the target triangular surface based on the pixel points in the first scene depth texture map corresponding to the vertices of the target triangular surface comprises: The spatial positions and UV coordinates of the pixels in the first scene depth texture map corresponding to the vertices of the target triangular surface are obtained as the spatial positions and UV coordinates of the vertices in the first clipping space corresponding to the vertices of the target triangular surface.

4. The method according to claim 2, characterized in that: The mapping the first spatial position to a second spatial position in a second clipping space based on the first camera parameter and the current second camera parameter includes: Obtain foreground pixels and background pixels corresponding to the vertices of the target triangular surface in the first scene depth texture map, wherein the foreground pixels are pixels with the closest depth among the pixels surrounding the positions corresponding to the vertices of the target triangular surface in the first scene depth texture map, and the background pixels are pixels with the farthest depth among the pixels surrounding the positions corresponding to the vertices of the target triangular surface in the first scene depth texture map; Based on the first camera parameter and the second camera parameter, respectively map the spatial position of the foreground pixel in the first cropping space and the spatial position of the background pixel in the first cropping space to the second cropping space to obtain the spatial position of the foreground pixel in the second cropping space and the spatial position of the background pixel in the second cropping space; In response to the distance between the spatial position of the foreground pixel in the second clipping space and the spatial position of the background pixel in the second clipping space being greater than a distance threshold, using the spatial position of the foreground pixel in the second clipping space and the UV coordinates of the background pixel as the vertex spatial positions and vertex UV coordinates corresponding to the vertices of the target triangle in the second clipping space; In response to the distance between the spatial position of the foreground pixel in the second clipping space and the spatial position of the background pixel in the second clipping space being no greater than the distance threshold, the spatial position and UV coordinate of the foreground pixel in the second clipping space are used as the vertex spatial positions and vertex UV coordinates of the vertices of the target triangle in the second clipping space.

5. The method according to claim 2, characterized in that: The step of determining the pixel point with the largest gradient change in the image block to obtain the vertex of the target triangle surface corresponding to the pixel point in the first scene depth texture map further includes: Obtaining the sum of squares of gradient differences of each pixel in the image block on two coordinate components as a gradient change value at each pixel in the image block; The two coordinate components are two components in the UV coordinate system of the first scene depth texture map.

6. The method according to claim 5, characterized in that The step of obtaining the sum of squares of gradient differences of each pixel in the image block on two coordinate components as the gradient change value of each pixel in the image block comprises: For each pixel in the image block, based on the difference between the depth of the pixel to the right of the pixel and the depth of the pixel, and the difference between the depth of the pixel below the pixel and the depth of the pixel, the sum of the squares of the gradient differences of the pixel on two coordinate components is calculated.

7. The method according to claim 1, characterized in that The step of generating a second scene depth texture map and a second scene color texture map of a second image frame based on the second spatial position and the first scene color texture map comprises: Based on the second spatial position, obtaining the screen space coordinates of each pixel in the target triangular surface corresponding to the first image frame, the screen space coordinates including UV coordinates and depth values, and mapping the screen space coordinates of each pixel in the target triangular surface corresponding to the first image frame to the screen space coordinates of each pixel in the target triangular surface corresponding to the second image frame, so as to obtain the second scene depth texture map; Based on UV coordinates corresponding to each pixel of the target triangle in the first image frame, color values ​​are sampled from the first scene color texture map to obtain the second scene color texture map.

8. The method according to claim 7, characterized in that The step of sampling color values ​​from the first scene color texture map based on UV coordinates corresponding to each pixel of the target triangle in the first image frame to obtain the second scene color texture map includes: Reversely remap the spatial position of each pixel in the target triangular surface in the second clipping space back to the first clipping space to obtain the reversely remapped position of each pixel in the target triangular surface in the first clipping space; Based on the reverse remapped position of each pixel in the target triangular surface in the first clipping space, obtaining the reverse remapped UV coordinates and reverse remapped depth of each pixel in the target triangular surface; Based on the inversely remapped UV coordinates and the inversely remapped depth of each pixel in the target triangle, sampling a color value from the first scene color texture map to obtain the second scene color texture map; In response to the difference between the inversely remapped depth of the first pixel and the depth obtained by sampling the first pixel from the first depth color texture map being less than a depth difference threshold, a color value is sampled from the first scene color texture map using the inversely remapped UV coordinate of the first pixel as the color value of the first pixel in the second scene color texture map; the first pixel is any one of the pixels in the target triangle; In response to the difference between the inversely remapped depth of the first pixel and the depth sampled by the first pixel from the first depth color texture map being not less than the depth difference threshold, a color value is sampled from the first scene color texture map using the UV coordinates of the vertex corresponding to the first pixel in the first image frame as the color value of the first pixel in the second scene color texture map.

9. The method according to claim 7 or 8, characterized in that: The target scene object comprises a stationary object displayed in the first image frame and stationary in the virtual scene; or, The target scene objects include all scene objects displayed in the first image frame.

10. The method according to claim 9, characterized in that In response to the target scene object including the stationary object, The step of acquiring, based on the second spatial position, screen space coordinates of each pixel in the target triangular surface corresponding to the first image frame, wherein the screen space coordinates include UV coordinates and depth values, and mapping the screen space coordinates of each pixel in the target triangular surface corresponding to the first image frame to the screen space coordinates of each pixel in the target triangular surface corresponding to the second image frame, so as to obtain the second scene depth texture map, includes: Based on the second spatial position, obtaining the screen space coordinates of each pixel in the target triangular surface corresponding to the first image frame, the screen space coordinates including UV coordinates and depth values, and mapping the screen space coordinates of each pixel in the target triangular surface corresponding to the first image frame to the screen space coordinates of each pixel in the target triangular surface corresponding to the second image frame, so as to obtain a scene depth texture map corresponding to the stationary object; Based on the second camera parameters, rendering the moving object in the virtual scene to obtain a scene depth texture map corresponding to the moving object; Based on the scene depth texture map corresponding to the stationary object and the scene depth texture map corresponding to the moving object, acquiring the second scene depth texture map; The step of sampling color values ​​from the first scene color texture map based on UV coordinates corresponding to each pixel of the target triangle in the first image frame to obtain the second scene color texture map includes: Based on the UV coordinates of each pixel of the target triangle surface in the first image frame, sampling color values ​​from the first scene color texture map to obtain the scene color texture map corresponding to the stationary object; Based on the second camera parameters, rendering the moving object in the virtual scene to obtain a scene color texture map corresponding to the moving object; Based on the scene color texture map corresponding to the stationary object and the scene color texture map corresponding to the moving object, the second scene color texture map is acquired.

11. The method according to claim 10, characterized in that The step of acquiring, based on the second spatial position, screen space coordinates of each pixel in the target triangular surface corresponding to the first image frame, wherein the screen space coordinates include UV coordinates and depth values, and mapping the screen space coordinates of each pixel in the target triangular surface corresponding to the first image frame to the screen space coordinates of each pixel in the target triangular surface corresponding to the second image frame, so as to obtain a scene depth texture map corresponding to the stationary object, includes: Obtaining, through the first rendering pipeline, screen space coordinates of each pixel in the target triangular surface corresponding to the first image frame based on the second spatial position, the screen space coordinates including UV coordinates and depth values, and mapping the screen space coordinates of each pixel in the target triangular surface corresponding to the first image frame to the screen space coordinates of each pixel in the target triangular surface corresponding to the second image frame, so as to obtain a scene depth texture map corresponding to the stationary object; The step of rendering the moving object in the virtual scene based on the second camera parameter to obtain a scene depth texture map corresponding to the moving object includes: Through the second rendering pipeline, based on the second camera parameters, the moving object in the virtual scene is rendered to obtain a scene depth texture map corresponding to the moving object; The step of sampling a color value from the first scene color texture map based on the UV coordinates of each pixel of the target triangle surface corresponding to the first image frame to obtain the scene color texture map corresponding to the stationary object includes: By means of the first rendering pipeline, based on UV coordinates of each pixel of the target triangle surface in the first image frame, a color value is sampled from the first scene color texture map to obtain a scene color texture map corresponding to the stationary object; The step of rendering the moving object in the virtual scene based on the second camera parameter to obtain a scene color texture map corresponding to the moving object includes: Through the second rendering pipeline, based on the second camera parameters, the moving object in the virtual scene is rendered to obtain a scene color texture map corresponding to the moving object.

12. An image display device for a virtual scene, characterized in that: The device comprises: A first texture acquisition module, used to acquire a first scene depth texture map and a first scene color texture map of a first image frame; the first image frame is an image frame generated by the most recent rendering of the virtual scene; A first position acquisition module, configured to acquire, based on the first scene depth texture map, a first spatial position of a vertex of a target triangular face in a first clipping space; the target triangular face is a triangular face of a target scene object in the first image frame; the first clipping space is a clipping space corresponding to a first camera parameter of the first image frame; a second position acquisition module, configured to map the first spatial position to a second spatial position in a second cropping space based on the first camera parameter and a current second camera parameter; the second cropping space is a cropping space corresponding to the second camera parameter; A second texture acquisition module, configured to generate a second scene depth texture map and a second scene color texture map of a second image frame based on the second spatial position and the first scene color texture map; A display module is used to display the second image frame based on the second scene depth texture map and the second scene color texture map.

13. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the image display method for a virtual scene as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer instruction, and the computer instruction is loaded and executed by a processor to implement the image display method for a virtual scene as described in any one of claims 1 to 11.

15. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; the computer instructions are read and executed by a processor of a computer device to implement the image display method for a virtual scene as described in any one of claims 1 to 11.

Images