Image display method, device, medium, and program product for a virtual scene

Abstract

The application discloses an image display method, device, medium and program product for a virtual scene, and belongs to the technical field of virtual worlds. The method comprises the following steps: acquiring a first scene depth texture map and a first scene color texture map of a first image frame; acquiring a first space position of a vertex of a target triangular face in a first clipping space based on the first scene depth texture map; mapping the first space position to a second space position in a second clipping space based on a first camera parameter and a current second camera parameter; and generating a second scene depth texture map and a second scene color texture map of a second image frame based on the second space position and the first scene color texture map. The above scheme can improve the display frame rate of the virtual scene, thereby improving the display effect of the virtual scene.

Description

Technical Field

[0001] This application relates to the field of virtual world technology, and in particular to an image display method, device, medium and program product for virtual scenes. Background Technology

[0002] In applications that include virtual scenes, elements or objects in the virtual scene are typically rendered as two-dimensional images, and the rendered two-dimensional images are displayed on the screen.

[0003] In related technologies, the display effect of virtual scenes is related to the frame rate of image rendering. The higher the frame rate, the better the display effect of virtual scenes, and vice versa.

[0004] However, when the display frame rate of a virtual scene is high, it requires a high level of terminal processing power. Due to limitations in the processing performance of the terminal, many terminals cannot support high frame rate displays, which limits the display effect of the virtual scene. Summary of the Invention

[0005] This application provides various embodiments of an image display method, device, medium, and program product for virtual scenes, the technical solutions of which are as follows:

[0006] According to one aspect of this application, an image display method for a virtual scene is provided, the method comprising:

[0007] Obtain the first scene depth texture map and the first scene color texture map of the first image frame; the first image frame is the image frame generated in the most recent rendering of the virtual scene.

[0008] Based on the first scene depth texture map, the first spatial position of the vertex of the target triangle in the first clipping space is obtained; the target triangle is the triangle of the target scene object in the first image frame; the first clipping space is the clipping space corresponding to the first camera parameters of the first image frame.

[0009] Based on the first camera parameters and the current second camera parameters, the first spatial position is mapped to the second spatial position in the second clipping space; the second clipping space is the clipping space corresponding to the second camera parameters.

[0010] Based on the second spatial location and the first scene color texture map, a second scene depth texture map and a second scene color texture map of the second image frame are generated;

[0011] The second image frame is displayed based on the second scene depth texture map and the second scene color texture map.

[0012] According to another aspect of this application, an image display device for a virtual scene is provided, the device comprising:

[0013] The first texture acquisition module is used to acquire the first scene depth texture map and the first scene color texture map of the first image frame; the first image frame is the image frame generated in the most recent rendering of the virtual scene.

[0014] The first position acquisition module is used to acquire the first spatial position of the vertex of the target triangle in the first clipping space based on the first scene depth texture map; the target triangle is the triangle of the target scene object in the first image frame; the first clipping space is the clipping space corresponding to the first camera parameters of the first image frame.

[0015] The second position acquisition module is used to map the first spatial position to a second spatial position in a second clipping space based on the first camera parameters and the current second camera parameters; the second clipping space is the clipping space corresponding to the second camera parameters.

[0016] The second texture acquisition module is used to generate a second scene depth texture map and a second scene color texture map of the second image frame based on the second spatial location and the first scene color texture map;

[0017] The display module is used to display the second image frame based on the second scene depth texture map and the second scene color texture map.

[0018] In another aspect, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the above-described image display method for virtual scenes.

[0019] In another aspect, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer storage medium, the computer program being loaded and executed by a processor to implement the above-described image display method for virtual scenes.

[0020] In another aspect, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the image display method for a virtual scene provided in the various optional implementations described above.

[0021] The beneficial effects of the technical solution provided in this application include at least the following:

[0022] After rendering the first image frame of the virtual scene, the first scene depth texture map and the first scene color texture map of the first image frame can be obtained. Then, based on the first scene depth texture map, the first spatial position of the vertices of the triangle face of the target scene object in the first image frame in the first clipping space is obtained. Based on the first spatial position, the first camera parameters corresponding to the first image frame, and the current second camera parameters, the first spatial position is mapped to the second spatial position. Then, based on the second spatial position and the first scene color texture map, the second scene depth texture map and the second scene color texture map of the second image frame are generated for display. In the above scheme, after rendering the first image frame of the virtual scene, for the target scene object in the first image frame, the pixel position, depth, and color of the target scene object in the next frame are obtained through spatial mapping and prediction based on camera parameters. It is not necessary to re-render the target scene object, thereby greatly reducing the workload of rendering the next image frame, thereby reducing the processing resource consumption of image display, increasing the display frame rate of the virtual scene, and thus improving the display effect of the virtual scene. Attached Figure Description

[0023] Figure 1 This is a structural block diagram of a computer system provided in an exemplary embodiment of this application;

[0024] Figure 2 This is a flowchart of the image display method for virtual scenes provided in this application;

[0025] Figure 3 This is a framework diagram of an image prediction process involved in this application;

[0026] Figure 4 This is a flowchart of the image display method for virtual scenes provided in this application;

[0027] Figure 5 This is a schematic diagram of the screen space set mesh in the scene depth involved in this application;

[0028] Figure 6 This is a schematic diagram of the screen space set grid in scene color involved in this application;

[0029] Figure 7 This is a flowchart of the image display method for virtual scenes provided in this application;

[0030] Figure 8 This is a diagram illustrating the implementation framework of frame prediction involved in this application;

[0031] Figure 9 This is a schematic diagram illustrating the process of outputting vertex information;

[0032] Figure 10This is a schematic diagram illustrating the process of outputting the spatial location and UV coordinates of pixels;

[0033] Figure 11 It is the process of outputting depth and color;

[0034] Figure 12 It is the scene color map output by the frame prediction;

[0035] Figure 13 This is the timing diagram for the screen upload;

[0036] Figure 14 This is a basic structural diagram of a "paired rendering pipeline";

[0037] Figure 15 This is a basic structural diagram of another type of "paired rendering pipeline";

[0038] Figure 16 This is a basic structural diagram of the rendering pipeline that interpolates intermediate frames in the rendering thread;

[0039] Figure 17 This is a structural block diagram of the image display device for virtual scenes provided in this application;

[0040] Figure 18 This is a structural block diagram of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0042] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0043] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0044] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the attack operations and other target behaviors involved in this application were all obtained under full authorization.

[0045] It should be understood that although the terms first, second, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, a first parameter may also be referred to as a second parameter without departing from the scope of this disclosure, and similarly, a second parameter may also be referred to as a first parameter. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0046] To facilitate understanding, the following explanations are provided for several terms used in this application.

[0047] 1) Virtual Scene: A virtual scene is a virtual scene displayed (or provided) by an application when it runs on a terminal. This virtual scene can be a simulation of a real-world environment, a semi-simulated / semi-fictional three-dimensional environment, or a purely fictional three-dimensional environment. A virtual scene can be any of a two-dimensional, 2.5-dimensional, or three-dimensional virtual scene. The following examples use a three-dimensional virtual scene as an illustration, but this is not a limitation. Optionally, the virtual scene can also be used for virtual scene battles between at least two virtual characters. Optionally, the virtual scene can also be used for battles between at least two virtual characters using virtual items. Optionally, the virtual scene can also be used for battles between at least two virtual characters using virtual items within a target area, where the target area shrinks over time within the virtual scene.

[0048] Virtual scenes are typically generated by applications on computer devices such as terminals and displayed based on the hardware (such as a screen) on those terminals. These terminals can be mobile devices such as smartphones, tablets, or e-readers; alternatively, they can be personal computer devices such as laptops or stationary computers.

[0049] 2) Rendering pipeline: refers to a series of processing steps in computer graphics and game development that convert 3D scene data into 2D images.

[0050] 3) Display on screen: This refers to the process by which the computer, after completing the rendering of a frame, presents the rendering result on a display device (such as a monitor, projector, etc.) so that it can be viewed by the user.

[0051] 4) Vertical Synchronization: After the software invokes the on-screen command, it still takes some time for the rendering result to be presented to the display device. Therefore, vertical synchronization will block the software from invoking the on-screen command again before the display device completes the presentation, causing the software to enter a waiting state until the current presentation is completed. This ensures that when the next on-screen command takes effect, the current presentation has already been completed. Due to the existence of the vertical synchronization mechanism, the interval between each frame will never be less than the device's vertical synchronization waiting time.

[0052] 5) Homogeneous coordinates: Homogeneous coordinates are a coordinate representation method used in geometry, computer graphics, and many other fields. In a homogeneous coordinate system, a point in n-dimensional space is represented by a point in n+1-dimensional space. For example, in three-dimensional space, a point is typically represented by three coordinates [x, y, z]. T In other words, in homogeneous coordinates, it would be represented by four coordinates [x, y, z, w]. T , where w is the homogeneous coordinate. Their main advantage is that they allow us to use matrix representations and manipulate affine transformations (such as translation, rotation, and scaling) and perspective projection. In non-homogeneous coordinates, translation cannot be represented by matrix multiplication, which complicates the combination and application of transformations. In homogeneous coordinates, all these transformations can be represented by matrix multiplication, making their combination and application much simpler. In this paper, all coordinates used for coordinate space transformations using transformation matrices are homogeneous coordinates.

[0053] 6) Model matrix (M matrix): This matrix describes the position, orientation, and size of a 3D model in the world, and is responsible for transforming coordinates from its local space (also known as object space) to world space. In other words, multiplying a coordinate value in local space by the M matrix yields its corresponding coordinate in world space.

[0054] 7) View matrix (V matrix): This matrix describes the position and orientation of the camera (or observer) in the world, and is responsible for transforming coordinates from world space to view space (also known as camera space). In other words, multiplying a coordinate value in world space by the V matrix yields the corresponding coordinate in view space.

[0055] 8) Projection matrix (P matrix): This matrix defines how the 3D world is mapped to the 2D screen, transforming coordinates from view space to the screen's clip space. In other words, multiplying a coordinate value in view space by the P matrix yields its corresponding coordinate in clip space.

[0056] 9) SceneColor: refers to the color information of each pixel in the frame buffer after the objects in the scene have been drawn during the rendering process.

[0057] 10) Scene Depth: During the rendering process, after objects in the scene are drawn, the depth information of each pixel in the frame buffer is obtained, which is the distance information of the pixel from the camera.

[0058] 11) Compute Shader Pass: This refers to the process of calling the GPU to perform the operations specified in the shader and output the results through the compute shader.

[0059] 12) Mesh Pass: This refers to the process by which the CPU sequentially renders the mesh data as a series of pixels through vertex shading, rasterization, pixel shading, and depth testing.

[0060] 13) UV Coordinates: In computer graphics, UV coordinates are a method for representing two-dimensional texture coordinates. In the UV coordinate system, the U-axis represents the horizontal direction, and the V-axis represents the vertical direction. The range of UV coordinates is usually between 0 and 1, where (0,0) represents the upper left corner of the texture, and (1,1) represents the lower right corner of the texture.

[0061] Figure 1 A structural block diagram of a computer system provided in an exemplary embodiment of this application is shown. The computer system 100 includes: a first terminal 110, a server 120, and a second terminal 130.

[0062] The first terminal 110 has a client 111 installed and running that supports virtual scenarios. This client 111 can be a multiplayer online battle program. When the first terminal runs the client 111, the user interface of the client 111 is displayed on the screen of the first terminal 110.

[0063] The second terminal 130 has a client 131 installed and running that supports virtual scenarios. This client 131 can be a multiplayer online battle program. When the second terminal 130 runs the client 131, the user interface of the client 131 is displayed on the screen of the second terminal 130.

[0064] Optionally, the clients installed on the first terminal 110 and the second terminal 130 are the same, or the clients installed on the two terminals are the same type of client on different operating system platforms (Android or iOS). The first terminal 110 can refer to one of multiple terminals, and the second terminal 130 can refer to another of multiple terminals. This embodiment only uses the first terminal 110 and the second terminal 130 as examples. The device types of the first terminal 110 and the second terminal 130 may be the same or different, and these device types include at least one of the following: smartphones, tablets, e-book readers, MP3 players, MP4 players, laptops, and desktop computers.

[0065] Figure 1 Only two terminals are shown in the diagram, but in different embodiments, multiple other terminals 140 can access the server 120. Optionally, one or more terminals 140 may also be terminals corresponding to developers, on which a development and editing platform for clients supporting virtual scenes is installed. Developers can edit and update the client on the terminal 140 and transmit the updated client installation package to the server 120 via wired or wireless network. The first terminal 110 and the second terminal 130 can download the client installation package from the server 120 to update the client.

[0066] The first terminal 110, the second terminal 130, and other terminals 140 are connected to the server 120 via a wireless network or a wired network.

[0067] Server 120 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Server 120 is used to provide backend services for clients supporting 3D virtual scenes. Optionally, server 120 undertakes the main computing work, and the terminal undertakes the secondary computing work; or, server 120 undertakes the secondary computing work, and the terminal undertakes the main computing work; or, server 120 and the terminal use a distributed computing architecture for collaborative computing.

[0068] In an illustrative example, server 120 includes processor 122, user account database 123, battle service module 124, and user-facing input / output interface (I / O interface) 125. Processor 122 loads instructions stored in server 120 and processes data in user account database 123 and battle service module 124. User account database 123 stores data about user accounts used by first terminal 110, second terminal 130, and other terminals 140, such as user account avatars, nicknames, combat power indices, and the service area where the user account is located. Battle service module 124 provides multiple battle rooms for users to play, such as 1v1, 3v3, and 5v5 battles. User-facing I / O interface 125 establishes communication and exchanges data with first terminal 110 and / or second terminal 130 via wireless or wired network.

[0069] Figure 2 A flowchart illustrating an exemplary embodiment of this application provides a method for displaying images in a virtual scene. This method can be executed by a computer device. The computer device may be... Figure 1 The system shown can be either the first terminal 110 or the second terminal 130, or the computer device itself can be... Figure 1 The server 120 in the system shown, or the computer device, may also include the above-mentioned... Figure 1 The system shown includes a first terminal 110, a server 120, and a second terminal 130; the method includes:

[0070] Step 210: Obtain the first scene depth texture map and the first scene color texture map of the first image frame; the first image frame is the image frame generated in the most recent rendering of the virtual scene.

[0071] In this embodiment, the rendering of the virtual scene refers to the process of mapping scene objects in the virtual scene from a three-dimensional model to a two-dimensional image through rendering. In other words, the first image frame is generated entirely through rendering, representing a two-dimensional image of the virtual scene.

[0072] For example, the first image frame is an image frame generated by the computer device after rasterizing and coloring the triangular faces of the unoccluded model within the view frustum and outputting the color and depth to the frame buffer.

[0073] In this embodiment of the application, when a computer device needs to generate an image frame by prediction during the display of a scene image of a virtual scene, it can retrieve the most recently rendered image frame (i.e., the aforementioned first image frame) and use that image frame to generate the next image frame to be displayed by prediction. Here, generating the next image frame by prediction means that part or all of the image frame is generated by prediction using the previous first image frame. The "prediction" process can be referred to in subsequent steps.

[0074] For the first image frame of the virtual scene, when the image frame is displayed on the screen, the scene depth texture map (SceneDepth) and scene color texture map (SceneColor) of the image frame are required. The scene depth texture map contains the depth corresponding to each pixel in the image frame of the virtual scene. The relationship between the depth in the scene depth texture map and the pixel is associated with the UV coordinates of the pixel. For example, the scene depth texture map contains the depth corresponding to the UV coordinates of each pixel.

[0075] Correspondingly, the scene color texture map contains the color corresponding to each pixel in the image frame of the virtual scene. The relationship between the color value in the scene color texture map and the pixel is associated with the UV coordinates of the pixel; for example, the scene color texture map contains the color value corresponding to the UV coordinates of each pixel.

[0076] Step 220: Based on the first scene depth texture map, obtain the first spatial position of the vertex of the target triangle in the first clipping space; the target triangle is the triangle of the target scene object in the first image frame; the first clipping space is the clipping space corresponding to the first camera parameters of the first image frame.

[0077] In a virtual scene, the model of each scene object can be enclosed by several triangular faces. The points at the three corners of each triangular face can be called the vertices of that triangular face.

[0078] To facilitate understanding, the following section introduces several spaces involved in the rendering process:

[0079] 1) Object space: also known as model space / local space / modeling space, is the coordinate system that defines the triangular faces of a model. Each model has its own coordinate system. For example, the geometric center of the model is the origin of this coordinate system. The positional attributes of the vertices of the triangular faces in this model are all relative to the coordinate values ​​of this origin. This coordinate space can be called object space.

[0080] 2) World Space: When placing a model into a virtual scene (such as a game world), a model matrix needs to be applied to the model. This matrix performs transformations on the model, including translation, rotation, and scaling. The transformed model will then be placed in world space with a specific angle, size, and position. Different objects do not have relationships between their object spaces, but world space provides a reference for the positions of different objects within a scene.

[0081] 3) Observation space: also known as view space, the coordinate system of this space is based on the camera; for example, with the camera position as the origin, the camera facing the positive Z-axis, the right side as the positive X-axis, and the top as the positive Y-axis, a coordinate system is established, and the coordinate space corresponding to this coordinate system is the observation space; the rendering pipeline is a series of processes that convert 3D scene data into 2D images. In world space, in addition to the model, there is also the camera that observes the model. In this case, defining the above observation space can correctly obtain the image observed by the camera and is convenient for calculation.

[0082] 4) Clip Space: Clip space is the space in which the world space model is projected onto the corresponding projection plane, and then clipping is performed. Since the models of all scene objects in the virtual scene are in the view space, and the camera has a certain viewing range (called a view frustum), objects inside the view frustum are visible to the camera (and can be displayed on the screen), while those outside the view frustum are invisible. To distinguish which objects are visible and which are not, a perspective projection matrix is ​​applied to the vertices of the model in the view space, transforming the vertices into a new space, which is the clip space. The calculation of the perspective matrix can be performed by the vertex shader. The output of the vertex shader is the coordinate value of the vertex within the clip space. If a vertex is within the view frustum, the xyz component values ​​of that vertex are in the range [-w, w]. If the absolute value of any xyz component exceeds w, then this vertex will be clipped (will not appear on the screen). The value of w is set to 1 before the transformation, and the transformed value corresponds to the depth before the transformation.

[0083] 5) Normalized device space: After the vertex shader performs matrix transformation, it can perform perspective division, that is, divide the three components of the vertex xyz by the w component, thereby transforming to the normalized device space, where all coordinates are normalized and fall within the range of [-1, 1].

[0084] 6) Screen Space: For coordinates in the standardized device space, the viewport transformation matrix can be applied to translate and scale these coordinates to the screen space. The screen space can be regarded as a coordinate system with (0,0) as the origin, the positive y-axis as the upward direction, and the positive x-axis as the right direction. Its coordinate range is determined by the screen resolution.

[0085] In this embodiment, the coordinates of pixels in the scene depth texture map and the scene color texture map can be obtained by matrix transformation from the coordinates in the clipping space. Correspondingly, for the coordinates of the vertices of the triangle face in the first scene depth texture map, the spatial coordinates of the vertex in the first clipping space corresponding to the first camera parameters (i.e., the aforementioned first spatial coordinates) can be derived in reverse.

[0086] The first camera parameters corresponding to the first image frame can also be referred to as the camera parameters at the time of rendering and displaying the first image frame.

[0087] Step 230: Based on the first camera parameters and the current second camera parameters, map the first spatial position to the second spatial position in the second clipping space; the second clipping space is the clipping space corresponding to the second camera parameters.

[0088] For two different moments, the parameters of the camera observing the virtual scene usually change (movement, rotation, etc.). Assuming that the position of the target scene object in the virtual scene remains unchanged, if we know the spatial coordinates of the vertices of the triangle face of the target scene object in the clip space of the previous moment, we can deduce (also known as mapping) the spatial coordinates of the vertices of the triangle face in the clip space of the current moment by combining the camera parameters of the previous moment and the current moment.

[0089] Step 240: Based on the second spatial location and the first scene color texture map, generate the second scene depth texture map and the second scene color texture map of the second image frame.

[0090] By deriving the spatial coordinates of the vertices of the triangle face on the target scene object in the virtual scene in the clip space at the current moment, we can determine the depth of each pixel on the triangle face at the current moment by using the spatial coordinates of the vertices of the triangle face on the target scene object in the clip space at the current moment. By assigning the color value of the pixel at the previous moment to the current moment, we can obtain the second scene depth texture map and the second scene color texture map at the current moment.

[0091] Step 250: Display the second image frame based on the second scene depth texture map and the second scene color texture map.

[0092] After obtaining the second scene depth texture map and the second scene color texture map at the current moment, the second image frame at the current moment can be displayed on the screen.

[0093] Since the second image frame is predicted based on the most recent rendered frame, it can also be called a predicted frame. In other words, some or all of the information in the second image frame is obtained by predicting the previous rendered frame, rather than by directly mapping the model of the scene object in the virtual scene.

[0094] In summary, the solution presented in this application, after rendering the first image frame of the virtual scene, can obtain the first scene depth texture map and the first scene color texture map of the first image frame. Then, based on the first scene depth texture map, the first spatial position of the vertices of the triangle face of the target scene object in the first image frame in the first clipping space is obtained. Based on the first spatial position, the first camera parameters corresponding to the first image frame, and the current second camera parameters, the first spatial position is mapped to the second spatial position. Then, based on the second spatial position and the first scene color texture map, the second scene depth texture map and the second scene color texture map of the second image frame are generated to display the second image frame. In the above solution, after rendering the first image frame of the virtual scene, for the target scene object in the first image frame, the pixel position, depth, and color of the target scene object in the next frame are obtained through spatial mapping and prediction based on camera parameters. It is not necessary to re-render the target scene object, thereby greatly reducing the workload of rendering the next image frame, thereby reducing the processing resource consumption of image display, increasing the display frame rate of the virtual scene, and thus improving the display effect of the virtual scene.

[0095] The above application Figure 2 The embodiment shown provides a scheme to improve the frame rate by predicting a predicted frame through rendering frames and inserting predicted frames between rendering frames. Since all or part of the information in the predicted frame can be directly obtained from the prediction of the previous frame, the complexity of predicting frame generation can be greatly reduced, thereby improving the frame rate while reducing the consumption of processing resources.

[0096] For example, please refer to Figure 3 This diagram illustrates a framework of an image prediction process related to this application. Figure 3As shown, during the process of displaying the scene image of the virtual scene 31, at the first moment, the scene elements 31b and 31c in the field of view of the camera 31a in the virtual scene are mapped into two-dimensional images through the rendering pipeline to obtain the first scene depth texture map 32 and the first scene color texture map 33. Then, the first image frame at the first moment is displayed according to the first scene depth texture map 32 and the first scene color texture map 33.

[0097] At the second moment after the first moment, the camera parameters of camera 31a change. At this time, instead of directly mapping the scene elements in the virtual scene within the field of view of camera 31a to a new two-dimensional image using the changed camera parameters, the first spatial position of the vertices of the triangles in the target scene elements (such as scene elements 31b and 31c) in the clipping space 34 corresponding to the camera parameters at the first moment is predicted based on the first scene depth texture map 32. Then, by combining the camera parameters at the first moment and the second moment, the first spatial position can be reprojected to the second spatial position in the clipping space 35 corresponding to the camera parameters at the second moment through reprojection, thereby obtaining the second scene depth texture map 36 at the second moment. Then, by combining the second spatial position and the first scene color texture map 33, the second scene color texture map 37 at the second moment can be obtained. Finally, the second image frame at the second moment is displayed based on the second scene depth texture map 36 and the second scene color texture map 37.

[0098] In the second image frame, at least some information corresponding to the target scene elements is predicted from the information in the first image frame that has already been rendered. Compared with the method of rendering directly from the virtual scene, the above scheme can reduce the resource consumption in the process of generating the second image frame.

[0099] Based on the above Figure 2 Please refer to the embodiments shown. Figure 4 This illustrates a flowchart of an image display method for a virtual scene provided in an exemplary embodiment of this application. Figure 4 As shown, step 220 above can be implemented as steps 220a, 220b, and 220c:

[0100] Step 220a: For each image block corresponding to the target scene object in the first scene depth texture map, determine the pixel point with the largest gradient change in the image block, so as to obtain the pixel point in the first scene depth texture map corresponding to the vertex of the target triangle.

[0101] Since the models of scene objects in a virtual scene are enclosed by triangular faces, the depth change between two adjacent points on the same triangular face will be relatively smooth, while the depth change between two adjacent points belonging to different triangular faces will be more abrupt. This situation can be reflected by the gradient change values ​​of adjacent pixels in the scene depth texture map.

[0102] Based on the above principle, in this embodiment of the application, for the existing first scene depth texture map, the first scene depth texture map can be divided into multiple image blocks, and it is assumed that in the first scene depth texture map, each image block in the region corresponding to the target scene object has a vertex of the triangle face on the target scene object. At this time, for the pixel point with the largest gradient change in the above image block, it can be considered that the pixel point corresponds to the vertex of the triangle face on the target scene object.

[0103] Step 220b: Based on the pixel points in the first scene depth texture map corresponding to the vertices of the target triangle, obtain the vertex spatial position and vertex UV coordinates in the first clipping space corresponding to the vertices of the target triangle.

[0104] After determining the pixel of the vertex of the triangle face on the target scene object in each image block within the region of the target scene object in the first scene depth texture map, since the first scene depth texture map contains the depth and UV coordinates of the pixel, the vertex spatial position and vertex UV coordinates of the triangle face on the target scene object in the first clipping space can be determined.

[0105] In some embodiments, based on the pixels in the first scene depth texture map corresponding to the vertices of the target triangle, obtaining the vertex spatial position and vertex UV coordinates in the first clipping space corresponding to the vertices of the target triangle includes:

[0106] Obtain the spatial position and UV coordinates of the pixels corresponding to the vertices of the target triangle in the first scene depth texture map, and use the vertex spatial position and vertex UV coordinates corresponding to the vertices of the target triangle in the first clipping space.

[0107] In one possible implementation of this application, the computer device can directly take the pixels of the vertices of the triangles on the target scene object in each image block within the region corresponding to the target scene object in the first scene depth texture map as the vertices of the triangles on the target scene object, and take their spatial position and UV coordinates as the vertex spatial position and vertex UV coordinates. This scheme is simple to execute, consumes few resources, and can ensure the efficiency of image prediction.

[0108] In some embodiments, mapping a first spatial position to a second spatial position in a second clipping space based on first camera parameters and current second camera parameters includes:

[0109] Obtain the foreground and background pixels corresponding to the vertices of the target triangle in the first scene depth texture map. The foreground pixel is the pixel with the closest depth among the pixels around the position of the vertex of the target triangle in the first scene depth texture map, and the background pixel is the pixel with the farthest depth among the pixels around the position of the vertex of the target triangle in the first scene depth texture map.

[0110] Based on the parameters of the first camera and the parameters of the second camera, the spatial positions of the foreground pixels in the first crop space and the spatial positions of the background pixels in the first crop space are mapped to the second crop space respectively, so as to obtain the spatial positions of the foreground pixels in the second crop space and the spatial positions of the background pixels in the second crop space.

[0111] In response to the spatial position of the foreground pixel in the second clipping space being greater than the distance threshold between the spatial position of the background pixel in the second clipping space, the spatial position of the foreground pixel in the second clipping space and the UV coordinates of the background pixel are used as the vertex spatial position and vertex UV coordinates of the target triangle in the second clipping space.

[0112] In response to the spatial position of the foreground pixel in the second clipping space, the distance between the foreground pixel and the spatial position of the background pixel in the second clipping space is not greater than a distance threshold. The spatial position and UV coordinates of the foreground pixel in the second clipping space are used as the vertex spatial position and vertex UV coordinates of the target triangle in the second clipping space.

[0113] If the vertex is located at the edge between the object and the background, the movement speed of the foreground object pixels is much greater than that of the background pixels when the perspective camera moves laterally. Therefore, directly using the vertex's position coordinates and UV coordinates may cause stretching and distortion. To address this, in this embodiment, based on the pixels corresponding to the vertices of the target triangle in the first scene depth texture map, during the process of obtaining the vertex spatial position and UV coordinates of the vertex corresponding to the target triangle in the second clipping space, the two pixels with the closest and furthest depths are selected from the pixels corresponding to the vertex in the first scene depth texture map and their surrounding pixels. These two pixels are then used as the foreground and background pixels, respectively. These two pixels are then projected into the second clipping space via reprojection. Specifically, the spatial positions (spatial coordinates, such as homogeneous coordinates) of these two pixels in the first clipping space are multiplied by the reprojection transformation matrix to obtain the spatial positions (spatial coordinates, such as homogeneous coordinates) of these two pixels in the second clipping space. The reprojection transformation matrix is ​​determined by the parameters of the first camera and the second camera. Then, the spatial positions of the two pixels after reprojection are compared. If the difference between the reprojected spatial positions is large, the two foreground and background pixels are considered to belong to different objects. The spatial position of the foreground pixel in the second clipping space and the UV coordinates of the background pixel in the second clipping space are taken as the vertex spatial position and vertex UV coordinates of the corresponding vertex. Otherwise, the spatial position of the foreground pixel in the second clipping space and the UV coordinates of the corresponding second clipping space are taken as the vertex spatial position and vertex UV coordinates of the corresponding vertex. This can suppress the stretching and aliasing of the edge position of the target scene object and improve the accuracy of image prediction.

[0114] The above scheme uses the pixel with the largest gradient change in the image patch of the scene depth texture map to locate the vertex in the image patch, thereby determining the vertex spatial position and vertex UV coordinate of the vertices of the triangle face on the target scene object in the first clip space, and then determining the spatial position of each point on the entire triangle face in the first clip space. This realizes the reverse derivation of the spatial position of the triangle face of the target scene object in the clip space from the scene depth texture map.

[0115] In some embodiments, before determining the pixel with the largest gradient change in the image patch to obtain the pixel corresponding to the vertex of the target triangle in the first scene depth texture map, the method further includes:

[0116] The sum of the squares of the gradient differences between the two coordinate components for each pixel in the image patch is obtained as the gradient change value at each pixel in the image patch.

[0117] Among them, the two coordinate components are two components in the UV coordinate system of the first scene depth texture map.

[0118] The scheme shown in this application provides an accurate way to represent the change in pixel depth by using the sum of the squares of the gradient differences of a pixel on two coordinate components as the gradient change value at that pixel, thus ensuring the accuracy of vertex prediction.

[0119] In some embodiments, obtaining the sum of squared gradient differences of each pixel in the image patch across the two coordinate components, as the gradient change value at each pixel in the image patch, includes:

[0120] For each pixel in the image patch, the sum of squares of the gradient differences of the pixel on the two coordinate components is calculated based on the difference between the depth of the pixel to the right of the pixel and the depth of the pixel to the bottom of the pixel.

[0121] In this embodiment of the application, the computer device can take the square of the difference between the depth of the pixel to the right of the pixel and the depth of the pixel below the pixel, and take the sum of the two squares as the sum of the squares of the gradient differences of the pixel on the two coordinate components. This reduces the consumption of computational resources and ensures the efficiency of vertex prediction while ensuring the accuracy of vertex prediction.

[0122] In the solution provided in this application embodiment, the scene color (SceneColor) and scene depth (SceneDepth) of the (k-1)th frame that has been cached, as well as the camera motion parameters of the kth frame, can be used to generate the scene color (SceneColor) and scene depth (SceneDepth) rendering result of the kth frame.

[0123] During normal scene object rendering, the unoccluded model triangles within the view frustum are rasterized, colored, and their color and depth are output to the frame buffer. In the frame prediction implemented in this application, the rendered scene color and scene depth are used to inversely reconstruct all the triangles of a stationary object appearing on the screen. To facilitate connecting vertices into a mesh and achieve a more uniform distribution of vertices, this application allocates one vertex per n×n pixel tile in the scene depth texture map, ensuring that the vertex is located at the pixel most likely to contain a vertex within that tile. This application uses a first-order nonlinear operator. —u represents the x-axis of the SceneDepthTexture, and v represents the y-axis—that is, the sum of the squares of the gradient components of the depth map, used to measure the magnitude of the depth field gradient change at each pixel in the SceneDepth depth map. The result of this operator is a scalar, making comparisons convenient; and the square makes the comparison unaffected by the sign of the gradient, thus only considering the rate of change of the gradient. After using this operator to find the pixel with the largest change in the sum of squared gradients within an n×n pixel block (Tile), the vertex can be considered to be located at that pixel's position. Then, connecting the vertices to form triangles allows the reconstruction of all triangles appearing on the screen. This application refers to the set of all reconstructed triangles on the screen as a "screen-space aggregated mesh." Please refer to [reference needed]. Figure 5 and Figure 6 It shows a schematic diagram of the screen space aggregated mesh in scene depth and a schematic diagram of the screen space aggregated mesh in scene color, which are related to this application.

[0124] After reconstructing the screen-space aggregated mesh, the VP matrix of the k-th frame (i.e., the product of the V matrix and the P matrix of the k-th frame, usually determined by the game engine's camera parameters, denoted as P) is used. k V k ) and the VP matrix of the (k-1)th frame (denoted as P) k-1 V k-1 Its inverse matrix is The following reprojection formula can be used to obtain the homogeneous coordinates of the triangle vertices in the clip space of the (k-1)th frame. Homogeneous coordinates of the reprojected spatial position in the k-th frame Since the vertices of the reprojected triangles all originate from stationary objects, the product of the object M matrices in the (k-1)th frame and the kth frame is the identity matrix (i.e., ...). Furthermore, the VP matrix is ​​camera-dependent, meaning that the reprojection result of the vertices of a stationary object using this method is accurate. In this embodiment, it can be assumed that the target scene object in the virtual scene is a stationary object. It is worth noting that the clip space position and its corresponding screen space UV coordinates can be converted to each other using perspective division and viewport transformation, as shown in the following formula:

[0125]

[0126] in, The matrix product is the reprojection transformation matrix, and the homogeneous coordinates of the clip spatial position of the (k-1)th frame. Multiplying by the reprojection transformation matrix yields the homogeneous coordinates of the spatial location in the k-th frame. The process of transforming the coordinates of the (k-1)th frame to the coordinates of the kth frame through matrix multiplication is called "reprojection"; correspondingly, the process of transforming the coordinates of the kth frame to the coordinates of the (k-1)th frame by multiplying them by the inverse of the reprojection transformation matrix is ​​called "inverse reprojection".

[0127] Based on the above Figure 2 or Figure 4 Please refer to the embodiments shown. Figure 7 This illustrates a flowchart of an image display method for a virtual scene provided in an exemplary embodiment of this application. Figure 7 As shown, step 240 above can be implemented as steps 240a and 240b:

[0128] Step 240a: Based on the second spatial position, obtain the screen space coordinates of each pixel in the target triangle in the first image frame. The screen space coordinates include UV coordinates and depth values. Map the screen space coordinates of each pixel in the target triangle in the first image frame to the screen space coordinates of each pixel in the target triangle in the second image frame to obtain the second scene depth texture map.

[0129] In this embodiment, the computer device can draw all the triangular faces formed by the vertices in the second clipping space to the frame buffer. The triangular faces are rasterized into pixels in the second image frame, and the UV coordinates of the first image frame recorded on the vertices are interpolated to each pixel. The UV coordinates of the first image frame recorded by the vertices interpolated to the pixels are used to sample depth values ​​from the first depth color texture map, and the UV coordinates and depth values ​​are combined to form the screen space coordinates of the first image frame. After scaling, the first clipping space coordinates are obtained, which are then mapped to the second image frame to obtain the second clipping space coordinates. After scaling, the second screen space coordinates are obtained, and their z-components are the depth values ​​of the pixels in the second image frame. After all the second pixel depth values ​​are cached in the texture, the second depth texture map can be obtained.

[0130] Step 240b: Based on the UV coordinates of each pixel of the target triangle in the first image frame, sample color values ​​from the first scene color texture map to obtain the second scene color texture map.

[0131] In this embodiment of the application, after obtaining the UV coordinates of each pixel of the target triangle in the first image frame, the computer device can sample the color values ​​of each pixel of the target triangle from the first scene color texture map, thereby obtaining the color values ​​of each pixel of the target triangle in the second scene color texture map.

[0132] In some embodiments, based on the UV coordinates of each pixel of the target triangle in the first image frame, color values ​​are sampled from the first scene color texture map to obtain the second scene color texture map, including:

[0133] The spatial positions of each pixel in the target triangle in the second clipping space are reverse-mapped back to the first clipping space to obtain the reverse-mapped positions of each pixel in the target triangle in the first clipping space.

[0134] Based on the inverse remapping position of each pixel in the target triangle in the first clipping space, obtain the UV coordinates and inverse remapping depth of each pixel in the target triangle.

[0135] Based on the UV coordinates and depth of the inverse remapping of each pixel in the target triangle, color values ​​are sampled from the first scene color texture map to obtain the second scene color texture map.

[0136] Wherein, in response to the depth of the inverse remapping of the first pixel, the difference between the depth sampled by the first pixel from the first depth color texture map and the depth is less than the depth difference threshold, the UV coordinates of the inverse remapping of the first pixel are used to sample the color value from the first scene color texture map as the color value of the first pixel in the second scene color texture map; the first pixel is any pixel among the pixels in the target triangle.

[0137] In response to the depth of the inverse remapping of the first pixel, if the difference between the depth of the first pixel sampled from the first depth color texture map and the depth of the first pixel is not less than the depth difference threshold, the color value of the first pixel is sampled from the first scene color texture map using the UV coordinates of the vertex corresponding to the first pixel in the first image frame, and used as the color value of the first pixel in the second scene color texture map.

[0138] In this embodiment, the UV coordinates and depth values ​​of pixels in the second image frame are combined to form a three-dimensional coordinate system. After scaling, the second position of the pixel in the second clipping space is obtained. Based on the first and second camera parameters, this system is remapped back to the first clipping space to obtain the remapped first position in the first clipping space. The remapped first position is then scaled to obtain the remapped first UV coordinates and the remapped first depth (i.e., the inverse remapped UV coordinates and inverse remapped depth of each pixel in the target triangle). Next, the pixel's remapped first depth is compared with the depth value sampled from the first depth color texture map. If the difference is less than a threshold, the color value is sampled from the first scene color texture map using the remapped first UV coordinates. If the difference is greater than the threshold, the color value is directly sampled from the first scene color texture map using the UV coordinates of the first image frame interpolated to the pixel's vertices. All second pixel color values ​​are cached in the texture to obtain the second color texture map. Through the above processing, clipping aliasing during color sampling can be suppressed, improving the prediction accuracy of the scene color texture map.

[0139] In some embodiments, the target scene object includes a stationary object that is displayed in the first image frame and is stationary in the virtual scene;

[0140] Alternatively, the target scene object includes all scene objects displayed in the first image frame.

[0141] In this embodiment of the application, the computer device can predict and generate corresponding image texture data only for static objects in the virtual scene, ensuring the accuracy of the predicted image.

[0142] Alternatively, computer devices can predict and generate corresponding image texture data for all objects in a virtual scene, ensuring the efficiency of image generation.

[0143] In some embodiments, in response to the target scene object including a stationary object, based on a second spatial position, the screen space coordinates corresponding to each pixel in the target triangle facet in the first image frame are obtained, the screen space coordinates including UV coordinates and depth values, and the screen space coordinates corresponding to each pixel in the target triangle facet in the first image frame are mapped to the screen space coordinates corresponding to each pixel in the target triangle facet in the second image frame to obtain a second scene depth texture map, including:

[0144] Based on the second spatial position, obtain the screen space coordinates of each pixel in the target triangle in the first image frame. The screen space coordinates include UV coordinates and depth values. Map the screen space coordinates of each pixel in the target triangle in the first image frame to the screen space coordinates of each pixel in the target triangle in the second image frame to obtain the scene depth texture map corresponding to the static object.

[0145] Based on the parameters of the second camera, the moving objects in the virtual scene are drawn and rendered to obtain the scene depth texture map corresponding to the moving objects;

[0146] A second scene depth texture map is obtained based on the scene depth texture map corresponding to the static object and the scene depth texture map corresponding to the moving object.

[0147] Based on the UV coordinates of each pixel on the target triangle in the first image frame, color values ​​are sampled from the first scene color texture map to obtain the second scene color texture map, including:

[0148] Based on the UV coordinates of each pixel in the first image frame corresponding to the target triangle, color values ​​are sampled from the first scene color texture map to obtain the scene color texture map corresponding to the static object.

[0149] Based on the parameters of the second camera, the moving objects in the virtual scene are drawn and rendered to obtain the scene color texture map corresponding to the moving objects;

[0150] A second scene color texture map is obtained based on the scene color texture map corresponding to the static object and the scene color texture map corresponding to the moving object.

[0151] In this embodiment of the application, when the target scene object includes a static object displayed in the first image frame and stationary in the virtual scene, the above-mentioned scheme (such as the above-mentioned) is used. Figure 2 , Figure 4 or Figure 7 The steps in the illustrated scheme can predict the scene depth texture map and scene color texture map of a stationary object. For a moving object in a virtual scene, the computer device can generate the scene depth texture map and scene color texture map corresponding to the moving object by mapping the moving object to a two-dimensional image. That is, according to the current second camera parameters, the moving object in the virtual scene is projected from the three-dimensional model to the two-dimensional plane to obtain the scene depth texture map and scene color texture map corresponding to the moving object. Then, the scene depth texture map and scene color texture map of the stationary object and the scene depth texture map and scene color texture map corresponding to the moving object are merged to obtain the complete second scene depth texture map and second scene color texture map of the second image frame.

[0152] In other words, in this embodiment of the application, during the rendering of the first image frame, the computer device can determine which scene objects in the virtual scene are stationary objects and which are moving objects, and mark or record the UV coordinates of the stationary or moving objects in the first image frame. Subsequently, when predicting and generating the second image frame, the computer device can predict the UV coordinates, depth, and color information of the stationary objects in the second image frame based on partial data of the UV coordinates corresponding to the stationary objects in the first image frame. For moving objects, the computer device can obtain the UV coordinates, depth, and color information of the moving objects in the second image frame through the rendering process. Finally, by combining the two, the second scene depth texture map and the second scene color texture map of the final second image frame can be obtained.

[0153] Please refer to Figure 8 The diagram illustrates the implementation framework of frame prediction involved in this application.

[0154] like Figure 8 As shown, firstly, a Compute Shader Pass calculates the position and UV of each vertex of the screen-space aggregated mesh based on the scene depth of frame k-1. This dispatches a GPU thread to each n×n pixel tile of the SceneDepth Texture, where it traverses all pixels in the tile and samples the depth value of each pixel. We can then use the difference ((Depth(u+1,v)-Depth(u,v))) 2 +(Depth(u,v+1)-Depth(u,v)) 2 To approximate the differential operator mentioned above In other words, the change in depth field gradient at each pixel is calculated by adding the square of the difference between the depth of the pixel to the right of the pixel and the depth of the pixel below the pixel. After traversing the entire tile and calculating the change in depth field gradient for each pixel (i.e., the result of the difference operator), the pixel position with the largest change in depth field gradient can be obtained. This position can be considered as the position of the vertex within the tile.

[0155] To avoid deformation caused by stretching, after finding the vertex position, the pixels with the closest and furthest depth values ​​are selected from the four adjacent pixels to its right, below, and to its lower right (corresponding to the pixels around the position of the vertex of the target triangle in the first scene depth texture map) as the foreground object pixel and the background pixel, respectively. If the selected foreground object pixel and background pixel are located on the same object, their position coordinates will still be relatively close after multiplying by the reprojection transformation matrix. In this case, the vertices can use the clip space position and UV coordinates of the foreground pixel. Conversely, if the position distance after reprojection is too far, the foreground object pixel and the background pixel can be considered to belong to different objects (such as the bus and the red building in the figure). In this case, the vertices will use the clip space position of the foreground pixel and the UV coordinates of the background pixel, thus ensuring that the pixel color of the foreground object does not appear in the stretching area. After the stretching deformation correction is completed, the vertex information after reprojection, including the clip space position and UV coordinates, is output to the texture for use in subsequent steps. The process of outputting vertex information is as follows: Figure 9 As shown.

[0156] After that, a mesh pass is executed to draw the screen space set of meshes onto the screen to generate the scene color (SceneColor) and scene depth (SceneDepth) for the k-th frame.

[0157] In the vertex shader, the vertex information output from the previously computed shader is first read to obtain the clip space position and UV coordinates of the vertices. Simultaneously, to prevent undrawn blank spaces at the screen edges, the positions and UV coordinates of vertices at the screen edges are always kept flush with the edges. Then, the hardware uses rasterization interpolation based on the vertex clip space position and UV coordinates to extract the pixel clip space position and UV coordinates, and outputs this to the pixel shader. The process of outputting the pixel's space position and UV coordinates is as follows: Figure 10 As shown.

[0158] In the pixel shader, the scene depth texture of frame k-1 is first sampled based on the rasterized interpolated UV coordinates to obtain the depth value of frame k-1. Then, the depth value is used as the z-component, and the rasterized interpolated UV coordinates are mapped from the [0,1] interval to [-1,1] to become the x and y components, which are then used as homogeneous coordinates. (The w component is directly set to 1). After multiplying the homogeneous coordinates by the reprojection transformation matrix, the z component of the coordinates is the depth value of the pixel in the k-th frame.

[0159] Similar to stretching distortion, if the method described above, "obtaining the value of frame k by reprojecting the value of frame k-1," is directly used, and the scene color (SceneColor) pixel value of frame k-1 is reprojected to obtain the scene color of frame k, clipping distortion will occur. To prevent clipping distortion, the depth value of frame k obtained earlier can be used as the z-component, and the UV value of the pixel processed by the pixel shader on the screen in frame k can be mapped from the [0,1] interval to [-1,1] and used as the xy component as homogeneous coordinates. (The w component is also directly set to 1). After multiplying it by the inverse of the reprojection transformation matrix and performing inverse reprojection, the z component of the transformed coordinates is compared with the depth value obtained in the (k-1)th frame. If they are approximately equal, the xy components of the coordinates obtained after inverse reprojection are mapped from the [-1,1] interval to [0,1] and used as the UV coordinates of the (k-1)th frame to sample the scene color (SceneColor) of the (k-1)th frame and output as the color of this pixel in the kth frame; otherwise, the scene color (SceneColor) of the (k-1)th frame is directly sampled using the UV coordinates obtained by rasterization interpolation and used as the output pixel color. The above process of outputting depth and color is as follows: Figure 11 As shown. At this point, the scene color (SceneColor) and scene depth (SceneDepth) of the predicted frame can be obtained.

[0160] During gameplay, the scene color (SceneColor) is predicted and output as the camera pans forward and rotates to the right. Figure 12 As shown, the frame prediction technique based on screen space vertex-by-vertex reprojection proposed in this application, when making predictions, ensures that the screen space set mesh remains continuous after reprojection, and therefore no pixel loss occurs after rasterization interpolation; moreover, it does not rely on any hardware or application programming interface (API) extensions, and can generate predicted frames with very low overhead while maintaining high compatibility.

[0161] The frame prediction technology based on screen-space vertex-by-vertex reprojection provided in this application can directly predict and generate the rendering result of the static object in the next frame at the software layer based on camera parameters and the rendering result of the static object in the previous frame. With the corresponding pipeline, it can meet the high frame rate requirements on most devices with low power consumption. Please refer to [reference needed]. Figure 13 The timing diagram shown shows that since the process of generating the rendering result of a static object through frame prediction no longer depends on the rendering result of the next frame, each rendering frame can be displayed on the screen immediately after it is rendered, without adding any extra waiting time for display.

[0162] Since frame prediction is only used to predict the rendering results of stationary objects, the pipeline proposed in this application renders moving objects and stationary objects separately. Moving objects will only be rendered after the rendering or prediction generation of stationary objects for each frame has been completed.

[0163] Based on the above Figure 2 , Figure 4 or Figure 7 In some embodiments, based on the second spatial location, the screen space coordinates of each pixel in the target triangle facet corresponding to the pixel in the first image frame are obtained. The screen space coordinates include UV coordinates and depth values. The screen space coordinates of each pixel in the target triangle facet corresponding to the pixel in the first image frame are mapped to the screen space coordinates of each pixel in the target triangle facet corresponding to the pixel in the second image frame to obtain the scene depth texture map corresponding to the static object, including:

[0164] Through the first rendering pipeline, based on the second spatial position, the screen space coordinates of each pixel in the target triangle face corresponding to the first image frame are obtained. The screen space coordinates include UV coordinates and depth values. The screen space coordinates of each pixel in the target triangle face corresponding to the first image frame are mapped to the screen space coordinates of each pixel in the target triangle face corresponding to the second image frame to obtain the scene depth texture map corresponding to the static object.

[0165] Based on the parameters of the second camera, moving objects in the virtual scene are rendered to obtain scene depth texture maps corresponding to the moving objects, including:

[0166] Using the second rendering pipeline, based on the parameters of the second camera, moving objects in the virtual scene are drawn and rendered to obtain the scene depth texture map corresponding to the moving objects;

[0167] Based on the UV coordinates of each pixel on the target triangle in the first image frame, color values ​​are sampled from the first scene color texture map to obtain the scene color texture map corresponding to the static object, including:

[0168] Through the first rendering pipeline, based on the UV coordinates of each pixel of the target triangle in the first image frame, color values ​​are sampled from the first scene color texture map to obtain the scene color texture map corresponding to the static object.

[0169] Based on the parameters of the second camera, moving objects in the virtual scene are rendered to obtain scene color texture maps corresponding to the moving objects, including:

[0170] Using the second rendering pipeline, based on the parameters of the second camera, moving objects in the virtual scene are drawn and rendered to obtain the scene color texture map corresponding to the moving objects.

[0171] In other embodiments, the above process can also be performed through the same rendering pipeline.

[0172] Frame prediction can efficiently generate rendering results for static objects. This application proposes two rendering pipelines adapted to frame prediction: one is a rendering pipeline with "rendered frame - predicted frame" as a pair (referred to as "paired rendering pipeline"), and the other is a rendering pipeline that directly interpolates intermediate frames in the rendering thread (referred to as "intermediate frame interpolation pipeline").

[0173] Please refer to Figure 14 This illustrates the basic structure of a "paired rendering pipeline": this pipeline treats every two frames of the game as a pair (referred to as "render frame" and "predict frame"), separating the rendering of dynamic and static objects into different passes. In a render frame, opaque and semi-transparent objects in the scene are first rendered in the opaque base scene rendering pass (BasePass Opaque) and the semi-transparent base scene rendering pass (BasePass Translucent), respectively. After rendering, the scene color (SceneColor) and scene depth (SceneDepth) at this time are cached into a cached texture (at this point, the cached texture only contains static objects). Then, opaque and semi-transparent dynamic objects are rendered on the render target texture (RenderTraget) through the opaque moving object rendering pass (Movable Pass Opaque) and the semi-transparent moving object rendering pass (MovablePass Translucent). After post-processing, the render target texture (RenderTraget) is then displayed on the screen. In the prediction frame, the rendering result of static objects is directly generated by frame prediction using cached scene color (SceneColor) and scene depth (SceneDepth) and output to the render target texture. Subsequently, the opaque and semi-transparent dynamic objects of the same frame are rendered through the opaque moving object rendering pass (Movable Pass Opaque) and the semi-transparent moving object rendering pass (Movable Pass Translucent). After post-processing, the render target texture is then displayed on the screen. As can be seen, in the "paired rendering pipeline," the ratio of graphics frames to logical frames is 1:1, which can improve performance and reduce battery power consumption while fully ensuring the player's operation latency and feel at high frame rates.

[0174] As can be seen, the method described above uses a paired rendering pipeline. Since the overhead of predicting and generating static scenes is much lower than that of rendering static scenes, it causes an imbalance in the load between rendered frames and predicted frames. This leads to errors in the GPU driver's load estimation, resulting in frequency reduction and a drop in frame rate. Because mobile device GPU drivers distinguish between two GPU frames based on the screen-up call boundary, this application proposes the following method to achieve a more balanced GPU load: Figure 15 The basic structure of the "paired rendering pipeline" is as follows: In the prediction frame, the results of frame prediction and rendering of moving objects are output to RenderTarget 1 (RT1), and are not displayed on the screen immediately after the prediction frame ends. In the rendering frame, the opaque base pass Opaque is divided into two parts (Part1 and Part2). At the beginning, Part1 is first rendered to RenderTarget 0 (RT0), and then the rendering results of the prediction frame, i.e., RenderTarget 1 (RT1), are displayed on the screen. After displaying, the remaining rendering of the rendering frame is completed on RenderTarget 0 (RT0) and displayed on the screen. In this way, the GPU load can be balanced by the number of objects rendered in the two parts (Part1 and Part2) of the opaque base pass Opaque.

[0175] On the other hand, if the game is not sensitive to the response time of player input and there is a need to further reduce performance overhead, this application also provides, for example... Figure 16The rendering pipeline shown directly interpolates intermediate frames on the rendering thread (referred to as the "intermediate frame interpolation pipeline"): At the start of the Render thread, the first part (Part 1) of the opaque base scene rendering pass (Base PassOpaque) is drawn. Then, the camera matrix and other parameters of the "intermediate frame" are interpolated using the camera parameters of the previous frame and the current frame. Next, the interpolated VP matrix and the scene color (SceneColor) and scene depth (SceneDepth) textures cached from the previous frame are used to generate the drawing results of all scene objects in the "intermediate frame" through frame prediction. Then, the motion parameter matrix of all moving objects is interpolated. At this point, two uniform parameter buffers can be used for all moving objects. When drawing the moving object rendering pass 1 (Moveable Pass 1), uniform parameter buffer 1 containing the interpolated parameters is used, while uniform parameter buffer 2, which directly uses the game logic output parameters, is used for the subsequent moving object rendering pass 2 (Moveable Pass 2). The interpolated intermediate frame parameters are used to complete the rendering of the moving object pass 1 (Moveable Pass 1). After drawing Pass 1, post-processing is performed, and the intermediate frames are displayed on the screen. Then, scene objects are drawn and cached in the second part of the opaque base pass and the translucent base pass. Finally, the Moving Pass 2 is rendered using Uniform Buffer 2 and post-processed before being displayed on the screen. As can be seen, while this high frame rate implementation cannot shorten player input response time to improve the feel of the controls, it makes camera movement and screen rotation very close to the effect of high frame rate rendering, without requiring modification to the game logic. For example, if the logical frame rate is set to 60Hz, the effect of camera movement can be very close to 120Hz, but most of the code logic runs at 60Hz, thus significantly improving the screen frame rate and smoothness with very low overhead.

[0176] The performance data of the frame prediction technology and its supporting rendering pipeline mentioned in this article on a device equipped with a certain chip, using the original pipeline, paired rendering pipeline, and intermediate frame interpolation pipeline, are shown in Table 1 below. It can be seen that, thanks to frame prediction reducing the rendering of static scene objects by one every two frames, the paired rendering pipeline achieves a 23.7% increase in frame rate and a 9.38% decrease in power consumption compared to the original pipeline. In the intermediate frame interpolation pipeline, since the game logic frame rate is only half of the screen frame rate, CPU overhead is further reduced, resulting in a 30.7% increase in frame rate and a 15.78% decrease in power consumption compared to the original pipeline.

[0177] Table 1

[0178]

[0179] On another device, the performance data obtained when limited to 90FPS is shown in Table 2 below: It can be seen that the average screen frame rate using the original pipeline, the paired rendering pipeline and the intermediate frame interpolation pipeline are roughly equal. The power consumption of the paired rendering pipeline is reduced by about 11% compared to the original pipeline, while the power consumption of the intermediate frame interpolation pipeline is reduced by about 21%.

[0180] Table 2

[0181]

[0182] Figure 17 A block diagram of an image display apparatus for a virtual scene provided in an exemplary embodiment of this application is shown. The apparatus includes:

[0183] The first texture acquisition module 1701 is used to acquire the first scene depth texture map and the first scene color texture map of the first image frame; the first image frame is the image frame generated in the most recent rendering of the virtual scene.

[0184] The first position acquisition module 1702 is used to acquire the first spatial position of the vertex of the target triangle in the first clipping space based on the first scene depth texture map; the target triangle is the triangle of the target scene object in the first image frame; the first clipping space is the clipping space corresponding to the first camera parameters of the first image frame.

[0185] The second position acquisition module 1703 is used to map the first spatial position to a second spatial position in a second clipping space based on the first camera parameters and the current second camera parameters; the second clipping space is the clipping space corresponding to the second camera parameters.

[0186] The second texture acquisition module 1704 is used to generate a second scene depth texture map and a second scene color texture map of the second image frame based on the second spatial location and the first scene color texture map.

[0187] Display module 1705 is used to display the second image frame based on the second scene depth texture map and the second scene color texture map.

[0188] In some embodiments, the first location acquisition module is used to,

[0189] For each image block corresponding to the target scene object in the first scene depth texture map, determine the pixel point with the largest gradient change in the image block to obtain the pixel point in the first scene depth texture map corresponding to the vertex of the target triangle.

[0190] Based on the pixels in the first scene depth texture map corresponding to the vertices of the target triangle, the vertex spatial position and vertex UV coordinates in the first clipping space corresponding to the vertices of the target triangle are obtained.

[0191] In some embodiments, the first position acquisition module is used to acquire the spatial position and UV coordinates of the pixel points corresponding to the vertices of the target triangle in the first scene depth texture map, as the vertex spatial position and vertex UV coordinates of the vertices of the target triangle in the first clipping space.

[0192] In some embodiments, the first location acquisition module is used to,

[0193] Obtain the foreground and background pixels corresponding to the vertices of the target triangle in the first scene depth texture map. The foreground pixel is the pixel with the closest depth among the pixels around the position of the vertex of the target triangle in the first scene depth texture map, and the background pixel is the pixel with the farthest depth among the pixels around the position of the vertex of the target triangle in the first scene depth texture map.

[0194] Based on the first camera parameters and the second camera parameters, the spatial positions of the foreground pixels in the first cropping space and the spatial positions of the background pixels in the first cropping space are mapped to the second cropping space, respectively, to obtain the spatial positions of the foreground pixels in the second cropping space and the spatial positions of the background pixels in the second cropping space.

[0195] In response to the fact that the distance between the spatial position of the foreground pixel in the second clipping space and the spatial position of the background pixel in the second clipping space is greater than a distance threshold, the spatial position of the foreground pixel in the second clipping space and the UV coordinates of the background pixel are used as the vertex spatial position and vertex UV coordinates of the target triangle in the second clipping space.

[0196] In response to the fact that the distance between the spatial position of the foreground pixel in the second clipping space and the spatial position of the background pixel in the second clipping space is not greater than the distance threshold, the spatial position and UV coordinates of the foreground pixel in the second clipping space are used as the vertex spatial position and vertex UV coordinates of the vertex of the target triangle in the second clipping space.

[0197] In some embodiments, the first position acquisition module is further configured to acquire the sum of squares of the gradient differences of each pixel in the image block on two coordinate components, as the gradient change value at each pixel in the image block; wherein the two coordinate components are two components in the UV coordinate system of the first scene depth texture map.

[0198] In some embodiments, the first position acquisition module is configured to, for each pixel in the image block, calculate the sum of squares of the gradient differences of the pixel in two coordinate components based on the difference between the depth of the pixel to the right of the pixel and the depth of the pixel, and the difference between the depth of the pixel to the bottom of the pixel and the depth of the pixel.

[0199] In some embodiments, the second texture acquisition module 1704 is used for,

[0200] Based on the second spatial location, the screen space coordinates of each pixel in the target triangle face corresponding to the first image frame are obtained. The screen space coordinates include UV coordinates and depth values. The screen space coordinates of each pixel in the target triangle face corresponding to the first image frame are mapped to the screen space coordinates of each pixel in the target triangle face corresponding to the second image frame to obtain the second scene depth texture map.

[0201] Based on the UV coordinates of each pixel of the target triangle in the first image frame, color values ​​are sampled from the first scene color texture map to obtain the second scene color texture map.

[0202] In some embodiments, the second texture acquisition module 1704 is used for,

[0203] The spatial positions of each pixel in the target triangle in the second clipping space are reverse-mapped back to the first clipping space to obtain the reverse-mapped positions of each pixel in the target triangle in the first clipping space.

[0204] Based on the inverse remapping position of each pixel in the target triangle in the first clipping space, obtain the UV coordinates and inverse remapping depth of each pixel in the target triangle.

[0205] Based on the UV coordinates and depth of the inverse remapping of each pixel in the target triangle, color values ​​are sampled from the first scene color texture map to obtain the second scene color texture map.

[0206] Wherein, in response to the depth of the inverse remapping of the first pixel, the difference between the depth sampled by the first pixel from the first depth color texture map and the depth is less than the depth difference threshold, the UV coordinates of the inverse remapping of the first pixel are used to sample the color value from the first scene color texture map as the color value of the first pixel in the second scene color texture map; the first pixel is any one of the pixels in the target triangle.

[0207] In response to the depth of the inverse remapping of the first pixel, if the difference between the depth of the first pixel sampled from the first depth color texture map and the depth of the first pixel is not less than the depth difference threshold, the color value of the first pixel is sampled from the first scene color texture map using the UV coordinates of the vertex corresponding to the first pixel in the first image frame, and used as the color value of the first pixel in the second scene color texture map.

[0208] In some embodiments, the target scene object includes a static object displayed in the first image frame and stationary in the virtual scene;

[0209] Alternatively, the target scene object may include all scene objects displayed in the first image frame.

[0210] In some embodiments, in response to the target scene object including the static object, the second texture acquisition module 1704 is configured to,

[0211] Based on the second spatial position, the screen space coordinates of each pixel in the target triangle face corresponding to the first image frame are obtained. The screen space coordinates include UV coordinates and depth values. The screen space coordinates of each pixel in the target triangle face corresponding to the first image frame are mapped to the screen space coordinates of each pixel in the target triangle face corresponding to the second image frame to obtain the scene depth texture map corresponding to the static object.

[0212] Based on the second camera parameters, the moving objects in the virtual scene are drawn and rendered to obtain the scene depth texture map corresponding to the moving objects;

[0213] Based on the scene depth texture map corresponding to the static object and the scene depth texture map corresponding to the moving object, the second scene depth texture map is obtained.

[0214] Based on the UV coordinates of each pixel of the target triangle in the first image frame, color values ​​are sampled from the first scene color texture map to obtain the scene color texture map corresponding to the static object.

[0215] Based on the second camera parameters, the moving objects in the virtual scene are drawn and rendered to obtain the scene color texture map corresponding to the moving objects;

[0216] The second scene color texture map is obtained based on the scene color texture map corresponding to the static object and the scene color texture map corresponding to the moving object.

[0217] In some embodiments, the second texture acquisition module 1704 is used for,

[0218] Through the first rendering pipeline, based on the second spatial position, the screen space coordinates of each pixel in the target triangle face corresponding to the first image frame are obtained. The screen space coordinates include UV coordinates and depth values. The screen space coordinates of each pixel in the target triangle face corresponding to the first image frame are mapped to the screen space coordinates of each pixel in the target triangle face corresponding to the second image frame to obtain the scene depth texture map corresponding to the static object.

[0219] Using the second rendering pipeline, based on the second camera parameters, the moving objects in the virtual scene are drawn and rendered to obtain the scene depth texture map corresponding to the moving objects;

[0220] Through the first rendering pipeline, based on the UV coordinates of each pixel of the target triangle in the first image frame, color values ​​are sampled from the first scene color texture map to obtain the scene color texture map corresponding to the static object.

[0221] Using the second rendering pipeline, based on the second camera parameters, the moving objects in the virtual scene are drawn and rendered to obtain the scene color texture map corresponding to the moving objects.

[0222] It should be noted that the device provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules according to actual needs, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0223] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments of the relevant method; the technical effects achieved by each module performing its operation are the same as the technical effects in the embodiments of the relevant method, and will not be elaborated here.

[0224] Figure 18 This illustration shows a structural block diagram of a computer device 1800 provided in an exemplary embodiment of this application. The computer device 1800 may be a portable mobile terminal, such as a smartphone or tablet computer. The computer device 1800 may also be referred to as a user device, portable terminal, or other names.

[0225] Typically, computer device 1800 includes a processor 1801 and a memory 1802.

[0226] Processor 1801 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1801 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1801 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1801 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1801 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0227] The memory 1802 may include one or more computer-readable storage media, which may be tangible and non-transitory. The memory 1802 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1802 is used to store at least one instruction, which is executed by the processor 1801 to implement the image display method for virtual scenes provided in the embodiments of this application.

[0228] In some embodiments, the computer device 1800 may also optionally include: a peripheral device interface 1803 and at least one peripheral device. Specifically, the peripheral device includes at least one of: a radio frequency circuit 1804, a touch display screen 1805, a camera 1806, an audio circuit 1807, and a power supply 1808.

[0229] In some embodiments, the computer device 1800 further includes one or more sensors 1809. The one or more sensors 1809 include, but are not limited to, an accelerometer 1810, a gyroscope 1811, a pressure sensor 1812, an optical sensor 1813, and a proximity sensor 1814.

[0230] Those skilled in the art will understand that the structure shown above does not constitute a limitation on the computer device 1800, and may include more or fewer components than shown, or combine certain components, or employ different component arrangements.

[0231] In an exemplary embodiment, a chip is also provided, the chip including programmable logic circuitry and / or program instructions, which, when the chip is run on a computer device, are used to implement the image display method for virtual scenes described above.

[0232] In an exemplary embodiment, a computer program product is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions to implement the image display method for a virtual scene provided in the above-described method embodiments.

[0233] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores a computer program that is loaded and executed by a processor to implement the image display method for virtual scenes provided in the above-described method embodiments.

[0234] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0235] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0236] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for displaying images in a virtual scene, characterized in that, The method includes: Obtain the first scene depth texture map and the first scene color texture map of the first image frame; the first image frame is the image frame generated in the most recent rendering of the virtual scene. For each image block corresponding to the target scene object in the first scene depth texture map, determine the pixel point with the largest gradient change in the image block to obtain the pixel point in the first scene depth texture map corresponding to the vertex of the target triangle. Based on the pixels in the first scene depth texture map corresponding to the vertices of the target triangle, the first spatial position of the vertices of the target triangle in the first clipping space is obtained. The first spatial position includes the vertex spatial position and vertex UV coordinates. The target triangle is the triangle of the target scene object in the first image frame. The first clipping space is the clipping space corresponding to the first camera parameters of the first image frame. Based on the first camera parameters and the current second camera parameters, the first spatial position is mapped to the second spatial position in the second clipping space; the second clipping space is the clipping space corresponding to the second camera parameters. Based on the second spatial location and the first scene color texture map, a second scene depth texture map and a second scene color texture map of the second image frame are generated; The second image frame is displayed based on the second scene depth texture map and the second scene color texture map.

2. The method according to claim 1, characterized in that, The step of obtaining the first spatial position of the vertex of the target triangle in the first clipping space based on the pixel points corresponding to the vertices of the target triangle in the first scene depth texture map includes: Obtain the spatial position and UV coordinates of the pixels corresponding to the vertices of the target triangle in the first scene depth texture map, and use them as the vertex spatial position and vertex UV coordinates of the vertices of the target triangle in the first clipping space.

3. The method according to claim 1, characterized in that, The step of mapping the first spatial position to a second spatial position in the second cropping space based on the first camera parameters and the current second camera parameters includes: Obtain the foreground and background pixels corresponding to the vertices of the target triangle in the first scene depth texture map. The foreground pixel is the pixel with the closest depth among the pixels around the position of the vertex of the target triangle in the first scene depth texture map, and the background pixel is the pixel with the farthest depth among the pixels around the position of the vertex of the target triangle in the first scene depth texture map. Based on the first camera parameters and the second camera parameters, the spatial positions of the foreground pixels in the first cropping space and the spatial positions of the background pixels in the first cropping space are mapped to the second cropping space, respectively, to obtain the spatial positions of the foreground pixels in the second cropping space and the spatial positions of the background pixels in the second cropping space. In response to the fact that the distance between the spatial position of the foreground pixel in the second clipping space and the spatial position of the background pixel in the second clipping space is greater than a distance threshold, the spatial position of the foreground pixel in the second clipping space and the UV coordinates of the background pixel are used as the vertex spatial position and vertex UV coordinates of the target triangle in the second clipping space. In response to the fact that the distance between the spatial position of the foreground pixel in the second clipping space and the spatial position of the background pixel in the second clipping space is not greater than the distance threshold, the spatial position and UV coordinates of the foreground pixel in the second clipping space are used as the vertex spatial position and vertex UV coordinates of the vertex of the target triangle in the second clipping space.

4. The method according to claim 1, characterized in that, The step of determining the pixel with the largest gradient change in the image patch to obtain the pixel corresponding to the vertex of the target triangle in the first scene depth texture map further includes: The sum of the squares of the gradient differences between each pixel in the image block and the two coordinate components is obtained as the gradient change value at each pixel in the image block. The two coordinate components are two components in the UV coordinate system of the first scene depth texture map.

5. The method according to claim 4, characterized in that, The step of obtaining the sum of the squares of the gradient differences between the two coordinate components for each pixel in the image block, as the gradient change value at each pixel in the image block, includes: For each pixel in the image block, the sum of squares of the gradient differences of the pixel in the two coordinate components is calculated based on the difference between the depth of the pixel to the right of the pixel and the depth of the pixel to the bottom of the pixel.

6. The method according to claim 1, characterized in that, The step of generating a second scene depth texture map and a second scene color texture map for a second image frame based on the second spatial location and the first scene color texture map includes: Based on the second spatial location, the screen space coordinates of each pixel in the target triangle face corresponding to the first image frame are obtained. The screen space coordinates include UV coordinates and depth values. The screen space coordinates of each pixel in the target triangle face corresponding to the first image frame are mapped to the screen space coordinates of each pixel in the target triangle face corresponding to the second image frame to obtain the second scene depth texture map. Based on the UV coordinates of each pixel of the target triangle in the first image frame, color values ​​are sampled from the first scene color texture map to obtain the second scene color texture map.

7. The method according to claim 6, characterized in that, The step of sampling color values ​​from the first scene color texture map to obtain the second scene color texture map based on the UV coordinates of each pixel corresponding to the target triangle in the first image frame includes: The spatial positions of each pixel in the target triangle in the second clipping space are reverse-mapped back to the first clipping space to obtain the reverse-mapped positions of each pixel in the target triangle in the first clipping space. Based on the inverse remapping position of each pixel in the target triangle in the first clipping space, obtain the UV coordinates and inverse remapping depth of each pixel in the target triangle. Based on the UV coordinates and depth of the inverse remapping of each pixel in the target triangle, color values ​​are sampled from the first scene color texture map to obtain the second scene color texture map. Wherein, in response to the depth of the inverse remapping of the first pixel, the difference between the depth sampled by the first pixel from the first depth color texture map and the depth is less than the depth difference threshold, the UV coordinates of the inverse remapping of the first pixel are used to sample the color value from the first scene color texture map as the color value of the first pixel in the second scene color texture map; the first pixel is any one of the pixels in the target triangle. In response to the depth of the inverse remapping of the first pixel, if the difference between the depth of the first pixel sampled from the first depth color texture map and the depth of the first pixel is not less than the depth difference threshold, the color value of the first pixel is sampled from the first scene color texture map using the UV coordinates of the vertex corresponding to the first pixel in the first image frame, and used as the color value of the first pixel in the second scene color texture map.

8. The method according to claim 6 or 7, characterized in that, The target scene object includes a static object displayed in the first image frame and stationary in the virtual scene; or, The target scene object includes all scene objects displayed in the first image frame.

9. The method according to claim 8, characterized in that, In response to the target scene object including the static object, Based on the second spatial location, the step of obtaining the screen space coordinates of each pixel in the target triangle in the first image frame, where the screen space coordinates include UV coordinates and depth values, and mapping the screen space coordinates of each pixel in the target triangle in the first image frame to the screen space coordinates of each pixel in the target triangle in the second image frame to obtain the second scene depth texture map includes: Based on the second spatial position, the screen space coordinates of each pixel in the target triangle face corresponding to the first image frame are obtained. The screen space coordinates include UV coordinates and depth values. The screen space coordinates of each pixel in the target triangle face corresponding to the first image frame are mapped to the screen space coordinates of each pixel in the target triangle face corresponding to the second image frame to obtain the scene depth texture map corresponding to the static object. Based on the second camera parameters, the moving objects in the virtual scene are drawn and rendered to obtain the scene depth texture map corresponding to the moving objects; Based on the scene depth texture map corresponding to the static object and the scene depth texture map corresponding to the moving object, the second scene depth texture map is obtained. The step of sampling color values ​​from the first scene color texture map to obtain the second scene color texture map based on the UV coordinates of each pixel corresponding to the target triangle in the first image frame includes: Based on the UV coordinates of each pixel of the target triangle in the first image frame, color values ​​are sampled from the first scene color texture map to obtain the scene color texture map corresponding to the static object. Based on the second camera parameters, the moving objects in the virtual scene are drawn and rendered to obtain the scene color texture map corresponding to the moving objects; The second scene color texture map is obtained based on the scene color texture map corresponding to the static object and the scene color texture map corresponding to the moving object.

10. The method according to claim 9, characterized in that, The step of obtaining the screen space coordinates of each pixel in the target triangle in the first image frame based on the second spatial position, wherein the screen space coordinates include UV coordinates and depth values, and mapping the screen space coordinates of each pixel in the target triangle in the first image frame to the screen space coordinates of each pixel in the target triangle in the second image frame to obtain the scene depth texture map corresponding to the static object, includes: Through the first rendering pipeline, based on the second spatial position, the screen space coordinates of each pixel in the target triangle face corresponding to the first image frame are obtained. The screen space coordinates include UV coordinates and depth values. The screen space coordinates of each pixel in the target triangle face corresponding to the first image frame are mapped to the screen space coordinates of each pixel in the target triangle face corresponding to the second image frame to obtain the scene depth texture map corresponding to the static object. The step of rendering moving objects in the virtual scene based on the second camera parameters to obtain a scene depth texture map corresponding to the moving objects includes: Using the second rendering pipeline, based on the second camera parameters, the moving objects in the virtual scene are drawn and rendered to obtain the scene depth texture map corresponding to the moving objects; The step of sampling color values ​​from the first scene color texture map to obtain the scene color texture map corresponding to the static object, based on the UV coordinates of each pixel of the target triangle in the first image frame, includes: Through the first rendering pipeline, based on the UV coordinates of each pixel of the target triangle in the first image frame, color values ​​are sampled from the first scene color texture map to obtain the scene color texture map corresponding to the static object. The step of rendering moving objects in the virtual scene based on the second camera parameters to obtain a scene color texture map corresponding to the moving objects includes: Using the second rendering pipeline, based on the second camera parameters, the moving objects in the virtual scene are drawn and rendered to obtain the scene color texture map corresponding to the moving objects.

11. An image display device for a virtual scene, characterized in that, The device includes: The first texture acquisition module is used to acquire the first scene depth texture map and the first scene color texture map of the first image frame; the first image frame is the image frame generated in the most recent rendering of the virtual scene. The first position acquisition module is used to determine the pixel with the largest gradient change in each image block corresponding to the target scene object in the first scene depth texture map, so as to obtain the pixel corresponding to the vertex of the target triangle in the first scene depth texture map; based on the pixel corresponding to the vertex of the target triangle in the first scene depth texture map, obtain the first spatial position of the vertex of the target triangle in the first clipping space, the first spatial position including vertex spatial position and vertex UV coordinates; the target triangle is the triangle of the target scene object in the first image frame; the first clipping space is the clipping space corresponding to the first camera parameters of the first image frame. The second position acquisition module is used to map the first spatial position to a second spatial position in a second clipping space based on the first camera parameters and the current second camera parameters; the second clipping space is the clipping space corresponding to the second camera parameters. The second texture acquisition module is used to generate a second scene depth texture map and a second scene color texture map of the second image frame based on the second spatial location and the first scene color texture map; The display module is used to display the second image frame based on the second scene depth texture map and the second scene color texture map.

12. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one computer instruction, which is loaded and executed by the processor to implement the image display method for a virtual scene as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer instruction, which is loaded and executed by a processor to implement the image display method for a virtual scene as described in any one of claims 1 to 10.

14. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium; the computer instructions are read and executed by a processor of a computer device to implement the image display method for a virtual scene as described in any one of claims 1 to 10.

Images