User interface display method and device, equipment, storage medium and program product

Abstract

The invention provides a user interface display method and device, equipment, a storage medium and a program product, a graphical user interface is provided through terminal equipment, the content displayed by the graphical user interface comprises a virtual scene, the virtual scene comprises a three-dimensional user interface, and the method comprises the steps of determining a depth map of the virtual scene, and displaying the three-dimensional user interface based on the depth map. Determining whether the three-dimensional user interface is shielded by a virtual object in the virtual scene; in response to determining that the three-dimensional user interface is shielded by the virtual object in the virtual scene, determining a visibility factor of pixels of the three-dimensional user interface based on the depth map; and rendering the pixels of the three-dimensional user interface based on the visibility factor to adjust the visibility of the pixels of the three-dimensional user interface, so that the part, shielded by the virtual object, of the three-dimensional user interface is invisible. According to the method and the device, the visibility of the three-dimensional user interface blocked by the virtual object is represented through semitransparent rendering, and anti-aliasing is realized on the basis of keeping the original effect of the three-dimensional user interface.

Description

Technical Field

[0001] The present disclosure relates to the field of electronic computers and Internet technology, and in particular to a method, device, equipment, storage medium and program product for displaying a user interface. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] In electronic games implemented based on electronic computers and Internet technologies, a user interface (UI) for interacting with users is usually displayed.

[0004] In the 3D virtual scene provided by electronic games, there is a three-dimensional user interface. Unlike a 2D UI that is either fully displayed or completely disappeared, the three-dimensional user interface may be blocked by other objects in the virtual scene and only partially displayed.

[0005] However, when the 3D user interface is blocked by other objects in the virtual scene and only part of it is displayed, the edge between the displayed part of the 3D user interface and the object blocking the 3D user interface is not smooth, usually called aliasing, and the display effect is poor. Summary of the invention

[0006] In view of this, the purpose of the present disclosure is to provide a method, device, equipment, storage medium and program product for displaying a user interface, which at least to a certain extent solve one of the technical problems in the related art.

[0007] Based on the above purpose, the first aspect of the exemplary embodiment of the present disclosure provides a method for displaying a user interface, wherein a graphical user interface is provided through a terminal device, and the content displayed by the graphical user interface includes a virtual scene, and the virtual scene includes a three-dimensional user interface. The method includes:

[0008] Determine a depth map of the virtual scene, and determine whether the three-dimensional user interface is blocked by a virtual object in the virtual scene based on the depth map;

[0009] In response to determining that the three-dimensional user interface is blocked by the virtual object in the virtual scene, determining a visibility factor of a pixel of the three-dimensional user interface based on the depth map;

[0010] The pixels of the three-dimensional user interface are rendered based on the visibility factor to adjust the visibility of the pixels of the three-dimensional user interface so that a portion of the three-dimensional user interface blocked by the virtual object is invisible.

[0011] Based on the same inventive concept, a second aspect of the exemplary embodiment of the present disclosure provides a display device for a user interface, which provides a graphical user interface through a terminal device, wherein the content displayed by the graphical user interface includes a virtual scene, and the virtual scene includes a three-dimensional user interface, and the device includes:

[0012] a depth map determination module, configured to determine a depth map of the virtual scene, and determine whether the three-dimensional user interface is blocked by a virtual object in the virtual scene based on the depth map;

[0013] a visibility factor determination module, configured to determine a visibility factor of a pixel of the three-dimensional user interface based on the depth map in response to determining that the three-dimensional user interface is blocked by the virtual object in the virtual scene;

[0014] The visibility rendering module is configured to render the pixels of the three-dimensional user interface based on the visibility factor to adjust the visibility of the pixels of the three-dimensional user interface so that the portion of the three-dimensional user interface blocked by the virtual object is invisible.

[0015] Based on the same inventive concept, a third aspect of the exemplary embodiment of the present disclosure provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in the first aspect is implemented.

[0016] Based on the same inventive concept, a fourth aspect of the exemplary embodiments of the present disclosure provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method described in the first aspect.

[0017] Based on the same inventive concept, a fifth aspect of the exemplary embodiments of the present disclosure provides a computer program product, including computer program instructions. When the computer program instructions are executed on a computer, the computer executes the method described in the first aspect.

[0018] As can be seen from the above, the user interface display method, device, equipment, storage medium and program product provided by the embodiments of the present disclosure provide a graphical user interface through a terminal device, the content displayed by the graphical user interface includes a virtual scene, and the virtual scene includes a three-dimensional user interface. The method includes: determining a depth map of the virtual scene, and determining whether the three-dimensional user interface is blocked by a virtual object in the virtual scene based on the depth map; in response to determining that the three-dimensional user interface is blocked by the virtual object in the virtual scene, determining the visibility factor of the pixels of the three-dimensional user interface based on the depth map; rendering the pixels of the three-dimensional user interface based on the visibility factor to adjust the visibility of the pixels of the three-dimensional user interface, so that the part of the three-dimensional user interface blocked by the virtual object is invisible. The present disclosure shows the visibility of the three-dimensional user interface after being blocked by the virtual object through semi-transparent rendering, and realizes anti-aliasing on the basis of maintaining the native effect of the three-dimensional user interface. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A schematic diagram comparing a two-dimensional user interface and a three-dimensional user interface provided for an exemplary embodiment of the present disclosure;

[0021] Figure 2a Another comparative schematic diagram of a two-dimensional user interface and a three-dimensional user interface provided for an exemplary embodiment of the present disclosure;

[0022] Figure 2b Another comparative schematic diagram of a two-dimensional user interface and a three-dimensional user interface provided for an exemplary embodiment of the present disclosure;

[0023] Figure 3 A schematic diagram of a sawtooth of a three-dimensional user interface provided by an exemplary embodiment of the present disclosure;

[0024] Figure 4 A schematic diagram of an application scenario of a method for displaying a user interface provided by an exemplary embodiment of the present disclosure;

[0025] Figure 5 A schematic diagram of a flow chart of a method for displaying a user interface provided by an exemplary embodiment of the present disclosure;

[0026] Figure 6aA schematic diagram of a sampling scenario provided for an exemplary embodiment of the present disclosure;

[0027] Figure 6b Another schematic diagram of a sampling scenario provided for an exemplary embodiment of the present disclosure;

[0028] Figure 7 A schematic diagram of a sampling process provided for an exemplary embodiment of the present disclosure;

[0029] Figure 8a A schematic diagram of the original effect of sawtooth of a three-dimensional user interface provided by an exemplary embodiment of the present disclosure;

[0030] Figure 8b A schematic diagram of an anti-aliasing effect of a three-dimensional user interface provided by an exemplary embodiment of the present disclosure;

[0031] Figure 8c Another schematic diagram of the anti-aliasing effect of a three-dimensional user interface provided by an exemplary embodiment of the present disclosure;

[0032] Fig. 9 A schematic diagram of a structure of a display device of a user interface provided by an exemplary embodiment of the present disclosure;

[0033] Fig.10 A schematic structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] It is understandable that before using the technical solutions disclosed in the embodiments of this application, the type, scope of use, usage scenarios, etc. of the personal information involved in this application should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0035] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present application according to the prompt message.

[0036] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0037] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation method of the present application. Other methods that meet the relevant laws and regulations may also be applied to the implementation method of the present application.

[0038] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0039] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, the principles and spirit of the present disclosure will be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0040] It should be understood herein that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction rather than having any limiting meaning.

[0041] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connecting" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. The article "one" or "a" before an element does not exclude the existence of multiple such elements.

[0042] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure.

[0043] As described in the background art, in electronic games implemented based on electronic computers and Internet technologies, a user interface (UI) for interacting with users is usually displayed.

[0044] As a specific example, the user interface is configured to display information such as the name of an NPC or the name of a monster, a health bar, etc.

[0045] In the three-dimensional (3D) virtual scene provided by electronic games, there is a three-dimensional user interface (3DUI). Different from a two-dimensional user interface (2DUI) that is either fully displayed or completely disappeared, the 3D UI may be blocked by other objects in the virtual scene and only partially displayed.

[0046] As a specific example, see Figure 1 , in a three-dimensional virtual scene:

[0047] 2DUI will not be blocked by other virtual objects;

[0048] The 3D UI is blocked by virtual objects such as buildings and only part of it is displayed.

[0049] Among them, compared with 2DUI, 3DUI is integrated into the three-dimensional virtual scene and may be blocked by other virtual objects in the three-dimensional virtual scene, or even by pixel-level precise occlusion of the three-dimensional user interface, which improves the three-dimensional game effect and enhances the sense of immersion.

[0050] However, the inventors of the present disclosure discovered that when a three-dimensional user interface is blocked by other objects in a virtual scene and only part of it is displayed, the edge between the displayed part of the three-dimensional user interface and the object blocking the three-dimensional user interface is not smooth, commonly known as aliasing, and the display effect is poor.

[0051] As a specific example:

[0052] The conventional rendering method of 3DUI is to render the prepared UI (including fonts, pictures, etc.) into a texture. The effect in this texture is the native effect of UI (that is, the effect of 2DUI directly pasted on the screen). Then paste this texture on a plane Mesh, and then integrate it into the rendering process of other Meshes through the plane Mesh for rendering. This is also the rendering method of 3DUI in the UE engine, which regards 3DUI as a Mesh in the scene. By observing and comparing the rendering process and effect of 2DUI with native production effect and 3DUI implemented in the UE engine based on this method, it is found that because 3DUI is rendered through the carrier of Mesh, it has more post-processing processes such as TSR / TAA anti-aliasing, Bloom, ToneMapping, etc. compared with 2DUI, and the final effect is significantly different. In fact, when the UI is natively rendered in 2D space, the fonts are rendered according to the corresponding font size and have post-processing effects such as anti-aliasing. The health bar and other image content are also produced as the final effect. Therefore, when pursuing native effects, the 3D Mesh post-processing process of the UI is redundant. The anti-aliasing process in the 3D engine post-processing will make the fonts that have been anti-aliased in the native font drawing process blurred. The Bloom process will illuminate other non-UI areas on the screen, and the ToneMapping process will change the original tone of the UI.

[0053] As a specific example, see Figure 2a and Figure 2b , taking the UI in a 3D virtual scene built by Unreal Engine (UE) as an example:

[0054] in, Figure 2a The same UI is shown in 3DUI and 2DUI in the daytime. It can be seen that 2DUI is the original effect, while 3DUI is more blurred and the color tone has changed after post-processing.

[0055] Figure 2b The following figure shows the display effect of the same UI set to 3DUI and 2DUI in a dark environment. It can be seen that the 3DUI self-luminous bloom will illuminate the surroundings, but the 3DUI is more blurred and the color tone has changed after post-processing.

[0056] As a specific example, see Figure 3 :

[0057] When pursuing the native effect of UI, the engine can be modified to extract the carrier plane mesh of 3DUI from all meshes for rendering to skip the post-processing process, so that the final screen can display the native rendering effect. However, since 3DUI needs to be integrated into the scene and have an accurate occlusion relationship with other 3D objects, it needs to perform a depth test with the scene depth composed of other 3D objects. The scene depth has jagged edges, which will cause jagged edges on the occluded edges of 3DUI.

[0058] Moreover, when the UE engine turns on temporal anti-aliasing TAA / TSR, the jitter of the scene depth in each frame will also cause obvious jitter on the occluded edges of the 3DUI that has not undergone unified anti-aliasing processing by the engine.

[0059] As a result, if the 3D engine's anti-aliasing processing is performed, the UI will become blurred, while if the 3D engine's anti-aliasing processing is not performed, the edges of the UI that are blocked will become jagged. It is difficult to retain the native effect of the UI while achieving pixel-level accurate occlusion effects for the three-dimensional user interface.

[0060] In order to solve the above problems, the present disclosure provides a user interface display solution, which specifically includes: determining the depth map of the virtual scene, and determining whether the three-dimensional user interface is blocked by the virtual object in the virtual scene based on the depth map; in response to determining that the three-dimensional user interface is blocked by the virtual object in the virtual scene, determining the visibility factor of the pixels of the three-dimensional user interface based on the depth map; rendering the pixels of the three-dimensional user interface based on the visibility factor to adjust the visibility of the pixels of the three-dimensional user interface so that the part of the three-dimensional user interface blocked by the virtual object is invisible. The present disclosure shows the visibility of the three-dimensional user interface after being blocked by the virtual object through semi-transparent rendering, and realizes anti-aliasing on the basis of maintaining the original effect of the three-dimensional user interface.

[0061] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0062] refer to Figure 4 , which is a schematic diagram of an application scenario of the user interface display method provided by an exemplary embodiment of the present disclosure.

[0063] This application scenario includes a terminal device 101, a server 102, and a data storage system 103. The terminal device 101, the server 102, and the data storage system 103 may be connected via a wired or wireless communication network to achieve data interaction.

[0064] The terminal device 101 may be an electronic device with data transmission and multimedia input / output functions close to the user side, including but not limited to a desktop computer, a mobile phone, a mobile computer, a tablet computer, a media player, a smart wearable device, a personal digital assistant (PDA) or other electronic devices capable of implementing the above functions. The electronic device may include a processor and a display screen with a touch input function, the display screen is used to present a graphical user interface, the graphical user interface can display an application interface, and the processor is used to process application data, generate a graphical user interface, and control the display of the graphical user interface on the display screen.

[0065] Server 102 and data storage system 103 can both be independent physical servers, or a server cluster or distributed system composed of multiple physical servers. They can also be cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0066] In some exemplary embodiments, the method for displaying a user interface provides a graphical user interface through the terminal device 101, and the content displayed by the graphical user interface includes a virtual scene, and the virtual scene includes a three-dimensional user interface.

[0067] In specific implementation, the graphical user interface is obtained by executing a software application on the processor of the terminal device 101 and rendering it on the display, which can be a display screen interface of the terminal device 101. The graphical user interface can present the entire game scene, or only a part of the game scene. The game scene includes multiple static virtual objects, including the ground, mountains, rocks, vegetation, buildings, etc. When the game scene is relatively large, only the partial content of the game scene is displayed on the graphical user interface of the terminal device 101 during the game.

[0068] The virtual scene is a scene displayed (or provided) when the application is running on the terminal device 101 or the server 102. Optionally, the virtual scene is a simulation environment of the real world, or a semi-simulation and semi-fictitious virtual environment, or a purely fictitious virtual environment. The virtual scene is any one of a two-dimensional virtual scene and a three-dimensional virtual scene, and the virtual environment can be the sky, land, ocean, etc., wherein the land includes environmental elements such as deserts and cities. Among them, the virtual scene is a scene with complete game logic of virtual objects such as user control. For example, for sandbox 3D shooting games, the virtual scene is a 3D game world for players to control virtual objects to fight. Example virtual scenes may include: at least one element of mountains, plains, rivers, lakes, oceans, deserts, skies, plants, buildings, and vehicles; for example, for 2D card games, the virtual scene is a scene for displaying released cards or displaying virtual objects corresponding to cards. Example virtual scenes may include: an arena, a decisive battlefield, or other "field" elements or other elements that can display the status of card battles; for 2D or 3D multiplayer online tactical competitive games, the virtual scene is a 2D or 3D terrain scene for virtual objects to fight. Example virtual scenes may include: canyon-style mountains, lines, rivers, classrooms, tables and chairs, podiums and other elements.

[0069] Three-dimensional user interface (3DUI) is a user interface created using three-dimensional graphics and interactive technology. Compared with traditional two-dimensional interfaces, 3DUI provides a richer visual experience and a more intuitive way of interaction. 3DUI can create an immersive environment, making users feel like they are in a three-dimensional space, enhancing the user's sense of participation and experience. By utilizing the characteristics of three-dimensional space, 3DUI can display information and data more effectively, allowing users to perceive information from different angles and depths. 3DUI allows users to interact with the interface through natural methods such as gestures, touch, and sound, improving the intuitiveness and ease of operation.

[0070] The data storage system 103 is used to assist the terminal device 101 and the server 102 in storing data.

[0071] Combine the following Figure 4 The display method of the user interface according to the exemplary embodiment of the present disclosure is described by using the application scenario. It should be noted that the above application scenario is only shown to facilitate understanding of the spirit and principle of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0072] refer to Figure 5 , which is a flow chart of a method for displaying a user interface provided by an exemplary embodiment of the present disclosure.

[0073] A method for displaying a user interface provides a graphical user interface through a terminal device, wherein the content displayed by the graphical user interface includes a virtual scene, and the virtual scene includes a three-dimensional user interface.

[0074] The method for displaying a user interface comprises the following steps:

[0075] Step S510: Determine a depth map of the virtual scene, and based on the depth map, determine whether the three-dimensional user interface is blocked by a virtual object in the virtual scene.

[0076] In specific implementation, a depth map is an image that records the distance information of each pixel in the virtual scene relative to the camera or sensor. In the depth map, color is usually used to represent the depth value, and the darker the color, the farther the distance, while the lighter the color, the closer the distance.

[0077] In specific implementation, the depth map can be generated by a dedicated depth camera (such as Microsoft's Kinect or Intel's RealSense), or obtained through other technologies such as stereo vision, time of flight (ToF), etc.

[0078] In a specific implementation, the virtual scene may include other virtual objects in addition to the three-dimensional user interface (3DUI). These virtual objects (including the three-dimensional user interface) may occlude each other, that is, one or more of these virtual objects may occlude the three-dimensional user interface.

[0079] The following describes a method for determining whether a 3D user interface is blocked by a virtual object in a virtual scene. Specifically:

[0080] In some exemplary embodiments, determining whether the three-dimensional user interface is blocked by a virtual object in the virtual scene based on the depth map includes:

[0081] Determine a first depth value of a pixel of the three-dimensional user interface in the depth map;

[0082] determining a depth buffer value for a pixel of the three-dimensional user interface;

[0083] In response to determining that the first depth value is less than the depth buffer value, determining that the three-dimensional user interface is obscured by the virtual object in the virtual scene;

[0084] or,

[0085] In response to determining that the first depth value is greater than or equal to the depth buffer value, it is determined that the three-dimensional user interface is not obstructed by the virtual object in the virtual scene.

[0086] Step S520: In response to determining that the three-dimensional user interface is blocked by the virtual object in the virtual scene, determine a visibility factor of a pixel of the three-dimensional user interface based on the depth map.

[0087] During specific implementation, the visibility factor of the pixels of the three-dimensional user interface is a variable used to adjust the visual visibility of the pixels of the three-dimensional user interface.

[0088] In a specific implementation, the visibility factor is a value between 0 (complete occlusion) and 1 (completely unoccluded).

[0089] Next, the method of determining the visibility factor of a pixel of a three-dimensional user interface is described, specifically:

[0090] In some exemplary embodiments, in response to determining that the 3D user interface is blocked by the virtual object in the virtual scene, determining the visibility factor of the pixel of the 3D user interface based on the depth map includes:

[0091] For each pixel of the three-dimensional user interface, determining a first coordinate value of the pixel of the three-dimensional user interface in the depth map;

[0092] Based on the first coordinate value, determining a number of sampling points within a preset range around a pixel of the three-dimensional user interface;

[0093] For each of the sampling points, determining a second depth value of the sampling point in the depth map, and obtaining a shadow contribution of the sampling point based on the first depth value and the second depth value;

[0094] An average value of the shadow contribution of all the sampling points corresponding to the pixels of the three-dimensional user interface is used as the visibility factor of the pixels of the three-dimensional user interface.

[0095] As a specific example, the visibility factor of a pixel of a three-dimensional user interface is determined by the following formula:

[0096] VisibityFactor=PCF(UV,z,SceneDepthTexture);

[0097] Among them, VisibityFactor is the visibility factor of the pixels of the three-dimensional user interface, PCF (Percentage-Closer Filtering) is the algorithm used to determine the visibility factor of the pixels of the three-dimensional user interface, UV is the first coordinate value of the pixels of the three-dimensional user interface in the depth map of the virtual scene, z is the first depth value of the pixels of the three-dimensional user interface in the depth map of the virtual scene, and SceneDepthTexture is the depth map of the virtual scene.

[0098] Step S530: Render the pixels of the three-dimensional user interface based on the visibility factor to adjust the visibility of the pixels of the three-dimensional user interface so that the portion of the three-dimensional user interface blocked by the virtual object is invisible.

[0099] In some exemplary embodiments, rendering the pixels of the three-dimensional user interface based on the visibility factor includes:

[0100] Multiplying the original alpha channel value of the pixel of the three-dimensional user interface by the visibility factor to obtain an updated alpha channel value;

[0101] Pixels of the three-dimensional user interface are rendered based on the updated alpha channel value.

[0102] In a specific implementation, the alpha channel of the 3D user interface is used to define the transparency of the 3D user interface. The alpha channel is an additional color channel that stores transparency information of each pixel of the 3D user interface.

[0103] As a specific example, the alpha channel value is updated using the following formula:

[0104] A' = A*VisibityFactor;

[0105] Wherein, A' is the updated alpha channel value, A is the original alpha channel value, and VisibityFactor is the visibility factor of the pixel of the three-dimensional user interface.

[0106] Through the above exemplary embodiments, the visibility of the three-dimensional user interface after being blocked by the virtual object is expressed through semi-transparent rendering.

[0107] In the above exemplary embodiments, on the one hand, since the anti-aliasing processing in the post-processing of the three-dimensional user interface in the related art is not adopted, the three-dimensional user interface will not appear blurry and the color tone will not change due to the anti-aliasing processing in the post-processing of the three-dimensional user interface, and the original effect of the three-dimensional user interface is maintained, that is, clarity and the color tone will not change; on the other hand, in order to solve the problem of aliasing caused by not adopting the anti-aliasing processing in the post-processing of the three-dimensional user interface in the related art, the present disclosure realizes anti-aliasing by expressing the visibility of the three-dimensional user interface after being blocked by a virtual object through translucent rendering.

[0108] In the above exemplary embodiments, the present disclosure implements anti-aliasing while maintaining the original effect of the 3D user interface (i.e., clarity and no change in color tone). However, the present disclosure is not limited thereto, and below, based on the above exemplary embodiments, the anti-aliasing effect will be further optimized.

[0109] Next, we will first introduce how to solve the jagged problem caused by the uneven placement of the 3D user interface itself and insufficient sampling due to scaling. Specifically:

[0110] To solve the jagged problem caused by the irregular placement of the 3D user interface, the present disclosure makes the 3D user interface always face the virtual camera, and makes the x-axis and y-axis of the 3D user interface parallel to the virtual camera. Specifically:

[0111] In some exemplary embodiments, before determining the depth map of the virtual scene, the method further includes:

[0112] Determining a first orientation of a virtual camera in the virtual scene;

[0113] determining a second orientation of the three-dimensional user interface based on the first orientation;

[0114] Determining a first local coordinate system of the virtual camera;

[0115] A second local coordinate system of the three-dimensional user interface is aligned with the first local coordinate system.

[0116] In specific implementation, the three-dimensional user interface is always oriented toward the front viewing direction of the virtual camera, and the x-axis and y-axis of the three-dimensional user interface are parallel to the virtual camera, which can be achieved by the following steps:

[0117] Get the virtual camera's viewing angle: First, you need to get the current virtual camera's viewing angle matrix, which contains the virtual camera's orientation information.

[0118] Calculate the orientation of the 3D user interface: Use the virtual camera's viewing matrix to calculate the direction the 3D user interface needs to face. Calculate the vector opposite to the virtual camera's forward viewing direction, and use this vector to set the rotation of the 3D user interface.

[0119] Set the rotation of the 3D user interface: Set the rotation of the 3D user interface so that it faces the virtual camera's forward viewing direction. Use quaternions or Euler angles to represent the rotation.

[0120] Keep the x-axis and y-axis parallel to the virtual camera: Ensure that the local coordinate system of the 3D user interface is aligned with the local coordinate system of the virtual camera. Perform matrix operations and use the inverse matrix of the virtual camera to transform the coordinates of the 3D user interface.

[0121] Update Transformation Matrix: Apply the above calculated rotation to the local transformation matrix of the 3D UI. This will ensure that the 3D UI is correctly oriented in the virtual camera's forward viewing direction in world space.

[0122] Rendering: In the rendering loop, use the updated transformation matrix to render the 3D user interface.

[0123] To solve the aliasing problem caused by insufficient sampling due to scaling, the present disclosure uses inverse perspective projection calculation to control the resolution of the 3D user interface rendered on the final screen to be the same as the native effect resolution. Specifically:

[0124] In some exemplary embodiments, before determining the depth map of the virtual scene, the method further includes:

[0125] Determining a resolution of a display screen of the terminal device, and determining a two-dimensional coordinate of the three-dimensional user interface on the display screen based on the resolution;

[0126] Determine a projection matrix of the virtual scene, determine an inverse matrix of the projection matrix, convert the two-dimensional coordinates into three-dimensional coordinates of the three-dimensional user interface in the virtual scene based on the inverse matrix, and adjust the size of the three-dimensional user interface based on the three-dimensional coordinates.

[0127] In specific implementation, the resolution of the 3D user interface rendered on the final screen is controlled to be the same as the native effect resolution, which can be achieved by the following steps:

[0128] Get the screen resolution: First, get the screen resolution, the width and height in pixels of the 3D user interface.

[0129] Get projection matrix: Get the projection matrix used by the current virtual scene. The projection matrix can be an orthogonal projection matrix or a perspective projection matrix, depending on the settings of the virtual scene.

[0130] Inverse matrix calculation: Calculate the inverse matrix of the projection matrix. The inverse matrix is ​​the inverse operation of the original matrix, which can be used to convert the two-dimensional screen coordinates back to three-dimensional coordinates.

[0131] 3D UI coordinate conversion: Use the inverse matrix to convert the screen coordinates of the 3D UI to coordinates in 3D space. Normalize the screen coordinates (i.e. divide by the screen resolution) and then apply the inverse matrix.

[0132] Adjust the size of 3D UI: According to the converted 3D coordinates, adjust the size of 3D UI to ensure that they appear the same as native effects on the screen.

[0133] Apply Transforms: Applies resized 3D UIs to the rendering pipeline, ensuring they appear at the correct resolution on the final rendered screen.

[0134] Test and adjust: Test the display of your 3D UI on different resolutions and screen sizes, and make adjustments as needed to ensure consistent display on all target devices.

[0135] In the above exemplary embodiments, a solution to the aliasing problem caused by the irregular arrangement of the 3D user interface itself and insufficient sampling due to scaling is introduced. Next, how to solve the aliasing problem caused by screen space jitter will be introduced, specifically:

[0136] When the 3D engine enables temporal anti-aliasing, it will encounter jitter problems caused by the original temporal anti-aliasing in the 3D engine rendering pipeline. For example, after the commonly used temporal anti-aliasing TAA / TSR method is turned on in the UE, each 3D object will generate random jitter in the screen space when the View space is transformed to the Clip space. In the related art, in the post-processing of 3DUI, the temporal anti-aliasing TAA / TSR can eliminate this jitter. However, in the present disclosure, since 3DUI has not been post-processed, it and other 3D objects will generate random jitters in each frame when judging the edges of occlusion in depth testing. Although the solution provided by the above exemplary embodiment can significantly reduce aliasing, the jitter will still cause the VisibityFactor value calculated by PCF to be unstable and produce a jitter effect.

[0137] For the jitter of the 3D user interface itself, the present disclosure finds the screen space jitter value and restores it to the position before the jitter, specifically:

[0138] In some exemplary embodiments, determining, for each pixel of the three-dimensional user interface, a first coordinate value of the pixel of the three-dimensional user interface in the depth map includes:

[0139] In response to screen space jitter occurring in the three-dimensional user interface, determining a screen space jitter value;

[0140] Based on the screen space jitter value, compensating a current position of the 3D user interface to restore the 3D user interface to a position before the screen space jitter occurs;

[0141] Based on the position before the screen space jitter occurs, the first coordinate value of the pixel of the three-dimensional user interface in the depth map is determined.

[0142] In specific implementation, restoring the shaking 3D user interface to the position before the shaking can be achieved by the following steps:

[0143] Analyze the jitter pattern: First, you need to analyze the jitter pattern of the 3D UI. You need to observe multiple jittering 3D UIs or multiple jitters of one 3D UI to determine whether the jitter is random, periodic, or follows a certain pattern.

[0144] Determine the range of jitter: Measures the deviation between the jittered 3D UI and the original 3D UI. Performs pixel-level analysis on the 3D UI to determine the maximum and minimum deviation of the jitter.

[0145] Build a jitter model: Based on the analysis results, try to build a jitter model that describes how the 3D user interface shifts in screen space.

[0146] Inverse dither model: Based on the dither model, reverse the model to find a way to restore the 3D UI to its original position. Apply pixel-level shifting to the 3D UI, or more complex 3D UI processing techniques such as filtering.

[0147] Apply Inverse Transformation: Applies an inverse transformation to the jittered 3D UI to try to restore its original position.

[0148] Evaluate results: After applying the inverse transformation, the quality of the resulting 3D user interface needs to be evaluated. If the results are not satisfactory, the dithering model or the inverse transformation method needs to be adjusted.

[0149] Optimize algorithm: Based on the evaluation results, optimize the inverse algorithm to improve the accuracy and quality of restoring the 3D user interface.

[0150] For the jitter of other virtual objects except the three-dimensional user interface, the present disclosure samples the depth of the current pixel before the jitter according to the offset of the screen space jitter vector of each frame for comparison, specifically:

[0151] In some exemplary embodiments, determining, for each of the sampling points, a second depth value of the sampling point in the depth map comprises:

[0152] In response to screen space jitter of the virtual object, determining a jitter vector of a target frame and a depth map of the target frame of the virtual scene;

[0153] Compensating the second coordinate value of the sampling point based on the jitter vector to obtain a compensated second coordinate value;

[0154] Sampling is performed from the depth map of the target frame based on the compensated second coordinate value to obtain the second depth value.

[0155] As a specific example, this is achieved through the following formula:

[0156] z=SceneDepthTexture_nojitter.sample(UV)=SceneDepthTexture.sample(UV+Jitter(i))

[0157] Among them, z is the depth value before pixel jitter, sample represents sampling, SceneDepthTexture_nojitter is the depth map of the virtual scene assumed to be jitter-free, SceneDepthTexture is the depth map of the virtual scene with jitter in the i-th frame, UV is the UV value of the sampling point, and Jitter(i) is the jitter vector of the current i-th frame.

[0158] In the above exemplary embodiments, a solution to the aliasing problem caused by screen space jitter is introduced. However, the inventors of the present disclosure have found that the method of directly restoring jitter combined with PCF anti-aliasing can alleviate both jitter and aliasing problems to a certain extent, but there are still some aliasing that can be observed by the naked eye. The reason is that the depth map sampling rendered only at a single screen resolution of the original scene rendering is not enough to achieve a more perfect anti-aliasing effect, especially when the magnification of the TSR process in post-processing is large, the resolution of the original scene rendering is not enough compared to the magnified screen resolution. Anti-aliasing algorithms such as TAA / TSR all need to logically expand the screen resolution in the time domain to achieve a good anti-aliasing effect, so it is also necessary to expand the resolution of the depth map sampled by PCF, and this process can just use the historical jitter of each frame combined with the principle of time domain anti-aliasing to construct a super-sampled depth map to improve the accuracy of the PCF process. Specifically:

[0159] In some exemplary embodiments, determining the depth map of the virtual scene includes:

[0160] Determine a jittered depth map of each frame of the virtual scene;

[0161] Based on the jittered depth map of each frame, a super-resolution depth map is obtained.

[0162] In some exemplary embodiments, obtaining a super-resolution depth map based on the jittered depth map of each frame includes:

[0163] Determining a depth map of the current frame jitter of the virtual scene;

[0164] Determining whether it is necessary to update the super-resolution depth map based on the jittered depth map of the current frame;

[0165] In response to determining that the super-resolution depth map needs to be updated based on the current frame jittered depth map, the super-resolution depth map is updated based on the current frame jittered depth map.

[0166] In some exemplary embodiments, the determining whether the super-resolution depth map needs to be updated based on the jittered depth map of the current frame includes:

[0167] Determine a third coordinate of the jittered current frame in the jittered depth map of the current frame and a fourth coordinate of the center of the pixel where the third coordinate is located;

[0168] In response to a distance between the third coordinate and the fourth coordinate being less than a distance threshold, determining that the super-resolution depth map needs to be updated based on the jittered depth map of the current frame;

[0169] or,

[0170] In response to the distance between the third coordinate and the fourth coordinate being greater than or equal to the distance threshold, it is determined that there is no need to update the super-resolution depth map based on the current frame jittered depth map.

[0171] In specific implementation, constructing a time-domain super-sampled depth map to improve the accuracy of PCF sampling can be achieved through the following steps:

[0172] Using the principle of temporal supersampling, it is necessary to construct a super-resolution (assuming 2*2 times the resolution) depth map (named TemporalUpsampleDepthTexture) for each frame in a temporal manner according to the jittered scene depth map SceneDepthTe xture of each frame, and use the super-resolution depth map to improve the accuracy of the above PCF sampling. In this process, each frame saves the depth map constructed by the previous frame (named PrevTemporalUpsampleDepthTexture).

[0173] Since TemporalUpsampleDepthTexture has a higher resolution and contains more information than the scene depth map of each frame, it is necessary to determine whether the value of each pixel in the TemporalUpsampleDepthTexture to be constructed in the current frame needs to be updated from the historical information in the SceneDepthTexture of the current frame. Since the formation principle of aliasing is that each pixel only uses the sampling result of the center point to represent the entire pixel area, the closer the sampling point is to the center point of the pixel, the higher the accuracy of using the information in the pixel to represent the sampling result (i.e., TemporalUpsampleDepthTexture has a large resolution but only the previous frame is not accurate enough, and SceneDepthTexture has a small resolution but the information representing the center position of the pixel is the latest and accurate). Therefore, the update judgment value (named bUpdateHistory) is set to whether the UV of the current pixel sampling point is close to the center point of the pixel in SceneDepthTexture.

[0174] The 3*3 pixel area near the sampling point K in SceneDepthTexture is as follows Figure 6a and Figure 6b As shown, O is the UV point after jitter (ie, UV+Jitter(i)), and K is the UV point at the center of the pixel where O is located.

[0175] Figure 6a and Figure 6b is the 3*3 pixel area around the sampling point K in SceneDepthTexture, Figure 6a The judgment case where O is far away from K (i.e. bUpdateHistory = false) is Figure 6b This is the case where O is closer to K (ie, bUpdateHistory = true).

[0176] Since the resolution of TemporalUpsampleDepthTexture is 2*2 times that of SceneDepthTexture, each pixel in SceneDepthTexture corresponds to 4 pixels in TemporalUpsampleDepth Texture. To determine whether the sampling point is close to the pixel center point of SceneDepthTexture, a pixel size in TemporalUpsampleDepthTexture is used as a measure. That is, if the distance |KO| between K and O is within the range of the measurement size, it is determined that the depth sampled in SceneDepthTexture is more accurate and can be used to update the history. Otherwise, the sampling result in SceneDepthTexture is rejected, and the corresponding value in PrevTemporalUpsampleDepthTexture is still used.

[0177] The value sampled from SceneDepthTexture with the same UV as the current pixel of TemporalUpsampleDepthTexture plus the jitter vector Jitter(i) of the current frame is set to new_z, as shown in the following formula:

[0178] new_z=SceneDepthTexture.sample(UV+Jitter(i))

[0179] The determination of whether to update history bUpdateHistory is as follows:

[0180] bUpdateHistory=PrevTemporalUpsampleDepthTexture is valid? (|KO|<(SceneDepthTexture_PixelSize / 4)):true

[0181] The specific method of using the corresponding value in PrevTemporalUpsampleDepthTexture is to first get the current frame clipping space coordinate ClipPos according to new_z and UV, as follows:

[0182] ClipPos=ClipPos(new_z,UV)

[0183] Then, according to the transformation of the camera viewport View from the current frame to the previous frame clipping space, the clipping space coordinates PrevClipPos in the previous frame are obtained, and the UV value Prev_UV in the previous frame is inferred as follows:

[0184] PrevClipPos=View.ClipToPrevClip(ClipPos)

[0185] Prev_UV=ClipPosToUV(PrevClipPos)

[0186] Sample the value Prev_z of Prev_UV in PrevTemporalUpsampleDepthTexture as follows:

[0187] Prev_z=PrevTemporalUpsampleDepthTexture.Sample(Prev_UV)

[0188] The obtained Prev_z is the depth in the previous frame. Its world coordinate PrevWorldPos needs to be inferred based on the spatial transformation of the camera viewport PrevView of the previous frame, as shown in the following formula:

[0189] PrevWorldPos=PrevView.DepthToWorldPos(Prev_z,Prev_UV)

[0190] According to the spatial transformation of the current frame camera viewport View, PrevWorldPos is converted into the depth Prev_z_now of the current frame, as follows:

[0191] Prev_z_now=View.WorldPosToDepth(PrevWorldPos)

[0192] Considering the situation where there may be moving objects at the same position, causing the depth to increase or decrease significantly, resulting in the complete failure of the depth of the previous frame, the latest information of SceneDepthTexture is needed to Clamp the value to the latest surrounding depth range. The correction process is as follows:

[0193] MinDepth,MaxDepth=SceneDepthTexture.SampleMinMax(K[3*3])

[0194] corrected_prev_z_now=Clamp(Prev_z_now,MinDepth,MaxDepth)

[0195] Where K[3*3] represents the 3*3 area centered on K in the above figure.

[0196] Finally, the depth value z of each pixel of the TemporalUpsampleDepthTexture that needs to be constructed in the current frame is updated as follows:

[0197] z=bUpdateHistory? new_z:corrected_prev_z_now

[0198] In summary, the generation process of each pixel in TemporalUpsampleDepthTexture can be expressed as follows Figure 7 .

[0199] Among them, from the 1st frame (t=1) to the current i-th frame (t=i), each frame uses the UV of the current pixel plus the jitter vector Jitter(t) to sample the depth after restoration of the jitter from the scene depth map SceneDepthTexture of the current frame to obtain new_z; when the history needs to be updated, that is, bUpdateHistory is true, the current pixel needs to be updated to new_z; when the history does not need to be updated, that is, bUpdateHistory is false, the current pixel needs to be updated to corrected_prev_z_now; the calculation of the corrected_prev_z_now value needs to find the corresponding UV from PrevTemporalUpsampleDepthTexture for sampling to obtain the depth and combine new_z and the transformation between the previous frame and the current frame of the camera viewport View to correct to the depth of the current frame.

[0200] After constructing a time-domain super-sampled depth map to improve the accuracy of PCF sampling, the final effect is almost invisible. The comparison of the before and after effects is as follows: Figure 8a , Figure 8b and Figure 8c As shown:

[0201] Figure 8a It shows the original jagged effect of the occluded edge after skipping the post-processing of 3DUI;

[0202] Figure 8b Shown in Figure 8a On the basis of PCF, semi-transparent rendering can achieve a certain degree of anti-aliasing effect;

[0203] Figure 8c Shown in Figure 8b On this basis, the final effect is achieved by constructing a time-domain super-sampled depth map to improve the accuracy of PCF sampling.

[0204] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0205] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0206] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a display device for a user interface.

[0207] refer to Fig. 9 , which is a structural schematic diagram of a display device of a user interface provided by an exemplary embodiment of the present disclosure.

[0208] A user interface display device provides a graphical user interface through a terminal device, wherein the content displayed by the graphical user interface includes a virtual scene, and the virtual scene includes a three-dimensional user interface.

[0209] The display device of the user interface includes the following modules:

[0210] A depth map determination module 910 is configured to determine a depth map of the virtual scene, and determine whether the three-dimensional user interface is blocked by a virtual object in the virtual scene based on the depth map;

[0211] A visibility factor determination module 920 is configured to determine a visibility factor of a pixel of the three-dimensional user interface based on the depth map in response to determining that the three-dimensional user interface is blocked by the virtual object in the virtual scene;

[0212] The visibility rendering module 930 is configured to render the pixels of the 3D user interface based on the visibility factor to adjust the visibility of the pixels of the 3D user interface so that the portion of the 3D user interface blocked by the virtual object is invisible.

[0213] In some exemplary embodiments, the depth map determination module 910 is configured to:

[0214] Determine a first depth value of a pixel of the three-dimensional user interface in the depth map;

[0215] determining a depth buffer value for a pixel of the three-dimensional user interface;

[0216] In response to determining that the first depth value is less than the depth buffer value, determining that the three-dimensional user interface is obscured by the virtual object in the virtual scene;

[0217] or,

[0218] In response to determining that the first depth value is greater than or equal to the depth buffer value, it is determined that the three-dimensional user interface is not obstructed by the virtual object in the virtual scene.

[0219] In some exemplary embodiments, the visibility factor determination module 920 is configured to:

[0220] For each pixel of the three-dimensional user interface, determining a first coordinate value of the pixel of the three-dimensional user interface in the depth map;

[0221] Based on the first coordinate value, determining a number of sampling points within a preset range around a pixel of the three-dimensional user interface;

[0222] For each of the sampling points, determining a second depth value of the sampling point in the depth map, and obtaining a shadow contribution of the sampling point based on the first depth value and the second depth value;

[0223] An average value of the shadow contribution of all the sampling points corresponding to the pixels of the three-dimensional user interface is used as the visibility factor of the pixels of the three-dimensional user interface.

[0224] In some exemplary embodiments, the visibility rendering module 930 is configured to:

[0225] Multiplying the original alpha channel value of the pixel of the three-dimensional user interface by the visibility factor to obtain an updated alpha channel value;

[0226] Pixels of the three-dimensional user interface are rendered based on the updated alpha channel value.

[0227] In some exemplary embodiments, the depth map determination module 910 is further configured to:

[0228] Determining a first orientation of a virtual camera in the virtual scene;

[0229] determining a second orientation of the three-dimensional user interface based on the first orientation;

[0230] Determining a first local coordinate system of the virtual camera;

[0231] A second local coordinate system of the three-dimensional user interface is aligned with the first local coordinate system.

[0232] In some exemplary embodiments, the depth map determination module 910 is further configured to:

[0233] Determining a resolution of a display screen of the terminal device, and determining a two-dimensional coordinate of the three-dimensional user interface on the display screen based on the resolution;

[0234] Determine a projection matrix of the virtual scene, determine an inverse matrix of the projection matrix, convert the two-dimensional coordinates into three-dimensional coordinates of the three-dimensional user interface in the virtual scene based on the inverse matrix, and adjust the size of the three-dimensional user interface based on the three-dimensional coordinates.

[0235] In some exemplary embodiments, the visibility factor determination module 920 is further configured to:

[0236] In response to screen space jitter occurring in the three-dimensional user interface, determining a screen space jitter value;

[0237] Based on the screen space jitter value, compensating a current position of the 3D user interface to restore the 3D user interface to a position before the screen space jitter occurs;

[0238] Based on the position before the screen space jitter occurs, the first coordinate value of the pixel of the three-dimensional user interface in the depth map is determined.

[0239] In some exemplary embodiments, the visibility factor determination module 920 is further configured to:

[0240] In response to screen space jitter of the virtual object, determining a jitter vector of a target frame and a depth map of the target frame of the virtual scene;

[0241] Compensating the second coordinate value of the sampling point based on the jitter vector to obtain a compensated second coordinate value;

[0242] Sampling is performed from the depth map of the target frame based on the compensated second coordinate value to obtain the second depth value.

[0243] In some exemplary embodiments, the depth map determination module 910 is further configured to:

[0244] Determine a jittered depth map of each frame of the virtual scene;

[0245] Based on the jittered depth map of each frame, a super-resolution depth map is obtained.

[0246] In some exemplary embodiments, the depth map determination module 910 is further configured to:

[0247] Determining a depth map of the current frame jitter of the virtual scene;

[0248] Determining whether it is necessary to update the super-resolution depth map based on the jittered depth map of the current frame;

[0249] In response to determining that the super-resolution depth map needs to be updated based on the current frame jittered depth map, the super-resolution depth map is updated based on the current frame jittered depth map.

[0250] In some exemplary embodiments, the depth map determination module 910 is further configured to:

[0251] Determine a third coordinate of the jittered current frame in the jittered depth map of the current frame and a fourth coordinate of the center of the pixel where the third coordinate is located;

[0252] In response to a distance between the third coordinate and the fourth coordinate being less than a distance threshold, determining that the super-resolution depth map needs to be updated based on the jittered depth map of the current frame;

[0253] or,

[0254] In response to the distance between the third coordinate and the fourth coordinate being greater than or equal to the distance threshold, it is determined that there is no need to update the super-resolution depth map based on the current frame jittered depth map.

[0255] For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0256] The device of the above embodiment is used to implement the corresponding user interface display method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0257] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments, the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for displaying the user interface described in any of the above embodiments is implemented.

[0258] Fig.10A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0259] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0260] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0261] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0262] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0263] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0264] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0265] The electronic device of the above embodiment is used to implement the corresponding user interface display method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0266] The memory 1020 stores machine-readable instructions executable by the processor 1010. When the electronic device is running, the processor 1010 communicates with the memory 1020 via the bus 1030, so that the processor 1010 executes the following instructions when running:

[0267] Determine a depth map of the virtual scene, and determine whether the three-dimensional user interface is blocked by a virtual object in the virtual scene based on the depth map;

[0268] In response to determining that the three-dimensional user interface is blocked by the virtual object in the virtual scene, determining a visibility factor of a pixel of the three-dimensional user interface based on the depth map;

[0269] The pixels of the three-dimensional user interface are rendered based on the visibility factor to adjust the visibility of the pixels of the three-dimensional user interface so that a portion of the three-dimensional user interface blocked by the virtual object is invisible.

[0270] In a possible implementation manner, in the instructions executed by the processor 1010, determining whether the three-dimensional user interface is blocked by a virtual object in the virtual scene based on the depth map includes:

[0271] Determine a first depth value of a pixel of the three-dimensional user interface in the depth map;

[0272] determining a depth buffer value for a pixel of the three-dimensional user interface;

[0273] In response to determining that the first depth value is less than the depth buffer value, determining that the three-dimensional user interface is obscured by the virtual object in the virtual scene;

[0274] or,

[0275] In response to determining that the first depth value is greater than or equal to the depth buffer value, it is determined that the three-dimensional user interface is not obstructed by the virtual object in the virtual scene.

[0276] In a possible implementation, in the instructions executed by the processor 1010, in response to determining that the three-dimensional user interface is blocked by the virtual object in the virtual scene, determining the visibility factor of the pixel of the three-dimensional user interface based on the depth map includes:

[0277] For each pixel of the three-dimensional user interface, determining a first coordinate value of the pixel of the three-dimensional user interface in the depth map;

[0278] Based on the first coordinate value, determining a number of sampling points within a preset range around a pixel of the three-dimensional user interface;

[0279] For each of the sampling points, determining a second depth value of the sampling point in the depth map, and obtaining a shadow contribution of the sampling point based on the first depth value and the second depth value;

[0280] An average value of the shadow contribution of all the sampling points corresponding to the pixels of the three-dimensional user interface is used as the visibility factor of the pixels of the three-dimensional user interface.

[0281] In a possible implementation manner, in the instructions executed by the processor 1010, rendering the pixels of the three-dimensional user interface based on the visibility factor includes:

[0282] Multiplying the original alpha channel value of the pixel of the three-dimensional user interface by the visibility factor to obtain an updated alpha channel value;

[0283] Pixels of the three-dimensional user interface are rendered based on the updated alpha channel value.

[0284] In a possible implementation manner, in the instructions executed by the processor 1010, before determining the depth map of the virtual scene, the method further includes:

[0285] Determining a first orientation of a virtual camera in the virtual scene;

[0286] determining a second orientation of the three-dimensional user interface based on the first orientation;

[0287] Determining a first local coordinate system of the virtual camera;

[0288] A second local coordinate system of the three-dimensional user interface is aligned with the first local coordinate system.

[0289] In a possible implementation manner, in the instructions executed by the processor 1010, before determining the depth map of the virtual scene, the method further includes:

[0290] Determining a resolution of a display screen of the terminal device, and determining a two-dimensional coordinate of the three-dimensional user interface on the display screen based on the resolution;

[0291] Determine a projection matrix of the virtual scene, determine an inverse matrix of the projection matrix, convert the two-dimensional coordinates into three-dimensional coordinates of the three-dimensional user interface in the virtual scene based on the inverse matrix, and adjust the size of the three-dimensional user interface based on the three-dimensional coordinates.

[0292] In a possible implementation, in the instruction executed by the processor 1010, determining, for each pixel of the three-dimensional user interface, a first coordinate value of the pixel of the three-dimensional user interface in the depth map includes:

[0293] In response to screen space jitter occurring in the three-dimensional user interface, determining a screen space jitter value;

[0294] Based on the screen space jitter value, compensating a current position of the 3D user interface to restore the 3D user interface to a position before the screen space jitter occurs;

[0295] Based on the position before the screen space jitter occurs, the first coordinate value of the pixel of the three-dimensional user interface in the depth map is determined.

[0296] In a possible implementation manner, in the instruction executed by the processor 1010, determining, for each of the sampling points, the second depth value of the sampling point in the depth map includes:

[0297] In response to screen space jitter of the virtual object, determining a jitter vector of a target frame and a depth map of the target frame of the virtual scene;

[0298] Compensating the second coordinate value of the sampling point based on the jitter vector to obtain a compensated second coordinate value;

[0299] Sampling is performed from the depth map of the target frame based on the compensated second coordinate value to obtain the second depth value.

[0300] In a possible implementation manner, in the instructions executed by the processor 1010, determining the depth map of the virtual scene includes:

[0301] Determine a jittered depth map of each frame of the virtual scene;

[0302] Based on the jittered depth map of each frame, a super-resolution depth map is obtained.

[0303] In a possible implementation manner, in the instructions executed by the processor 1010, obtaining a super-resolution depth map based on the jittered depth map of each frame includes:

[0304] Determining a depth map of the current frame jitter of the virtual scene;

[0305] Determining whether it is necessary to update the super-resolution depth map based on the jittered depth map of the current frame;

[0306] In response to determining that the super-resolution depth map needs to be updated based on the current frame jittered depth map, the super-resolution depth map is updated based on the current frame jittered depth map.

[0307] In a possible implementation manner, in the instruction executed by the processor 1010, the determining whether it is necessary to update the super-resolution depth map based on the jittered depth map of the current frame includes:

[0308] Determine a third coordinate of the jittered current frame in the jittered depth map of the current frame and a fourth coordinate of the center of the pixel where the third coordinate is located;

[0309] In response to a distance between the third coordinate and the fourth coordinate being less than a distance threshold, determining that the super-resolution depth map needs to be updated based on the jittered depth map of the current frame;

[0310] or,

[0311] In response to the distance between the third coordinate and the fourth coordinate being greater than or equal to the distance threshold, it is determined that there is no need to update the super-resolution depth map based on the current frame jittered depth map.

[0312] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the user interface display method described in any of the above embodiments.

[0313] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0314] The above-mentioned non-transitory computer-readable storage medium can be any available medium or data storage device that can be accessed by a computer, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)), etc.

[0315] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the user interface display method described in any embodiment in the above exemplary method part, and have the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0316] Based on the same inventive concept, corresponding to the method for displaying a user interface described in any of the above embodiments, the present disclosure further provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer so that the computer and / or the processor execute the method for displaying a user interface. Corresponding to the execution subject corresponding to each step in each embodiment of the method for displaying a user interface, the processor that executes the corresponding step may belong to the corresponding execution subject.

[0317] The computer program product of the above embodiment is used to enable the computer and / or the processor to execute the user interface display method described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0318] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, method, or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." In addition, in some embodiments, the present disclosure may also be implemented in the form of a computer program product in one or more computer-readable media, which contains computer-readable program code.

[0319] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive examples) of computer-readable storage media may include, for example: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device.

[0320] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0321] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0322] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0323] It should be understood that each box in the flowchart and / or block diagram and the combination of boxes in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine, and these computer program instructions are executed by a computer or other programmable data processing device to produce a device that implements the functions / operations specified in the boxes in the flowchart and / or block diagram.

[0324] These computer program instructions may also be stored in a computer-readable medium that enables a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable medium produce a product that includes an instruction device that implements the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0325] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby enabling the instructions executed on the computer or other programmable device to provide a process for implementing the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0326] In addition, although the operations of the disclosed method are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted in the flow chart can be performed in a different order. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0327] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0328] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0329] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0330] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0331] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0332] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

[0333] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims. The scope of the attached claims conforms to the broadest interpretation, thereby including all such modifications and equivalent structures and functions.

Claims

1. A method for displaying a user interface, characterized in that: A graphical user interface is provided by a terminal device, wherein the content displayed by the graphical user interface includes a virtual scene, and the virtual scene includes a three-dimensional user interface. The method includes: Determine a depth map of the virtual scene, and determine whether the three-dimensional user interface is blocked by a virtual object in the virtual scene based on the depth map; In response to determining that the three-dimensional user interface is blocked by the virtual object in the virtual scene, determining a visibility factor of a pixel of the three-dimensional user interface based on the depth map; The pixels of the three-dimensional user interface are rendered based on the visibility factor to adjust the visibility of the pixels of the three-dimensional user interface so that a portion of the three-dimensional user interface blocked by the virtual object is invisible.

2. The method according to claim 1, characterized in that The determining, based on the depth map, whether the three-dimensional user interface is blocked by a virtual object in the virtual scene comprises: Determine a first depth value of a pixel of the three-dimensional user interface in the depth map; determining a depth buffer value for a pixel of the three-dimensional user interface; In response to determining that the first depth value is less than the depth buffer value, determining that the three-dimensional user interface is obscured by the virtual object in the virtual scene; or, In response to determining that the first depth value is greater than or equal to the depth buffer value, it is determined that the three-dimensional user interface is not obstructed by the virtual object in the virtual scene.

3. The method according to claim 2, characterized in that In response to determining that the 3D user interface is blocked by the virtual object in the virtual scene, determining a visibility factor of a pixel of the 3D user interface based on the depth map includes: For each pixel of the three-dimensional user interface, determining a first coordinate value of the pixel of the three-dimensional user interface in the depth map; Based on the first coordinate value, determining a number of sampling points within a preset range around a pixel of the three-dimensional user interface; For each of the sampling points, determining a second depth value of the sampling point in the depth map, and obtaining a shadow contribution of the sampling point based on the first depth value and the second depth value; An average value of the shadow contribution of all the sampling points corresponding to the pixels of the three-dimensional user interface is used as the visibility factor of the pixels of the three-dimensional user interface.

4. The method according to claim 1, characterized in that: The rendering of the pixels of the three-dimensional user interface based on the visibility factor includes: Multiplying the original alpha channel value of the pixel of the three-dimensional user interface by the visibility factor to obtain an updated alpha channel value; Pixels of the three-dimensional user interface are rendered based on the updated alpha channel value.

5. The method according to claim 1, characterized in that Before determining the depth map of the virtual scene, the method further includes: Determining a first orientation of a virtual camera in the virtual scene; determining a second orientation of the three-dimensional user interface based on the first orientation; Determining a first local coordinate system of the virtual camera; A second local coordinate system of the three-dimensional user interface is aligned with the first local coordinate system.

6. The method according to claim 1, characterized in that Before determining the depth map of the virtual scene, the method further includes: Determining a resolution of a display screen of the terminal device, and determining a two-dimensional coordinate of the three-dimensional user interface on the display screen based on the resolution; Determine a projection matrix of the virtual scene, determine an inverse matrix of the projection matrix, convert the two-dimensional coordinates into three-dimensional coordinates of the three-dimensional user interface in the virtual scene based on the inverse matrix, and adjust the size of the three-dimensional user interface based on the three-dimensional coordinates.

7. The method according to claim 3, characterized in that The step of determining, for each pixel of the three-dimensional user interface, a first coordinate value of the pixel of the three-dimensional user interface in the depth map comprises: In response to screen space jitter occurring in the three-dimensional user interface, determining a screen space jitter value; Based on the screen space jitter value, compensating a current position of the 3D user interface to restore the 3D user interface to a position before the screen space jitter occurs; Based on the position before the screen space jitter occurs, the first coordinate value of the pixel of the three-dimensional user interface in the depth map is determined.

8. The method according to claim 3, characterized in that The step of determining, for each of the sampling points, a second depth value of the sampling point in the depth map comprises: In response to screen space jitter of the virtual object, determining a jitter vector of a target frame and a depth map of the target frame of the virtual scene; Compensating the second coordinate value of the sampling point based on the jitter vector to obtain a compensated second coordinate value; Sampling is performed from the depth map of the target frame based on the compensated second coordinate value to obtain the second depth value.

9. The method according to claim 1, characterized in that: Determining the depth map of the virtual scene includes: Determine a jittered depth map of each frame of the virtual scene; Based on the jittered depth map of each frame, a super-resolution depth map is obtained.

10. The method according to claim 9, characterized in that The step of obtaining a super-resolution depth map based on the jittered depth map of each frame includes: Determining a depth map of the current frame jitter of the virtual scene; Determining whether it is necessary to update the super-resolution depth map based on the jittered depth map of the current frame; In response to determining that the super-resolution depth map needs to be updated based on the current frame jittered depth map, the super-resolution depth map is updated based on the current frame jittered depth map.

11. The method according to claim 10, characterized in that The determining whether it is necessary to update the super-resolution depth map based on the jittered depth map of the current frame includes: Determine a third coordinate of the jittered current frame in the jittered depth map of the current frame and a fourth coordinate of the center of the pixel where the third coordinate is located; In response to a distance between the third coordinate and the fourth coordinate being less than a distance threshold, determining that the super-resolution depth map needs to be updated based on the jittered depth map of the current frame; or, In response to the distance between the third coordinate and the fourth coordinate being greater than or equal to the distance threshold, it is determined that there is no need to update the super-resolution depth map based on the current frame jittered depth map.

12. A display device for a user interface, characterized in that: A graphical user interface is provided through a terminal device, wherein the content displayed by the graphical user interface includes a virtual scene, wherein the virtual scene includes a three-dimensional user interface, and the apparatus includes: a depth map determination module, configured to determine a depth map of the virtual scene, and determine whether the three-dimensional user interface is blocked by a virtual object in the virtual scene based on the depth map; a visibility factor determination module, configured to determine a visibility factor of a pixel of the three-dimensional user interface based on the depth map in response to determining that the three-dimensional user interface is blocked by the virtual object in the virtual scene; The visibility rendering module is configured to render the pixels of the three-dimensional user interface based on the visibility factor to adjust the visibility of the pixels of the three-dimensional user interface so that the portion of the three-dimensional user interface blocked by the virtual object is invisible.

13. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 11 is implemented.

14. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The method comprises computer program instructions, and when the computer program instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 11.

Images