Method and system for rendering video graphics

Abstract

The invention relates to a graphics rendering system and method. In an exemplary embodiment, a user interface (UI) is detected prior to an image rendering process that renders two-dimensional (2D) UI elements on three-dimensional (3D) elements per frame. The rendered UI may be temporarily stored in a buffer memory for later reuse in multiple frames. Some other embodiments are also provided.

Description

Background Art

[0001] The present invention relates to a graphics processing system and method.

[0002] With the increasing popularity of online games over the past decade, the demand for an enhanced gaming experience on all types of user devices, especially mobile devices, continues to grow rapidly. Important design considerations and challenges for mobile gaming platforms include processing power, memory usage, screen size, and power consumption. Existing mobile gaming platforms have been inadequate due to various limitations, as will be further explained below.

[0003] Therefore, there is a need for new and improved mobile gaming systems and methods. Summary of the invention

[0004] The present invention relates to a graphics rendering system and method. In an exemplary embodiment, a user interface (UI) is detected before an image rendering process in which two-dimensional (2D) UI elements are rendered on three-dimensional (3D) elements in each frame. The rendered UI can be temporarily stored in a buffer memory for later reuse in multiple frames. There are also some other embodiments.

[0005] A system of one or more computers may be configured to perform a specific operation or action by means of software, firmware, hardware, or a combination thereof installed on the system, which software, firmware, hardware, or a combination thereof causes or results in the system performing an action in operation. One or more computer programs may be configured to perform a specific operation or action by means of instructions contained therein, which, when executed by a data processing device, causes the device to perform an action. A general aspect includes a method for rendering graphics. The method includes processing an object stream for display on a plurality of frames. The object stream includes a first plurality of objects and a second plurality of objects. The second plurality of objects is associated with a user interface (UI). The method also includes detecting the second plurality of objects. The method also includes rendering a first frame using the first plurality of objects, the first frame may include a three-dimensional texture. The method also includes rendering a first UI texture using the second plurality of objects. The method also includes storing the first UI texture in a buffer memory. The method also includes superimposing the first UI texture on the first frame. The method also includes rendering a second frame using the first plurality of objects. The method also includes superimposing the first UI texture on the second frame. Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the operations of the method.

[0006] The implementation scheme may include one or more of the following features. In some embodiments, the first plurality of objects may include three-dimensional objects and the second plurality of objects may include two-dimensional objects. The method may include mixing the first UI texture with the first frame. The method may include rendering a third frame using the first plurality of objects, rendering a second UI texture using the second plurality of objects, storing the second UI texture in a buffer memory, and superimposing the second UI texture on the third frame. The method may include determining the ratio between the UI texture and the frame based on the output frame rate. The object stream is updated at a predetermined rate. The predetermined rate is greater than or equal to 60fps. The method may include: updating the first UI texture based on detecting a third plurality of objects, and the third plurality of objects are associated with the second UI texture. The method may include displaying a first output frame, the first output frame including at least the first frame and the first UI texture. Implementation of the described technology may include hardware, methods or processes, or computer software on a computer-accessible medium.

[0007] A general aspect includes a graphics rendering device. The device includes a data storage device configured to store a first plurality of objects and a second plurality of objects. The second plurality of objects is associated with a user interface (UI). The device may include a processor coupled to the data storage device. The processor is configured to detect the second plurality of objects and render a first frame using the first plurality of objects. The first frame may include a three-dimensional texture. The processor is also configured to render a first UI texture using the second plurality of objects. The processor is also configured to overlay the first UI texture on the first frame. The processor is also configured to render a second frame using the first plurality of objects. The processor is also configured to overlay the first UI texture on the second frame. The device also includes a buffer memory coupled to the processor. The buffer memory may include a first frame buffer and a second frame buffer. The first frame buffer is configured to store the first frame and the second frame, and the second frame buffer is configured to store the first UI texture. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the operations of the method.

[0008] Embodiments may include one or more of the following features. The device may include a display configured to display a first output frame including at least the first frame and the first UI texture. The processor includes a central processing unit (CPU). The processor is also configured to receive one or more graphics library commands from a software application. The one or more graphics library commands include commands for rendering the first frame and the first UI texture. The one or more graphics library commands also include commands for storing the first UI texture to the second frame buffer of the buffer memory. The processor also includes a graphics processing unit (GPU) configured to render the first frame. Implementations of the described techniques may include hardware, methods or processes, or computer software on a computer-accessible medium.

[0009] A general aspect includes a graphics rendering method. The method includes receiving one or more graphics library commands, the graphics library commands being used to process an object stream for display on multiple frames. The object stream includes a first plurality of objects and a second plurality of objects. The second plurality of objects is associated with a user interface (UI). The method also includes detecting the second plurality of objects. The method also includes rendering a first frame to a first frame buffer using the first plurality of objects. The first frame may include a three-dimensional texture. The method also includes rendering a first UI texture to a second frame buffer using the second plurality of objects. The method also includes storing the first UI texture in the second frame buffer. The method also includes superimposing the first UI texture on the first frame. The method also includes rendering a second frame using the first plurality of objects. The method also includes superimposing the first UI texture on the second frame. Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the operations of the method.

[0010] Embodiments may include one or more of the following features. The method may include executing, for the first frame, a color channel for rendering a color into the first frame buffer. The method also includes detecting the second plurality of objects based at least on the output of the color channel. Updating the first frame at a first rate and updating the first UI texture at a second rate. The first rate is associated with the second rate. The first rate is greater than the second rate. Implementation of the described techniques may include hardware, a method or process, or computer software on a computer accessible medium.

[0011] It should be appreciated that embodiments of the present invention have many advantages over prior art. In addition, the graphics rendering technology according to the embodiments of the present invention can reduce power consumption and computing costs while maintaining smooth animation effects in online game applications. For some embodiments, the technology according to the embodiments of the present invention can achieve high frame rates without sacrificing video quality. The present invention can also exempt application / game developers from the maintenance workload of video processing in some aspects. In addition, the present invention provides a new performance improvement technology that can be implemented by, for example, OpenGL's application programming interface (APIs), and implemented on a mobile platform.

[0012] The embodiments of the present invention can be implemented in combination with existing systems and processes. For example, the graphics rendering method and system according to the present invention can be used in a variety of applications, including mobile game platforms, video streaming, client media players, and online media platforms. In addition, the embodiments of the present invention effectively reduce the use of the GPU by offloading part of the rendering work to the GPU in high frame rate scenarios (e.g., 60fps or higher), thereby providing a better gaming experience. The present invention also has other advantages.

[0013] The present invention achieves these and other advantages in the context of known technology. However, the nature and advantages of the present invention may be further understood by referring to the following portions of the specification and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a simplified diagram illustrating a system 100 for rendering an object using an overlay graphical user interface in accordance with an embodiment of the present invention.

[0015] Figure 2 is a simplified diagram showing an image 200 with an overlaid graphical user interface in accordance with an embodiment of the present invention.

[0016] Figure 3 is a simplified flow chart illustrating a method 300 for displaying a graphical user interface on a three-dimensional object according to an embodiment of the present invention.

[0017] Figure 4 is a simplified diagram illustrating a pipeline 400 for displaying a graphical user interface on a three-dimensional object according to an embodiment of the present invention.

[0018] Figure 5A and Figure 5B is a simplified diagram illustrating a pipeline 500 for displaying a graphical user interface on a three-dimensional object according to an embodiment of the present invention.

[0019] Figure 6is a simplified diagram illustrating a pipeline 600 for displaying a graphical user interface on a three-dimensional object according to an embodiment of the present invention.

[0020] Figure 7 is a simplified flow chart illustrating a pipeline 700 for displaying a graphical user interface on a three-dimensional object according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The present invention relates to a graphics rendering system and method. In an exemplary embodiment, a user interface (UI) is detected before an image rendering process in which two-dimensional (2D) UI elements are rendered on three-dimensional (3D) elements in each frame. The rendered UI can be temporarily stored in a buffer memory for later reuse in multiple frames. There are also some other embodiments.

[0022] Online mobile games have undergone tremendous changes in recent years, especially with the upgrade of 5G networks, online mobile games have now become the competitive center of smartphone competitiveness. Existing methods rely on cross-platform 3D mobile game engines to develop mobile 3D games. The core functional modules of game engines include 3D graphics rendering, physics engine, artificial intelligence (AI), memory management, etc. In recent years, especially with the emergence of deep learning technology, game engines have made great progress. However, due to limited resources, improving video frame rate to obtain smooth visual effects in mobile game applications remains a challenging task; resource constraints usually require a trade-off between visual quality and frame rate. Various traditional methods focus on hardware upgrades (e.g., CPU, GPU, etc.), which leads to high cost, additional power consumption, and poor heat dissipation. Some game engine developers try to improve the performance of games by modifying their own platforms, which greatly limits the versatility and usability of graphics rendering methods.

[0023] The following description is intended to enable one skilled in the art to make and use the invention and to put the invention into the context of a particular application. Various modifications and various uses in different applications will be apparent to one skilled in the art, and the general principles defined herein can be applied to a wide range of embodiments. Therefore, the present invention is not intended to be limited to the embodiments presented, but should have the widest scope consistent with the principles and novel features disclosed herein.

[0024] In the following detailed description, many specific details are set forth to provide a more thorough understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention can be practiced without these specific details. In other cases, well-known structures and devices will be presented in block diagram form rather than in detail to avoid obscuring the present invention.

[0025] The reader is reminded that all papers and documents filed at the same time as this specification, and all papers and documents open to the public together with this specification, are hereby incorporated by reference into this specification. Unless otherwise expressly stated, all features disclosed in this specification (including any attached claims, abstracts and drawings) can be replaced by alternative features for the same, equivalent or similar purposes. Therefore, unless otherwise expressly stated, each feature disclosed is only an example in a series of equivalent or similar features.

[0026] Furthermore, any element in a claim that does not expressly state a “means” for performing a specified function or a “step” for performing a specific function shall not be construed as a “means” or “step” provision under 35 U.S.C. § 112, paragraph 6. In particular, the use of “step” or “action” in the claims herein is not intended to invoke the provisions of 35 U.S.C. § 112, paragraph 6.

[0027] Note that if left, right, front, back, up, down, forward, reverse, clockwise, and counterclockwise labels are used, these labels are for convenience only and do not represent any particular fixed direction. Instead, these labels are used to reflect the relative position and / or orientation of the various parts of the object.

[0028] Figure 1 1 is a simplified diagram showing a system 100 for rendering an object using an overlay graphical user interface according to an embodiment of the present invention. This diagram is merely an example and should not unduly limit the scope of the claims. Those of ordinary skill in the art will recognize many variations, substitutions, and modifications.

[0029] As shown, user equipment 100n includes a system 100 for rendering an object using an overlay graphical user interface. In various embodiments, user equipment 100n includes but is not limited to at least one of a portable gaming device, a smart phone, a tablet computer, a laptop computer, a desktop computer, a server computer and / or the like. For example, user equipment 100n is configured to enable one or more software applications (e.g., 3D game applications, applications based on 2D UI, 3D guidance applications and / or similar applications) to provide an interactive visual experience to the user. According to some embodiments, system 100 includes but is not limited to at least one of a data memory 105, a processor 110, a buffer memory 125, an audio interface 140, a display 145, a communication interface 150 and / or the like. The components of system 100 may be configured to execute a graphics rendering process in response to one or more graphics library commands. For example, one or more graphics library commands include commands for intercepting graphics library commands (e.g., OpenGL API calls or similar graphics library draw calls) sent from a 3D application to a processor 110.

[0030] In various embodiments, the system 100 is configured to receive and process an object stream for display on multiple frames. For example, the object stream includes a first plurality of objects and a second plurality of objects. The first plurality of objects includes one or more 3D objects associated with a 3D element. The second plurality of objects may include one or more 2D objects associated with a user interface. For example, the user interface includes, but is not limited to, one of a 2D UI image element (e.g., user score, game progress, etc.), a 2D polygonal image element (e.g., a player image, a game object, etc.), and / or the like. In various embodiments, the object stream may be stored in a data memory 105 and may be retrieved by a processor 110 for further processing. The data memory 155 may include a dynamic random access memory (DRAM) and / or a non-volatile memory. For example, the object stream may be temporarily stored in a DRAM for processing, and executable instructions (e.g., a graphics rendering algorithm) may be stored in a non-volatile memory. In various embodiments, the data memory 155 may be implemented as part of the processor 110 in a system-on-chip (SoC) arrangement.

[0031] In some embodiments, the processor 110 may be communicatively coupled (e.g., via a bus, a wired connector, or electrical pathways (e.g., traces and / or pads, etc.) of a printed circuit board (PCB) or an integrated circuit (IC), and / or the like) to each of one or more of the data storage 105, the processor 110, the buffer memory 125, the audio interface 140, the display 145, the communication interface 150, and / or the like. For example, the processor 110 may retrieve a stream of objects stored in the data storage 105 to render a 3D scene with an overlaid graphical user interface. In some cases, the processor 110 includes, but is not limited to, at least one of a central processing unit (CPU) 115 and a graphics processing unit (GPU) 120, and / or the like. For example, the CPU 115 and the GPU 120 may each include multiple processing cores. Different types of processing units are optimized for different types of computations. For example, the CPU 115 processes various types of system functions, such as executing and modifying one or more graphics library commands for graphics rendering, detecting a second plurality of objects associated with a UI, rendering 2D image elements, moving rendered UI textures to a buffer memory 125 for storage, and / or the like. The GPU 120 may be specifically configured to operate graphics creation and image processing, which is advantageous for processing object streams for rendering video graphics. For example, the GPU 120 is configured to render one or more image frames including a 3D texture using a first plurality of objects.

[0032] In various embodiments, the buffer memory 125 includes one or more data structures for storage. For example, the buffer memory 125 includes a first frame buffer 130 and a second frame buffer 135. During the graphics rendering process, 3D image elements can be drawn to the first frame buffer 130 using at least a first plurality of objects, and 2D image elements (e.g., UI textures) can be drawn to the second frame buffer 135 using at least a second plurality of objects. Then, the processor 110 can perform superimposing the 2D image elements (e.g., UI textures) on the 3D elements (e.g., the first frame). In some cases, the rendered UI texture can be temporarily stored in the buffer memory 125 and can be reused to merge with other 3D elements (e.g., the second frame, etc.). In a specific example, the buffer memory 125 includes a random-access memory (RAM) or a similar memory storage device.

[0033] The system 100 may also include an audio interface 140 and a display 145 to facilitate user interaction. The audio interface 140 may include, but is not limited to, a microphone, a sound sensor, a noise sensor, and / or the like, and may be used to receive or capture voice signals, sound signals, and / or noise signals, etc. The display 145 may be used to display a rendered image frame in which a 2D user interface overlays a 3D image element. For example, the display 145 may include, but is not limited to, an integrated display screen, a touch screen display screen, a non-touch screen display screen, an external display screen, a monitor, and / or any combination thereof. The communication interface 150 provides wired or wireless communication with other devices and / or networks. For example, the communication interface 150 may be connected to a computer for tether operations, wherein the computer provides the processing power required for graphics-intensive applications. Other embodiments of the system include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each of which is configured to perform the operations of the method.

[0034] Figure 2 2 is a simplified diagram showing an image 200 with an overlaid graphical user interface according to an embodiment of the present invention. This diagram is merely an example and should not unduly limit the scope of the claims. Those of ordinary skill in the art will recognize many variations, substitutions, and modifications.

[0035] As shown, image 200 may include at least 3D image element 205 and 2D image element 210. For example, 3D element 205 is associated with a 3D scene in a mobile gaming application and may be rendered by a 3D renderer (e.g., Figure 1 120). The 2D image element 210 may be associated with a user interface in a mobile gaming application and may be rendered by a 2D renderer (e.g., Figure 1 The 3D elements in the video game are updated in each frame (e.g., at a predetermined rate such as 60fps or higher), while the 2D image elements related to the UI can remain static for multiple frames and are only updated in response to user input. Existing methods generally render and update 2D image elements as well as 3D elements in each frame, which unnecessarily increases GPU usage and overall power consumption.

[0036] Figure 3is a simplified flow chart illustrating a method 300 for displaying a graphical user interface on a three-dimensional object according to an embodiment of the present invention. This diagram is merely an example and should not unduly limit the scope of the claims. One of ordinary skill in the art will recognize many variations, substitutions, and modifications. For example, one or more steps may be added, deleted, repeated, modified, replaced, overlapped, and / or rearranged, which should not limit the scope of the claims.

[0037] At step 302, the method includes processing an object stream for display over a plurality of frames (e.g., using Figure 1 For example, the system 100 receives one or more graphics library commands (e.g., OpenGL draw calls) for processing an object stream including a first plurality of objects and a second plurality of objects. In some cases, the first plurality of objects includes three-dimensional objects and the second plurality of objects includes two-dimensional objects. The second plurality of objects may be associated with a user interface (e.g., Figure 2 2D image element 210 is shown. According to a specific embodiment, the object stream may be temporarily stored in the data storage 105 and retrieved by the processor 110 for further processing.

[0038] At step 304, the method includes detecting a second plurality of objects. For example, in response to one or more graphics library commands sent by a software application (e.g., a 3D application) to detect a second plurality of objects, processor 110 performs detection of the second plurality of objects within the object stream. The processor 110 may perform detection of the second plurality of objects within the object stream based at least on the color rendering to the frame buffer (e.g., Figure 1 The output of the color channel (color pass) pipeline of the first frame buffer 130) is used to perform detection of the second plurality of objects, as further described below.

[0039] At step 306, the method includes rendering a first frame using the first plurality of objects. The first frame may include a three-dimensional texture. For example, a 3D renderer (e.g., GPU 120) renders the first frame by executing a color channel pipeline for rendering colors to a first frame buffer 130. It should be understood that one or more frames (e.g., a second frame, a third frame, etc.) may be rendered by the same rendering process to form a plurality of constant frames of a video output (e.g., a video game), wherein each frame is updated at a predetermined rate (e.g., 60fps, 90fps, 120fps, etc.).

[0040] At step 308, the method includes rendering out the first UI texture using a second plurality of objects. The second plurality of objects may include two-dimensional objects. For example, a 2D renderer (e.g., CPU 115) renders the first UI texture by rendering the second plurality of objects to a second frame buffer 135. Similar to the multiple constant frames explained above, the UI texture is updated at a specific rate associated with the refresh rate of the frame. In various embodiments, the ratio between the UI texture and the frame is determined based on the output frame rate to relieve part of the UI rendering workload, thereby reducing the use of the GPU / CPU and the overall power consumption. For example, the UI texture can be updated at a second rate (e.g., 30fps), the frame can be updated at a first rate (e.g., 120fps or higher), and the first rate can be greater than the first rate. Therefore, compared with existing methods, the UI rendering workload can be reduced by up to 75%.

[0041] At step 310, the method includes storing the first UI texture in the buffer memory. For example, in response to one or more graphics library commands including a command to store the first UI texture in the second frame buffer, the first UI texture is stored in the second frame buffer 135 of the buffer memory 125. It is advantageous to render the first frame and the first UI texture into separate frame buffers (e.g., the first frame buffer 130 and the second frame buffer 135), which allows the frame and the UI texture to be updated at different rates, thereby maximizing graphics rendering efficiency.

[0042] At step 312, the method further includes overlaying the first UI texture on the first frame. In the graphics rendering pipeline, the UI texture rendering is performed after the 3D rendering (e.g., color channels and / or the like) is completed to overlay the UI texture on the rendered 3D image element (e.g., the first frame).

[0043] At step 314, the method includes rendering a second frame using the first plurality of objects. Similar to the first frame, the second frame may be rendered by a 3D renderer (e.g., GPU 120) by executing a color channel pipeline for rendering colors to the first frame buffer 130. It should be understood that one or more frames (e.g., the third frame) may be rendered by a rendering process to form a plurality of constant frames of a video output (e.g., a video game), wherein each frame is updated at a predetermined rate (e.g., 60fps, 90fps, 120fps, etc.).

[0044] In step 316, the method includes overlaying the first UI texture on the second frame. In some cases, the method also includes blending the first UI texture with the first frame to generate a first output frame. The first output frame including at least the first frame and the first UI texture can be output to a display device (e.g., Figure 1Display 145). As described above, the UI texture can remain unchanged over multiple frames, while the image frame is updated on each frame. The processor 110 can perform a previously stored UI texture superimposed on a subsequent frame (e.g., a second frame, a third frame, etc.) to avoid unnecessary calculations, rather than rendering a repeated UI texture on each frame. In this case, the ratio between the UI texture and the frame can be determined based on the output frame rate. According to an embodiment, the UI texture can be updated when a new UI element (e.g., a third plurality of objects) is detected in the object stream. In some cases, the UI texture can be updated based on user input.

[0045] In a specific example, the method may also include rendering a third frame using the first plurality of objects, and rendering a second UI texture using the second plurality of objects. The second UI texture may also be stored in a buffer memory and may be processed by the processor 110 to be superimposed on the third frame and / or one or more subsequent frames. In various embodiments, a new UI texture may be rendered only after a new UI-related object is detected. For example, the system 100 may receive and detect a third plurality of objects associated with a new UI texture, and update the UI texture based on detecting the third plurality of objects.

[0046] Figure 4 4 is a simplified diagram of a pipeline 400 for displaying a graphical user interface on a three-dimensional object according to an embodiment of the present invention. This figure is merely an example and should not unduly limit the scope of the claims. Those of ordinary skill in the art will recognize many variations, substitutions, and modifications. As shown, in order to process an object stream for display on multiple frames, a computing system (e.g., system 100) can detect UI elements within the object stream by detecting UI channel 405. In some cases, the UI elements include two-dimensional objects. Based on the detection of the UI, system 100 can then cache the UI channel 410 to a custom frame buffer (e.g., Figure 1 The cached UI may then be blended with a previously rendered image frame (e.g., a 3D image frame) to generate an output frame through a blended UI channel 415. In various implementations, the cached UI may be reused to be merged with one or more image frames to generate one or more output frames.

[0047] FIG. 5A to FIG. 5B 5 is a simplified diagram showing a pipeline 500 for displaying a graphical user interface on a three-dimensional object according to an embodiment of the present invention. These diagrams are merely examples and should not unduly limit the scope of the claims. Those of ordinary skill in the art will recognize many variations, substitutions, and modifications. Figure 5A As shown, in the graphics rendering pipeline (for example, using Figure 1In the system 100 implementation of ), the 3D image elements are rendered before the 2D UI is rendered. For example, the color can be rendered to the default frame buffer (e.g., Figure 1 3D image elements are rendered in the first frame buffer 130 of the image processing unit to generate one or more image frames (e.g., a first frame, a second frame, etc.). In various embodiments, in addition to the color channel 505, the 3D image elements can also be rendered through one or more 3D rendering channels, and the 3D rendering channels include but are not limited to a shadow channel, an illumination channel, a secondary color channel, one or more post-processing channels (e.g., a blur channel, a sharpening channel, a filtering channel, etc.) and / or similar channels. One or more rendering channels can be processed sequentially or in parallel. Once the 3D rendering (e.g., the color channel 505) is completed, the UI rendering channel 510 begins rendering 2D UI elements on the 3D elements. A final frame 515 can be generated in response to the completion of the UI rendering channel 510. In some cases, the final frame 515 includes at least the first frame and the first UI texture covering the first frame. The generated final frame 515 can then be output to a display device (e.g., Figure 1 The display 145) is displayed.

[0048] In various embodiments, because each frame buffer reset indicates the start of a new pass, the system 100 can identify one or more rendering passes by detecting the occurrence of a frame buffer reset. In some cases, the UI rendering pass 510 can be detected based on the output of the color channel 505. For example, the UI rendering pass 510 renders the output of the color channel 505 to the default frame buffer. Figure 5B , the system 100 can detect the start of the UI rendering channel 510, which will bind the default frame buffer at block 520 (in Figure 5B 520) and draws the UI texture at block 525 to generate a final frame 530. It will be appreciated that the end of the UI rendering channel 510 may be defined by the final frame 530.

[0049] Figure 6 6 is a simplified diagram of a pipeline 600 for displaying a graphical user interface on a three-dimensional object according to an embodiment of the present invention. This diagram is merely an example and should not unduly limit the scope of the claims. Those of ordinary skill in the art will recognize many variations, substitutions, and modifications. In various embodiments, the 3D image elements (e.g., the first frame) and the 2D UI elements may be rendered into separate frame buffers. For example, the system 100 may render the 3D image elements into a default frame buffer (e.g., Figure 1 ), and renders the 2D UI elements to a custom frame buffer (e.g., Figure 1In some cases, the 2D UI elements may be cached in a custom frame buffer and may be reused over one or more frames.

[0050] According to some embodiments, 2D UI elements can be cached for intercepted drawing calls 610 by using an additional texture bound to a custom frame buffer 605 (referred to as "FB UI"), which retains the above-mentioned 2D UI elements in a texture representing the entire screen. The cached UI elements can be blended with the default frame buffer 615 including the rendered 3D elements to generate a final frame. For example, in response to one or more graphics library commands, the graphics library commands include a command for blending the UI texture with the 3D color texture, and the blending UI channel 620 can blend the custom frame buffer 605 with the default frame buffer 615 into the final frame buffer 625. When the 2D UI elements remain static and do not need to be updated, the system 100 can skip the UI rendering task and use the cached UI elements to blend with one or more subsequent 3D frames to generate one or more subsequent final frames, so that the GPU / CPU has more bandwidth to handle 3D rendering scenes under high frame rate scenes (e.g., 60fps or higher frame rates), and may reduce overall power consumption. In some implementations, the ratio between UI elements and image frames can be determined based on the output frame rate to determine the portion of the UI workload that can be skipped.

[0051] Figure 7 7 is a simplified flow chart illustrating a pipeline 700 for displaying a graphical user interface on a three-dimensional object according to an embodiment of the present invention. This diagram is merely an example and should not unduly limit the scope of the claims. One of ordinary skill in the art will recognize many variations, substitutions, and modifications. For example, one or more steps may be added, deleted, repeated, modified, replaced, overlapped, and / or rearranged, which should not limit the scope of the claims.

[0052] As shown, to process an object stream for display over multiple frames, a computing system (e.g., system 100) may detect UI elements within the object stream at block 705. For example, the UI elements may be detected based on at least a previous 3D rendering pass (e.g., Figure 5A The output of the color channel 505) is used to detect UI elements, and in the 3D rendering channel, the 3D image elements are rendered into the default frame buffer.

[0053] At block 710, system 100 determines whether to update UI elements. For example, if a new UI element is detected at block 705, system 100 continues to render the new UI element into a custom frame buffer separated from the default frame buffer at blocks 715 and 720. Therefore, the UI element can be saved in a texture representing the entire screen for further processing. At block 725, the previously rendered 3D image element can be stored in the default frame buffer. At block 725, the new UI element can be mixed with the previously rendered 3D image element to generate a final frame 745. Alternatively, if a new UI element is not detected at block 705, system 100 can determine not to update the UI at block 710, and skip the UI rendering pass after completing 3D rendering at block 735. At block 740, the previously cached UI element can be used again to mix with the newly rendered 3D element to generate a final frame 745.

[0054] Although specific embodiments are fully described above, various modifications, alternative constructions and equivalents may be used. Therefore, the above descriptions and illustrations should not be taken as limiting the scope of the invention, which is defined by the appended claims.

Claims

1. A graphics rendering method, the method comprises: Processing an object stream for display on multiple frames, the object stream including a first plurality of objects and a second plurality of objects, the second plurality of objects being associated with a user interface (UI); Detecting the second plurality of objects; Rendering a first frame using the first plurality of objects, the first frame including a three-dimensional texture; Rendering a first UI texture using the second plurality of objects; Storing the first UI texture in a buffer memory; Overlaying the first UI texture on the first frame; Rendering a second frame using the first plurality of objects; And Overlaying the first UI texture on the second frame.

2. The method according to claim 1, wherein, The first plurality of objects includes three-dimensional objects, and the second plurality of objects includes two-dimensional objects.

3. The method according to claim 1, further comprises: Blending the first UI texture with the first frame.

4. The method according to claim 1, further comprises: Rendering a third frame using the first plurality of objects; Rendering a second UI texture using the second plurality of objects; Storing the second UI texture in a buffer memory; And Overlaying the second UI texture on the third frame.

5. The method according to claim 1, further comprises: Determining a ratio between the UI texture and the frame based on the output frame rate.

6. The method according to claim 1, wherein, The object stream is updated at a predetermined rate.

7. The method according to claim 1, wherein, The predetermined rate is greater than or equal to 60 fps.

8. The method according to claim 1, further comprises: Updating the first UI texture based on detecting a third plurality of objects, the third plurality of objects being associated with a second UI texture.

9. The method according to claim 1, further comprises: Displaying a first output frame including at least the first frame and the first UI texture.

10. A graphics rendering device, the device comprises: A data memory configured to store a first plurality of objects and a second plurality of objects, the second plurality of objects being associated with a user interface (UI); A processor coupled to the data memory, the processor being configured to: Detect the second plurality of objects; Render a first frame using the first plurality of objects, the first frame including a three-dimensional texture; Render a first UI texture using the second plurality of objects; Overlay the first UI texture on the first frame; Render a second frame using the first plurality of objects; And Overlay the first UI texture on the second frame; And A buffer memory coupled to the processor, the buffer memory including a first frame buffer and a second frame buffer, the first frame buffer being configured to store the first frame and the second frame, and the second frame buffer being configured to store the first UI texture.

11. The device according to claim 10, further comprises: A display configured to display a first output frame including at least the first frame and the first UI texture.

12. The apparatus according to claim 10, wherein, the processor includes a central processing unit (CPU).

13. The apparatus according to claim 10, wherein, the processor is further configured to receive one or more graphics library commands from a software application, the one or more graphics library commands including commands for rendering the first frame and the first UI texture.

14. The apparatus according to claim 13, wherein, the one or more graphics library commands further include commands for storing the first UI texture in the second frame buffer of the buffer memory.

15. The apparatus according to claim 10, wherein, the processor further includes a graphics processing unit (GPU) configured to render the first frame.

16. A graphics rendering method, the method comprising: receiving one or more graphics library commands for processing an object stream for display on a plurality of frames, the object stream including a first plurality of objects and a second plurality of objects, the second plurality of objects being associated with a user interface (UI); detecting the second plurality of objects; rendering a first frame including a three-dimensional texture to a first frame buffer using the first plurality of objects; rendering a first UI texture to a second frame buffer using the second plurality of objects; storing the first UI texture in the second frame buffer; overlaying the first UI texture on the first frame; rendering a second frame using the first plurality of objects; and overlaying the first UI texture on the second frame.

17. The method according to claim 16, further comprising: executing, for the first frame, rendering of colors to color channels in the first frame buffer.

18. The method according to claim 17, further comprising: detecting the second plurality of objects based at least on an output of the color channels.

19. The method according to claim 16, wherein, the first frame is updated at a first rate, the first UI texture is updated at a second rate, and the first rate is associated with the second rate.

20. The method according to claim 19, wherein, the first rate is greater than the second rate.

Images