A rendering instruction processing method and related device

By obtaining and replacing rendering instructions that are not based on target expansion into extension-based instructions in the terminal device, the game frame drop and power consumption problems are solved, and more efficient graphics rendering and lower device load are achieved.

CN113838180BActive Publication Date: 2024-11-29HUAWEI TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202010587225.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-06-24
Publication Date
2024-11-29
Estimated Expiration
2040-06-24

AI Technical Summary

Technical Problem

Game frame drop and power consumption problems are common in mobile terminal devices, especially before graphics rendering technology has improved but the expansion has not entered the standard, making it difficult for game engines to quickly adapt to the latest expansion, resulting in poor performance loss and rendering effects.

Method used

A rendering instruction processing method is proposed, which triggers the GPU to perform rendering tasks by obtaining a first rendering instruction not implemented based on the target extension and obtaining a second rendering instruction based on the target extension according to the instruction. This method uses preset conditions to judge and replace rendering instructions to reduce device power consumption and memory data copying amount, and reduce GPU and CPU load.

Benefits of technology

By replacing non-extended instructions with GPU extension-based rendering instructions, the performance loss and power consumption of terminal devices are significantly reduced, rendering efficiency is improved, and performance improvement and compatibility are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113838180B_ABST
    Figure CN113838180B_ABST
Patent Text Reader

Abstract

The embodiment of the present application discloses a rendering instruction processing method, the method is applied to a terminal device, the graphics processor GPU in the terminal device supports the target extension in openGL ES; in the present application, the CPU can obtain a first rendering instruction that is not implemented based on the target extension, the first rendering instruction is used to implement a rendering task, the second rendering instruction is a rendering instruction implemented based on the target extension, and the second rendering instruction is used to implement the rendering task, and the GPU is triggered to execute the rendering task according to the second rendering instruction. The present application uses a rendering instruction based on the GPU extension to replace the rendering instruction that is not originally based on the GPU extension, thereby reducing the performance loss of the terminal device.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a rendering instruction processing method and related equipment. Background Art

[0002] OpenGL (open graphics library) refers to a professional graphics programming interface that defines a cross-programming language and cross-platform programming interface specification. It is used in industries including content creation, energy, entertainment, game development, manufacturing, pharmaceuticals, and virtual reality. OpenGL can help programmers develop high-performance, high-visual-expression graphics processing software on hardware devices such as personal computers (PCs), workstations, and supercomputers. OpenGL ES (openGL for embedded systems) is a subset of the OpenGL 3D graphics application programming interface (API), designed for embedded devices such as mobile phones and game consoles.

[0003] Game frame drops and power consumption issues have always been a pain point for mobile phone manufacturers. Graphics rendering technology is also constantly improving. OpenGL ES has developed from 1.0 to 3.2, supporting more features and continuously improving rendering efficiency. GPU manufacturers are also actively promoting various technologies such as load reduction and image quality improvement, and launching OpenGL ES extensions based on their own GPU features. The purpose of launching new extensions is to improve rendering efficiency, promote GPU's own features, etc.

[0004] However, extensions have a long evolutionary path before they become standards. Before becoming standards or being approved by standards organizations, game engines generally do not quickly adapt to use the latest extensions, which results in additional performance loss on terminal devices and poor rendering effects. Summary of the invention

[0005] In a first aspect, the present application provides a rendering instruction processing method, the method is applied to a terminal device, the terminal device includes a graphics processor GPU, the GPU supports the target extension in the embedded system open graphics library openGL ES, the method includes:

[0006] Acquire a first rendering instruction that is not implemented based on the target extension, and the first rendering instruction is used to implement a rendering task; acquire a second rendering instruction according to the first rendering instruction; wherein the second rendering instruction is a rendering instruction implemented based on the target extension, and the second rendering instruction is used to implement the rendering task;

[0007] Taking a game as an example of an application that requires graphics rendering, when the game is started, a rendering instruction for a rendering task in the game can be obtained, and then the rendering task corresponding to the rendering instruction can be executed; wherein the first rendering instruction is one of the obtained rendering instructions; in an embodiment of the present application, the terminal device can pre-configure some rules, which define preset conditions that the instructions should meet. If the first rendering instruction meets the preset conditions, it can be indicated that the current rendering scene needs to be optimized; in this embodiment, the first rendering instruction is configured not to be implemented based on the target extension, and the first rendering instruction and the second rendering instruction are used to implement the same rendering task;

[0008] That is, the current GPU supports the target extension that the second rendering instruction depends on, but the application does not use the second rendering instruction for the rendering task at this time, but uses the first rendering instruction that does not depend on the target extension implementation. At this time, the second rendering instruction can be obtained as the rendering instruction for subsequent execution of the rendering task;

[0009] In an optional implementation, the device power consumption when the GPU is triggered to perform the rendering task according to the first rendering instruction is greater than the device power consumption when the GPU is triggered to perform the rendering task according to the second rendering instruction; and / or, the corresponding amount of memory data copying when the GPU is triggered to perform the rendering task according to the first rendering instruction is greater than the corresponding amount of memory data copying when the GPU is triggered to perform the rendering task according to the second rendering instruction; and / or, the corresponding GPU load when the GPU is triggered to perform the rendering task according to the first rendering instruction is greater than the corresponding GPU load when the GPU is triggered to perform the rendering task according to the second rendering instruction; and / or, the corresponding central processing unit CPU load when the GPU is triggered to perform the rendering task according to the first rendering instruction is greater than the corresponding CPU load when the GPU is triggered to perform the rendering task according to the second rendering instruction.

[0010] That is, under the condition of executing the same rendering task, the rendering efficiency of the second rendering instruction is better than that of the first rendering task;

[0011] triggering the GPU to execute the rendering task according to the second rendering instruction;

[0012] In an optional implementation, after the CPU obtains the second rendering instruction, it passes the second rendering instruction to the GPU driver. For example, the GPU driver can compile the second rendering instruction into an object or machine code executable by the GPU. After compilation, these second rendering instructions can be executed by the GPU. After receiving the compiled second rendering instruction, the GPU can execute the corresponding rendering task.

[0013] In the embodiment of the present application, the game can reduce the terminal performance loss caused by frequent memory mapping or reduce the CPU / GPU load by executing instructions based on GPU extension without modifying the code. At the same time, the embodiment of the present application is a general extension optimization framework based on GPU extension, which can improve the performance of mobile phones that support the extension, and do not enable optimization for mobile phones that do not support the extension, and has better compatibility.

[0014] In an optional implementation, obtaining a second rendering instruction according to the first rendering instruction includes:

[0015] Based on a mapping relationship, the second rendering instruction corresponding to the first rendering instruction is obtained, where the mapping relationship includes a preset corresponding relationship between the first rendering instruction and the second rendering instruction.

[0016] In which, the CPU can obtain the second rendering instruction corresponding to the first rendering instruction based on a mapping relationship, and the mapping relationship includes a preset correspondence relationship between the first rendering instruction and the second rendering instruction, wherein the mapping relationship includes a correspondence relationship between rendering instructions, and in the mapping relationship, the first rendering instruction corresponds to the second rendering instruction.

[0017] In an optional implementation, obtaining the second rendering instruction according to the first rendering instruction includes: when the first rendering instruction meets a preset condition, obtaining the second rendering instruction according to the first rendering instruction; the preset condition includes: the first rendering instruction belongs to a preset instruction set.

[0018] The terminal device may pre-specify some instructions that are not implemented based on the target extension and have poor rendering efficiency, and the current GPU supports a certain extension, based on which there may be another instruction that can implement the same rendering task. At this time, the terminal device may pre-specify some instructions as candidate optimized instructions (such as the first rendering instruction in this embodiment), and accordingly, may pre-specify optimized instructions as candidate optimized instructions (such as the second rendering instruction in this embodiment).

[0019] It should be noted that the first rendering instruction may be an instruction set including multiple instructions. In this case, the preset instruction set may include multiple instructions and the timing sequence between the instructions. The first rendering instruction belonging to the preset instruction set can be understood as the first rendering instruction including the names of multiple instructions and the order of their execution belong to the preset instruction set.

[0020] In an optional implementation, the first rendering instruction is used to operate a target cache, and the preset condition includes at least one of the following:

[0021] The cache type of the target cache meets a preset condition;

[0022] The historical information of the target cache being operated by the first rendering instruction meets a preset condition; or,

[0023] The context information of the target cache meets a preset condition.

[0024] It should be noted that, for the cache type of the target cache to meet the preset conditions, it may be whether the type of data stored in the target cache meets the conditions, or whether some attributes of the target cache itself meet the conditions. For example, when the first rendering instruction is the glBufferData instruction, it can be determined whether the target cache is used to store vertex or index data, and whether its purpose is dynamically updated. If so, it is determined that the cache type of the target cache meets the preset conditions.

[0025] It should be noted that, for the target cache, the number of historical operations by the first rendering instruction is greater than the preset value; it can be obtained whether the target cache is operated by the first rendering instruction within a certain period of time. For example, when the first rendering instruction is the glBlitFramebuffer instruction, it can be determined whether the number of times (frequency) that the bound ReadFramebuffer is called within a certain period of time in history is greater than the preset value.

[0026] It should be noted that the context information can represent the rendering state. For example, when the first rendering instruction is a glBlitFramebuffer instruction, the color and depth template attachments of the currently bound Read Framebuffer can be determined to determine whether their sampling values ​​are greater than 1, and so on.

[0027] In an optional implementation, the first rendering instruction is used to operate a texture object, and the preset condition includes that context information of the texture object satisfies the preset condition.

[0028] In an optional implementation, the second rendering instruction is configured to call an interface of the target extension.

[0029] Exemplarily, the first rendering instruction may be a glBufferData instruction, and the second rendering instruction may be a glBufferStorageEXT instruction implemented based on the EXT_buffer_storage extension. It should be noted that the GL_MAP_PERSISTENT_BIT_EXT, GL_MAP_COHERENT_BIT_EXT, and GL_DYNAMIC_STORAGE_BIT_EXT attributes may also be additionally specified in the glBufferStorageEXT instruction to create a buffer that supports persistent address mapping.

[0030] In an optional implementation, the first rendering instruction is located in a target function table, and the first rendering instruction includes a plurality of first sub-instructions. Accordingly, if the first rendering instruction satisfies a preset condition, it includes:

[0031] If at least one first sub-instruction among the multiple first sub-instructions included in the first rendering instruction meets a preset condition.

[0032] It should be noted that the preset condition satisfied by the at least one first sub-instruction may refer to the preset condition satisfied by the first rendering instruction.

[0033] In an optional implementation, the second rendering instruction includes a plurality of second sub-instructions, each second sub-instruction corresponds to a first sub-instruction; and in response to the first rendering instruction, executing the second rendering instruction includes:

[0034] In response to each first sub-instruction, a second sub-instruction corresponding to each first sub-instruction is executed.

[0035] In an optional implementation, at least one second sub-instruction among the plurality of second sub-instructions is configured to call an interface of the target extension.

[0036] In an optional implementation, at least one second sub-instruction among the plurality of second sub-instructions is configured to call an interface of the target extension and is an instruction related to implementing a function of the target extension.

[0037] In an optional implementation, the first rendering instruction includes a first target rendering instruction and a second target rendering instruction; wherein the first target rendering instruction is a rendering instruction corresponding to a first frame image, the second target rendering instruction is a rendering instruction corresponding to a second frame image, the first frame image is an image frame before the second frame image, and if the first rendering instruction satisfies a preset condition, it includes:

[0038] If the first target rendering instruction meets the preset condition, and the second target rendering instruction meets the preset condition.

[0039] Specifically, before each frame (first frame image) ends, the scene is identified based on the analysis of the first target rendering instruction to determine whether to enter the optimization. For the intercepted first target rendering instruction, the specific parameters, number of calls, current OpenGL ES state information, recorded historical information, and current context characteristics, etc. are analyzed to match the preset scene in the extended optimization algorithm library (specifically, please refer to the part of how to determine whether the first rendering instruction meets the preset conditions in the above embodiment). If the first target rendering instruction meets the preset conditions, the second target rendering instruction intercepted in the second frame image can be analyzed to match the preset scene in the extended optimization algorithm library (specifically, please refer to the part of how to determine whether the first rendering instruction meets the preset conditions in the above embodiment). If the second target rendering instruction meets the preset conditions, the second rendering instruction is obtained.

[0040] In a second aspect, the present application provides a rendering instruction processing device, the device is applied to a terminal device, the terminal device includes a graphics processor GPU, the GPU supports the target extension in the open graphics library openGL ES of the embedded system, and the device includes:

[0041] An instruction acquisition module, used for acquiring a first rendering instruction that is not implemented based on the target extension, wherein the first rendering instruction is used for implementing a rendering task;

[0042] Obtaining a second rendering instruction according to the first rendering instruction; wherein the second rendering instruction is a rendering instruction implemented based on the target extension, and the second rendering instruction is used to implement the rendering task; and,

[0043] An instruction execution module is used to trigger the GPU to execute the rendering task according to the second rendering instruction.

[0044] In an optional implementation, the instruction acquisition module is used to:

[0045] Based on a mapping relationship, the second rendering instruction corresponding to the first rendering instruction is obtained, where the mapping relationship includes a preset corresponding relationship between the first rendering instruction and the second rendering instruction.

[0046] In an optional implementation, the device power consumption when the GPU is triggered to perform the rendering task according to the first rendering instruction is greater than the device power consumption when the GPU is triggered to perform the rendering task according to the second rendering instruction; and / or,

[0047] The amount of memory data copied when the GPU is triggered to execute the rendering task according to the first rendering instruction is greater than the amount of memory data copied when the GPU is triggered to execute the rendering task according to the second rendering instruction; and / or

[0048] The GPU load corresponding to when the GPU is triggered to execute the rendering task according to the first rendering instruction is greater than the GPU load corresponding to when the GPU is triggered to execute the rendering task according to the second rendering instruction; and / or,

[0049] The corresponding central processing unit (CPU) load when the GPU is triggered to execute the rendering task according to the first rendering instruction is greater than the corresponding CPU load when the GPU is triggered to execute the rendering task according to the second rendering instruction.

[0050] In an optional implementation, the preset condition includes: the first rendering instruction belongs to a preset instruction set.

[0051] In an optional implementation, the first rendering instruction is used to operate a target cache, and the preset condition includes at least one of the following:

[0052] The cache type of the target cache meets a preset condition;

[0053] The historical information of the target cache being operated by the first rendering instruction meets a preset condition; or,

[0054] The context information of the target cache meets a preset condition.

[0055] In an optional implementation, the first rendering instruction is used to operate a texture object, and the preset condition includes that context information of the texture object satisfies the preset condition.

[0056] In an optional implementation, the second rendering instruction is configured to call an interface of the target extension.

[0057] In an optional implementation, the first rendering instruction includes multiple first sub-instructions, and accordingly, the instruction acquisition module is used to acquire the second rendering instruction if at least one first sub-instruction among the multiple first sub-instructions included in the first rendering instruction meets a preset condition.

[0058] In an optional implementation, the second rendering instruction includes a plurality of second sub-instructions, each second sub-instruction corresponds to a first sub-instruction; and the instruction execution module is used to:

[0059] In response to each first sub-instruction, a second sub-instruction corresponding to each first sub-instruction is executed.

[0060] In an optional implementation, at least one second sub-instruction among the plurality of second sub-instructions is configured to call an interface of the target extension.

[0061] In an optional implementation, at least one second sub-instruction among the plurality of second sub-instructions is configured to call an interface of the target extension and is an instruction related to implementing a function of the target extension.

[0062] In an optional implementation, the first rendering instruction includes a first target rendering instruction and a second target rendering instruction; wherein the first target rendering instruction is a rendering instruction corresponding to a first frame image, the second target rendering instruction is a rendering instruction corresponding to a second frame image, the first frame image is an image frame before the second frame image, and if the first rendering instruction satisfies a preset condition, it includes:

[0063] If the first target rendering instruction meets the preset condition, and the second target rendering instruction meets the preset condition.

[0064] In a third aspect, the present application provides a terminal device, comprising a processor and a memory, wherein the processor obtains a code stored in the memory to execute the first aspect and any one of its optional implementation methods.

[0065] In a fourth aspect, the present application provides a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium contains computer instructions for executing the rendering instruction processing method described in any one of the first aspect and its optional implementations.

[0066] In a fifth aspect, the present application also provides a computer program product, comprising computer instructions, which are executed by a processor of a host device to perform operations performed by the processor in any one of the possible implementations of this embodiment.

[0067] The embodiment of the present application provides a rendering instruction processing method, which is applied to a terminal device, wherein the terminal device includes a graphics processor GPU, and the GPU supports a target extension in an open graphics library for embedded systems openGL ES, including: obtaining a first rendering instruction that is not implemented based on the target extension, and the first rendering instruction is used to implement a rendering task; obtaining a second rendering instruction according to the first rendering instruction; wherein the second rendering instruction is a rendering instruction implemented based on the target extension, and the second rendering instruction is used to implement the rendering task; and triggering the GPU to execute the rendering task according to the second rendering instruction. In the above manner, for a specific scenario, using a rendering instruction based on a GPU extension instead of a rendering instruction that is not originally based on a GPU extension can reduce the performance loss of the terminal device. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 A schematic diagram of the system architecture;

[0069] Figure 2 A block diagram illustrating a computing device that may implement the techniques described in this disclosure;

[0070] Figure 3 A flowchart of a rendering instruction processing method provided in an embodiment of the present application;

[0071] Figure 4 This is a schematic diagram of a rendering instruction replacement in this embodiment;

[0072] Figure 5 An illustration of a rendering instruction processing method provided in an embodiment of the present application;

[0073] Figure 6a An illustration of a rendering instruction processing provided by an embodiment of the present application;

[0074] Figure 6b An illustration of a rendering instruction processing provided by an embodiment of the present application;

[0075] Figure 7 An illustration of a rendering instruction processing method provided in an embodiment of the present application;

[0076] Figure 8 An illustration of a rendering instruction processing method provided in an embodiment of the present application;

[0077] Fig. 9 A schematic diagram of the structure of a rendering instruction processing device provided in this application;

[0078] Fig.10 A schematic diagram of the structure of a terminal device provided in this application. DETAILED DESCRIPTION

[0079] The following describes the embodiments of the present application in conjunction with the accompanying drawings. Obviously, the described embodiments are only embodiments of a part of the present application, rather than all embodiments. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0080] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described here can be implemented in an order other than that illustrated or described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The naming or numbering of steps in this application does not mean that the steps in the method flow must be executed in the time / logical sequence indicated by the naming or numbering. The process steps that have been named or numbered can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved.

[0081] OpenGL (open graphics library) refers to a professional graphics programming interface that defines a cross-programming language and cross-platform programming interface specification. It is used in industries including content creation, energy, entertainment, game development, manufacturing, pharmaceuticals, and virtual reality. OpenGL can help programmers develop high-performance, high-visual-expression graphics processing software on hardware devices such as personal computers (PCs), workstations, and supercomputers.

[0082] OpenGL ES (openGL for embedded systems) is a subset of the OpenGL 3D graphics application programming interface (API), designed for embedded devices such as mobile phones and game consoles. Among them, OpenGL ES is based on OpenGL, removing many non-essential features such as glBegin / glEnd, quadrilaterals (GL_QUADS), polygons (GL_POLYGONS) and other complex primitives. After years of development, there are now two main versions, OpenGL ES 1.x for fixed pipeline hardware, and OpenGL ES 2.x for programmable pipeline hardware. Among them, OpenGL ES 1.0 is based on the OpenGL 1.3 specification, and OpenGL ES 1.1 is based on the OpenGL 1.5 specification. OpenGL ES 2.0 is defined with reference to the OpenGL 2.0 specification.

[0083] The enabling scenarios of OpenGL ES include but are not limited to: image processing, such as image tone conversion and beautification; camera preview effect processing, such as beauty camera, video processing, 3D games, etc.

[0084] In one implementation, OpenGL is implemented as a client-server system, where the application is the client and OpenGL is the server. Figure 1 As shown, the client program needs to call the OpenGL interface to implement 3D rendering. Then the OpenGL commands and data will be cached in the random access memory (RAM). Under certain conditions, these commands and data will be sent to the video random access memory (VRAM) through the CPU clock. Under the control of the GPU, the data and commands in the VRAM are used to complete the graphics rendering and store the results in the frame buffer. The frames in the frame buffer will eventually be sent to the display to display the results. In modern graphics hardware systems, it is also supported to send data directly from RAM to VRAM or directly from the frame buffer to RAM without using the CPU clock (such as VBO and PBO in OpenGL).

[0085] In some OpenGL implementations, such as those associated with the X Window System, the client and server execute on different machines, connected by a network. In this case, the client sends OpenGL commands, which are converted into the window system's protocol and sent to the server over the shared network.

[0086] In the field of gaming, game frame drops and power consumption issues have always been a pain point for terminal manufacturers. Correspondingly, graphics rendering technology is also constantly improving. OpenGL ES has developed from 1.0 to 3.2, supporting more features and continuously improving rendering efficiency. GPU manufacturers are also actively promoting various technologies such as load reduction and image quality improvement, and launching more efficient OpenGL ES extensions. New OpenGL ES extensions can improve rendering efficiency. However, before extensions enter the standard, there is a long evolutionary path. For example, a GPU manufacturer proposes a new extension that can improve the rendering efficiency of a specific scene, promotes it to other GPU manufacturers, becomes a general extension, and then promotes the approval of the Khronos organization, and finally promotes it to be incorporated into the standard.

[0087] However, before becoming a standard or being approved by a standards organization, game engines generally do not quickly adapt to use the latest extensions, that is, even if the GPU of the terminal device currently running the game supports a non-standard OpenGL ES extension, because the extension has not become a standard, for the same rendering task, the rendering instructions in the game are not based on the extension, but are implemented based on other instructions, which reduces the final rendering efficiency (for example, the display effect after rendering is poor or the power consumption caused to the terminal device is high, etc.). In order to solve the above problems, the present application proposes a rendering instruction processing method.

[0088] Next, a schematic diagram of a system architecture of an embodiment of the present application is described.

[0089] Figure 2 1 is a block diagram illustrating a computing device 30 that can implement the technology described in the present invention. The computing device 30 can be a rendering instruction processing device in the embodiment of the present application, and examples of the computing device 30 include but are not limited to wireless devices, mobile or cellular phones (including so-called smart phones), personal digital assistants (PDAs), video game consoles including video displays, mobile video game devices, mobile video conferencing units, laptop computers, desktop computers, TV set-top boxes, tablet computing devices, e-book readers, fixed or mobile media players, and the like.

[0090] exist Figure 2 In the example of FIG. 3 , the computing device 30 includes a central processing unit (CPU) 32 having a CPU memory 34, a graphics processing unit (GPU) 36 having a GPU memory 38 and one or more shader units 40, a display unit 42, a display buffer unit 44, a user interface unit 46, and a storage unit 48. In addition, the storage unit 48 can store a GPU driver 50 having a compiler 54, a GPU program 52, and a natively compiled GPU program 56.

[0091] Examples of CPU 32 include, but are not limited to, a digital signal processor (DSP), a general purpose microprocessor, an application specific integrated circuit (ASIC), a field programmable logic array (FPGA), or other equivalent integrated or discrete logic circuits. Figure 232 are illustrated as separate units in the examples of FIG. 34 , but in some examples, CPU 32 and GPU 36 may be integrated into a single unit. CPU 32 may execute one or more applications. Examples of applications may include a web browser, an email application, a spreadsheet, a video game, an audio and / or video capture, playback or editing application, or other application that initiates the generation of image data to be presented via display unit 42.

[0092] exist Figure 2 In the example shown, CPU 32 includes CPU memory 34. CPU memory 34 may represent an on-chip storage device or memory used when executing machine or object code. CPU memories 34 may each include hardware memory registers capable of storing a fixed number of digital bits. CPU 32 may be able to read values ​​from or write values ​​to local CPU memory 34 more quickly than reading values ​​from or writing values ​​to storage unit 48 (which may be accessed, for example, via a system bus).

[0093] GPU 36 represents one or more dedicated processors for performing graphics operations. That is, for example, GPU 36 can be a dedicated hardware unit with fixed functions and programmable components for rendering graphics and executing GPU applications. GPU 36 can also include a DSP, a general-purpose microprocessor, an ASIC, an FPGA, or other equivalent integrated or discrete logic circuits.

[0094] GPU 36 also includes GPU memory 38, which may represent on-chip storage devices or memory used when executing machine or object code. GPU memories 38 may each include hardware memory registers capable of storing a fixed number of digital bits. GPU 36 may be able to read values ​​from or write values ​​to local GPU memory 38 more quickly than reading values ​​from or writing values ​​to storage unit 48 (which may be accessed, for example, via a system bus).

[0095] GPU 36 also includes a shading unit 40. As described in more detail below, shading unit 40 may be configured as a programmable pipeline of processing components. In some examples, shading unit 40 may be referred to as a "shader processor" or a "unified shader" and may perform geometry, vertex, pixel, or other shading operations to render graphics. Shading unit 40 may include Figure 2 For the sake of clarity, one or more components not specifically shown in the figure, such as components for fetching and decoding instructions, one or more arithmetic and logic units (ALUs) for performing arithmetic calculations, and one or more memories, caches, or registers.

[0096] The display unit 42 represents a unit capable of displaying video data, images, text, or any other type of data. The display unit 42 may include a liquid crystal display (LCD), a light emitting diode (LED) display, an organic LED (OLED), an active-matrix organic light-emitting diode (AMOLED) display, and the like.

[0097] Display buffer unit 44 represents a memory or storage device dedicated to storing data for display unit 42 for presenting an image (e.g., a photo or video frame). Display buffer unit 44 may represent a two-dimensional buffer including a plurality of storage locations. The number of storage locations within display buffer unit 44 may be substantially similar to the number of pixels to be displayed on display unit 42. For example, if display unit 42 is configured to include 640x480 pixels, then display buffer unit 44 may include 640x480 storage locations. Display buffer unit 44 may store the final pixel value of each of the pixels processed by GPU 36. Display unit 42 may retrieve the final pixel value from display buffer unit 44 and display the final image based on the pixel value stored in display buffer unit 44.

[0098] The user interface unit 46 represents a unit that a user can use to interact with other units of the computing device 30 (e.g., CPU 32) or otherwise interface to communicate with other units of the computing device 30. Examples of the user interface unit 46 include, but are not limited to, a trackball, a mouse, a keyboard, and other types of input devices. The user interface unit 46 may also be a touch screen and may be incorporated as part of the display unit 42.

[0099] The storage unit 48 may include one or more computer-readable storage media. Examples of the storage unit 48 include, but are not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer or processor.

[0100] In some example embodiments, storage unit 48 may include instructions that cause CPU 32 and / or GPU 36 to perform the functions of the present invention for implementing CPU 32 and GPU 36. In some examples, storage unit 48 may be considered a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier wave or propagation signal. However, the term "non-transitory" should not be interpreted to mean that storage unit 48 is not removable. As an example, storage unit 48 may be removed from computing device 30 and moved to another device. As another example, a storage unit substantially similar to storage unit 48 may be inserted into computing device 30. In some examples, non-transitory storage media may store data that may change over time (e.g., in RAM).

[0101] Storage unit 48 stores GPU driver 50 and compiler 54, GPU program 52, and natively compiled GPU program 56. GPU driver 50 represents a computer program or executable code that provides an interface for accessing GPU 36. CPU 32 executes GPU driver 50 or some portion thereof to connect with GPU 36, and for this reason, GPU driver 50 is executed in Figure 2 In the example of FIG. 3 , a GPU driver 50 is shown as a dashed box within the CPU 32. The GPU driver 50 can access programs or other executable files executed by the CPU 32, including the GPU program 52.

[0102] The GPU program 52 may include code written in a high-level (HL) programming language (e.g., using an application programming interface (API)). An example of an API includes the Open Graphics Library (OpenGL). Generally speaking, an API includes a predetermined, standardized set of commands that are executed by associated hardware. The API commands allow a user to instruct the hardware components of the GPU to execute the commands without the user having to know the specifics of the hardware components.

[0103] The GPU program 52 may call or otherwise include one or more functions provided by the GPU driver 50. The CPU 32 generally executes the program in which the GPU program 52 is embedded, and upon encountering the GPU program 52, passes the GPU program 52 to the GPU driver 50 (e.g., in the form of a command stream). The CPU 32 executes the GPU driver 50 in this context to process the GPU program 52. For example, the GPU driver 50 may process the GPU program 52 by compiling the GPU program into an object or machine code that is executable by the GPU 36. This object code is in Figure 2 The example shown is a natively compiled GPU program 56.

[0104] In some examples, the compiler 54 may operate in real time or near real time to compile the GPU program 52 during execution of a program in which the GPU program 52 is embedded. For example, the compiler 54 generally represents a module that condenses HL instructions defined according to the HL programming language into LL instructions of a low-level (LL) programming language. After compilation, these LL instructions can be executed by a specific type of processor or other type of hardware (e.g., FPGA, ASIC, etc., including, for example, CPU 32 and GPU 36).

[0105] exist Figure 2 In the example of , compiler 54 may receive GPU program 52 from CPU 32 while executing HL code including GPU program 52. Compiler 54 may compile GPU program 52 into a native compiled GPU program 56 conforming to the LL programming language. Compiler 54 then outputs native compiled GPU program 56 including LL instructions.

[0106] The GPU 36 generally receives a natively compiled GPU program 56 (as shown by the dashed box labeled “Natively Compiled GPU Program 56” within the GPU 36), after which, in some examples, the GPU 36 immediately renders an image and outputs the rendered portion of the image to the display buffer unit 44. For example, the GPU 36 may generate a plurality of primitives to be displayed at the display unit 42. A primitive may include one or more lines (including curves, splines, etc.), points, circles, ellipses, polygons (where a polygon is generally defined as a collection of one or more triangles), or any other two-dimensional (2D) primitives. The term “primitive” may also refer to three-dimensional (3D) primitives, such as a cube, cylinder, sphere, cone, pyramid, torus, etc. In general, the term “primitive” refers to any geometric shape or element that is rendered by the GPU 36 for display as an image (or frame in the context of video data) via the display unit 42.

[0107] The GPU 36 may transform primitives or other state data of primitives (e.g., which defines the texture, lighting, camera configuration, or other aspects of the primitive) into so-called "world space" by applying one or more model transformations (which may also be specified in the state data). Once transformed, the GPU 36 may apply the view transformation of the effective camera (which may also be specified in the state data defining the camera) to transform the coordinates of the primitives and lights into camera or eye space. The GPU 36 may also perform vertex shading to render the appearance of the primitives in the view of any effective light. The GPU 36 may perform vertex shading in one or more of the above-mentioned model, world, or view spaces (although vertex shading is typically performed in world space).

[0108] Once the primitives are shaded, the GPU 36 may perform a projection to project the image into (as an example) a unit cube with poles at (-1, -1, -1) and (1, 1, 1). This unit cube is often referred to as a canonical view volume. After transforming the model from eye space to the canonical view volume, the GPU 36 may perform clipping to remove any primitives that do not reside at least partially in the view volume. In other words, the GPU 36 may remove any primitives that are not within the camera frame. The GPU 36 may then map the coordinates of the primitives from the view volume to screen space, effectively reducing the 3D primitives of the primitives to 2D coordinates of the screen.

[0109] Given the transformed and projected vertices that define the primitives with their associated shading data, the GPU 36 may then rasterize the primitives. For example, the GPU 36 may calculate and set the color of the pixels of the screen covered by the primitives. During rasterization, the GPU 36 may apply any textures associated with the primitives (wherein the textures may include state data). The GPU 36 may also perform a Z-buffer algorithm (also known as a depth test) during rasterization to determine if any primitives and / or objects are obscured by any other objects. The Z-buffer algorithm sorts the primitives according to their depth so that the GPU 36 knows the order in which each primitive should be drawn to the screen. The GPU 36 outputs the rendered pixels to the display buffer unit 44.

[0110] Display buffer unit 44 may temporarily store rendered pixels of a rendered image until the entire image is rendered. In this context, display buffer unit 44 may be viewed as an image frame buffer. Display buffer unit 44 may then transmit the rendered image to be displayed on display unit 42. In some alternative examples, GPU 36 may output the rendered portion of the image directly to display unit 42 for display, rather than temporarily storing the image in display buffer unit 44. Display unit 42 may then display the image stored in display buffer unit 78.

[0111] In the embodiment of the present application, when a program that needs to perform graphics rendering is running, the CPU 32 can obtain rendering instructions for each frame image from the central processing unit memory, and execute the rendering instructions to achieve graphics rendering. In particular, the CPU 32 can obtain a function table (hook table) for each frame image from the central processing unit memory, wherein the function table may include multiple rendering instructions, and the CPU can achieve graphics rendering by executing each rendering instruction in the function table (the specific execution process can be referred to Figure 2 The description in the corresponding embodiment will not be repeated here).

[0112] In particular, the GPU can support multiple extensions in OpenGL ES. However, when application developers write programs, they may not use more efficient extensions to write programs for graphics rendering because the extensions have not been extended to the standard.

[0113] In order to solve the above problems, a rendering instruction processing method provided in an embodiment of the present application is described in detail below.

[0114] Reference Figure 3 , Figure 3 A flowchart of a rendering instruction processing method provided in an embodiment of the present application is shown as follows: Figure 3 As shown in the figure, the rendering instruction processing method provided by the embodiment of the present application includes:

[0115] 301. Obtain a first rendering instruction that is not implemented based on the target extension, where the first rendering instruction is used to implement a rendering task.

[0116] Taking a game as an example, when the game is started, the rendering instructions for the rendering tasks in the game can be obtained, and then the rendering tasks corresponding to the rendering instructions can be executed; wherein the first rendering instruction is one of the obtained rendering instructions.

[0117] In the embodiment of the present application, the first rendering instruction is not implemented based on the target extension.

[0118] 302. Obtain a second rendering instruction according to the first rendering instruction; wherein the second rendering instruction is a rendering instruction implemented based on the target extension, and the second rendering instruction is used to implement the rendering task.

[0119] In an embodiment of the present application, if the first rendering instruction meets a preset condition, a second rendering instruction is obtained according to the first rendering instruction; wherein the second rendering instruction is a rendering instruction implemented based on the target extension.

[0120] In the embodiment of the present application, in the process of replacing instructions in the application code that do not adopt the target extension with instructions implemented based on the extension, it is first necessary to identify which instructions need to be converted.

[0121] In an embodiment of the present application, after obtaining the first rendering instruction, the first rendering instruction will be analyzed to determine whether the current scene meets the preset requirements. If the first rendering instruction meets the preset conditions, it is determined that the first rendering instruction needs to be converted.

[0122] In an embodiment of the present application, the terminal device can pre-configure some rules, which define preset conditions that instructions should meet, wherein the first rendering instruction is used to operate the target cache, and the preset conditions include at least one of the following: the first rendering instruction belongs to a preset instruction set; the cache type of the target cache meets the preset conditions; the target cache has been historically operated a number of times by the first rendering instruction is greater than a preset value; or, the context information of the target cache meets the preset conditions.

[0123] The first rendering instruction includes a plurality of first sub-instructions. Accordingly, it is necessary to determine whether at least one first sub-instruction among the plurality of first sub-instructions included in the first rendering instruction meets a preset condition.

[0124] Specifically, for the preset condition that the first rendering instruction belongs to a preset instruction set, the terminal device can pre-specify some instructions that are not implemented based on the target extension and have poor rendering efficiency, and the current GPU supports a certain extension, based on which there may be another instruction that can implement the same rendering task. At this time, the terminal device can pre-specify some instructions as candidate optimization instructions (such as the first rendering instruction in this embodiment), and correspondingly, it can also pre-specify optimization instructions as candidate optimization instructions (such as the second rendering instruction in this embodiment).

[0125] In one implementation, the first rendering instruction may be an independent instruction.

[0126] For example, the glMapBufferRange instruction and the glBufferData instruction in openGL ES, when the vertex and index data are frequently updated, there are a lot of map operations when executing the above instructions, which will cause a high GPU load and CPU load. For another example, when the glBlitFramebuffer instruction is executed, the amount of data copied to the DDR memory is large. The preset instruction set in the embodiment of the present application can be an instruction set including the above instructions.

[0127] It should be noted that whether the instruction belongs to the preset instruction set can be directly determined by identifying the instruction name.

[0128] In one implementation, the first rendering instruction may be an instruction set including multiple instructions. In this case, the preset instruction set may include multiple instructions and the timing sequence between the instructions. The first rendering instruction belonging to the preset instruction set can be understood as the first rendering instruction including the names of multiple instructions and the execution order of the instructions all belong to the preset instruction set. For example, the preset instruction set includes an instruction subset {glBindBuffer; glBufferData; glMapBufferRange; memcpy; glUnmapBuffer}.

[0129] It should be noted that the first rendering instruction belongs to a preset instruction set, and may be just a condition that the first rendering instruction satisfies a preset condition.

[0130] In one implementation, the first rendering instruction is used to operate a target buffer, wherein the target buffer may be one of a frame buffer, a buffer, and a render buffer.

[0131] Specifically, for the cache type of the target cache to meet the preset conditions, it may be whether the type of data stored in the target cache meets the conditions, or whether some attributes of the target cache itself meet the conditions. For example, when the first rendering instruction is the glBufferData instruction, it can be determined whether the target cache is used to store vertex or index data, and whether its purpose is dynamically updated. If so, it is determined that the cache type of the target cache meets the preset conditions.

[0132] Specifically, if the target cache is operated by the first rendering instruction more than a preset value, it can be determined whether the target cache is operated by the first rendering instruction more than a preset value within a certain period of time. For example, when the first rendering instruction is a glBlitFramebuffer instruction, it can be determined whether the number of times (frequency) that the bound ReadFramebuffer is called within a certain period of time in history is greater than a preset value.

[0133] Specifically, the context information of the target cache meets the preset conditions; the context information of the target cache can be obtained, and the context information can represent the rendering status. For example, when the first rendering instruction is the glBlitFramebuffer instruction, the color and depth template attachments of the currently bound Read Framebuffer can be judged, and its sampling value can be judged whether it is greater than 1, and so on.

[0134] In an embodiment of the present application, whether the current scene meets the conditions can be judged based on whether the first rendering instruction belongs to a preset instruction set; whether the cache type of the target cache meets the preset conditions; whether the historical number of times the target cache has been operated by the first rendering instruction is greater than a preset value; and whether the context information of the target cache meets at least one of the preset conditions, thereby determining whether the first rendering instruction meets the preset conditions.

[0135] In an embodiment of the present application, if the first rendering instruction satisfies a preset condition, a second rendering instruction may be obtained; wherein the second rendering instruction is configured to be implemented based on the target extension.

[0136] Specifically, a mapping relationship between the first rendering instruction and the second rendering instruction can be pre-configured, and the mapping relationship includes a preset correspondence between the first rendering instruction and the second rendering instruction; after determining that the first rendering instruction meets the preset conditions, the second rendering instruction corresponding to the first rendering instruction can be obtained in the mapping relationship.

[0137] In one implementation, the second rendering instruction is configured to call an interface of the target extension.

[0138] Exemplarily, the first rendering instruction may be a glBufferData instruction, and the second rendering instruction may be a glBufferStorageEXT instruction implemented based on the EXT_buffer_storage extension. It should be noted that the GL_MAP_PERSISTENT_BIT_EXT, GL_MAP_COHERENT_BIT_EXT, and GL_DYNAMIC_STORAGE_BIT_EXT attributes may also be additionally specified in the glBufferStorageEXT instruction to create a buffer that supports persistent address mapping.

[0139] In one implementation, the second rendering instruction includes a plurality of second sub-instructions, and each second sub-instruction corresponds to a first sub-instruction.

[0140] In one implementation, the first rendering instruction is used to operate a texture object, and the preset condition includes that context information of the texture object satisfies the preset condition.

[0141] For example, refer to Figure 4 , Figure 4 This is a schematic diagram of a rendering instruction replacement in this embodiment, such as Figure 4As shown in, at the first glMapBufferRange, the first rendering instruction can be an instruction set {glBindBuffer; glBufferData; glMapBufferRange; memcpy; glUnmapBuffer}, and correspondingly, the second rendering instruction can be an instruction set {glBindBuffer; glBufferStorageEXT; glMapBufferRange; memcpy; return}; wherein, glBindBuffer corresponds to glBindBuffer, glBufferData corresponds to glBufferStorageEXT, glMapBufferRange corresponds to glMapBufferRange, memcpy corresponds to memcpy, and glUnmapBuffer corresponds to return. It can be seen that at least one of the multiple second sub-instructions is configured to call the interface of the target extension. At least one of the multiple second sub-instructions is configured to call the interface of the target extension and is an instruction related to implementing the functions of the target extension.

[0142] When it is not the first time glMapBufferRange, the first rendering instruction can be the instruction set {glBindBuffer; glBufferData; glMapBufferRange; memcpy; glUnmapBuffer}, and accordingly, the second rendering instruction can be the instruction set {glBindBuffer; return; return address; memcpy; return}; among which, glBindBuffer corresponds to glBindBuffer, glBufferData corresponds to return, glMapBufferRange corresponds to return address, memcpy corresponds to memcpy, and glUnmapBuffer corresponds to return.

[0143] For another example, the first rendering instruction may be a glBlitFramebuffer instruction, and the second rendering instruction may be a glRenderbufferStorageMultisampleEXT instruction implemented based on the EXT_multisampled_render_to_texture extension.

[0144] In an embodiment of the present application, the device power consumption when the GPU is triggered to perform the rendering task according to the first rendering instruction is greater than the device power consumption when the GPU is triggered to perform the rendering task according to the second rendering instruction; and / or, the corresponding memory data copy amount when the GPU is triggered to perform the rendering task according to the first rendering instruction is greater than the corresponding memory data copy amount when the GPU is triggered to perform the rendering task according to the second rendering instruction; and / or, the corresponding GPU load when the GPU is triggered to perform the rendering task according to the first rendering instruction is greater than the corresponding GPU load when the GPU is triggered to perform the rendering task according to the second rendering instruction; and / or, the corresponding central processing unit CPU load when the GPU is triggered to perform the rendering task according to the first rendering instruction is greater than the corresponding CPU load when the GPU is triggered to perform the rendering task according to the second rendering instruction.

[0145] In an embodiment of the present application, the first rendering instruction includes a first target rendering instruction and a second target rendering instruction; wherein, the first target rendering instruction is a rendering instruction corresponding to the first frame image, the second target rendering instruction is a rendering instruction corresponding to the second frame image, and the first frame image is an image frame before the second frame image, then if the first target rendering instruction satisfies a preset condition and the second target rendering instruction satisfies a preset condition.

[0146] Specifically, before each frame (first frame image) ends, the scene is identified based on the analysis of the first target rendering instruction to determine whether to enter the optimization. For the intercepted first target rendering instruction, the specific parameters, number of calls, current OpenGL ES state information, recorded historical information, and current context characteristics, etc. are analyzed to match the preset scene in the extended optimization algorithm library (specifically, please refer to the part of how to determine whether the first rendering instruction meets the preset conditions in the above embodiment). If the first target rendering instruction meets the preset conditions, the second target rendering instruction intercepted in the second frame image can be analyzed to match the preset scene in the extended optimization algorithm library (specifically, please refer to the part of how to determine whether the first rendering instruction meets the preset conditions in the above embodiment). If the second target rendering instruction meets the preset conditions, the second rendering instruction is obtained.

[0147] 303. Trigger the GPU to execute the rendering task according to the second rendering instruction.

[0148] In the embodiment of the present application, after the CPU obtains the second rendering instruction, it passes the second rendering instruction to the GPU driver. For example, the GPU driver can compile the second rendering instruction into an object or machine code executable by the GPU. After compilation, these second rendering instructions can be executed by the GPU. After receiving the compiled second rendering instruction, the GPU can execute the corresponding rendering task.

[0149] In an optional implementation, the first rendering instruction may be used as a pointer to execute the code corresponding to the second rendering instruction.

[0150] In an embodiment of the present application, the second rendering instruction may include multiple second sub-instructions, each second sub-instruction corresponding to a first sub-instruction; then, the second sub-instruction corresponding to each first sub-instruction may be executed in response to each first sub-instruction. That is, after obtaining the first rendering instruction from the application, the code corresponding to the first rendering instruction is not executed, but the code corresponding to the second rendering instruction is executed with the first rendering instruction as a pointer. Among them, executing the second rendering instruction can be understood as sending the second rendering instruction to the corresponding driver, and executing the rendering task corresponding to the second rendering instruction based on the GPU.

[0151] Taking the game rendering scene as an example, in the embodiment of the present application, the game can reduce the terminal performance loss caused by frequent memory mapping or reduce the CPU / GPU load by executing instructions based on GPU extension without modifying the code. At the same time, the embodiment of the present application is a general extension optimization framework based on GPU extension, which can improve the performance of mobile phones that support the extension, and do not enable optimization for mobile phones that do not support the extension, and has better compatibility.

[0152] The embodiment of the present application provides a rendering instruction processing method, which is applied to a terminal device, wherein the terminal device includes a graphics processor GPU, and the GPU supports the target extension in the open graphics library for embedded systems openGL ES, including: obtaining a first rendering instruction; if the first rendering instruction meets a preset condition, obtaining a second rendering instruction; wherein the second rendering instruction is configured to be implemented based on the target extension; and, in response to the first rendering instruction, executing the second rendering instruction. In the above manner, for a specific scene, using a rendering instruction based on GPU extension instead of a rendering instruction that is not originally based on GPU extension can reduce the performance loss of the terminal device.

[0153] Next, taking the game rendering scene as an example, we will give a comparison Figure 3 The corresponding embodiments include more detailed embodiments.

[0154] Reference Figure 5 , Figure 5A schematic diagram of a rendering instruction processing method 500 provided in an embodiment of the present application is shown as follows: Figure 5 As shown in, when the game starts 501, initialization 502 can be performed, and the initialization process may include: traversing the directory where the extended optimization algorithm is located, loading all installed dynamic library files, wherein the extended optimization algorithm in the embodiment of the present application must include the following functions: CheckExtEnabled: used to check whether the extended optimization algorithm is supported; OnEnterHookMode: called after the extended optimization algorithm is enabled, it will replace the hook table of OpenGL ES; OnExitHookMode: called after the extended optimization algorithm is closed, it will restore the hook table of OpenGL ES; OnInterceptEndCommand: called before each frame ends, completes scene recognition, and determines whether subsequent frames enter optimization.

[0155] Based on GetProcAddress 503, function pointers of external interfaces of all extended optimization algorithms are obtained, and the algorithm enable detection function CheckExtEnabled 504 of each extended optimization algorithm is called to determine whether the algorithm is enabled. Specifically, all openGL ES extensions supported by the current device GPU can be obtained to check whether the extensions that the current extended optimization algorithm depends on are supported. If supported, OnEnterHookMode 505 is enabled.

[0156] Before each frame ends, the OnInterceptEndCommand function 506 is called to identify the scene and decide whether to enter optimization. All OpenGL ES and EGL function calls are intercepted and the command analysis module of the extended optimization framework is jumped to. For the intercepted commands, the specific parameters, the number of calls, the current OpenGL ES state information, the recorded historical information and the current context characteristics are analyzed to match the preset scene 507 in the extended optimization algorithm library.

[0157] Each algorithm has preset supported scenarios, which is a collection of rules. For example, it may include:

[0158] 1. Instruction name: Each scenario is a combination of a series of instructions, and the general scenario can be inferred through the instructions;

[0159] 2. Command parameters: Based on the specific parameter information of the command, the scope of the scene is further narrowed;

[0160] 3. Number of command calls: Analyze the command calls within a period of time and only optimize common scenarios. Do not optimize occasional scenarios.

[0161] 4. Current context state information: Get the current OpenGL ES context state information and only optimize scenes that meet the expected range;

[0162] 5. Historical information: It is necessary to record instruction calls over a period of time as a reference for matching information, etc.

[0163] For intercepted instructions, match the preset scenes in the extended optimization algorithm library. If the scenes match, start to perform instruction conversion optimization. Call the algorithm enable function OnEnterHookMode of each extended optimization algorithm to replace the GL function table of the current rendering thread.

[0164] In the next frame, if optimization is available, instruction conversion 508 is performed and the next frame is rendered 509 .

[0165] Then turn off optimization and restore function table 510

[0166] Next, we give a comparison with Figure 3 The corresponding embodiments include more detailed embodiments.

[0167] Reference Figure 6a , Figure 6a An example of a rendering instruction processing provided in an embodiment of the present application is as follows: Figure 6a As shown in FIG. 1 , the conversion of rendering instructions can be achieved through the extended optimization framework in this embodiment, and Figure 3 The difference between the embodiments is that Figure 6a A process is given that can perform multiple optimization judgments and optimization executions.

[0168] like Figure 6a As shown in the figure, the first rendering instruction (glFunction) that the extended optimization framework can obtain is based on scene matching and conversion enable (for details, please refer to Figure 5 Corresponding embodiment description) and instruction conversion, to achieve one optimization, then can be based on scene matching, conversion enable (for details, please refer to Figure 5 The corresponding embodiment description) and instruction conversion are used to achieve another optimization, wherein the difference between the two optimizations is that based on different scene recognitions, the instructions are replaced with rendering instructions that support different extensions.

[0169] For more details, see Figure 6b , Figure 6b An example of a rendering instruction processing provided in an embodiment of the present application is as follows: Figure 6b As shown in , for glFunction1, it is based on scene matching and conversion enable (for details, please refer to Figure 5Corresponding embodiment description) and instruction conversion, only the optimization corresponding to Algorithm 2 is implemented (Algorithm 1 may be due to scene matching failure). For glFunction2, it implements the optimization corresponding to Algorithm 1 and Algorithm 2 based on scene matching, conversion enabling and instruction conversion. For glFunction3, it does not implement optimization based on scene matching, conversion enabling and instruction conversion (Algorithm 1 and Algorithm 2 may be due to scene matching failure). For glFunction4, it only implements the optimization corresponding to Algorithm 1 based on scene matching, conversion enabling and instruction conversion (Algorithm 2 may be due to scene matching failure).

[0170] Next, we give a comparison with Figure 3 The corresponding embodiments include more detailed embodiments, which are aimed at scenes where vertex and index data are frequently updated in game rendering.

[0171] In this embodiment, the glMapBufferRange instruction can be identified to check whether the currently bound buffer is vertex or index data and whether it is dynamically writable. If it is satisfied, if the map instruction call reaches a certain ratio, the scene is matched, and then the extended conversion is enabled.

[0172] In this embodiment, a buffer dictionary may be maintained to record all buffer information, including target, size, usage and other information.

[0173] Reference Figure 7 , when the glBufferData instruction 701 is obtained, it can be checked whether the target buffer of its operation is vertex or index data, whether the purpose is dynamic update, and whether the size meets the limit 702. If the conditions are met, call the glBufferStorageEXT interface 703, additionally specify the GL_MAP_PERSISTENT_BIT_EXT, GL_MAP_COHERENT_BIT_EXT and GL_DYNAMIC_STORAGE_BIT_EXT attributes to create a buffer that supports persistent address mapping. If the interface has been converted before and the size and attributes have not changed, return directly. When glMapBufferRange704 is used for the first time, GL_MAP_PERSISTENT_BIT_EXT and GL_MAP_COHERENT_BIT_EXT attributes need to be added to the access parameters to obtain the persistent memory address 705. The next time glMapBufferRange704 is used for the same buffer, the previously obtained address 706 is directly returned. When glUnmapBuffer, return directly.

[0174] Through the above method, in scenarios where there are a large number of map operations on vertex and index data, the call ratio of glBufferData instructions, glMapBufferRange instructions and glUnmapBuffer instructions is reduced, which can effectively reduce the CPU and GPU load.

[0175] Next, we give a comparison with Figure 3 The corresponding embodiments include more detailed embodiments. They are aimed at anti-aliasing scenarios in the rendering process. Figure 8 , the glBlitFramebuffer instruction 801 can be identified, and the color and depth template attachments of the currently bound ReadFramebuffer are checked to see if the sampling value is greater than 1. If satisfied, if the Blit instruction call reaches a certain ratio, the scene is matched 802, and then the extended conversion is enabled.

[0176] First, you can create a new framebuffer803, create a new single-sample color texture804 and allocate space805, create a new depth template attachment806, and use the extension to provide interfaces to bind single-sample color and depth template attachments. RenderBuffer allocates spaceglRenderbufferStorageMultisampleEXT807, binds the texture to the FramebufferglframebufferTexture2DMultisampleEXT808, binds the RBO to glFramebufferRenderBuffer809, and renders directly to the new Framebuffer in the next frame810.

[0177] When binding a multi-sampled framebuffer, replace it with the newly created framebuffer ID so that the multi-sampled result is output to the single-sampled attachment. When binding a framebuffer where Blit is located, skip binding the framebuffer and blit operations. When binding to a subsequent framebuffer, since the single-sampled texture output by the previous Blit is used, the bound texture here is replaced with the texture corresponding to the newly created framebuffer.

[0178] In this embodiment, for anti-aliasing scenes, the game can be expanded through GPU without modifying the code, thereby reducing 60% of DDR data copies and reducing the power consumption of the mobile phone.

[0179] Reference Fig. 9 , Fig. 9The present application provides a schematic diagram of the structure of a rendering instruction processing device 900, the device is applied to a terminal device, the terminal device includes a graphics processor GPU, the GPU supports the target extension in the open graphics library openGL ES for embedded systems, and the device includes:

[0180] The instruction acquisition module 901 is used to acquire a first rendering instruction that is not implemented based on the target extension, and the first rendering instruction is used to implement the rendering task; acquire a second rendering instruction according to the first rendering instruction; wherein the second rendering instruction is a rendering instruction implemented based on the target extension, and the second rendering instruction is used to implement the rendering task;

[0181] The instruction execution module 902 is used to trigger the GPU to execute the rendering task according to the second rendering instruction.

[0182] In an optional implementation, the instruction acquisition module is used to:

[0183] Based on a mapping relationship, the second rendering instruction corresponding to the first rendering instruction is obtained, where the mapping relationship includes a preset corresponding relationship between the first rendering instruction and the second rendering instruction.

[0184] In an optional implementation, the device power consumption when the GPU is triggered to perform the rendering task according to the first rendering instruction is greater than the device power consumption when the GPU is triggered to perform the rendering task according to the second rendering instruction; and / or,

[0185] The amount of memory data copied when the GPU is triggered to execute the rendering task according to the first rendering instruction is greater than the amount of memory data copied when the GPU is triggered to execute the rendering task according to the second rendering instruction; and / or

[0186] The GPU load corresponding to when the GPU is triggered to execute the rendering task according to the first rendering instruction is greater than the GPU load corresponding to when the GPU is triggered to execute the rendering task according to the second rendering instruction; and / or,

[0187] The corresponding central processing unit (CPU) load when the GPU is triggered to execute the rendering task according to the first rendering instruction is greater than the corresponding CPU load when the GPU is triggered to execute the rendering task according to the second rendering instruction.

[0188] In an optional implementation, the preset condition includes: the first rendering instruction belongs to a preset instruction set.

[0189] In an optional implementation, the first rendering instruction is used to operate a target cache, and the preset condition includes at least one of the following:

[0190] The cache type of the target cache meets a preset condition;

[0191] The historical information of the target cache being operated by the first rendering instruction meets a preset condition; or,

[0192] The context information of the target cache meets a preset condition.

[0193] In an optional implementation, the first rendering instruction is used to operate a texture object, and the preset condition includes that context information of the texture object satisfies the preset condition.

[0194] In an optional implementation, the second rendering instruction is configured to call an interface of the target extension.

[0195] In an optional implementation, the first rendering instruction includes multiple first sub-instructions, and accordingly, the instruction acquisition module is used to acquire the second rendering instruction if at least one first sub-instruction among the multiple first sub-instructions included in the first rendering instruction meets a preset condition.

[0196] In an optional implementation, the second rendering instruction includes a plurality of second sub-instructions, each second sub-instruction corresponds to a first sub-instruction; and the instruction execution module is used to:

[0197] In response to each first sub-instruction, a second sub-instruction corresponding to each first sub-instruction is executed.

[0198] In an optional implementation, at least one second sub-instruction among the plurality of second sub-instructions is configured to call an interface of the target extension.

[0199] In an optional implementation, at least one second sub-instruction among the plurality of second sub-instructions is configured to call an interface of the target extension and is an instruction related to implementing a function of the target extension.

[0200] In an optional implementation, the first rendering instruction includes a first target rendering instruction and a second target rendering instruction; wherein the first target rendering instruction is a rendering instruction corresponding to a first frame image, the second target rendering instruction is a rendering instruction corresponding to a second frame image, the first frame image is an image frame before the second frame image, and if the first rendering instruction satisfies a preset condition, it includes:

[0201] If the first target rendering instruction meets the preset condition, and the second target rendering instruction meets the preset condition.

[0202] Reference Fig.10 , Fig.10 A schematic diagram of the structure of a terminal device 1000 provided in this application is shown as follows: Fig.10 As shown, the terminal device includes a processor 1001 and a memory 1003. The processor includes a central processing unit CPU and a graphics processing unit GPU. The GPU supports the target extension in the open graphics library openGL ES for embedded systems. The CPU is used to obtain the code of the memory to execute:

[0203] Acquire a first rendering instruction that is not implemented based on the target extension, and the first rendering instruction is used to implement a rendering task; acquire a second rendering instruction according to the first rendering instruction; wherein the second rendering instruction is a rendering instruction implemented based on the target extension, and the second rendering instruction is used to implement the rendering task;

[0204] triggering the GPU to execute the rendering task according to the second rendering instruction;

[0205] In the embodiment of the present application, the game can reduce the terminal performance loss caused by frequent memory mapping or reduce the CPU / GPU load by executing instructions based on GPU extension without modifying the code. At the same time, the embodiment of the present application is a general extension optimization framework based on GPU extension, which can improve the performance of mobile phones that support the extension, and do not enable optimization for mobile phones that do not support the extension, and has better compatibility.

[0206] In an optional implementation, obtaining a second rendering instruction according to the first rendering instruction includes:

[0207] Based on a mapping relationship, the second rendering instruction corresponding to the first rendering instruction is obtained, where the mapping relationship includes a preset corresponding relationship between the first rendering instruction and the second rendering instruction.

[0208] In an optional implementation, obtaining the second rendering instruction according to the first rendering instruction includes: when the first rendering instruction meets a preset condition, obtaining the second rendering instruction according to the first rendering instruction; the preset condition includes: the first rendering instruction belongs to a preset instruction set.

[0209] In an optional implementation, the first rendering instruction is used to operate a target cache, and the preset condition includes at least one of the following:

[0210] The cache type of the target cache meets a preset condition;

[0211] The historical information of the target cache being operated by the first rendering instruction meets a preset condition; or,

[0212] The context information of the target cache meets a preset condition.

[0213] In an optional implementation, the first rendering instruction is used to operate a texture object, and the preset condition includes that context information of the texture object satisfies the preset condition.

[0214] In an optional implementation, the device power consumption when the GPU is triggered to perform the rendering task according to the first rendering instruction is greater than the device power consumption when the GPU is triggered to perform the rendering task according to the second rendering instruction; and / or,

[0215] The amount of memory data copied when the GPU is triggered to execute the rendering task according to the first rendering instruction is greater than the amount of memory data copied when the GPU is triggered to execute the rendering task according to the second rendering instruction; and / or

[0216] The GPU load corresponding to when the GPU is triggered to execute the rendering task according to the first rendering instruction is greater than the GPU load corresponding to when the GPU is triggered to execute the rendering task according to the second rendering instruction; and / or,

[0217] The corresponding central processing unit (CPU) load when the GPU is triggered to execute the rendering task according to the first rendering instruction is greater than the corresponding CPU load when the GPU is triggered to execute the rendering task according to the second rendering instruction.

[0218] In an optional implementation, the second rendering instruction is configured to call an interface of the target extension.

[0219] In an optional implementation, the first rendering instruction is located in a target function table, and the first rendering instruction includes a plurality of first sub-instructions. Accordingly, if the first rendering instruction satisfies a preset condition, it includes:

[0220] If at least one first sub-instruction among the multiple first sub-instructions included in the first rendering instruction meets a preset condition.

[0221] It should be noted that the preset condition satisfied by the at least one first sub-instruction may refer to the preset condition satisfied by the first rendering instruction.

[0222] In an optional implementation, the second rendering instruction includes a plurality of second sub-instructions, each second sub-instruction corresponds to a first sub-instruction; and in response to the first rendering instruction, executing the second rendering instruction includes:

[0223] In response to each first sub-instruction, a second sub-instruction corresponding to each first sub-instruction is executed.

[0224] In an optional implementation, at least one second sub-instruction among the plurality of second sub-instructions is configured to call an interface of the target extension.

[0225] In an optional implementation, at least one second sub-instruction among the plurality of second sub-instructions is configured to call an interface of the target extension and is an instruction related to implementing a function of the target extension.

[0226] In an optional implementation, the first rendering instruction includes a first target rendering instruction and a second target rendering instruction; wherein the first target rendering instruction is a rendering instruction corresponding to a first frame image, the second target rendering instruction is a rendering instruction corresponding to a second frame image, the first frame image is an image frame before the second frame image, and if the first rendering instruction satisfies a preset condition, it includes:

[0227] If the first target rendering instruction meets the preset condition, and the second target rendering instruction meets the preset condition.

[0228] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0229] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0230] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0231] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0232] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, server, or other network device, etc.) to execute the present application. Figure 2 All or part of the steps of the method described in the embodiment. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0233] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A rendering instruction processing method, the method is applied to a terminal device, the terminal device includes a graphics processor GPU, the GPU supports the target extension in the embedded system open graphics library OpenGL ES, characterized in that: The method comprises: Acquire a first rendering instruction that is not implemented based on the target extension, where the first rendering instruction is used to implement a rendering task; Obtaining a second rendering instruction according to the first rendering instruction; wherein the second rendering instruction is a rendering instruction implemented based on the target extension, and the second rendering instruction is used to implement the rendering task; and, triggering the GPU to execute the rendering task according to the second rendering instruction; The acquiring the second rendering instruction according to the first rendering instruction comprises: Based on a mapping relationship, the second rendering instruction corresponding to the first rendering instruction is obtained, where the mapping relationship includes a preset corresponding relationship between the first rendering instruction and the second rendering instruction.

2. The method according to claim 1, characterized in that The device power consumption when the GPU is triggered to execute the rendering task according to the first rendering instruction is greater than the device power consumption when the GPU is triggered to execute the rendering task according to the second rendering instruction; and / or, The amount of memory data copied when the GPU is triggered to execute the rendering task according to the first rendering instruction is greater than the amount of memory data copied when the GPU is triggered to execute the rendering task according to the second rendering instruction; and / or The GPU load corresponding to when the GPU is triggered to execute the rendering task according to the first rendering instruction is greater than the GPU load corresponding to when the GPU is triggered to execute the rendering task according to the second rendering instruction; and / or, The corresponding central processing unit (CPU) load when the GPU is triggered to execute the rendering task according to the first rendering instruction is greater than the corresponding CPU load when the GPU is triggered to execute the rendering task according to the second rendering instruction.

3. The method according to claim 1 or 2, characterized in that: The acquiring the second rendering instruction according to the first rendering instruction includes: when the first rendering instruction satisfies a preset condition, acquiring the second rendering instruction according to the first rendering instruction; the preset condition is one or more of the following situations: The first rendering instruction belongs to a preset instruction set; The first rendering instruction is used to operate a target cache, and a cache type of the target cache meets a preset condition; The historical information of the target cache being operated by the first rendering instruction meets a preset condition; The context information of the target cache meets a preset condition; or, The first rendering instruction is used to operate a texture object, and the preset condition includes that context information of the texture object meets the preset condition.

4. The method according to claim 1 or 2, characterized in that: The second rendering instruction is an interface for calling the target extension.

5. The method according to claim 1 or 2, characterized in that: The first rendering instruction includes a plurality of first sub-instructions. Accordingly, if the first rendering instruction satisfies a preset condition, it includes: If at least one first sub-instruction among the multiple first sub-instructions included in the first rendering instruction meets a preset condition.

6. The method according to claim 5, characterized in that The second rendering instruction includes a plurality of second sub-instructions, each of which corresponds to a first sub-instruction; and in response to the first rendering instruction, executing the second rendering instruction includes: In response to each first sub-instruction, a second sub-instruction corresponding to each first sub-instruction is executed.

7. The method according to claim 5, characterized in that At least one second sub-instruction among the plurality of second sub-instructions is configured to call an interface of the target extension.

8. The method according to claim 5, characterized in that At least one second sub-instruction among the plurality of second sub-instructions is configured not to call an interface of the target extension, and is an instruction related to implementing a function of the target extension.

9. The method according to claim 1 or 2, characterized in that: The first rendering instruction includes a first target rendering instruction and a second target rendering instruction; wherein the first target rendering instruction is a rendering instruction corresponding to a first frame image, the second target rendering instruction is a rendering instruction corresponding to a second frame image, the first frame image is an image frame before the second frame image, and if the first rendering instruction satisfies a preset condition, it includes: If the first target rendering instruction meets the preset condition, and the second target rendering instruction meets the preset condition.

10. A rendering instruction processing device, the device is applied to a terminal device, the terminal device includes a graphics processor GPU, the GPU supports the target extension in the open graphics library openGL ES of embedded systems, characterized in that: The device comprises: An instruction acquisition module, used for acquiring a first rendering instruction that is not implemented based on the target extension, wherein the first rendering instruction is used for implementing a rendering task; Based on a mapping relationship, obtaining the second rendering instruction corresponding to the first rendering instruction, the mapping relationship including a preset correspondence relationship between the first rendering instruction and the second rendering instruction; wherein the second rendering instruction is a rendering instruction implemented based on the target extension, and the second rendering instruction is used to implement the rendering task; and, An instruction execution module is used to trigger the GPU to execute the rendering task according to the second rendering instruction.

11. The device according to claim 10, characterized in that The device power consumption when the GPU is triggered to execute the rendering task according to the first rendering instruction is greater than the device power consumption when the GPU is triggered to execute the rendering task according to the second rendering instruction; and / or, The amount of memory data copied when the GPU is triggered to execute the rendering task according to the first rendering instruction is greater than the amount of memory data copied when the GPU is triggered to execute the rendering task according to the second rendering instruction; and / or The GPU load corresponding to when the GPU is triggered to execute the rendering task according to the first rendering instruction is greater than the GPU load corresponding to when the GPU is triggered to execute the rendering task according to the second rendering instruction; and / or, The corresponding central processing unit (CPU) load when the GPU is triggered to execute the rendering task according to the first rendering instruction is greater than the corresponding CPU load when the GPU is triggered to execute the rendering task according to the second rendering instruction.

12. The device according to claim 10 or 11, characterized in that The instruction acquisition module is used to: When the first rendering instruction meets a preset condition, the second rendering instruction is obtained according to the first rendering instruction; the preset condition is one or more of the following situations: The first rendering instruction belongs to a preset instruction set; The first rendering instruction is used to operate a target cache, and a cache type of the target cache meets a preset condition; The historical information of the target cache being operated by the first rendering instruction meets a preset condition; The context information of the target cache meets a preset condition; or, The first rendering instruction is used to operate a texture object, and the preset condition includes that context information of the texture object meets the preset condition.

13. The device according to claim 10 or 11, characterized in that The second rendering instruction is configured to call an interface of the target extension.

14. The device according to claim 10 or 11, characterized in that The first rendering instruction includes a plurality of first sub-instructions. Accordingly, the instruction acquisition module is used to acquire a second rendering instruction if at least one first sub-instruction among the plurality of first sub-instructions included in the first rendering instruction meets a preset condition.

15. The device according to claim 14, characterized in that The second rendering instruction includes a plurality of second sub-instructions, each of which corresponds to a first sub-instruction; and the instruction execution module is used to: In response to each first sub-instruction, a second sub-instruction corresponding to each first sub-instruction is executed.

16. The device according to claim 14, characterized in that At least one second sub-instruction among the plurality of second sub-instructions is configured to call an interface of the target extension.

17. The device according to claim 14, characterized in that At least one second sub-instruction among the plurality of second sub-instructions is configured to call an interface of the target extension, and is an instruction related to implementing a function of the target extension.

18. The device according to claim 10 or 11, characterized in that The first rendering instruction includes a first target rendering instruction and a second target rendering instruction; wherein the first target rendering instruction is a rendering instruction corresponding to a first frame image, the second target rendering instruction is a rendering instruction corresponding to a second frame image, the first frame image is an image frame before the second frame image, and if the first rendering instruction satisfies a preset condition, it includes: If the first target rendering instruction meets the preset condition, and the second target rendering instruction meets the preset condition.

19. A non-volatile computer-readable storage medium, characterized in that: The non-volatile readable storage medium contains computer instructions for executing the rendering instruction processing method described in any one of claims 1 to 9.

20. A computing device, characterized in that: The computing device includes a memory and a processor, wherein the memory stores codes, and the processor is used to obtain the codes to execute the rendering instruction processing method according to any one of claims 1 to 9.

Citation Information

Patent Citations

  • Rending method and apparatus

    CN105741227A

  • OpenGL-based image processor and function expansion method thereof

    CN108133453A