Method and related device for graphics rendering based on vulkan

Abstract

Embodiments of the present application provide a method and related device for implementing graphics rendering based on Vulkan, the method comprising: processing, by a first thread, asynchronously executed OpenGL ES instructions into a first event, the first event containing description information of the asynchronously executed OpenGL ES instructions; processing, by a second thread, the first event to obtain a second task, the second task containing one first event or multiple first events, the multiple first events being obtained according to the same or different asynchronously executed OpenGL ES instructions; saving, by a third thread, the second task to a command buffer; and sending, by a Vulkan interface, the second task in the command buffer to a GPU image processor, wherein the GPU is configured to complete graphics rendering according to the second task.

Description

Technical Field

[0001] The present application relates to the field of computer graphics technology, and in particular to a method and related apparatus for implementing graphics rendering based on Vulkan. Background Art

[0002] With the rapid development of terminal technology, computers and mobile phones can now install various applications to achieve a variety of functions. For example, an application can be equipped with features such as beauty, makeup, and magic emojis. Due to years of code accumulation, these functions are currently based on the corresponding renderers and graphics engines written in the Open Graphics Library (OpenGL) or OpenGL for Embedded Systems (OpenGL ES). However, many manufacturers are planning to adopt Vulkan to replace OpenGL ES applications. For most applications, OpenGL or OpenGL ES will still exist to maintain forward compatibility.

[0003] To provide OpenGL ES functionality in computer systems that don't use the OpenGL ES driver, a simple approach is to directly implement OpenGL ES using Vulkan as the underlying layer. However, the efficiency of directly implementing OpenGL ES using Vulkan as the underlying layer is far lower than that of the native OpenGL ES driver, which in turn affects GPU rendering performance. Summary of the Invention

[0004] The embodiments of the present application disclose a method and related apparatus for implementing graphics rendering based on Vulkan, which can improve the rendering efficiency of the GPU.

[0005] In a first aspect, an embodiment of the present application discloses a method for implementing graphics rendering based on Vulkan. In this method, an asynchronously executed OpenGL ES instruction is processed into a first event through a first thread, and the first event may include description information of the asynchronously executed OpenGL ES instruction; then, the first event may be processed by a second thread to obtain a second task (job), and the second task may include one first event or multiple first events, and the multiple first events are obtained according to the same or different asynchronously executed OpenGL ES instructions; then, the second task may be saved to a command buffer through a third thread; finally, the second task in the command buffer may be sent to a graphics processor (GPU) through a Vulkan interface, wherein the GPU is used to complete graphics rendering according to the second task.

[0006] In the embodiment of the present application, Vulkan is used as the backend to implement OpenGL ES graphics rendering. Because Vulkan supports multi-threaded parallel processing, the embodiment of the present application takes advantage of Vulkan's multi-threading advantages and processes different events or tasks in different threads (e.g., a first thread, a second thread, and a third thread). For example, the first thread processes an asynchronously executed OpenGL ES instruction into a first event, the second thread receives the first event and processes it into a second task job, the third thread saves the second task job into a command buffer, and then sends the command buffer to the GPU through the Vulkan interface, and the GPU completes the graphics rendering. Moreover, while the third thread is processing the second task, the second thread is also processing other received first events, and the first thread is also processing other received asynchronously executed OpenGL ES instructions. It can be seen that each thread is processed in parallel, thereby improving processing efficiency. Therefore, it is also possible to increase the frequency of submission to the GPU, thereby achieving the purpose of improving rendering efficiency.

[0007] In a possible implementation of the first aspect, before the asynchronously executed OpenGL ES instructions are processed into the first event by the first thread, multiple OpenGL ES instructions may be classified into asynchronously executed OpenGL ES instructions and synchronously executed OpenGL ES instructions according to the type of the OpenGL ES application program interface API.

[0008] It can be seen that by classifying OpenGL ES instructions into asynchronously executed OpenGL ES instructions and synchronously executed OpenGL ES instructions, the OpenGL ES instructions can be processed differently in a targeted manner to improve processing efficiency.

[0009] In a possible implementation of the first aspect, state setting and obtaining type OpenGL ES instructions are synchronously executed OpenGL ES instructions; object creation and destruction type OpenGL ES instructions are also synchronously executed OpenGL ES instructions; data upload type OpenGL ES instructions and drawing type OpenGL ES instructions are both asynchronously executed OpenGL ES.

[0010] In a possible implementation of the first aspect, after classifying multiple OpenGL ES instructions into asynchronously executed OpenGL ES instructions and synchronously executed OpenGL ES instructions according to the type of the OpenGL ES application program interface API, the description information of the synchronously executed OpenGL ES instructions can also be saved to a data structure through the first thread, and the saved result can be returned after the saving is completed.

[0011] In a possible implementation of the first aspect, after classifying multiple OpenGL ES instructions into asynchronously executed OpenGL ES instructions and synchronously executed OpenGL ES instructions according to the type of the OpenGL ES application program interface API, the first thread can also obtain description information of the synchronously executed OpenGL ES instructions from the data structure and send it to the image processor, and the image processor is used to complete graphics rendering according to the description information of the synchronously executed OpenGL ES instructions; wherein the first thread can be a rendering thread of the OpenGL ES application.

[0012] It can be seen that the first thread may be a thread that executes a rendering thread that calls an OpenGL ES application. Therefore, when processing synchronously executed OpenGL ES instructions in the first thread, it can ensure that the OpenGL ES API is visible to a single thread of the upper-layer application.

[0013] In a possible implementation of the first aspect, obtaining the second task job by processing the first event through the second thread can be specifically manifested as: when the first event currently processed belongs to the preset first type, executing the step of obtaining the second task job by processing the first event through the second thread.

[0014] It can be seen that by controlling the frequency of the first event processing becoming the second task in the second thread through the type of the first event, the frequency of submission to the GPU can be controlled, thereby avoiding unstable GPU rendering performance and improving the efficiency of GPU rendering applications.

[0015] In a possible implementation of the first aspect, obtaining the second task job through processing by the second thread can be specifically manifested as: when the first event currently processed does not belong to the preset first type, if the number of first events currently accumulated in the cache is greater than the first threshold, executing the step of obtaining the second task by processing the first event through the second thread.

[0016] It can be seen that in the second thread, the frequency of first event processing becoming the second task is controlled by the type of the first event and the number of cached first events. Furthermore, the frequency of submission to the GPU can be controlled, thereby avoiding unstable GPU rendering performance and improving the efficiency of GPU rendering applications.

[0017] In a possible implementation of the first aspect, obtaining the second task job through the second thread processing can be specifically manifested as follows: when the first event currently processed does not belong to the preset first type, and the number of first events currently accumulated and cached is greater than the second threshold and less than or equal to the first threshold, if the GPU load is less than the first load threshold, it means that the GPU can process at this time, and then the step of obtaining the second task through the second thread processing is executed, wherein the second threshold is less than the first threshold; the first load threshold can be set based on experience, or it can be comprehensively determined based on the amount of data processed during graphics rendering. It is easy to understand that the "equal" situation can also be placed in another branch of the judgment, such as the situation where it is greater than or equal to the second threshold and less than the first threshold. It should be noted that the current load of the GPU is less than the first load can also be understood as the operating frequency of the GPU at this time is lower than the frequency during normal operation, or the electronic device running this GPU does not have the problem of severe heat generation and rapid power consumption. Among them, this load is mainly the load during the GPU rendering process.

[0018] It can be seen that in the second thread, the frequency of first event processing becoming the second task is controlled by the type of the first event, the number of cached first events, and the GPU load. Furthermore, the frequency of submission to the GPU can be controlled, thereby avoiding unstable GPU rendering performance and improving the efficiency of GPU rendering applications.

[0019] In one possible implementation of the first aspect, processing the first event by the second thread to obtain the second task (job) can be specifically manifested as: if the currently processed first event does not belong to the preset first type, and the number of currently cached first events is greater than a second threshold and less than or equal to the first threshold, and if the GPU load is greater than or equal to the first load threshold, indicating that the GPU is unable to process any further, then the first event is cached for processing, wherein the second threshold is less than the first threshold. It is readily understood that the "equal" condition can also be placed in another branch of the judgment, such as a condition greater than or equal to the second threshold and less than the first threshold. It should be noted that the GPU load being greater than or equal to the first load can also be understood as meaning that the GPU's operating frequency is higher than its normal operating frequency, or that the electronic device running the GPU is experiencing severe overheating and rapid power consumption. This load primarily refers to the GPU's rendering load. It can be seen that in the second thread, the frequency with which the first event is processed as the second task is controlled by the type of the first event, the number of cached first events, and the GPU load. For example, if the GPU is unable to process any further, the first event can be cached and not submitted to the GPU, thereby preventing unstable GPU rendering performance.

[0020] In a possible implementation of the first aspect, the second task may include multiple first events, and caching the second task in the command buffer area through the third thread is specifically manifested as: sorting the multiple first events in the second task through the third thread; and caching the sorted second task in the command buffer area.

[0021] It can be seen that the GPU needs to complete graphics rendering according to the second task in the command buffer, and the second task contains multiple first events. The first event contains the description information of the asynchronously executed OpenGL ES instructions. The GPU ultimately completes graphics rendering according to the description information of the asynchronously executed OpenGL ES instructions, so the multiple first events in the second task are sorted in the third thread. Using Vulkan, the underlying optimization of the GPU hardware can be directly operated to integrate the randomly arranged first events, thereby more effectively utilizing the GPU hardware resources.

[0022] In a possible implementation of the first aspect, sorting the multiple first events in the second task by the third thread can be specifically performed as follows: sorting the multiple first events in the second task by the third thread based on pipeline states corresponding to the multiple first events in the second task.

[0023] It can be seen that the second task contains multiple first events, and multiple first events may correspond to the same pipeline state. Each pipeline state requires drawing to perform image rendering. Therefore, Vulkan can directly operate the underlying optimization of the GPU hardware. By using the optimization strategy that multiple first events may correspond to the same pipeline state, the multiple first events in the second task are sorted in the third thread, which can more effectively utilize GPU hardware resources.

[0024] In a possible implementation of the first aspect, multiple first events in the second task corresponding to first events of the same pipeline state are sequenced together.

[0025] It can be seen that because each pipeline state requires drawing instructions to render the image, when multiple first events that may correspond to the same pipeline state are sorted together, only one drawing instruction can be called to complete the graphics rendering. Therefore, the rearranged first events can reduce the number of switches during the drawing process and improve the operating efficiency of the GPU.

[0026] In a possible implementation of the first aspect, the second task includes multiple first events, and caching the second task in the command cache area through the third thread is specifically manifested as: sorting the first events in the second task corresponding to the same variable that are modified through the third thread; modifying the sorted first events in the second task based on the variable; and caching the sorted and modified second task in the command cache area.

[0027] It can be seen that the GPU needs to complete graphics rendering according to the second task in the command buffer, and the second task contains multiple first events. The first event contains the description information of the asynchronously executed OpenGL ES instructions. The GPU ultimately completes the graphics rendering according to the description information of the asynchronously executed OpenGL ES instructions, and multiple first events may correspond to the same variable for modification. Each modified first event needs to be uploaded to the GPU to complete the graphics rendering. Therefore, Vulkan can be used to directly operate the underlying optimization of the GPU hardware. Through the optimization strategy of multiple first events corresponding to the same variable for modification, the multiple first events in the second task are sorted in the third thread, and the randomly arranged first events are integrated, which can more effectively utilize the GPU hardware resources.

[0028] In a possible implementation of the first aspect, first events corresponding to modifications of the same variable in the multiple events are sorted together.

[0029] It can be seen that because each modified first event needs to be uploaded to the GPU to complete graphics rendering, when multiple first events may correspond to first events that modify the same variable and are sorted together, there is no need to upload them to the GPU once they are modified. They can be uploaded to the GPU after all modifications are completed, reducing the switching frequency and improving the operating efficiency of the GPU.

[0030] In a possible implementation of the first aspect, processing the asynchronous instruction into a first event through the first thread includes: obtaining description information of the asynchronous OpenGL ES instruction through the first thread; and encapsulating the description information of the asynchronous OpenGL ES instruction and the asynchronous OpenGL ES instruction into the first event.

[0031] A second aspect of an embodiment of the present application discloses a device for implementing graphics rendering based on Vulkan, comprising:

[0032] A context module, configured to process the asynchronously executed OpenGL ES instruction into a first event through a first thread, wherein the first event includes description information of the asynchronously executed OpenGL ES instruction;

[0033] An event management module is configured to obtain a second task job by processing the first event through a second thread, where the second task includes one or more first events, and the multiple first events are obtained by processing the same or different asynchronously executed OpenGLES instructions;

[0034] An event packaging module, configured to save the second task to a command buffer via a third thread;

[0035] The task scheduling module is used to send the second task in the command buffer to the GPU graphics processor through the Vulkan interface, wherein the graphics processor is used to complete graphics rendering according to the second task.

[0036] In a possible implementation of the second aspect, the context module is further configured to classify the plurality of OpenGL ES instructions into asynchronously executed OpenGL ES instructions and synchronously executed OpenGL ES instructions according to the type of the OpenGL ES application program interface API.

[0037] In a possible implementation of the second aspect, state setting and obtaining type OpenGL ES instructions and object creation and destruction type OpenGL ES instructions are both synchronously executed OpenGL ES instructions; data upload type OpenGL ES instructions and drawing type OpenGL ES instructions are both asynchronously executed OpenGL ES.

[0038] In a possible implementation of the second aspect, the context module is further configured to save, through the first thread, description information of the synchronously executed OpenGL ES instructions to a data structure.

[0039] In a possible implementation of the second aspect, the context module is further configured to obtain, through the first thread, description information of the synchronously executed OpenGL ES instructions from the data structure, and send the information to the image processor, and the image processor is configured to complete graphics rendering according to the description information of the synchronously executed OpenGL ES instructions.

[0040] In a possible implementation of the second aspect, the event management module is specifically configured to: when the first event belongs to a preset first type, execute a step of obtaining a second task job through a second thread processing.

[0041] In a possible implementation of the second aspect, the event management module is specifically used to: when the first event does not belong to the preset first type, if the number of first events currently cached cumulatively is greater than a first threshold, execute the step of processing the first event through a second thread to obtain a second task.

[0042] In a possible implementation of the second aspect, the event management module is specifically used to: when the first event does not belong to the preset first type and the number of first events currently accumulated in the cache is greater than the second threshold and less than the first threshold, if the load of the GPU is less than the first load threshold, execute the step of processing the first event through the second thread to obtain the second task, and the second threshold is less than the first threshold.

[0043] In a possible implementation of the second aspect, the event management module is specifically used to: when the first event does not belong to the preset first type and the number of first events currently cached is greater than the second threshold and less than or equal to the first threshold, if the load of the GPU is greater than the first preset threshold, cache the first event.

[0044] In a possible implementation of the second aspect, the event packaging module is specifically configured to: sort the multiple first events in the second task through a third thread; and cache the sorted second tasks in a command buffer.

[0045] In a possible implementation of the second aspect, the event packaging module is specifically configured to sort the multiple first events in the second task based on pipeline states corresponding to the multiple first events through a third thread.

[0046] In a possible implementation of the second aspect, multiple first events corresponding to first events of the same pipeline state are sequenced together.

[0047] In a possible implementation of the second aspect, the event packaging module is specifically used to: sort the first events in the second task that correspond to the same variable and are modified through a third thread; modify the sorted first events in the second task based on the variable; and cache the sorted and modified second tasks into a command buffer area.

[0048] In a possible implementation of the second aspect, first events corresponding to modifications of the same variable in the multiple events are sorted together.

[0049] In a possible implementation of the second aspect, the context module is specifically configured to: obtain description information of the asynchronous OpenGL ES instruction through the first thread; and encapsulate the description information of the asynchronous OpenGL ES instruction and the asynchronous OpenGL ES instruction into a first event.

[0050] Regarding the technical effects brought about by the second aspect or specific implementation method, reference may be made to the introduction to the technical effects of the first aspect or the corresponding implementation method.

[0051] A third aspect of an embodiment of the present application provides an electronic device, which includes a processor and a memory; the processor is used to execute computer instructions stored in the memory, so that the electronic device implements the method described in the first aspect or any possible implementation method of the first aspect.

[0052] The fourth aspect of an embodiment of the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are prompted to implement the method described in the first aspect or any possible implementation method of the first aspect.

[0053] A fifth aspect of an embodiment of the present application provides a computer program product, characterized in that computer instructions are stored in the computer program product, which, when executed by one or more processors, prompt the one or more processors to implement the method described in the first aspect or any possible implementation method of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The following is an introduction to the drawings used in the embodiments of this application.

[0055] Figure 1 This is a schematic diagram of an architecture for implementing OpenGL ES provided in an embodiment of the present application;

[0056] Figure 2 This is a schematic diagram of an architecture for implementing graphics rendering based on Vulkan, provided in an embodiment of the present application;

[0057] Figure 3 is a flowchart of a method for implementing a graphics rendering based on Vulkan according to an embodiment of the present application;

[0058] Figure 4 is a component structure diagram of a method for implementing a graphics rendering based on Vulkan according to an embodiment of the present application;

[0059] Figure 5A is a flowchart of a method for classifying OpenGL ES instructions in an OpenGL ES implementation layer according to an embodiment of the present application;

[0060] Figure 5B is a flowchart of a method for processing OpenGL ES instructions executed synchronously according to an embodiment of the present application;

[0061] Figure 5C is a flowchart of a method for processing OpenGL ES instructions executed asynchronously according to an embodiment of the present application;

[0062] Figure 5D is a flowchart of a method for processing a second task according to an embodiment of the present application;

[0063] Figure 5E is a flowchart of a method for processing a second task according to an embodiment of the present application;

[0064] Figure 5F is a diagram of rearranging according to a first strategy according to an embodiment of the present application;

[0065] Figure 5G is a diagram of rearranging according to a second strategy according to an embodiment of the present application;

[0066] Figure 5H is a flowchart of a method for sending a second task in a command buffer to a GPU according to an embodiment of the present application;

[0067] Figure 6 is a structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0068] The embodiments of the present application will be described in detail with reference to the drawings. It should be noted that the terms “exemplary” or “for example” in the present application are used to mean “by way of example,” “like this,” or “thus.” Any embodiment or design scheme described as “exemplary” or “for example” in the present application should not be interpreted as being more preferred or advantageous than other embodiments or design schemes. The terms “exemplary” or “for example” are used in the present application to express the relevant concept in a specific manner.

[0069] The following is a brief introduction to the relevant technologies and professional terms involved in this application to facilitate understanding.

[0070] 1. Open Graphics Library or Open Graphics Library (OpenGL)

[0071] OpenGL is a cross-language, cross-platform application programming interface (API) for rendering 2D and 3D vector graphics. This API consists of nearly 350 different function calls used to create complex three-dimensional scenes from simple graphics bits. Another API system, Direct3D, is available only on Microsoft Windows. OpenGL is commonly used in virtual reality, scientific visualization programs, and video game development.

[0072] Efficient implementations of OpenGL (taking advantage of graphics acceleration hardware) exist for Windows, some UNIX platforms, and macOS. These implementations are typically provided by display device manufacturers and are highly dependent on the hardware provided by those manufacturers. The OpenGL API defines several functions that can be called by client programs, as well as named integer constants (for example, the constant GL_TEXTURE_2D corresponds to the decimal integer 3553). While these function definitions superficially resemble those in the C programming language, they are language-independent.

[0073] 2. OpenGL for Embedded Systems (OpenGL ES)

[0074] OpenGL ES is a subset of the OpenGL 3D graphics API, designed for embedded devices such as mobile phones, PDAs, and game consoles. The API is defined and promoted by the Khronos Group, an industry association for graphics hardware and software that focuses on open standards for graphics and multimedia.

[0075] OpenGL ES is a custom cut from OpenGL, removing many non-essential features such as glBegin / glEnd, quads (GL_QUADS), polygons (GL_POLYGONS), and other complex primitives. After years of development, there are now two main versions: OpenGL ES1.x for fixed-pipeline hardware, and OpenGL ES2.x for programmable-pipeline hardware. OpenGL ES1.0 is based on the OpenGL 1.3 specification, and OpenGL ES1.1 is based on the OpenGL 1.5 specification. They support two profiles, common and common lite, respectively. The lite profile only supports fixed-point real numbers, while the common profile supports both fixed-point and floating-point numbers. OpenGL ES2.0 was defined with reference to the OpenGL 2.0 specification. The common profile was released in August 2005 and introduced support for programmable pipelines.

[0076] 3. Vulkan (a cross-platform graphics API)

[0077] Vulkan is a cross-platform 2D and 3D graphics application programming interface (API) first announced by the Khronos Group at the 2015 Game Developers Conference (GDC).

[0078] Khronos initially referred to the Vulkan API as the "next generation OpenGL initiative" or "glNext," but those names were dropped after the official announcement. Like OpenGL, Vulkan is designed for real-time 3D applications (such as video games) and aims to provide high performance with low CPU overhead.

[0079] Like OpenGL, Vulkan was developed by the Khronos Group and is the successor to AMD Mantle. It inherits the former's powerful low-overhead architecture, enabling software developers to fully exploit the performance, efficiency, and capabilities of Radeon GPUs and multi-core CPUs.

[0080] Compared to OpenGL, Vulkan significantly reduces the CPU's "API overhead" (the background work performed by the CPU when analyzing the game's hardware requirements) in providing important features, performance, and image quality, and can use GPU hardware features that are not normally accessible through OpenGL.

[0081] 4. EGL

[0082] EGL is the interface between rendering APIs (such as OpenGL ES) and the underlying native platform window system. The EGL API is an independent API that is independent of the various versions of the OpenGL ES standard. Its main function is to create contexts for OpenGL instructions. Generally speaking, OpenGL is an API for operating the GPU. It sends relevant instructions to the GPU through the driver to control the operating state of the graphics rendering pipeline state machine. However, when it comes to interacting with the native window system, such an intermediate layer is required, and it is best to be platform-independent. Therefore, EGL was designed as a bridge between OpenGL and the native window system.

[0083] 5. Application Programming Interface (API)

[0084] An API is a set of predefined functions or conventions for connecting different components of a software system. It is used to provide a set of routines that applications and developers can access based on certain software or hardware without having to access the source code or understand the details of the internal working mechanism.

[0085] This interface is used by programmers during programming. Through this interface, the system and applications can access system resources and obtain OS services during execution. It is also the only way for programs to obtain operating system services. The programming interface of most operating systems consists of a set of system calls, each of which is a subroutine that performs a specific function.

[0086] An application programming interface (API), also known as an application programming interface (API), is a set of definitions, procedures, and protocols that enable communication between computer software. A primary function of an API is to provide a common set of functions. APIs also serve as a form of middleware, enabling data sharing across various platforms. In programming practice, the design of programming interfaces must prioritize the proper division of responsibilities within the software system. Well-designed interfaces can reduce interdependencies between system components, enhance the cohesion of component units, and reduce coupling between them, thereby improving system maintainability and scalability.

[0087] 6. Backend

[0088] In software engineering and programming, the backend operates behind the scenes, controlling the frontend's content. It's primarily responsible for programming architecture and database management. The backend primarily interacts with the database to handle business logic, focusing on implementing functionality, data access, and platform stability and performance.

[0089] 7. Frontend

[0090] In software engineering and programming, the front end is the part of a software system that directly interacts with users. For web front ends, this typically refers to the front-end portion of a website, including both the presentation and structure layers: the structure of the web page, the visual appearance of the web page, and how interactions are implemented at the web level.

[0091] Front-end technology is generally divided into front-end design and front-end development. Front-end design can generally be understood as the visual design of the website, while front-end development is the front-end code implementation of the website.

[0092] 8. Driver (Device)

[0093] Driver, a computer software term, refers to the program that drives software in a computer. A driver, also known as a device driver, is a special program added to the operating system that contains information about hardware devices. This information enables the computer to communicate with the corresponding device. A driver is a configuration file written by the hardware manufacturer for the operating system. Without a driver, the hardware in the computer will not function.

[0094] 9. Render Pass

[0095] Rendering channels usually refer to multi-pass rendering technology. In multi-pass rendering technology, an object needs to be rendered multiple times, and the results of each rendering process are accumulated into the final rendering result. These rendering processes are generally: lighting, shadows, reflections, highlights, and global illumination. Multiple passes are used to achieve effects that cannot be achieved by a single pass. Passes are interdependent. Later passes will use the data of previous passes (depth, geometry information), and the data output by the final pass is the data in the frame buffer. Therefore, the relationship between passes can be figuratively likened to a series of processes.

[0096] 10. Rendering Pipeline

[0097] The rendering pipeline, also known as the rendering pipeline, is a collection of independent, parallel processing units within the display chip that handles graphics signals. The rendering pipeline transforms a series of vertex data, textures, and other information into a visible image. This process is performed jointly by the CPU and GPU.

[0098] 11. Command buffer

[0099] A command buffer is a collection of commands. It is submitted to the appropriate hardware queue for GPU processing. The driver then extracts the command buffer, validates it, and compiles it before actual GPU processing begins. A command buffer records the various Vulkan API commands that the application intends to execute. Once a command buffer has been baked, it can be reused over and over again. Commands are recorded in the order specified by the application. These commands are used to perform various tasks, including binding vertex buffers, binding pipelines, recording render pass commands, setting viewports and clipping, specifying drawing commands, and controlling the copying of images and buffer contents.

[0100] 12. Uniform variables

[0101] A uniform is a data type used in the rendering process of a graphics rendering system. A uniform is a modifier of a variable type and is a constant value used by shaders in OpenGL ES. It is used to store data required by various shaders, such as transformation matrices, lighting parameters, or colors.

[0102] 13. Draw call

[0103] A draw call is a command from the CPU to the GPU, simply pointing to a list of primitives to be rendered. A "draw instruction" can be a single instruction, such as an API call, or a collection of instructions. For example, in OpenGL, a draw call typically includes multiple API calls. This single draw call can be considered a single drawing instruction, completing a single drawing. A single draw call can draw a single target, multiple draw calls can draw a single target, or a single draw call can draw multiple targets.

[0104] 14. Asynchronous Execution

[0105] Asynchronous execution means that statements don't necessarily finish executing in the same order they started. For example, in a query operation, after a client application issues a query instruction to the server, the next statement in the query instruction will be executed immediately, without waiting for the server to return the query results to the client. In other words, when executing a task asynchronously, it is possible to move on to another task before the current one completes. Asynchronous execution frees applications from the constraints of a single task, improving flexibility and efficiency.

[0106] 15. Synchronous Execution

[0107] Synchronous execution is any process consisting of multiple tasks, in which the tasks need to be executed in sequence and the next task can only be executed after the previous task returns a result.

[0108] At present, the development status of computer image technology is as follows:

[0109] In terminal and desktop systems, the API layer (or driver) of the graphics system is constantly changing. Although OpenGL ES and Vulkan currently provide support for the application layer at the same time, the future trend is that Vulkan may gradually replace OpenGL ES. In order to maintain forward compatibility, OpenGL ES will still exist.

[0110] Due to upgrades to GPU hardware and operating systems, the graphics system's API layer will also be continuously updated synchronously, which may bring the following problems: 1. Maintaining OpenGL ES and Vulkan at the same time will cause a bloated software system; 2. The overlap of the two API modules will lead to engineering complexity; 3. Maintaining two API systems at the same time will also lead to a large waste of engineering resources.

[0111] In order to provide the OpenGL ES API functionality in a system that does not use the OpenGL ES API driver, the commonly used method is to directly use Vulkan as the underlying implementation of OpenGL ES. Figure 1 , Figure 1 This is a schematic diagram of an architecture for implementing OpenGL ES provided in an embodiment of the present application. Figure 1 It can be seen that architecture 100 is configured with one or more backends, such as OpenGL 105A, Vulkan 105B, Direct3D 105C, etc. Each backend has its corresponding driver, for example, OpenGL 105A corresponds to OpenGL Driver 106A, Vulkan 105B corresponds to Vulkan Driver 106B, and Direct3D 105C corresponds to Direct3D Driver 106C. Through architecture 100, different backends can be used to implement OpenGL ES based on different platforms and configurations. For example, if the application is installed on an electronic device deployed with the Android system, Vulkan Driver 106B can be used to implement OpenGL ES graphics rendering.

[0112] pass Figure 1In the illustrated architecture 100, an electronic device can obtain OpenGL ES instructions from an application program calling an API from the EGL / OpenGL entry layer, and obtain state data corresponding to the OpenGL ES instructions through a processing module 103. Since the architecture 100 supports one or more backends, the state data corresponding to the OpenGL ES instructions can be first stored in the front end 104. Based on the adopted back end (e.g., Vulkan 105B), the state data stored in the front end 104 is converted into state data corresponding to the adopted back end, and then the corresponding back end instructions are generated from the state data corresponding to the back end and submitted to the corresponding driver (e.g., Vulkan Driver 106B). The corresponding driver then submits the instructions to the GPU to complete the call, thereby achieving OpenGL ES graphics rendering.

[0113] The inventors of this application have discovered that, from acquiring the call data of the OpenGL ES API, to the front-end saving the call data, and then converting the call data saved by the front-end into the corresponding data of the back-end, the OpenGL ES API is converted and run in a single thread, which cannot give full play to the advantages of Vulkan multi-threading. As a result, compared with the native OpenGL ES driver implementation, performing the conversion operation in a single thread may increase the additional CPU running time, resulting in reduced rendering efficiency. In addition, although the architectural design meets the versatility (supporting multiple back-ends), it lacks a specific architectural design and suitable optimization methods for a certain back-end (such as Vulkan), resulting in the failure to give full play to the advantages of Vulkan and limitations in resource utilization.

[0114] To solve the above problems, this application proposes a method and related device for implementing graphics rendering based on Vulkan, which includes:

[0115] An electronic device receives OpenGL ES instructions invoked by an application, classifies multiple OpenGL ES instructions into asynchronously executed OpenGL ES instructions and synchronously executed OpenGL ES instructions, processes the asynchronously executed OpenGL ES instructions into a first event through a first thread, saves description information of the synchronously executed OpenGL ES instructions into a data structure, or obtains description information of the synchronously executed OpenGL ES instructions from the data structure through the first thread; sends the first event to a second thread through a lock-free queue, and processes the first event through the second thread to obtain a second task, wherein the second task contains one or more first events; then, caches the second task in a command buffer through a third thread, and sends the second task in the command buffer to the GPU through a Vulkan interface. Finally, the GPU completes graphics rendering based on the second task. Based on the proposed multi-threaded architecture, various optimization techniques (such as dynamic control of GPU instruction submission and GPU instruction reordering) can be adopted to fully utilize CPU and GPU resources and improve operational efficiency.

[0116] Through the embodiment of the present application, Vulkan is used as the backend to implement OpenGL ES graphics rendering. Because Vulkan supports multi-threaded parallel processing, the multi-threaded advantage of Vulkan is utilized in the embodiment of the present application to process different events or tasks in different threads (such as the first thread, the second thread, and the third thread). At the same time, by using Vulkan as the backend to implement OpenGL ES graphics rendering, the OpenGL ES API can be used on more devices and systems, increasing the versatility of the OpenGL ES API, and allowing devices that do not support the OpenGL ES API to run OpenGL ES applications. For device manufacturers and systems, the maintenance cost of the OpenGL ES driver can be reduced, and there is no need to maintain multiple graphics API drivers on devices that support multiple graphics APIs, streamlining the software system.

[0117] See Figure 2 , Figure 2 This is a schematic diagram of an architecture for implementing graphics rendering based on Vulkan provided by an embodiment of the present application. Figure 2It can be seen that the architecture 200 may include an application 201, an EGL / OpenGL entry 202, a context module 203, an event management module 204, an event packaging module 205, a task scheduling module 206, and a Vulkan Driver 207. When the application 201 is running, the application 201 may call the OpenGL ES API and obtain the called OpenGL ES instructions and related parameters through the EGL / OpenGL entry 202. Then, the called OpenGL ES instructions and related parameters are processed by the context module 203, which converts them into corresponding state data and GPU instructions and then sends them to the event management module 204, which packages them in batches. After that, the event packaging module 205 converts the packaged instructions and state information and other data into a Vulkan command buffer. Finally, the task scheduling module 206 sends the data in the command buffer to the GPU, and the GPU completes the graphics rendering. The functions of each module are introduced as follows:

[0118] Applications 201 include, but are not limited to, game applications, video applications, chat and communication applications, and camera applications. It should be noted that the aforementioned applications are applications that use the OpenGL ES API. The aforementioned applications can be installed on electronic devices, which may include, but are not limited to, laptop computers, desktop computers, mobile phones, mobile terminals (e.g., smartphones), wearable devices, in-vehicle devices, IoT devices, or other devices capable of performing graphics rendering processing. The electronic devices may be devices running Android, iOS, Windows, or other operating systems.

[0119] The EGL / OpenGL ES entry 202 is used to obtain OpenGL ES instructions of the application 201 when the application 201 makes an API call.

[0120] The context module 203 runs in the first thread. Because multiple OpenGL ES instructions can be classified into asynchronous OpenGL ES instructions and synchronous OpenGL ES instructions, for synchronous OpenGL ES instructions, in one possible implementation, the context module 203 is configured to save the description information of the synchronous OpenGL ES instruction to a data structure based on the type of the synchronous OpenGL ES instruction (e.g., if the OpenGL ES instruction is a state setting type), or to obtain the description information of the synchronous OpenGL ES instruction from the data structure based on the type of the synchronous OpenGL ES instruction (e.g., if the OpenGL ES instruction is a state obtaining type). The context module 203 can also send the relevant description information of the synchronous OpenGL ES instruction to the GPU via the Vulkan Driver 207, so that the GPU can perform image rendering.

[0121] For asynchronous OpenGL ES commands, the context module 203 can process the asynchronous OpenGL ES commands into first events through a first thread based on the type of the asynchronous OpenGL ES commands (for example, if the OpenGL ES commands are of the drawing type), and then send the first events to the event manager module 204 through a lock-free queue. The first events include description information of the asynchronous OpenGL ES commands.

[0122] In one possible implementation, the context module 203 can obtain the description information of the asynchronous OpenGL ES instruction in the corresponding data module through the first thread, and then encapsulate the description information of the asynchronous OpenGL ES instruction and the asynchronous OpenGL ES instruction into a first event. It should be noted that the description information includes the state value of the OpenGL ES instruction. OpenGL ES is a state set, so various states are involved in drawing. These states may all have default values. These states can be changed through OpenGL ES, and these state values ​​will be saved in the context. For example, the blending mode, texture image, program and other information used by OpenGL ES.

[0123] Event manager module 204, running on the second thread, dynamically controls the frequency of submitting first events to event packaging module 205 based on one or more of the first event type, the number of first event caches, or GPU load. If a first event needs to be sent to event packaging module 205, the second thread processes the first event to obtain a second task (job), which is then sent to packaging module 205. The second task contains one or more first events.

[0124] In a possible implementation, when the first event currently processed belongs to the preset first type, the event manager module 204 executes the step of obtaining the second task job through the second thread processing.

[0125] In one possible implementation, when the first event currently processed does not belong to the preset first type, if the number of first events currently cached is greater than a first threshold, the event manager module 204 executes a step of obtaining a second task through a second thread.

[0126] In one possible implementation, if the currently processed first event does not belong to the preset first type, and the number of currently cached first events is greater than the second threshold and less than or equal to the first threshold, and if the current GPU load is less than the first load threshold, indicating that the GPU is now capable of processing, the event manager module 204 executes the step of processing the first event via a second thread to obtain a second task. The second threshold is less than the first threshold; the first load threshold can be set empirically or determined comprehensively based on the amount of data processed during graphics rendering. It is readily understood that the "equal to" condition can also be placed in another branch of the judgment, such as a condition greater than or equal to the second threshold and less than the first threshold. It should be noted that the current GPU load being less than the first load can also be understood to mean that the GPU's operating frequency is lower than its normal operating frequency, or that the electronic device running the GPU does not experience severe overheating or rapid power consumption. This load primarily refers to the GPU's rendering load. In one possible implementation, if the currently processed first event does not belong to the preset first type and the number of currently cached first events is greater than a second threshold and less than or equal to the first threshold, and if the current GPU load is greater than the first load threshold, the event manager module 204 caches the first event. The second threshold is less than the first threshold. It is readily understood that the "equal" condition can also be placed in another branch of the judgment, such as a condition greater than or equal to the second threshold and less than the first threshold. It should be noted that a GPU load greater than or equal to the first load can also be understood as meaning that the GPU's operating frequency is higher than its normal operating frequency, or that the electronic device running the GPU is experiencing severe overheating and rapid power consumption. This load primarily refers to the GPU's rendering load. The event wrapper module 205, running in a third thread, is responsible for sorting the first events contained in the second task according to a specific optimization strategy, then caching the sorted second task in the command buffer, and then sending the second task in the command buffer to the task scheduling module 206.

[0127] In one possible implementation, the event wrapper module 205 uses a third thread to sort the first events in the second task based on the pipeline states corresponding to the first events in the second task. First events in the second task that correspond to the same pipeline state are sorted together.

[0128] In one possible implementation, the event wrapper module event wrapper 205 uses a third thread to sort multiple first events in the second task corresponding to the first event of modifying the same variable; then modifies the sorted first events in the second task based on the variable; and finally caches the sorted and modified second task into the command buffer area.

[0129] The task scheduling unit job dispatcher 207 runs on the second thread and is used to send the second task in the command buffer to the GPU through the Vulkan interface, wherein the GPU is used to complete graphics rendering according to the second task.

[0130] It should be noted that the first thread, second thread, and third thread mentioned in the embodiment of the present application are three independent threads, and different processing processes are performed in different threads. For example, in the first thread, the asynchronous OpenGL ES instruction is processed into a first event event, in the second thread, the first event event is processed into a second task job, and in the third thread, the second task job is cached in the command buffer. Moreover, when the third thread is processing the second task, the second thread is also processing other first events received, and the first thread is also processing other asynchronously executed OpenGL ES instructions received. It can be seen that each thread is processed in parallel, thereby improving processing efficiency, and therefore the frequency of submission to the GPU can also be increased, thereby achieving the purpose of improving rendering efficiency.

[0131] See Figure 3 , Figure 3 This is a flow chart of a method for implementing graphics rendering based on Vulkan provided in an embodiment of the present application. Optionally, the method can be based on Figure 2 The architecture shown is implemented. Figure 3 The Vulkan-based graphics rendering method shown includes at least the following steps:

[0132] Step S301: the electronic device classifies a plurality of OpenGL ES instructions into asynchronous OpenGL ES instructions and synchronous OpenGL ES instructions.

[0133] Specifically, the electronic device can be a terminal device that supports platforms such as Windows, Android, Unix-like, Linux, MacOS, etc., and can be but not limited to laptop computers, desktop computers, mobile phones, smart phones, tablet computers, multimedia players, e-readers, smart car devices, smart home appliances, artificial intelligence devices, wearable devices, Internet of Things devices, or virtual reality / augmented reality / mixed reality devices, etc.

[0134] For the OpenGL ES API, there are generally four types: state setting and obtaining, object generation and destruction, data upload and download, and drawing. Please refer to Table 1, which is a description of the OpenGL ES API provided in the embodiment of the present application. As can be seen from Table 1, the embodiment of the present application can classify OpenGL ES instructions into asynchronous OpenGL ES instructions and synchronous OpenGL ES instructions according to the type of OpenGL ES API. For example, the OpenGL ES instructions corresponding to the state setting and obtaining type API and the object generation and destruction type API are classified as synchronous OpenGL ES instructions; the OpenGL ES instructions corresponding to the data upload type API and the drawing type API are classified as asynchronous OpenGL ES instructions.

[0135] Table 1 Description of the OpenGL ES API

[0136]

[0137] Therefore, after the electronic device obtains the OpenGL ES instructions of the application through the EGL / OpenGL ES entry layer, it can process the synchronous OpenGL ES instructions through step S302A and the asynchronous OpenGL ES instructions through step S302B according to the classification set for the OpenGL ES instructions in advance.

[0138] Step S302A: The electronic device processes the synchronized OpenGL ES instructions through the first thread.

[0139] Specifically, when the electronic device processes synchronized OpenGL ES instructions (e.g., state setting and obtaining, object creation and destruction) through the first thread in the context module, it saves the description information of the synchronized OpenGL ES instructions to the data structure corresponding to the context module, or obtains the description information of the synchronized OpenGL ES instructions from the data structure corresponding to the context module. After the electronic device calls the OpenGL ES API to complete the corresponding work, it needs to return a result indicating that the call is complete before it can continue to call the next API. The returned result can be an empty result, that is, a result that does not contain any information.

[0140] For example, if the electronic device is in the context module, the first OpenGL ES instruction received is of the state setting type, and the content carried by the OpenGL ES instruction is to write A=1 and B=2. Therefore, after A=1 and B=2 are saved in the data structure corresponding to the context module, a result indicating that the saving is completed is returned; if the electronic device is in the context module, the second OpenGL ES instruction received is of the state acquisition type, and the content carried by the OpenGL ES instruction is to read A and B, then after obtaining the state values ​​of A and B from the data structure corresponding to the context module, it sends them to the image processor GPU, where the image processor can be used to complete graphics rendering according to the description information of the synchronously executed OpenGL ES instructions; if the electronic device is in the context module, the third OpenGL ES instruction received is of the object destruction type, and the content carried by the OpenGL ES instruction is to delete A and B, then after deleting the state values ​​of A and B from the data structure corresponding to the context module, a result indicating that the deletion is completed is returned.

[0141] Step S302B: Process the asynchronous OpenGL ES instruction into a first event through the first thread.

[0142] Specifically, after the electronic device calls the OpenGL ES API, when the Context module processes the asynchronous OpenGL ES instruction (for example, data upload type, drawing type) through the first thread, a first event event can be generated based on the state value and OpenGL ES instruction saved in the data structure corresponding to the Context module. It can be understood that the first event event includes the state value of the asynchronous OpenGL ES instruction at this time and the content information of the OpenGL ES instruction. For example, if the OpenGL ES instruction received by the electronic device in the context module is of the drawing type, and the content carried by the OpenGL ES instruction is A+B=C, the state values ​​of A and B are obtained from the data structure corresponding to the context module, and then the first event is generated based on the state values ​​of A and B, and A+B=C, and other descriptive information.

[0143] Then, after the electronic device sends the first event to the Event Manager module, the API call can return. Since the Context module runs on the first thread and the Event Manager module runs on the second thread, data can be sent or received between the two modules running on different threads through a lock-free first-in-first-out queue.

[0144] Different first events have different properties (such as whether they need to wait for the execution result). For example, for the eglSwapBuffers interface, it is necessary to wait for the GPU to complete rendering of the frame and return the result. Therefore, the Context module in the first thread needs to block and wait for the Event Manager module and Job Dispatcher module in the second thread to complete sending instructions to the GPU, and then wait for the GPU to complete rendering of the frame, and finally present the content of the frame on the screen.

[0145] For another example, for the first event processed by the OpenGL ES API in the Context module, such as data upload or drawing instructions, the electronic device can directly send the first event to the Event Manager module through the Context module, without waiting for the return result, and can continue to receive and execute the next OpenGL ES API called by the application. It can be seen that the operation of sending instructions to the GPU is not in the first thread, so this part of the CPU load is shared by other threads. Therefore, the application logic in the first thread can obtain more sufficient computing resources, improving the performance of the entire application.

[0146] It should be noted that the first thread may be a rendering thread corresponding to the application.

[0147] Step S303: The electronic device obtains a second task through the second thread processing.

[0148] Specifically, when the electronic device receives a first event from the Context module in the Event Manager module, the Event Manager module can determine, based on the type of the first event, whether the first event needs to be processed to obtain a second task (job). If so, the first event is packaged into the second task; otherwise, the first event is cached.

[0149] In one possible implementation, if the first event currently being processed belongs to a preset first type, which may be an OpenGL ES API instruction such as glFinish / glFlush / glReadPixels, or an OpenGL ES API instruction of the bulk data upload type with the uploaded data greater than a preset threshold, such as glBufferData or glSubBufferData, then in the event manager module, the first event is immediately processed by a second thread to obtain a second task. Immediate processing means that the first event requiring immediate processing is prioritized. That is, upon receiving the first event requiring immediate processing, the first event is processed first, if possible. For example, if the first event received by the electronic device in the event manager module contains asynchronous OpenGL ES instructions such as one or more of glFinish / glFlush / glReadPixels, the first event currently being processed is processed as a second task. The second task may include the first event currently being processed, or the first event currently being processed and the first event cached.

[0150] In one possible implementation, if the first event currently being processed does not belong to the preset first type, it means that the first event currently being processed does not need to be processed immediately to obtain the second task. Therefore, the first event can be cached in the Event manager module, and when processing is required, the first event cached in the Event manager module can be packaged into the second task through the first thread. For example, if the asynchronous OpenGL ES instruction contained in the first event received by the electronic device in the Event manager module does not belong to any of glFinish / glFlush / glReadPixels, or does not belong to glBufferData or glSubBufferData in the large block data upload type, the first event can be cached first.

[0151] In one possible implementation, when the first event currently processed does not belong to the preset first type, if the number of first events currently cached is greater than the first threshold, the cached first event can be packaged into a second task through the first thread. It should be noted that the first threshold can be set according to actual conditions, and the first threshold can be the maximum number of packaged first events, which is not limited in the embodiments of the present application. For example, when the asynchronous OpenGL ES instruction contained in the first event received by the electronic device in the Event manager module does not belong to any of glFinish / glFlush / glReadPixels, or does not belong to glBufferData or glSubBufferData in the bulk data upload type, if the number of first events cached in the Event manager module is greater than the first threshold, the first event currently being processed can be packaged into a second task, or the first event currently being processed and the cached second time can be packaged together into a second task.

[0152] In one possible implementation, if a first event currently being processed does not belong to a preset first type, and the number of first events currently cached is greater than a second threshold but less than the first threshold, and if the current GPU load is too light, that is, less than the first load threshold, the first event currently being processed may be packaged into a second task via the first thread. The second threshold is less than the first threshold. The first and second thresholds may be set based on actual circumstances and are not limited in the embodiments of the present application. For example, if an asynchronous OpenGL ES instruction included in a first event received by an electronic device in an event manager module does not belong to any of glFinish / glFlush / glReadPixels, or does not belong to glBufferData or glSubBufferData in the bulk data upload type, and if the number of first events cached in the event manager module is within the allowed cacheable amount, but the GPU load is relatively light at this time, the first event currently being processed may be packaged into a second task, or the first event currently being processed and the cached second event may be packaged together into a second task.

[0153] In one possible implementation, if the number of currently accumulated cached first events is greater than a second threshold and less than a first threshold, and if the current GPU load is excessive, that is, greater than or equal to the first load threshold, the first events may be cached in the Event Manager module. When processing is required, the first events cached in the Event Manager module are packaged into a second task via the first thread. For example, if the asynchronous OpenGL ES instruction contained in the first event received by the electronic device in the Event Manager module does not belong to any of glFinish / glFlush / glReadPixels, or does not belong to glBufferData or glSubBufferData in the bulk data upload type, and if the number of first events cached in the Event Manager module is within the allowed cache limit, but the GPU load is excessive and unsuitable for submission to the GPU, the currently processed first event may be cached and submitted later. It should be noted that the first load threshold can be set based on experience, or can be determined based on the time and amount of data processed during graphics rendering, or can be determined based on the amount of data processed during graphics rendering. Different GPUs may correspond to different first load thresholds, and the setting can be based on actual conditions, and the present embodiment does not impose any limitation.

[0154] It should be noted that if there is an API that can detect the GPU load, the API can be called to obtain the GPU load. If there is no API that can detect the GPU load, the GPU load can be calculated using an estimation method. For example, the GPU load can be estimated based on the time of the last submission (which can be the time of the second task last submitted to the GPU), the number of last submissions (which can be the number of triangle drawing instructions in the first event contained in the second task last submitted to the GPU), the number of triangle drawing instructions in the first event in the cache, the time of the previous frame (which can be the time of the last frame of triangle graphics that was drawn), and the number of the previous frame (which can be the number of triangles in the last frame of graphics that was drawn). Assuming that the time it takes for the GPU to process the first event is proportional to the number of triangle drawing instructions it contains, the current GPU load can be simply expressed as:

[0155] GPU load = time of last submission - number of last submissions / (number of last frames * time of last frame)

[0156] After the second task is obtained through the second thread processing, the second task is sent to the Event wrapper module.

[0157] The frequency of submission to the GPU is determined by when the electronic device packages and processes the first event in the Event Manager module. Therefore, submission to the GPU needs to be as frequent as possible to ensure that the GPU has no idle time. However, a higher submission frequency also increases the I / O burden. By dynamically controlling the submission frequency to the GPU, the GPU can be loaded as high and stable as possible to fully utilize the GPU's hardware resources. Therefore, in the Event Manager module, the electronic device packages a certain number of first events into second tasks and sends them to the Event Wrapper module.

[0158] Step S304: the electronic device caches the second task into the command buffer area through the third thread.

[0159] Specifically, the electronic device can receive the second task sent from the Event manager module in the Event wrapper module, save the second task to the command buffer through the third thread, and then send the second task in the command buffer to the Job dispatcher module.

[0160] In one possible implementation, in the process of saving the first event in the second task to the command buffer, the first event in the second task can be sorted according to the optimization rules, and the sorted first event is still saved in the Command buffer. Therefore, the first event in the second task sent to the Job dispatcher module is optimized and sorted. For example, if the order of the first events in the second task received by the Job dispatcher module of the electronic device is first event 1, first event 2, first event 3, first event 4, the first events are sorted according to the optimization rules, and the order of the first events in the sorted second task can be first event 2, first event 4, first event 1, first event 3, and then the sorted first event in the second task is saved in the Command buffer.

[0161] Step S305: The electronic device sends the second task in the command buffer to the GPU through the Vulkan interface.

[0162] Specifically, in the Job dispatcher module, the electronic device can receive the second task in the command buffer sent from the Event wrapper module, and then the electronic device can send the second task in the command buffer to the GPU through the corresponding Vulkan API, and the GPU will complete the graphics rendering.

[0163] based on Figure 2The architecture 200 shown can be implemented on the Android system. Figure 4 , Figure 4 This is a schematic diagram of a component structure for implementing graphics rendering based on Vulkan provided by an embodiment of the present application. Figure 4 It can be seen that on an electronic device using the Android system, in the software framework of an OpenGL ES application running with Vulkan as the back-end implementation of the OpenGL ES API, the modules involved in the embodiments of the present application include one or more of the following: an application APP401 using OpenGL ES, an EGL / OpenGL ES entry layer 402, an OpenGL ES implementation layer 403, a Vulkan driver 404, and a GPU 405.

[0164] The OpenGL ES implementation layer 403 includes one or more of the following: a context module 403A, an event management module 403B, an event packaging module 403C, and a task scheduling module 403D. Furthermore, the OpenGL ES implementation layer 403 includes three threads: a first thread, a second thread, and a third thread. The first thread is the OpenGL ES rendering thread. The context module 403A runs on the first thread, the same thread as the OpenGL ES application's OpenGL ES API calls. The event management module 403B runs on the second thread, the event packaging module 403C runs on the third thread, and the task scheduling module 403D runs on the second thread.

[0165] pass Figure 4 The component structure shown implements the OpenGL ES API based on a multi-threaded architecture with Vulkan as the backend in the following specific implementations:

[0166] After OpenGL ES API is invoked by OpenGL ES application (APP 401), in OpenGL ES implementation 403, different flow processes can be performed according to different OpenGL ES APIs. OpenGL ES APIs can be divided into at least four categories: state setting and obtaining (such as glViewport, glGetInteger, etc.), object generation and destruction (such as glGenBuffer, glDeletBuffers, etc.), data uploading and downloading (such as glBufferData, glReadPixels, etc.), and drawing (such as glDrawElements). The four categories of APIs can be divided into two categories for execution. The execution mode of the state category and the object category is synchronous execution, that is, still executed in the thread of the application program. Data and drawing are asynchronously executed, that is, will not be executed in the thread of the application program, but will be executed in another independent thread. For example, see Figure 5A , Figure 5A is a flow diagram provided by an embodiment of the present application for classifying OpenGL ES instructions in an OpenGL ES implementation layer. From Figure 5A It can be seen that after receiving an OpenGL ES instruction obtained by invoking an OpenGL ES API by an APP, it is determined whether the OpenGL ES instruction is a synchronous OpenGL ES instruction. If yes, the processing flow of the synchronous OpenGL ES instruction is performed. If not, the processing flow of the asynchronous OpenGL ES instruction is performed.

[0167] See Figure 5B , Figure 5B is a flow diagram provided by an embodiment of the present application for processing a synchronous OpenGL ES instruction. Figure 5BOpenGL ES instructions that do not involve synchronization of multithreading processing are processed in the context module 403A. For state reading and writing type or resource management type API (such as object generation and destruction, and data uploading and downloading), the state management module or the object management module in the context module 403A can be passed through, and a return is made after the corresponding work is completed. The so-called synchronous execution is because the calculation amount of the two types of OpenGL ES API calls is small, and synchronous execution is beneficial to the performance of the overall application. In terms of module functionality, object management is to establish a 1:n mapping relationship between OpenGL ES and Vulkan (one OpenGL ES object can need the support of several Vulkan objects). In addition, the object generation and destruction type of OpenGL ES API also acts on the object management module. The state management can save the current entire system state, and manage the state necessary for the OpenGL ES specification through the creation of a context. From the perspective of the OpenGL ES specification, the state management can be divided into the following two types: state setting and state obtaining. The state setting type of API is to set the value of the state in the OpenGL ES specification, and the state obtaining type of API is to obtain the value of the state in the OpenGL ES specification. Figure 5B It can be seen that, if it is a state setting and obtaining type of API, because there is a data structure corresponding to the state in the context module 403A, since the context module 403A runs in the rendering thread of the OpenGL ES of the APP 401, the state value of the synchronous OpenGL ES instruction can be directly saved in the data structure of the context module 403A; or the current OpenGL ES instruction state value can be obtained from the data structure, and the corresponding state value currently saved in the context module 403A is written into the address of the API call.

[0168] Referring to Figure 5C , Figure 5C is a processing flow schematic diagram of an asynchronous execution of an OpenGL ES instruction provided by the embodiment of the present application, and is a process of processing data uploading and drawing type of OpenGL ES API in the context module 403A. The two types of API of the OpenGL ES instruction are packaged and processed into a first event object of the system. In order to ensure that each first event of the packaged processing is independent of each other, the object management and the state management modules are accessed to obtain the state value of the asynchronous OpenGL ES instruction when each first event is generated, and the state value is saved in the first event. After the first event is sent to the event management module 403B, the API call is returned. Because the first event is not processed at this time, the processing of the two types of API is asynchronous.

[0169] Referring to Figure 5D , Figure 5Dis a flowchart of a process of obtaining a second task by the embodiment of the present application, i.e. the process of processing the first event from the context module 403A by the event management module 403B. The event management module 403B runs in a second thread, which is another independent thread relative to the context module 403A. Therefore, the event management module 403B constantly detects the first event sent by the context module 403A running in a first thread. From the detection, the event management module 403B can obtain the second task. Figure 5D It can be seen that if the first event currently obtained for processing belongs to a preset first type, the preset first type can be an instruction of an OpenGL ES API, such as glFinish / glFlush / glReadPixels, or the OpenGL ES API is a bulk data uploading type and the uploaded data is greater than a preset threshold, such as glBufferData, glSubBufferData, etc. In the event manager module, the first event is immediately processed to obtain the second task by the second thread. The so-called immediate processing can be considered that the processing order of the first event that needs to be immediately processed is arranged in the first place, i.e. after receiving the first event that needs to be immediately processed, the above-mentioned first event is processed first in possible cases.

[0170] In the case that the first event currently obtained for processing does not belong to the preset first type, i.e. the case that the first event does not need to be immediately packaged into the second task, if the number of the first events currently accumulated and cached is greater than a first threshold N, the cached first events can be packaged into the second task by the first thread. It should be noted that the first threshold N can be set according to actual conditions, and the first threshold N can be a maximum value of the packaging number of the first events, which is not limited by the embodiment of the present application.

[0171] In the case that the first event currently obtained for processing does not belong to the preset first type, and in the case that the number of the first events currently accumulated and cached is greater than a second threshold M and less than the first threshold N, if the current load of the GPU is too small and the GPU can continue to process, the first event currently obtained for processing can be packaged into the second task by the first thread. The second threshold M is less than the first threshold N, and the first threshold N can be a maximum value of the packaging number of the first events, and the second threshold M can be a minimum value of the packaging number of the first events. The first threshold and the second threshold can be set according to actual conditions, which is not limited by the embodiment of the present application.

[0172] In the case that the number of the first events in the current cumulative cache is greater than the second threshold M and less than the first threshold, if the current GPU load is too large and the GPU cannot continue to process, the first event can be cached in the Event manager module, and the first event cached in the Event manager module can be packed into the second task by the first thread when it needs to be processed.

[0173] The event management module 403B dynamically controls the frequency of submission to the GPU by dynamically controlling the packing number and timing of the first event, which can ensure the smooth running state of the GPU processing high load and can fully utilize the hardware resources of the GPU.

[0174] Please refer to Figure 5E , Figure 5E is a flowchart of processing a second task provided by an embodiment of the present application, that is, the process of processing the second task from the event management module by the event packaging module. In the third thread running in the event packaging module 403C, processing the second task mainly includes: extracting and processing all the first events in the second task, analyzing the information in the first event, judging whether the order of the first event can be rearranged according to the rearrangement optimization strategy, and then saving the OpenGL ES instructions contained in the first event to the command buffer using the Vulkan API. Command buffer and then sending the command buffer to the task scheduling module 403D.

[0175] Among them, the rearrangement optimization strategy includes one or more of the following: adjusting the order based on the pipeline state corresponding to the drawing instruction (strategy one) and adjusting the order of the uniform upload instruction and the drawing instruction based on the scene of repeated modification and use of uniform (strategy two).

[0176] The arrangement rule of strategy one is: if there is a drawing instruction Draw call in the OpenGL ES instruction in the first event contained in the second task, because each drawing instruction corresponds to a pipeline state, multiple drawing instructions corresponding to the same pipeline state can be combined together. For example, please refer to Figure 5F , Figure 5F is a schematic diagram of rearranging according to strategy one provided by an embodiment of the present application, from Figure 5FIt can be seen that before reordering, the order of the drawing instructions in the OpenGL ES instructions in the first event contained in the second task is drawing instruction 1, drawing instruction 2, drawing instruction 3, and drawing instruction 4, and drawing instruction 1 corresponds to pipeline state 1, drawing instruction 2 corresponds to pipeline state 2, drawing instruction 3 corresponds to pipeline state 1, and drawing instruction 4 corresponds to pipeline state 2. After reordering, the order of the drawing instructions in the OpenGL ES instructions in the first event contained in the second task is drawing instruction 1, drawing instruction 3, drawing instruction 2, and drawing instruction 4, and drawing instruction 1 and drawing instruction 3 both correspond to pipeline state 1, and drawing instruction 2 and drawing instruction 4 both correspond to pipeline state 2. Before reordering, the four drawing instructions required four pipeline state switches, but after reordering according to strategy one, only two pipeline state switches are required.

[0177] The arrangement rule of strategy 1 is: if the OpenGL ES instructions in the first event included in the second task need to modify the same uniform multiple times, you can use a uniform with larger memory capacity, save the modified data of each time at a different memory offset of the uniform with larger memory capacity, and then upload multiple uniform modified values ​​at once before using the drawing instructions to draw. For example, see Figure 5G , Figure 5G This is a schematic diagram of rearrangement according to Strategy 2 provided in an embodiment of the present application, from Figure 5G As can be seen, before reordering, uniform1 in the OpenGL ES instructions in the first event contained in the second task was modified, and then the modified data was used for drawing using the drawing instructions. Uniform1 was then modified again, and then the modified data was used again for drawing. It can be seen that modifying the same uniform and then using the modified data for drawing immediately after modification, and then modifying it multiple times, can lead to excessive uploads and waste of resources. After reordering, uploading multiple uniform modified values ​​at once and then using the drawing instructions for drawing, reduces the number of uniform data uploads to just one.

[0178] See Figure 5H , Figure 5His a flowchart of a process of sending a second task in a command buffer to a GPU, i.e., a process of sending a second task in a command buffer to a GPU by a task scheduling module. The event wrapper module 403C running in the third thread continuously sends the second task saved in the command buffer to the lock-free queue shared with the task scheduling module 403D. The task scheduling module 403D running in the second thread takes the command buffer from the shared lock-free queue, submits the command buffer to the GPU through the Vulkan interface, and finally completes the graphic rendering by the GPU according to the second task.

[0179] Please refer to Figure 6 , Figure 6 is a structural diagram of an electronic device. The electronic device 60 includes at least one memory 601, at least one processor 602, and a communication interface 604. Optionally, it can also include a bus 603, wherein the memory 601, the processor 602, and the communication interface 604 are connected through the bus 603.

[0180] The memory 601 is used to provide a storage space, and the storage space can store data such as an operating system and computer instructions. The memory 601 can be one or a combination of RAM, ROM, EPROM, or CD-ROM.

[0181] The processor 602 is a module for arithmetic operation and / or logical operation, and can be one or a combination of CPU, GPU, MPU, ASIC, FPGA, CPLD, etc.

[0182] The communication interface 604 is used to receive externally transmitted data and / or transmit data to the outside, and can be a wired link interface including an Ethernet cable, etc., or a wireless link (Wi-Fi, Bluetooth, general wireless transmission, vehicle-mounted short-range communication technology, etc.) interface. Optionally, the communication interface 604 can also include a transmitter (such as a radio frequency transmitter, an antenna, etc.) coupled with the interface, or a receiver, etc.

[0183] The processor 602 in the electronic device 60 is used to read the computer instructions stored in the memory 601, and is used to execute the foregoing method for implementing graphic rendering based on Vulkan, for example Figure 3 The method for implementing graphic rendering based on Vulkan described in the embodiments. The processor 602 in the electronic device 60 is used to read the computer instructions stored in the memory 601, and is used to perform the following operations:

[0184] The asynchronously executed OpenGL ES instruction is processed into a first event by a first thread, the first event can include description information of the asynchronously executed OpenGL ES instruction; then a second task can be processed by a second thread, the second task includes a first event or a plurality of first events (event), the plurality of first events are obtained according to the same or different asynchronously executed OpenGL ES instructions; then the second task can be saved to a command buffer by a third thread; finally, the second task in the command buffer can be sent to a GPU image processor through a Vulkan interface, wherein the GPU is used to complete graphic rendering according to the second task.

[0185] In the embodiments of the present application, Vulkan is used as a backend to realize OpenGL ES graphic rendering. Because Vulkan supports multi-thread parallel processing, the multi-thread advantage of Vulkan is utilized in the embodiments of the present application, and different events or tasks are processed in different threads (for example, a first thread, a second thread and a third thread). For example, the asynchronously executed OpenGL ES instruction is processed into a first event by a first thread, the first event is received by a second thread and processed into a second task, the second task is saved to a command buffer in a third thread, and the command buffer is sent to a GPU through a Vulkan interface, and the GPU is used to complete graphic rendering. When the second task is processed in the third thread, the second thread also processes other received first events, and the first thread also processes other received asynchronously executed OpenGL ES instructions. It can be seen that the threads are processed in parallel, so that the processing efficiency can be improved, and therefore the frequency of submitting to the GPU can be improved, so that the purpose of improving the rendering efficiency can be achieved.

[0186] In a possible implementation, before the asynchronously executed OpenGL ES instruction is processed into a first event by a first thread, the plurality of OpenGL ES instructions are classified into asynchronously executed OpenGL ES instructions and synchronously executed OpenGL ES instructions according to the type of an OpenGL ES application program interface (API).

[0187] In a possible implementation, the state setting and obtaining type of OpenGL ES instruction and the object generation and destruction type of OpenGL ES instruction belong to synchronously executed OpenGL ES instructions; the data uploading type of OpenGL ES instruction and the drawing type of OpenGL ES instruction belong to asynchronously executed OpenGL ES.

[0188] In a possible implementation, after classifying multiple OpenGL ES instructions into asynchronously executed OpenGL ES instructions and synchronously executed OpenGL ES instructions according to the type of the OpenGL ES application program interface API, the method further includes: saving description information of the synchronously executed OpenGL ES instructions to a data structure through the first thread.

[0189] In one possible implementation, after classifying multiple OpenGL ES instructions into asynchronously executed OpenGL ES instructions and synchronously executed OpenGL ES instructions according to the type of the OpenGL ES application program interface API, the method further includes: obtaining description information of the synchronously executed OpenGL ES instructions from a data structure through a first thread, and sending the information to the image processor, wherein the image processor is configured to complete graphics rendering according to the description information of the synchronously executed OpenGL ES instructions; wherein the first thread may be a rendering thread of an OpenGL ES application.

[0190] In a possible implementation, obtaining the second task job through the second thread processing includes: when the first event currently processed belongs to a preset first type, executing the step of obtaining the second task job through the second thread processing the first event.

[0191] In one possible implementation, obtaining the second task job through processing by a second thread includes: when the first event currently processed does not belong to the preset first type, if the number of first events currently accumulated in the cache is greater than a first threshold, executing the step of obtaining the second task through processing the first event by the second thread.

[0192] In one possible implementation, obtaining the second task job through processing by a second thread includes: when the first event currently processed does not belong to the preset first type, and the number of first events currently accumulated and cached is greater than the second threshold and less than the first threshold, if the current GPU load is heavy, executing the step of obtaining the second task through processing the first event by the second thread.

[0193] In one possible implementation, obtaining a second task job through a second thread includes: when the first event currently processed does not belong to the preset first type, and the number of first events currently cached is greater than the second threshold and less than the first threshold, if the current GPU load is small, caching the first event.

[0194] In a possible implementation, the second task includes multiple first events, and caching the second task in the command buffer area through a third thread includes: sorting the multiple first events in the second task through the third thread; and caching the sorted second task in the command buffer area.

[0195] In a possible implementation, sorting the multiple first events in the second task by the third thread includes: sorting the multiple first events in the second task by the third thread based on pipeline states corresponding to the multiple first events in the second task.

[0196] In a possible implementation, multiple first events in the second task corresponding to first events of the same pipeline state are sequenced together.

[0197] In one possible implementation, the second task includes multiple first events, and the second task is cached in the command cache area through a third thread, including: sorting the first events in the second task corresponding to the same variable that are modified through the third thread; modifying the sorted first events in the second task based on the variable; and caching the sorted and modified second task in the command cache area.

[0198] In a possible implementation, the first events corresponding to modifications of the same variable in the multiple events are sorted together.

[0199] In a possible implementation, processing the asynchronous instruction into a first event through the first thread includes: obtaining description information of the asynchronous OpenGL ES instruction through the first thread; and encapsulating the description information of the asynchronous OpenGL ES instruction and the asynchronous OpenGL ES instruction into the first event.

[0200] The present application also provides a computer-readable storage medium in which computer instructions are stored. When the computer instructions are executed on one or more processors, Figure 3 The illustrated embodiment describes a graphics rendering method based on Vulkan.

[0201] The present application also provides a computer program product, wherein computer instructions are stored in the computer program product, and when the computer instructions are executed by one or more processors, the one or more processors are prompted to implement Figure 3 The illustrated embodiment describes a graphics rendering method based on Vulkan.

[0202] The embodiment of the present application also provides a chip system, which includes at least one processor, a memory and an interface circuit, wherein the interface circuit is used to provide information input / output for at least one processor, and the at least one memory stores computer instructions. When the computer instructions are executed on one or more processors, the system realizes Figure 3 The illustrated embodiment describes a graphics rendering method based on Vulkan.

[0203] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer instruction product. The computer instruction product includes one or more computer instructions. When the computer instruction is loaded and executed on a computer, the process or function described in the embodiment of the present application can be implemented in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instruction can be stored in a computer-readable storage medium or transmitted via a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. Available media can be magnetic media, (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid-state drive (SSD)), etc.

[0204] The steps in the method embodiments of the present application can be adjusted in sequence, combined, or deleted according to actual needs.

[0205] The modules in the embodiment of the device of the present application can be merged, divided and deleted according to actual needs.

Claims

1. A method for implementing graphics rendering based on Vulkan, characterized in that: include: Processing the asynchronously executed OpenGL ES instruction into a first event through a first thread, where the first event includes description information of the asynchronously executed OpenGL ES instruction; Processing the first event by a second thread to obtain a second task, where the second task includes a plurality of first events, and the plurality of first events are obtained by processing the same or different asynchronously executed OpenGL ES instructions; sorting the plurality of first events in the second task by a third thread, and caching the sorted second tasks into a command buffer; The second task in the command buffer is sent to the graphics processor through a Vulkan interface, wherein the graphics processor is configured to complete graphics rendering according to the second task.

2. The method according to claim 1, characterized in that Before obtaining the description information of the asynchronously executed OpenGLES instruction through the first thread, the method further includes: A plurality of OpenGL ES instructions are classified into asynchronously executed OpenGL ES instructions and synchronously executed OpenGL ES instructions according to the type of the OpenGL ES application program interface.

3. The method according to claim 2, characterized in that State setting and obtaining type OpenGL ES instructions and object creation and destruction type OpenGL ES instructions are all synchronously executed OpenGL ES instructions; data upload type OpenGL ES instructions and drawing type OpenGL ES instructions are all asynchronously executed OpenGL ES.

4. The method according to claim 2 or 3, characterized in that After classifying the plurality of OpenGL ES instructions into asynchronously executed OpenGL ES instructions and synchronously executed OpenGL ES instructions according to the type of the OpenGL ES application program interface, the method further includes: The description information of the synchronously executed OpenGL ES instructions is saved in a data structure through the first thread.

5. The method according to claim 2 or 3, characterized in that After classifying the plurality of OpenGL ES instructions into asynchronously executed OpenGL ES instructions and synchronously executed OpenGL ES instructions according to the type of the OpenGL ES application program interface API, the method further includes: After obtaining the description information of the synchronously executed OpenGL ES instructions from the data structure through the first thread, the description information is sent to the image processor, and the image processor is used to complete graphics rendering according to the description information of the synchronously executed OpenGL ES instructions.

6. The method according to claim 1 or 2, characterized in that The step of processing the first event by a second thread to obtain a second task includes: When the first event belongs to a preset first type, the step of processing the first event by a second thread to obtain a second task is executed.

7. The method according to claim 1 or 2, characterized in that The step of processing the first event by a second thread to obtain a second task includes: When the first event does not belong to the preset first type and if the cumulative number of cached first events is greater than a first threshold, the step of processing the first event by a second thread to obtain a second task is executed.

8. The method according to claim 1 or 2, characterized in that The step of processing the first event by the second thread to obtain the second task job includes: When the first event does not belong to the preset first type and the number of first events currently accumulated in the cache is greater than the second threshold and less than or equal to the first threshold, if the load of the GPU is less than the first load threshold, the step of obtaining a second task by processing the first event through a second thread is executed, and the second threshold is less than the first threshold.

9. The method according to claim 1 or 2, characterized in that The second task job is obtained by the second thread processing, including: When the first event does not belong to the preset first type and the number of first events currently cached is greater than the second threshold and less than or equal to the first threshold, if the GPU load is greater than or equal to the first load threshold, the first event is cached.

10. The method according to claim 1 or 2, characterized in that The sorting of the plurality of first events in the second task by the third thread includes: The plurality of first events in the second task are sorted by a third thread based on pipeline states corresponding to the plurality of first events.

11. The method according to claim 10, characterized in that First events corresponding to the same pipeline state among the plurality of first events are sorted together.

12. A device for implementing graphics rendering based on Vulkan, characterized in that: include: a context module, configured to process the asynchronously executed OpenGL ES instruction into a first event through a first thread, wherein the first event includes description information of the asynchronously executed OpenGL ES instruction; an event management module, configured to process the first event through a second thread to obtain a second task, wherein the second task includes a plurality of first events, and the plurality of first events are obtained by processing the same or different asynchronously executed OpenGL ES instructions; an event packaging module, configured to sort the plurality of first events in the second task through a third thread, and cache the sorted second tasks in a command buffer; The task scheduling module is used to send the second task in the command buffer to the GPU graphics processor through the Vulkan interface, wherein the graphics processor is used to complete graphics rendering according to the second task.

13. The device according to claim 12, characterized in that The context module is further configured to classify a plurality of OpenGL ES instructions into asynchronously executed OpenGL ES instructions and synchronously executed OpenGL ES instructions according to the type of the OpenGL ES application program interface API.

14. The device according to claim 13, characterized in that State setting and obtaining type OpenGL ES instructions and object creation and destruction type OpenGL ES instructions are both synchronously executed OpenGL ES instructions; data upload type OpenGL ES instructions and drawing type OpenGL ES instructions are both asynchronously executed OpenGL ES.

15. The device according to claim 13 or 14, characterized in that The context module is further configured to save the description information of the synchronously executed OpenGL ES instructions to a data structure through the first thread.

16. The device according to claim 12 or 13, characterized in that The context module is further configured to obtain description information of the synchronously executed OpenGL ES instructions from the data structure through the first thread and send the information to the image processor. The image processor is configured to complete graphics rendering according to the description information of the synchronously executed OpenGL ES instructions.

17. The device according to claim 12 or 13, characterized in that The event management module is specifically used to: When the first event belongs to a preset first type, the step of obtaining a second task job through the second thread processing is performed.

18. The device according to claim 12 or 13, characterized in that The event management module is specifically used to: When the first event does not belong to the preset first type, if the number of first events currently cached cumulatively is greater than a first threshold, the step of processing the first event by a second thread to obtain a second task is executed.

19. The device according to claim 12 or 13, characterized in that The event management module is specifically used to: When the first event does not belong to the preset first type and the number of first events currently accumulated in the cache is greater than the second threshold and less than or equal to the first threshold, if the load of the GPU is less than the first load threshold, the step of obtaining a second task by processing the first event through a second thread is executed, and the second threshold is less than the first threshold.

20. The device according to claim 12 or 13, characterized in that The event management module is specifically used to: When the first event does not belong to the preset first type and the number of first events currently cached is greater than the second threshold and less than or equal to the first threshold, if the GPU load is greater than or equal to the first load threshold, the first event is cached.

21. The device according to claim 12 or 13, characterized in that The event packaging module is specifically configured to sort the multiple first events in the second task based on pipeline states corresponding to the multiple first events through a third thread.

22. The device according to claim 21, characterized in that The multiple first events corresponding to the first events of the same pipeline state are sequenced together.

23. An electronic device, characterized in that: The electronic device includes a processor and a memory; the processor is configured to execute computer instructions stored in the memory, so that the electronic device implements the method according to any one of claims 1 to 11.

24. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which, when executed by one or more processors, prompt the one or more processors to implement the method according to any one of claims 1 to 11.

25. A computer program product, characterized in that The computer program product stores computer instructions, which, when executed by one or more processors, prompt the one or more processors to implement the method according to any one of claims 1 to 11.

Images