Graph rendering method and device and electronic equipment

Abstract

The invention discloses a graph rendering method and device and electronic equipment, and relates to the technical field of data processing. The graph rendering method comprises the following steps: generating data to be rendered; and calling a GPU to perform parallel rendering on the to-be-rendered data based on a Vulkan driving module included in the driving layer to obtain rendered data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a graphics rendering method and device and electronic equipment. BACKGROUND

[0002] At present, users have a high frequency of using relatively complex functions such as atomic islands, real-time blur effects, multitasking, and live streaming in the process of using electronic equipment such as mobile phones in daily life, and users usually hope to feel no lag in the interaction process to bring a more extreme smooth experience. However, with the continuous evolution of these complex scenarios, the continuous increase in load has brought great challenges to the optimization of the graphics rendering link. The existing graphics rendering framework mechanism is usually difficult to adapt to the existing complex high-load application scenarios, thereby causing problems such as lag in the process of graphics rendering of electronic equipment in some complex high-load application scenarios. SUMMARY

[0003] Embodiments of the present application provide a graphics rendering method, device and electronic equipment, which can solve the problem of lag in the process of graphics rendering of electronic equipment in some complex high-load application scenarios in the related art.

[0004] In a first aspect, a graphics rendering method is provided, and the method comprises:

[0005] generating to-be-rendered data;

[0006] performing parallel rendering on the to-be-rendered data by calling a GPU based on a Vulkan driver module included in a driver layer to obtain rendered data.

[0007] In a second aspect, a graphics rendering device is provided, and the device comprises:

[0008] a generating module configured to generate to-be-rendered data;

[0009] a rendering module configured to perform parallel rendering on the to-be-rendered data by calling a GPU based on a Vulkan driver module included in a driver layer to obtain rendered data.

[0010] In a third aspect, an electronic device is provided, which comprises a processor and a memory, and the memory stores programs or instructions executable on the processor, and the programs or instructions are executed by the processor to implement the steps of the method according to the first aspect.

[0011] In a fourth aspect, an embodiment of the present application provides a readable storage medium, which stores programs or instructions, and the programs or instructions are executed by a processor to implement the steps of the method according to the first aspect.

[0012] In a fifth aspect, an embodiment of the present application provides a chip, the chip comprising a processor and a communication interface, the communication interface and the processor being coupled, the processor being configured to run programs or instructions to implement steps of the method according to the first aspect.

[0013] In a sixth aspect, an embodiment of the present application provides a computer program product stored in a storage medium, the program product being executed by at least one processor to implement steps of the method according to the first aspect.

[0014] In the embodiment of the present application, since the Vulkan has the multi-thread and parallel rendering capability, by calling the GPU to perform parallel rendering on the to-be-rendered data based on the Vulkan driver module, the efficiency of rendering can be improved, and the probability of frame freezing in the process of performing graphic rendering in a complex high-load application scenario can be reduced. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is a flowchart of an image rendering method provided by an embodiment of the present application;

[0016] Figure 2 is a framework diagram of a graphic rendering system in the present application;

[0017] Figure 3 is a flowchart of a graphic rendering process of a graphic rendering system in an embodiment of the present application;

[0018] Figure 4 is a rendering pipeline link diagram of a graphic rendering process of a graphic rendering system in an embodiment of the present application;

[0019] Figure 5 is a rendering pipeline parallelization flowchart of a graphic rendering process of a graphic rendering system in an embodiment of the present application;

[0020] Figure 6 is a rendering drawing instruction multi-thread API switching diagram of a graphic rendering process of a graphic rendering system in an embodiment of the present application;

[0021] Figure 7 is an inter-frame parallel asynchronous flowchart of a graphic rendering process of a graphic rendering system in an embodiment of the present application;

[0022] Figure 8 is an intra-frame parallel drawing diagram of a graphic rendering process of a graphic rendering system in an embodiment of the present application;

[0023] Figure 9 is a 2D engine Skia multi-thread model diagram in a graphic rendering system in an embodiment of the present application;

[0024] Figure 10 is a 2D engine refactoring GPU resource in a graphics rendering system in the embodiments of the present application, which is uniformly managed by a Globalcontext class diagram;

[0025] Figure 11 is an inter-frame and intra-frame parallel rendering flowchart of a graphics rendering process in the graphics rendering system in the embodiments of the present application;

[0026] Figure 12 is a Vsyn synthesis cycle schematic diagram in the embodiments of the present application;

[0027] Figure 13 is a structural schematic diagram of an image rendering device provided by some embodiments of the present application;

[0028] Figure 14 is a structural schematic diagram of an electronic device provided by some embodiments of the present application;

[0029] Figure 15 is a hardware structural schematic diagram of an electronic device provided by some embodiments of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art belong to the scope of protection of the present application.

[0031] The terms “first”, “second”, and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by “first”, “second” are usually a category and do not limit the number of objects, for example, the first object can be one or more. In addition, “or” in the present application means at least one of the connected objects. For example, the protection scope of “A or B” at least covers three schemes, namely, scheme one: including A and not including B; scheme two: including B and not including A; scheme three: including A and B. In addition, the terms “A and / or B”, “at least one of A and B”, “at least one of A or B” also at least cover the above three schemes, respectively. The character “ / ” generally represents that the objects before and after are in an “or” relationship.

[0032] The graphics rendering method, device and electronic device provided by the embodiments of the present application will be described in detail below in combination with the drawings and some embodiments and application scenarios.

[0033] Please refer to Figure 1 The embodiment of the present application provides a graphics rendering method, and the method comprises the following steps:

[0034] Step 101, generating to-be-rendered data;

[0035] Step 102, calling a graphics processing unit (GPU) to perform parallel rendering on the to-be-rendered data based on a Vulkan driver module included in a driving layer, to obtain rendered data.

[0036] The image rendering method can be applied to an operating system of an electronic device, and the operating system can call a GPU in the electronic device to execute the image rendering method based on a related software architecture pre-deployed in the electronic device. For example, please refer to Figure 2 In some embodiments of the present application, the software architecture in the operating system can comprise an application layer, a framework layer, a system image framework service layer, a driving layer and a hardware layer connected in sequence, the application layer is used to generate to-be-rendered data, the system image framework service layer comprises a rendering service module, and the driving layer comprises a Vulkan driver module; the rendering service module is used to call a CPU in the hardware layer to perform parallel rendering on the to-be-rendered data based on the Vulkan API driver module.

[0037] The software architecture in the operating system can be referred to as a graphics rendering architecture in the electronic device. The application layer can comprise various rendering scenes of applications installed in the electronic device, for example, please refer to Figure 2The application layer can include the following rendering scenes: animation / animation effect scene, real-time blur scene, live broadcast scene, multi-task scene, etc. The above framework layer can include a View layer, which can include a View module, an Animation module, a Draw module, etc. The above system image framework service layer further includes a UI hardware rendering (HWUI) module, a SurfaceFlinger module, an OpenGL pipeline (OpenGL Pipeline) module, a Vulkan pipeline (Vulkan Pipeline) module, an application programming interface (Application Programming Interface, API) module, and a hardware design (Hardware Composer, HWC) module. The HWUI module can include sub-modules such as a render thread (RenderThread), a bitmap (Bitmap), and a canvas (Canvas). The SurfaceFlinger module includes a vertical synchronization (Vsync) module, a Layer module, and a render pipeline engine (RenderEngine) module, etc. The API module includes an OpenGL ES interface and a Vulkan interface. The HWC module includes a direct rendering manager user space library (libdrm). The above driving layer can be a driver development kit (Driver Development Kit, DDK) layer. The DDK layer can include an OpenGL driver module and the Vulkan driver module. The above hardware layer can include central processing units (CPU), graphics processing units (GPU), and neural network processing units (Neural network Processing Unit, NPU), etc.

[0038] It can be understood that the above step 101 can include generating to-be-rendered data based on the application layer. The above step 102 can include that the rendering service module calls a graphics processing unit (GPU) to perform parallel rendering on the to-be-rendered data based on the Vulkan driver module included in the driving layer, to obtain rendered data.

[0039] Please refer to Figure 3 and Figure 4In some embodiments of the present application, the process of graphics rendering based on the above graphics rendering architecture can mainly include the following procedures: a user can interact with an application in an electronic device in various application scenarios to generate corresponding to-be-rendered data in the application layer, and the to-be-rendered data generated in the application layer can be transmitted to the Renderservice through Inter Process Communication (IPC), and the Renderservice can call the GPU for parallel rendering through the ukia API based on the SKia engine in the operating system to obtain rendered data.

[0040] The above parallel rendering can include at least one of inter-frame parallel rendering and intra-frame parallel rendering. The inter-frame parallel rendering can refer to the rendering of more than two frames of images synchronously. The intra-frame parallel rendering can refer to the synchronous rendering of different contents in the same frame of images through more than two sub-tasks.

[0041] Please refer to Figure 5 The rendering process of the above to-be-rendered data can include two parallel rendering threads: a vSync-app thread and a vSync-SF thread, wherein the vSync-app thread and the vSync-SF thread can both use the inter-frame parallel and / or intra-frame parallel mechanism in the embodiments of the present application for rendering. In order to facilitate understanding, the rendering process of the vSync-SF thread is taken as an example to further explain the rendering process in the embodiments of the present application.

[0042] The above to-be-rendered data can be data generated in response to user interaction with the application in the application layer, such as to-be-rendered data generated in the user operation live broadcast, launcher and other interaction processes.

[0043] In the rendering process, the system graphics framework service layer can split the rendering link time-consuming rendering task into multi-thread concurrent execution. For example Figure 4 As shown in the figure, the RendeService process can be rendered based on the inter-frame parallel and intra-frame parallel mechanisms. As shown in the figure Figure 5 The RenderThread rendering thread and the RenderEngine rendering thread can implement the parallelization of time-consuming operations such as Layout / input / Draw / texture upload.

[0044] In the embodiments of the present application, by switching the rendering mode of SurfaceFlinger from OpenGL to the more efficient Vulkan mode, it is beneficial to realize the multi-thread and parallel rendering capability of the image rendering architecture. For example Figure 6As shown, the RenderThread rendering thread performs instruction drawing and the like, the RenderEngine rendering thread performs instruction execution and the like, and both the RenderThread rendering thread and the RenderEngine rendering thread have the ability to invoke the Vulkan API rendering.

[0045] It can be understood that, in some embodiments of the present application, after obtaining the rendered data, the CPU and GPU hardware capabilities can be invoked through the kernel DDK Vulkan driver layer to synthesize and display the rendered data.

[0046] In this embodiment, since Vulkan has multi-thread and parallel rendering capabilities, the GPU is invoked based on the Vulkan driver module to perform parallel rendering on the to-be-rendered data, which is beneficial to improve the rendering efficiency, so that the probability of freezing of the electronic device in the process of performing graphic rendering in a complex high-load application scenario can be reduced.

[0047] Optionally, the Vulkan driver module included in the driver layer invokes the GPU to perform parallel rendering on the to-be-rendered data to obtain rendered data, including:

[0048] The to-be-rendered data is rendered in parallel based on a main thread and a rendering thread, wherein the main thread is used for occlusion culling, dirty area calculation, and synchronization of processed data to the rendering thread, and the rendering thread is used for rendering processing of the data synchronized from the main thread.

[0049] The rendering thread can also be used for design of creation and management of GPU resources, and the resources of the rendering thread can be managed by a unified thread resource pool.

[0050] The to-be-rendered data is rendered in parallel, including rendering the to-be-rendered data in parallel by a main thread and a rendering thread, wherein the main thread can be used for occlusion culling, dirty area calculation, and synchronization of processed data to the rendering thread, and the rendering thread can be used for rendering processing of the data synchronized from the main thread.

[0051] It can be understood that the main thread and the rendering thread can be synchronized.

[0052] In the embodiment, the to-be-rendered data is rendered in parallel based on the main thread and the rendering thread, and compared with the prior art in which rendering is performed by only one thread, the rendering efficiency of the to-be-rendered data can be improved because the to-be-rendered data is rendered by the main thread and the rendering thread in synchronization.

[0053] Optionally, the parallel rendering of the to-be-rendered data based on the main thread and the rendering thread to obtain the rendered data comprises:

[0054] In the process of processing the Nth frame data based on the main thread, the (N-1)th frame data is processed in synchronization based on the rendering thread to obtain (N-1)th frame rendering data.

[0055] After the processing of the Nth frame data is completed by the main thread, the Nth frame data is synchronized to the rendering thread based on the main thread, wherein in the process of synchronizing the Nth frame data to the rendering thread by the main thread, the Nth frame data received is rendered by the rendering thread to obtain Nth frame rendering data, the rendered data comprises the (N-1)th frame rendering data and the Nth frame rendering data, and N is an integer greater than 1.

[0056] The main thread and the rendering thread can render the to-be-rendered data in parallel based on an asynchronous inter-frame parallel mechanism. Figure 7 The rendering of one frame of image by the main thread can sequentially include the following three stages: a Prepare stage, a Process stage and a Sync stage. The rendering of one frame of image by the rendering thread can sequentially include the following three stages: a Sync stage, a Process stage and a Flush stage, wherein the Prepare stage is a stage of realizing resource preparation and command recording, the Process stage is a stage of realizing GPU execution of rendering commands, and the Flush stage is a stage of submitting commands to GPU for execution and displaying results. The Sync stage is a stage of avoiding unnecessary synchronization waiting and communicating parallel tasks, wherein the main thread synchronizes the processed data to the rendering thread in the Sync stage, and correspondingly, the rendering thread receives the data synchronized by the main thread in the Sync stage.

[0057] The asynchronous inter-frame parallel mechanism can also be referred to as an asynchronous parallel pipeline mechanism. Specifically, the N-1th frame data is referred to as FrameN-1, and the Nth frame data is referred to as FrameN. When the main thread is rendering FrameN-1, the rendering thread can start drawing FrameN-1 in synchronization when the main thread reaches the Sync stage. Correspondingly, when the main thread is rendering FrameN, the rendering thread can start drawing FrameN in synchronization when the main thread reaches the Sync stage. In this way, the main thread and the rendering thread can efficiently draw in parallel.

[0058] Please refer to Figure 7 In some embodiments of the present application, after the main thread completes the Process stage of FrameN-1 and enters the Sync stage, the main thread can synchronize the processed data to the rendering thread. The rendering thread can process the received data in the Process stage in synchronization when the main thread processes FrameN-1. After the main thread completes the Sync stage of FrameN-1, the main thread can continue to process FrameN, while the rendering thread can process FrameN-1. In this way, the main thread and the rendering thread can process FrameN and FrameN-1 in synchronization to improve the rendering efficiency of the to-be-rendered data.

[0059] It can be understood that FrameN-1 and FrameN are any two adjacent frames of the to-be-rendered data.

[0060] In this embodiment, the main thread and the rendering thread perform parallel rendering on the to-be-rendered data based on the asynchronous inter-frame parallel mechanism. In this way, the rendering efficiency of the to-be-rendered data can be improved.

[0061] Optionally, when the main thread receives an N+1th refresh signal, the main thread completes the processing of the Nth frame data, where the N+1th refresh signal is used to instruct the main thread to perform rendering processing on N+1th frame data.

[0062] The N+1th refresh signal can be a signal issued by the application layer. Please refer to Figure 7 When the vSync-app completes the rendering processing of the N+1th frame data, the application layer can issue the N+1th refresh signal.

[0063] In some embodiments of the present application, when the refresh signal N arrives, the main thread completes the drawing of the N-1th frame, and then the rendering thread draws the N-1th frame. Correspondingly, when the refresh signal N+1 arrives, the main thread draws the Nth frame and the rendering thread draws the Nth frame in turn. In this way, the drawing in the period is ensured, and the rendering time is reduced.

[0064] In the embodiment, the main thread completes the processing of the Nth frame data when the N+1th refresh signal is received. In this way, the main thread and the rendering thread can complete the drawing in the refresh period, so as to reduce the rendering time.

[0065] Optionally, the rendering processing of the Nth frame data by the rendering thread includes:

[0066] The rendering task corresponding to the Nth frame data is divided into at least two decomposition tasks;

[0067] The at least two decomposition tasks are synchronously executed by at least two rendering sub-threads in the rendering thread, to obtain at least two drawing information corresponding to the at least two decomposition tasks, wherein the at least two rendering sub-threads correspond to the at least two decomposition tasks one by one, and the Nth frame rendering data includes the at least two drawing information;

[0068] The at least two drawing information are submitted to the GPU.

[0069] In the embodiment, the main thread and the rendering thread can perform parallel rendering processing on the to-be-rendered data based on the intra-frame parallel mechanism. Specifically, the main thread can perform rendering processing on the to-be-rendered data based on two or more sub-threads or sub-tasks synchronously; and the rendering thread can perform rendering processing on the received data based on two or more sub-threads or sub-tasks synchronously.

[0070] Please refer to Figure 8 The decomposition task is a sub-task of the rendering task, and is relatively independent and has granularity, so as to facilitate load balancing.

[0071] In the embodiment, the at least two decomposition tasks can correspond to at least two rendering sub-threads, so that each rendering sub-thread can record the command buffer (DrawCall) of each sub-tree at the same time, and then the command buffer (DrawCall) of each sub-tree is unified to a Canvas, and the Canvas is submitted to the GPU.

[0072] After the rendering task is split into multiple decomposition tasks, the rendering order between the decomposition tasks can be determined according to the rendering tree. The main task can be understood as the root node of the rendering tree, and the content on the screen is drawn layer by layer from the root node to the leaf node of the tree, so the rendering order of the decomposition tasks can be determined by traversing the rendering tree from the root node of the rendering tree in a depth-first or breadth-first manner. The splitting rule for splitting the rendering task can be to split according to the elements in the page corresponding to the to-be-rendered data, for example, the desktop, wallpaper, status bar, each application, each application icon, application interface, background, layout element on the application interface, and the like on the screen in the rendering task can be decomposed into independent elements, and then the elements are combined to obtain the decomposition tasks, and each decomposition task includes at least one independent element.

[0073] The process of generating a GPU drawcall by each rendering sub-thread is as follows: the decomposition task generates a rendering command required by the GPU by ① process, ② flush, and ③ cmdBuffer. The rendering command required by the GPU is submitted to the GPU, and the drawing task is performed.

[0074] Each Canvas is drawn on the same surface, and the submission order of the drawing instruction drawcall is consistent with the rendering queue. Finally, the main task and the sub-task are submitted to the GPU in sequence in parallel, and the display is sent. The drawing information can include the drawing instruction drawcall.

[0075] In this embodiment, the rendering task corresponding to the Nth frame data is split into at least two decomposition tasks, the at least two decomposition tasks are synchronously executed based on the at least two rendering sub-threads in the rendering thread, at least two drawing information corresponding to the at least two decomposition tasks is obtained, and then the at least two drawing information is submitted to the GPU. Since more than two rendering sub-threads can be used to synchronously process the same frame data to realize intra-frame parallel processing, the rendering efficiency is further improved.

[0076] Optionally, the electronic device further includes an SKia engine module in the operating system, the SKia engine module includes a Globalcontext management class, the Globalcontext management class includes a GPU resource cache (GPUResourceCache) class, the GPUResourceCache is used to control the at least two rendering sub-threads to synchronously execute the at least two decomposition tasks, and the GPUResourceCache is used to manage data corresponding to the rendering sub-threads.

[0077] The Globalcontext management class further includes a buffer management BufferMange class, a pipeline management pipelineManager class, and a rendering pass management renderpassmanger class.

[0078] See Figure 9 and Figure 10 In order to implement parallel rendering, the Skia engine in the operating system of the electronic device is optimized as follows in the embodiments of the present application.

[0079] As Figure 9 For the 2D engine multithreading model, the Skia engine is optimized to enable the Skia engine to support multiple threads to generate a vulkan command buffer simultaneously, and then submit the vulkan command buffer to a GPU commandQueue. Correspondingly, the 2D engine multithreading model supports subtask parallel GPU rendering.

[0080] As Figure 10 is a class diagram design, GPU resources are uniformly held by the Globalcontext, cross-thread reuse is supported, and the life cycle is guaranteed by reference technology.

[0081] In the related art, each rendering thread independently manages corresponding rendering data, and therefore, the relationship and conflicts between contexts cannot be effectively processed. See Figure 9 and Figure 10 In the embodiments of the present application, a Globalcontext management class is newly added relative to the framework in the related art, the contexts of each rendering thread are uniformly managed by the Globalcontext, and the Globalcontext class is designed to realize unified coordination of each thread state synchronization and data management by the GPUResourceCache, so that unified Canvas data resource management submission can be achieved.

[0082] The Globalcontext management class holds rendering pipeline link management classes such as BufferMange, pipelineManager, and renderpassmanger, and the Globalcontext management class holds a GPU resource data cache. In this way, the Globalcontext management class can uniformly manage the commandbuffer (DrawCall) and data resources of each sub-tree recorded by each rendering thread at the same time, and uniformly submit to the GPU. Thus, the life cycle of each thread can be effectively controlled whether it is cross-thread.

[0083] The asynchronous interframe parallel and intraframe parallel asynchronous rendering pipeline effect in the embodiment of the application is as shown in Figure 11

[0084] The system service layer rendering management mechanism is optimized, a multi-thread parallel rendering scheme is implemented, and an interframe parallel rendering mechanism and strategy are implemented. When the previous frame is about to end, the next vSync arrives, and the next frame starts to implement the parallel rendering interval. The parallel rendering interval between multiple frames is implemented, and the time consumption accumulation phenomenon caused by interframe queuing is eliminated.

[0085] The intraframe parallel mechanism fully utilizes the CPU multi-core computing power, splits the multiple tasks in a frame into multiple subtasks, finally manages and combines the resources and data of the multiple subtasks into a buffer, and uniformly composes and sends the display, so that the rendering of a frame is more efficient.

[0086] The rendering of a frame can be quickly completed in a short time, and the rendering of multiple frames can also be sequentially and efficiently completed, so that the phenomenon of lag is avoided, and a more smooth experience is brought.

[0087] It should be noted that in the embodiment of the application, the parallel mechanism trigger should ensure that the main thread does not involve the creation and use of GPU resources during splitting. The main thread mainly performs data synchronization, occlusion culling, and dirty area calculation work from the application to the RenderService, and creates and manages GPU resources using the rendering thread.

[0088] Optionally, the submitting the at least two rendering information to the GPU comprises:

[0089] According to the order indicated by the rendering tree corresponding to the rendering task, the at least two rendering information is submitted to the GPU.

[0090] It can be understood that after the GPU completes the processing of the rendering instruction, it can be based on the Sika mechanism to synthesize and send the display.

[0091] In this embodiment, by submitting the at least two rendering instructions to the GPU according to the order indicated by the rendering tree corresponding to the rendering task, the rendering of each element in the rendering task can be sequentially performed according to the order indicated by the rendering tree, thereby facilitating the sequential performance of the rendering task.

[0092] Optionally, the main thread includes a first sub-thread and at least one second sub-thread. The first sub-thread, when the to-be-rendered data is acquired, encapsulates the to-be-rendered data as a rendering task, and stores the rendering task in the thread resource pool. The second sub-thread is used to acquire the rendering task from the thread resource pool for rendering processing. ​

[0093] In some embodiments of the present application, the threads are further optimized as follows: a thread pool is added to manage multiple threads, supporting scalability.

[0094] The first sub-thread can traverse the rendering tree / rendering node once in a depth-first or breadth-first manner, encapsulate the drawing instructions in the rendering tree into a plurality of self-contained rendering tasks, and put the rendering tasks into a single-thread resource management task pool, while preparing to calculate the overlap relationship information between the rendering tasks.

[0095] The second sub-thread can obtain and execute the self-contained rendering tasks from the single-thread resource management task pool, and each task can be independently and out-of-order executed. After execution, the generated GPU object resources are packed and placed in a multi-thread resource management pool.

[0096] The GPU objects are obtained from the multi-thread resource management task pool, and the binding order of the GPU objects to the GPU command is maintained.

[0097] In this embodiment, by making the main thread include a first sub-thread and at least one second sub-thread, the first sub-thread is used to encapsulate the to-be-rendered data into a rendering task and store the rendering task in the thread resource pool when the to-be-rendered data is obtained, and the second sub-thread is used to obtain the rendering task from the thread resource pool for rendering processing. In this way, since the thread resource pool can manage multiple threads, the scalability is improved.

[0098] The embodiment of the present application provides a graphics rendering method, which is based on the graphics rendering serial drawing pipeline in the related art and realizes parallel rendering of multiple tasks and multiple threads, for example, switches the underlying graphics rendering engine capability from OpenGL API to Vulkan API, and realizes integration of an OS system rendering link from bottom to top.

[0099] In the embodiment of the present application, the graphics rendering API is replaced by Vukan API: in a high-load multi-task scene and an interactive scene, the multi-thread and parallel rendering capability of the Vulkan API is used to improve the rendering efficiency of the multi-task, so that the rendering time is controlled within a reasonable range, so as to realize upgrading and optimization of an Android graphics rendering link.

[0100] The parallel rendering mechanism and strategy are established in the embodiments of the present application, which fully adapt to the utilization of the underlying Vulkan API parallel rendering capability: from the UI thread to the rendering thread, from the application to the system rendering link, the following is achieved: ① inter-frame parallel, parallel strategy: the rendering process is changed from serial to inter-frame rendering asynchronous flow parallel, which reduces the frame loss problem in heavy load scenarios, and the parallel mechanism is based on the current Skia engine implementation. RenderService splits the processing pipeline into multiple tasks, which becomes a pipeline, and is accelerated by multi-threading to achieve inter-frame parallel. And the thread resource pool is used to manage multiple threads, which has better scalability. ② intra-frame parallel, parallel strategy: make full use of CPU multi-core power, split the rendering node into sub-nodes, and realize intra-frame fine-grained high parallel GPU drawing. That is, each thread simultaneously records the command buffer (DrawCall) of each sub-node, and then merges them into a Canvas in the main task, and submits them to the GPU.

[0101] Among them, the parallel mechanism: the skia engine is optimized, and the GPU resources are uniformly managed by multiple threads, solving the problem of cross-thread unsafe GPU resources in related technologies where different threads have their own context.

[0102] The software architecture in the operating system provided by the embodiments of the present application at least has the following beneficial effects:

[0103] The embodiments of the present application provide a graphics rendering method, which meets the extreme smoothness requirement experience of high-load drawing scenes such as atomic island, live broadcast scene, multi-window multi-task scene, real-time blur dynamic effect scene, etc.

[0104] Firstly, in the system graphics framework and service layer, the inter-frame parallel rendering mechanism is realized, based on the skia engine, the processing pipeline is split into multiple sub-tasks by creating a new RenderService process, which becomes a pipeline, and is accelerated by multi-threading to achieve inter-frame parallel, solving the low efficiency and high drawing time caused by current thread serial drawing. In multi-frame drawing, parallel rendering is realized, the rendering task is split, multi-core concurrency is realized, and pipeline execution is realized to reduce rendering time.

[0105] Secondly, in the system graphics framework and service layer, the intra-frame parallel rendering mechanism is realized, by making full use of CPU multi-core power, the rendering node is split into sub-nodes, each thread simultaneously records the command buffer (DrawCall) of each sub-node, and then merges them into a Canvas in the main task, and then submits them to the GPU to realize intra-frame fine-grained high parallel GPU drawing. By allowing multiple intra-frame tasks to be executed in parallel, the current sub-task dependency problem is solved and the load balancing problem is optimized.

[0106] Again, switching the underlying rendering capability application rendering and SurfaceFlinger rendering synthesis from OpenGles api to Vulkan api, providing parallel rendering and multi-threaded rendering capability, while modifying Skia, uniformly managing GPU resources in multiple threads, solving the problem of different threads owning their own context in the existing design mode, and the cross-thread safety problem of GPU resources, efficiently supporting parallel GPU drawing of sub-tasks, from bottom to top, breaking through the rendering link, and realizing the high-efficiency rendering mechanism.

[0107] The atomic island scene, the live broadcast scene, and the blur dynamic effect high-load scene pose new challenges to rendering load and efficiency, which increase the rendering time consumption by about 1-4ms, and different time consumption values on different models.

[0108] The smooth requirement of graphic display requires that rendering and synthesis cannot exceed one vsync period, and when the frame rate is 120HZ, the vsync period is 8.3ms, as shown in Figure 12 .

[0109] The inter-frame and intra-frame parallel asynchronous flow strategy and mechanism of the embodiment of the present application: rendering task splitting, multi-core concurrency, pipeline execution, reducing rendering time consumption by about 1ms.

[0110] The efficient rendering of the embodiment of the present application: switching to efficient rendering Vulkan API, efficient rendering pipeline, reducing rendering time consumption by 0.5-1ms.

[0111] In summary, through the embodiment of the present application, the rendering time consumption can be effectively reduced for high-load scene requirements, and the user experience of using the system is more smooth.

[0112] The graphic rendering method provided by the embodiment of the present application, the execution subject can be a graphic rendering device. In the embodiment of the present application, the graphic rendering device is taken as an example to execute the graphic rendering method, and the graphic rendering device provided by the embodiment of the present application is described.

[0113] Please refer to Figure 13 , Figure 13 The image rendering device provided by the embodiment of the present application, the image rendering device comprises:

[0114] The generating module 1301 is configured to generate to-be-rendered data.

[0115] The rendering module 1302 is configured to call the GPU to perform parallel rendering on the to-be-rendered data based on a Vulkan driving module included in a driving layer, and obtain rendered data.

[0116] Optionally, the rendering module 1302 is specifically configured to perform parallel rendering on the to-be-rendered data based on a main thread and a rendering thread, to obtain rendered data, wherein the main thread is configured to perform occlusion culling, dirty area calculation on the to-be-rendered data, and synchronize the processed data to the rendering thread, and the rendering thread is configured to perform rendering processing on the data synchronized from the main thread.

[0117] Optionally, the rendering module 1302 is specifically configured to, in a process of processing the Nth frame data based on the main thread, process the (N-1)th frame data based on the rendering thread, to obtain (N-1)th frame rendering data.

[0118] The rendering module 1302 is further specifically configured to, after the main thread completes processing of the Nth frame data, synchronize the Nth frame data from the main thread to the rendering thread, wherein in a process of synchronizing the Nth frame data from the main thread to the rendering thread, the rendering thread performs rendering processing on the received Nth frame data, to obtain Nth frame rendering data, the rendered data includes the (N-1)th frame rendering data and the Nth frame rendering data, and N is an integer greater than 1.

[0119] Optionally, in a case where the main thread receives an N+1th refresh signal, the main thread completes processing of the Nth frame data, wherein the N+1th refresh signal is used to instruct the main thread to perform rendering processing on an N+1th frame data.

[0120] Optionally, the rendering module 1302 includes:

[0121] The decomposition submodule is configured to split a rendering task corresponding to the Nth frame data into at least two decomposition tasks.

[0122] The execution submodule is configured to synchronize execution of the at least two decomposition tasks based on at least two rendering sub-threads in the rendering thread, to obtain at least two drawing information corresponding to the at least two decomposition tasks one by one, wherein the at least two rendering sub-threads correspond to the at least two decomposition tasks one by one, and the Nth frame rendering data includes the at least two drawing information.

[0123] The submission submodule is configured to submit the at least two drawing information to the GPU.

[0124] Optionally, the operating system of the graphics rendering apparatus 1300 further comprises an SKia engine module, the SKia engine module comprises a Globalcontext management class, the Globalcontext management class comprises a GPUResourceCache class, the GPUResourceCache is used to control the at least two rendering sub-threads to synchronously execute the at least two decomposition tasks, and the GPUResourceCache is used to manage data corresponding to the rendering sub-threads.

[0125] The Globalcontext management class further comprises a buffer management BufferMange class, a pipeline management pipelineManager class, and a rendering pass management renderpassmanger class.

[0126] In this embodiment, since Vulkan has multi-thread and parallel rendering capabilities, the GPU is called based on the Vulkan driver module to perform parallel rendering on the to-be-rendered data, which is beneficial to improve the rendering efficiency, and thus the probability of frame freezing of the electronic device in the process of performing graphics rendering in a complex high-load application scenario can be reduced.

[0127] The graphics rendering apparatus 1300 in the embodiment of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), and the like, and can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, and the like, and the embodiment of the present application is not limited thereto.

[0128] The graphic rendering apparatus 1300 in the embodiments of the present application can be an apparatus with an operating system. The operating system can be an Android operating system, can be an ios operating system, and can also be other possible operating systems, which are not limited in the embodiments of the present application.

[0129] The graphic rendering apparatus 1300 provided by the embodiments of the present application can implement Figure 1 The method embodiments achieve the same technical effects as the above processes, which are not repeated here.

[0130] In some embodiments, as shown in Figure 14 The embodiments of the present application also provide an electronic device 1400, which includes a processor 1401, a memory 1402, a program or instruction stored in the memory 1402 and executable on the processor 1401. When the processor 1401 executes the program or instruction, the above processes of the graphic rendering method embodiments are implemented, and the same technical effects are achieved. To avoid repetition, the above processes are not repeated here.

[0131] Figure 15 The hardware structure schematic diagram of an electronic device according to the embodiments of the present application.

[0132] The electronic device 1500 includes but is not limited to a radio frequency unit 1501, a network module 1502, an audio output unit 1503, an input unit 1504, a sensor 1505, a display unit 1506, a user input unit 1507, an interface unit 1508, a memory 1509, and a processor 1510.

[0133] The processor 1510 is configured to generate to-be-rendered data.

[0134] The processor 1510 is configured to call the GPU to perform parallel rendering on the to-be-rendered data based on the Vulkan driver module included in the driver layer, to obtain rendered data.

[0135] Optionally, the processor 1510 is configured to perform parallel rendering on the to-be-rendered data based on a main thread and a rendering thread, to obtain rendered data. The main thread is configured to perform occlusion culling, dirty area calculation, and synchronization of processed data to the rendering thread. The rendering thread is configured to perform rendering processing on the data synchronized from the main thread.

[0136] Optionally, the processor 1510 is configured to, in a process of processing the Nth frame data based on the main thread, perform processing on the (N-1)th frame data based on the rendering thread, to obtain (N-1)th frame rendering data.

[0137] The processor 1510 is configured to synchronize the Nth frame data to the rendering thread based on the main thread after the main thread completes processing of the Nth frame data, wherein, in the process of synchronizing the Nth frame data to the rendering thread based on the main thread, the rendering thread performs rendering processing on the received Nth frame data to obtain Nth frame rendering data, and the rendering data includes the N-1th frame rendering data and the Nth frame rendering data, and N is an integer greater than 1.

[0138] Optionally, in the case that the main thread receives an N+1th refresh signal, the main thread completes processing of the Nth frame data, wherein the N+1th refresh signal is used to instruct the main thread to perform rendering processing on N+1th frame data.

[0139] Optionally, the processor 1510 is configured to split the rendering task corresponding to the Nth frame data into at least two decomposition tasks.

[0140] The processor 1510 is configured to synchronize execution of the at least two decomposition tasks based on at least two rendering sub-threads in the rendering thread to obtain at least two drawing information corresponding to the at least two decomposition tasks, wherein the at least two rendering sub-threads correspond to the at least two decomposition tasks one by one, and the Nth frame rendering data includes the at least two drawing information.

[0141] The processor 1510 is configured to submit the at least two drawing information to the GPU.

[0142] Optionally, the graphic rendering method is applied to an electronic device, and an operating system of the electronic device further includes an SKia engine module, the SKia engine module includes a Globalcontext management class, the Globalcontext management class includes a GPUResourceCache class, the GPUResourceCache is used to control synchronization execution of the at least two decomposition tasks by the at least two rendering sub-threads, and the GPUResourceCache is used to manage data corresponding to the rendering sub-threads.

[0143] The Globalcontext management class further includes a buffer management BufferMange class, a pipeline management pipelineManager class, and a rendering channel management renderpassmanger class.

[0144] Those skilled in the art can understand that the electronic device 1500 can further include a power supply (such as a battery) for supplying power to each component, and the power supply can be logically connected to the processor 1510 through a power management system, so that the power management system can realize the functions of managing charging, discharging, and power consumption management. Figure 15 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and the electronic device can include more or fewer components than the figure, or combine certain components, or different component arrangements, which are not described here.

[0145] It should be understood that in the embodiments of the present application, the input unit 1504 can include a graphics processor (GPU) 15041 and a microphone 15042. The graphics processor 15041 processes image data of a still picture or a video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1506 can include a display panel 15061, which can be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 1507 includes a touch panel 15071 and other input devices 15072. The touch panel 15071 is also called a touch screen. The touch panel 15071 can include two parts of a touch detection device and a touch controller. The other input devices 15072 can include, but are not limited to, a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), a trackball, a mouse, a joystick, etc., which are not described here.

[0146] The memory 1509 can be used to store software programs and various data. The memory 1509 can mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), and the like. In addition, the memory 1509 can include a volatile memory or a non-volatile memory, or the memory 1509 can include both volatile and non-volatile memories. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synch link DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM). The memory 1509 in the embodiments of the present application includes but is not limited to these and any other suitable types of memories.

[0147] The processor 1510 can include one or more processing units; optionally, the processor 1510 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and an application program, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 1510.

[0148] The embodiments of the present application also provide a readable storage medium, the readable storage medium stores programs or instructions, the programs or instructions are executed by a processor to realize the processes of the above-mentioned graphic rendering method embodiments, and the same technical effects can be achieved. To avoid repetition, details are not described here.

[0149] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0150] The chip provided in the embodiments of the present application includes a processor and a communication interface, the communication interface is coupled with the processor, the processor is used to run programs or instructions, realizes the processes of the above-mentioned graphic rendering method embodiments, and can achieve the same technical effects. To avoid repetition, details are not described here.

[0151] It should be understood that the chip mentioned in the embodiments of the present application can also be referred to as a system-level chip, a system chip, a chip system, or a system-on-chip chip, etc.

[0152] It should be noted that in this document, the term "comprising" or "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of additional identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to the order of performing the functions as shown or discussed, but can also include performing the functions in a substantially simultaneous manner or in reverse order, for example, the described method can be performed in an order different from that described, and various steps can also be added, omitted or combined. In addition, the features described with reference to some examples can be combined in other examples.

[0153] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a plurality of instructions for making a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the methods described in various embodiments of the present application.

[0154] The embodiments of the present application are described above with reference to the accompanying drawings, but the present application is not limited to the specific embodiments described above, and the specific embodiments described above are merely illustrative, but not restrictive, and a person of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims.

Claims

1. A graphics rendering method, characterized in that, The method includes: Generate data to be rendered; The Vulkan driver module included in the driver layer calls the GPU to perform parallel rendering of the data to be rendered, and the rendered data is obtained.

2. The graphics rendering method according to claim 1, characterized in that, The Vulkan driver module, included in the driver layer, calls the GPU to perform parallel rendering of the data to be rendered, obtaining the rendered data, including: The data to be rendered is rendered in parallel using the main thread and the rendering thread to obtain the rendered data. The main thread is used to perform occlusion culling, dirty area calculation, and synchronize the processed data to the rendering thread. The rendering thread is used to render the data synchronized from the main thread.

3. The graphics rendering method according to claim 2, characterized in that, The parallel rendering of the data to be rendered based on the main thread and the rendering thread to obtain the rendered data includes: During the processing of the Nth frame data based on the main thread, the N-1th frame data is processed synchronously based on the rendering thread to obtain the N-1th frame rendering data. After the main thread completes the processing of the Nth frame data, the main thread synchronizes the Nth frame data with the rendering thread. During the synchronization process, the rendering thread renders the received Nth frame data to obtain the Nth frame rendering data. The rendered data includes the (N-1)th frame rendering data and the Nth frame rendering data, where N is an integer greater than 1.

4. The graphics rendering method according to claim 3, characterized in that, When the main thread receives the (N+1)th refresh signal, the main thread completes the processing of the Nth frame data, wherein the (N+1)th refresh signal is used to instruct the main thread to render the (N+1)th frame data.

5. The graphics rendering method according to claim 3, characterized in that, The rendering process based on the received Nth frame data by the rendering thread to obtain the Nth frame rendering data includes: The rendering task corresponding to the Nth frame data is split into at least two decomposition tasks; Based on at least two rendering sub-threads in the rendering thread, the at least two decomposition tasks are executed synchronously to obtain at least two drawing information corresponding to the at least two decomposition tasks. The at least two rendering sub-threads correspond to the at least two decomposition tasks, and the Nth frame rendering data includes the at least two drawing information. The at least two drawing information are submitted to the GPU.

6. The graphics rendering method according to claim 5, characterized in that, The system is applied to electronic devices, and the operating system of the electronic devices also includes a SKia engine module. The SKia engine module includes a GlobalContext management class, which includes a GPUResourceCache class. The GPUResourceCache is used to control the at least two rendering sub-threads to execute the at least two decomposed tasks synchronously, and the GPUResourceCache is used to manage the data corresponding to the rendering sub-threads. The GlobalContext management class also includes a BufferManager class, a pipelineManager class, and a renderpassmanger class.

7. An image rendering apparatus, characterized in that, The device includes: The generation module is used to generate the data to be rendered; The rendering module is used to call the GPU to perform parallel rendering of the data to be rendered based on the Vulkan driver module included in the driver layer, so as to obtain the rendered data.

8. The graphics rendering apparatus according to claim 7, characterized in that, The rendering module is specifically used to perform parallel rendering of the data to be rendered based on the main thread and the rendering thread to obtain the rendered data. The main thread is used to perform occlusion culling, dirty area calculation, and synchronize the processed data to the rendering thread. The rendering thread is used to perform rendering processing on the data synchronized from the main thread.

9. The graphics rendering apparatus according to claim 8, characterized in that, The rendering module is specifically used to process the N-1th frame data synchronously based on the rendering thread during the process of processing the Nth frame data based on the main thread, so as to obtain the N-1th frame rendering data. The rendering module is further configured to synchronize the Nth frame data to the rendering thread based on the main thread after the main thread completes the processing of the Nth frame data. During the process of synchronizing the Nth frame data from the main thread to the rendering thread, the rendering thread performs rendering processing on the received Nth frame data to obtain Nth frame rendering data. The rendered data includes the (N-1)th frame rendering data and the Nth frame rendering data, where N is an integer greater than 1.

10. The graphics rendering apparatus according to claim 9, characterized in that, When the main thread receives the (N+1)th refresh signal, the main thread completes the processing of the Nth frame data, wherein the (N+1)th refresh signal is used to instruct the main thread to render the (N+1)th frame data.

11. The graphics rendering apparatus according to claim 9, characterized in that, The rendering module includes: The decomposition submodule is used to split the rendering task corresponding to the Nth frame data into at least two decomposition tasks; An execution submodule is used to synchronously execute the at least two decomposition tasks based on at least two rendering sub-threads in the rendering thread, and obtain at least two drawing information corresponding one-to-one with the at least two decomposition tasks, wherein the at least two rendering sub-threads correspond one-to-one with the at least two decomposition tasks, and the Nth frame rendering data includes the at least two drawing information. The submission submodule is used to submit the at least two drawing information to the GPU.

12. The graphics rendering apparatus according to claim 11, characterized in that, The operating system of the image rendering device also includes an SKia engine module. The SKia engine module includes a GlobalContext management class, which includes a GPUResourceCache class. The GPUResourceCache is used to control the at least two rendering sub-threads to execute the at least two decomposed tasks synchronously, and the GPUResourceCache is used to manage the data corresponding to the rendering sub-threads. The GlobalContext management class also includes a BufferManager class, a pipelineManager class, and a renderpassmanger class.

13. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a program or instructions that can run on the processor, and the program or instructions, when executed by the processor, implement the steps of the graphics rendering method as described in any one of claims 1-6.

Images