Pretreatment system and method thereof

Abstract

A pre-processing system for a pre-compiled shader includes the following elements. And the first persistent cache is arranged in a rendering engine under the surface-flinger. And the second persistent cache is arranged in the HWUI. A generation module generates a text file, a binary file, and a descriptor associated with a shader for rendering a plurality of pages of an application. And the installation module is used for storing the text file and the binary file into the scanning file path of the first persistent cache and / or the second persistent cache based on the descriptor.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a data processing mechanism, in particular, to a pre-processing system and a pre-processing method of a shader operating in a mobile device platform. BACKGROUND

[0002] With the improvement of the computing capability of mobile devices equipped with the Android platform, the refresh rate of the mainstream screen of the mobile device has been greatly increased. For example, the refresh rate can be increased to 120Hz or 144Hz. Given such a high refresh rate (e.g., 120Hz), it needs to complete the processing of a frame in a very short time (e.g., 8.3ms).

[0003] In the graphics processing of an application program (APP) on the Android platform, a graphics processing unit (GPU) is used to render the pages, animations, images or schemes of the APP. Each frame of the pages, animations, images or schemes needs a certain rendering time. In addition, in the current graphics rendering process, the shader needs to be compiled. This means that during the rendering of each frame, the GPU driver needs to compile the shader code into executable instructions to present the required visual effects on the screen. Therefore, the shader compilation is a time-consuming process.

[0004] However, at a high refresh rate, each frame must be completed in a short time, which will greatly reduce the rendering time of each frame. Therefore, the shader compilation will seriously block the rendering process. If the rendering process is blocked by the shader compilation, frame loss or stuttering will occur.

[0005] In view of the above problems, it is necessary to improve the shader compilation mechanism to reduce the blocking in the rendering pipeline, so as to ensure smooth graphics rendering and avoid performance problems caused by shader compilation. SUMMARY

[0006] According to one embodiment of the present application, a pre-processing system for a pre-compiled shader is provided. The pre-processing system includes a first persistent cache, a second persistent cache, a generating module and an installing module. The first persistent cache is disposed in a rendering engine under Surface-flinger. The second persistent cache is disposed in a Hardware Accelerated Rendering Engine for UI (HWUI). The generating module is configured to generate an output file including a text file, a binary file and a descriptor associated with the shader, wherein the shader is used to render a plurality of pages of an application program (APP). The installing module is configured to store the text file and the binary file into a scan file path of the first persistent cache or the second persistent cache based on the descriptor.

[0007] According to another embodiment of the present application, a pre-processing method for a pre-compiled shader is provided. The pre-processing method includes the following steps: generating an output file including a text file, a binary file and a descriptor associated with the shader, wherein the shader is used to render a plurality of pages of an application program (APP). Storing the text file and the binary file into a scan file path of a first persistent cache or a second persistent cache based on the descriptor. The first persistent cache is disposed in a rendering engine under Surface-flinger.

[0008] The present application provides a pre-processing system and a method thereof, which achieves the technical effect that the compilation of a shader does not block the rendering process through a pre-compiled shader. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 is a block diagram of a pre-processing system according to an embodiment of the present application.

[0010] Figure 2 is a block diagram showing a generating module and its operation.

[0011] Figure 3 is a block diagram showing an installing module and its operation.

[0012] Figure 4 is a block diagram showing the operation of various caches of the pre-processing system.

[0013] Figure 5 is a flowchart showing a pre-processing method according to an embodiment of the present application.

[0014] In the following detailed description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the disclosed embodiments. It will be apparent, however, that one or more embodiments can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to simplify the figures. DETAILED DESCRIPTION

[0015] Reference Figure 1 FIG. 1 shows a block diagram of a pre-processing system 1000 according to an embodiment of the present application. The pre-processing system 1000 is applied to an operating system of a local device. The local device is, for example, a smartphone, and the operating system is, for example, an Android system. The pre-processing system 1000 is used to process a shader in the operating system, and the shader is used for graphic rendering of a page, animation, image or scheme of an APP 10 (i.e., an application program) in the operating system. The pre-processing system 1000 can be a software program implemented by a set of instruction codes stored in a non-volatile storage medium of the local device. When loaded from the non-volatile storage medium, the pre-processing system 1000 can be executed by a processor (e.g., a CPU or a GPU) of the local device.

[0016] The pre-processing system 1000 includes a generation module 100, an installation module 200, a first persistent cache 310 and a second persistent cache 410. The generation module 100 and the installation module 200 can be sub-modules of the software program of the pre-processing system 1000. The first persistent cache 310 and the second persistent cache 410 can be memory caches in the local device.

[0017] More specifically, the pre-processing system 1000 is used to perform pre-compilation on a shader, and the shader can be associated with the APP 10 in the operating system. The pre-compilation on the shader can be performed at an early development stage of the APP 10 (e.g., when first activated in the operating system). The shader is used to perform graphic rendering of a page, scheme or content of the APP 10 based on graphic rendering commands of an upper layer graphic interface (e.g., “Canvas”, “View” or the like of the Android graphic interface). Alternatively, the shader can be associated with a Surface-Flinger 11 to perform graphic rendering based on images superimposed with a GPU.

[0018] To perform the pre-compilation of the shader, first, the generation module 100 queries (e.g., polls) a page, a scheme, or a content of interest of the APP 10. Thereafter, the generation module 100 generates an output file 20 of the graphical rendering command related to the shader with the upper layer graphical interface in a run-time manner. The output file 20 can have a format including a first header, a second header, and a payload, and such an output file 20 can be stored using a data exchange language (e.g., json, xml, etc.), which is a domain-specific language for describing the shader in the Skia graphics library (i.e., the Skia graphics library is a third-party open source 2D graphics library). The first header can include statistical information related to the shader. Further, the second header can include a key of the shader. The key can be descriptive information of the shader property, which can be used as a feature value matching the shader, and for finding the shader. Further, the payload can include main information of the shader, which can be in a text form or a binary form.

[0019] The payload of the output file 20 can include a text file 21 or a binary file 22. The text file 21 is suitable for different types of local devices, on the other hand, the binary file 22 includes an executable binary code (e.g., a shader binary code) suitable for a hardware or software configuration of the local device, and thus the binary file 22 can be directly executed by such a type of local device. The installation module 200 acquires the text file 21 and / or the binary file 22 from the generation module 100, and installs the shader into an operating system of the local device based on the text file 21 and / or the binary file 22.

[0020] The text file 21 and / or the binary file 22 can be stored in a static manner in a first persistent cache 310 and / or a second persistent cache 410. The first persistent cache 310 is a persistent cache associated with a rendering engine 300 under the Surface-flinger of the local device, and the second persistent cache 410 is a persistent cache associated with a HWUI 400 (i.e., a hardware-accelerated rendering engine for UI) of the local device. When the local device is turned off, the data of the text file 21 and / or the binary file 22 in the first persistent cache 310 and / or the second persistent cache 410 can be retained. For example, when the shader is associated with the APP 10, the text file 21 and / or the binary file 22 of the shader can be stored in the second persistent cache 410 in the HWUI 400. Alternatively, when the shader is associated with the display compositing system (Surface-Flinger) 11, the text file 21 and / or the binary file 22 of the shader can be stored in the first persistent cache 310 in the rendering engine 300 thereof.

[0021] Reference Figure 2 which shows a block diagram of the generation module 100 and its operation. The APP 10 can act as a "caller" role and use a package name (or process name, etc.) to inform the generation module 100 of which shader is of interest (i.e., the shader of interest). In addition, the APP 10 can utilize the generation module 100 to query the page, animation, or scheme of interest that the shader of interest is to render. For example, the APP 10 can inform the generation module 100 of information about the shader through a layer stack of an Android graphical interface 600 (i.e., an upper layer graphical interface such as Canvas, View, etc.), a Java-native-interface (JNI) 610, and a HWUI 400. Alternatively, when the shader is to perform graphical rendering based on an image overlaid with a GPU, a Surface-Flinger 11 can inform and use the generation module 100 through a rendering engine 300.

[0022] The generation module 100 includes a text generation unit 110, a text conversion unit 120, a compilation unit 130, a linking unit 140, and a statistics unit 150. The generation module 100 performs pre-compilation of the shader under the following conditions: (1) when the APP 10 is first activated, (2) when the APP 10 first uses a page and causes a change in graphical drawing, and (3) when the content of the APP 10 is updated and causes a change in graphical drawing.

[0023] More specifically, the Skia graphics library can provide a "make SKSL program" function, and the text generation unit 110 of the generation module 100 is formed by modifying the make SKSL program function. The text generation unit 110 is used to generate a text file 21 (e.g., an SKSL text file) of the shader, and the text generation unit 110 can directly output the text file 21 to the installation module 200. Figure 1 In addition, the text conversion unit 120 is used to convert the text file 21 from the SKSL format to the GLSL (OpenGL Shading Language) format.

[0024] In addition, for a GPU that supports loading of binary code, the compilation unit 130 is used to compile the GLSL text file into a binary file 22, and the compilation unit 130 can directly output the binary file 22 to the installation module 200. Figure 1In the installation module 200. In addition, the link unit 140 is used to link the binary file 22, and the statistical unit 150 is used to perform statistical calculations to generate a descriptor 23. The descriptor 23 may include the usage frequency of the shader during pre-compilation. Alternatively, the descriptor 23 may also include GPU hardware information, driver version, and Android system version corresponding to the binary file 22. As shown below Figure 3 As described above, the descriptor 23 can be written into the header of the text file 21 and the binary file 22 to facilitate the subsequent installation process of the shader.

[0025] refer to Figure 3 , which shows a block diagram of the installation module 200 and its operation. The installation module 200 performs the installation of the shader into the operating system of the local device under the following conditions: (1) when the APP 10 is installed to the local device, (2) when the APP 10 is activated for the first time, and (3) when the surface-flinger 11 is activated.

[0026] The installation module 200 includes a selection unit 210, an environment creation unit 220, a pre-compilation unit 230, and a binary acquisition unit 240. The selection unit 210 can obtain a descriptor 23 from a header of a binary file 22 and / or a text file 21. Based on the descriptor 23, the selection unit 210 selects a binary file 22 or a text file 21 to install the shader. In one example, when a binary file 22 corresponding to the shader exists, the binary file 22 can be preferentially selected and directly stored in the scan file path 600. The scan file path 600 can include Figure 1 The binary file 22 may be stored in the first persistent cache 310 and / or the second persistent cache 410 of the shader. For example, when the shader is associated by the APP 10, the binary file 22 may be stored in the scan file path 600 included in the second persistent cache 410. Alternatively, when the shader is associated by the surface-flinger 11, the binary file 22 may be stored in the scan file path 600 included in the first persistent cache 310.

[0027] On the other hand, if no binary file exists, or if the relevant information of the binary file 22 (e.g., the driver version or the Android system version) does not match, the binary file 22 may not be selected. Instead, the selection unit 210 may select the text file 21 to install the shader. The selection unit 210 may first send the text file 21 to the environment creation unit 220, and the environment creation unit 220 is used to provide a rendering context for the shader. The rendering context is, for example, an environment scheme for the shader installation process, not the rendering process.

[0028] In one example using OpenGL (Open Graphics Library), the pre-compile unit 230 is used to take advantage of the "pre-compiler" function provided by the Skia graphics library, which can convert the text file 21 from the SKSL format to the GLSL format through the Skia graphics library interface. Then, the GLSL format is compiled into the binary file 22 by the binary acquisition unit 240. The binary acquisition unit 240 utilizes the "Get-Program-Binary" function provided through the GPU DDK (Driver Development Kit) interface, thereby capturing the compiled binary code from the GPU and generating the binary file 22. Then, the binary acquisition unit 240 stores the compiled binary code to the scan file path 600.

[0029] In another example using the skia-VK (Vulkan implementation) backend, the pre-compile unit 230 is used to take advantage of the "shader compile" function provided by the Skia graphics library, which can convert the text file 21 from the SKSL format to the SPIR-V format. Then, the binary acquisition unit 240 stores the SPIR-V code to the scan file path 600. More specifically, for the case of the skia-VK backend, the generation module 100 can further include a unit of catch pipeline cache (not shown in FIG. 2). When each frame associated with the APP is processed through the skia-VK interface, the catch pipeline cache unit is used to obtain updated information of the pipelineCache by taking advantage of the "vkGetPipelineCacheData" function of the skia-VK backend. Thereafter, the updated information of the pipelineCache can be included in the binary file 22. Figure 3

[0030] In addition, for the case of the skia-VK backend, when the pipelineCache information is included in the binary file 22, the installation module 200 can store such pipelineCache information to the scan file path 600. At the time of operation of the APP 10, the above-mentioned pipelineCache information can be obtained from the scan file path 600 and then loaded through the "vkCreateGraphicsPipelines" function of the skia-VK interface.

[0031] ​The selection unit 210 can partially install the pre-compiled shader based on the frequency of use of the shader to reduce the installation time and save the space of the first persistent cache 310 and the second persistent cache 410. In another example, when the shader is created, the text file 21 and / or the binary file 22 can be integrated into the installation package. The installation package can include information of the text file 21 and the binary file 22 corresponding to various contents. In another example, the binary file 22 can be updated and downloaded from a remote server based on the following information before the APP 10 is installed or used: GPU hardware information, driver version, and Android system version. In another alternative example, the surface-flinger 11 can be directly placed into a directory corresponding to the system image.

[0032] Referring to Figure 4 which shows the internal architecture of the first persistent cache 310 and the second persistent cache 410. In each of the first persistent cache 310 and the second persistent cache 410, there are a scan file path 600, a file cache 710, and a shader cache 720. The file cache 710 is, for example, a "fileBlob" cache.

[0033] When the APP 10 is activated and / or the surface-flinger 11 is activated in the main cache phase, the pre-processing system 1000 can read the file data 30 from the scan file path 600. The file data 30 can include the pre-compiled shader and its text file 21 and binary file 22, and further include updates of other shaders in operation. Each of the first persistent cache 310 and the second persistent cache 410 can be a read only memory (ROM) cache. Therefore, the file data 30 stored in the scan file path 600 can be preserved even when the local device is powered off.

[0034] Then, the file data 30 is loaded into the file cache 710 and the shader cache 720. The file cache 710 has a cache entry list, and the file data 30 can be recorded in the cache entry. The Skia graphics library can look up the cache entry in the file cache 710 to check whether the cache entry records any item corresponding to the key of the shader of interest (i.e., any shader). The cache entry list records the matching relationship of the shader description and the shader file for fast lookup in the memory. The cache entry list is looked up using the shader description for fast data acquisition from the memory of the device. The memory can be a RAM, a DRAM of a SOC, and the like.

[0035] Subsequent queries of the file data 30 can be performed in the file cache 710 and the shader cache 720. Due to the RAM cache fast access characteristics of the file cache 710 and the shader cache 720, the efficiency of the subsequent queries of the file data 30 can be improved.

[0036] At the runtime of the APP 10 and / or the surface-flinger 11, when there are any new shaders other than the pre-compiled shaders, these new shaders can be updated and stored in the shader cache 720. Then, these new shaders can be periodically flushed into the scan file path 600 of the first persistent cache 310 and / or the second persistent cache 410. By this periodic flushing of the shaders, the frequent access to the I / O interface (not shown in the figure) of the local device can be avoided.

[0037] Reference Figure 5 which shows a flowchart of a pre-processing method according to an embodiment of the content of the present application. The pre-processing method can be performed using the pre-processing system 1000 of various embodiments shown in Figures 1-4 which shows a flowchart of a pre-processing method according to an embodiment of the content of the present application. The pre-processing method can be performed using the pre-processing system 1000 of various embodiments shown in

[0038] Then, in step S102, the text file 21 is compiled by the compiling unit 130 to obtain a binary file 22, and then the binary file 22 is directly provided to the installation module 200. The binary file 22 includes a plurality of shader binary codes. Then, in step S104, the descriptor 23 is generated by the statistics unit 122. The descriptor 23 is written into the header of the text file 21 and the binary file 22. The text file 21, the binary file 22 and the descriptor 23 are associated with the shaders used for rendering the pages of the APP 10.

[0039] Then, in step S106, the selection unit 210 determines whether to select the binary file 22 or the text file 21 to store into the scan file path 600 of the first persistent cache 310 and / or the second persistent cache 410 based on the descriptor 23. In one example, step S106 can be triggered when the APP 10 is installed to the local device, when the APP 10 is activated for the first time, or when the surface-flinger 11 is activated. If the binary file 22 is selected to store in step S106, step S108 is performed: the binary file 22 is directly stored into the scan file path 600.

[0040] On the other hand, if the text file 21 is selected to store in step S106, step S110 is performed: a rendering context is provided for the shader. The rendering context is an environment scheme in which the shader is installed, rather than for rendering. Then, in step S112, a pre-compilation is performed on the text file 21. For example, for the case of OpenGL, the text file 21 is converted from the SKSL format to the GLSL format using the “Pre-compiled-Program” function (provided by the Skia graphics library) by the pre-compilation unit 230. Then, in step S114, the binary acquisition unit 240 captures the compiled binary code from the GPU and generates the binary file 22 using the “Get-Program-Binary” function provided by the GPU DDK interface.

[0041] Alternatively, for the case of the skia-VK (Vulkan implementation) backend, in step S112, the pre-compilation unit 230 converts the text file 21 from the SKSL format to the SPIR-V format using the “Shader Compile” function provided by the Skia graphics library. Then, step S116 is performed.

[0042] In step S116, the Skia graphics library updates and flushes the file data 30 (e.g., the binary file 22) to the scan file path 600 of the first persistent cache 310 and / or the second persistent cache 410.

[0043] In some relative examples (not shown in Figures 1-5 In some relative examples (not shown in

[0044] In contrast to the above relative example, in the present summary, the pre- compilation process of the shader is performed in advance. The pre-compilation process can be performed in advance in the early development stage of the "debug mode" or "capture mode" of the APP 10 (e.g., the pre-compilation process can be performed when the APP 10 is first activated). Thus, even if a new shader is encountered that is not recorded in any cache, the compilation of the shader does not block the rendering process.

[0045] The use of ordinal terms such as "first", "second", in the claims to indicate a claim element by the order of recitation is not to imply that a requirement burden is placed on the application as a result of such a recitation. Such a recitation merely identifies a claim element as a "first" or "second" for identification only and does not affect the validity of the claim element nor modify the meaning of any claim element in anyway.

[0046] Various modifications and changes can be made as would be obvious to a person skilled in the art having the benefit of this disclosure. It is intended that the specification and examples be considered as illustrative only, with the true scope of the disclosure being indicated by the following claims, and their equivalents.

Claims

1. A pre-processing system, comprising: a first persistent cache disposed in a rendering engine under Surface-flinger; a second persistent cache disposed in HWUI; a generation module configured to generate an output file including a text file, a binary file and a descriptor associated with a shader used to render a plurality of pages of an application; and an installation module configured to store the text file and the binary file into a scan file path of the first persistent cache and / or the second persistent cache based on the descriptor. The text file is a Skia shader language text file, and the binary file includes a plurality of shader binary codes.

2. The pre-treatment system of claim 1, wherein, The generation module includes:

3. The pre-treatment system of claim 1, wherein, a text generation unit configured to generate the text file and provide the text file to the installation module; and a compilation unit configured to compile the text file to obtain the binary file and provide the binary file to the installation module. The generation module includes:

4. The pre-treatment system of claim 1, wherein, a statistics unit configured to generate the descriptor written into a header of each of the text file and the binary file. The descriptor includes at least a frequency of use of the shader, and hardware information of a graphics processing unit corresponding to the binary file, a driver version and an Android system version.

5. The pre-treatment system of claim 4, wherein, The installation module includes:

6. The pre-treatment system of claim 1, wherein, a selection unit configured to select to store the binary file or the text file into a scan file path of a corresponding persistent cache based on the descriptor. The installation module further includes:

7. The pre-treatment system of claim 6, wherein, an environment creation unit configured to provide a rendering context for the shader when selecting to store the text file into the scan file path, a pre-compilation unit configured to pre-compile the text file; and a binary acquisition unit configured to capture a set of compiled binary codes from a GPU to form the binary file. Further comprising:

8. The pre-treatment system of claim 1, wherein, a file cache configured to record a list of cache entries, wherein the text file and the binary file are read from the scan file path and stored into the file cache; wherein the list of cache entries records a matching relationship between a shader descriptor and a shader file for fast lookup in memory. The generation module is used in a debugging mode or a capturing mode in an early development stage of the application, and is executed when the application is first activated or when a page of the application is first used and causes a change in graphics rendering.

9. The pre-treatment system of claim 1, wherein, The installation module operates when the application is installed to a local device or operates when the application is first activated.

10. The pre-treatment system of claim 1, wherein, 11.A pre-processing method, comprising: generating an output file including a text file, a binary file and a descriptor associated with a shader used to render a plurality of pages of an application; and storing the text file and the binary file into a scan file path of a first persistent cache and / or a second persistent cache based on the descriptor, wherein the first persistent cache is disposed in a rendering engine under Surfaceflinger and the second persistent cache is disposed in HWUI. ​ 12. The pre-treatment method of claim 11, wherein, The text file is a Skia shader language text file, and the binary file includes a plurality of shader binary codes.

13. The pre-treatment method of claim 11, wherein, The steps of generating the text file and the binary file include: generating the text file by a text generation unit and providing the text file; compiling the text file by a compiling unit to obtain the binary file and providing the binary file.

14. The pre-treatment method of claim 11, wherein, The steps of generating the descriptor include: generating the descriptor by a statistics unit; wherein the descriptor is written into a header of each of the text file and the binary file.

15. The pre-treatment method of claim 14, wherein, The descriptor at least includes a usage frequency of the shader, and hardware information of a graphics processing unit corresponding to the binary file, a driver version, and an Android system version.

16. The pretreatment method according to claim 11, wherein: The steps of storing the text file and the binary file into the scan file path include: selecting to store the binary file or the text file into the scan file path of the corresponding persistent cache based on the descriptor.

17. The pre-treatment method of claim 16, wherein, When selecting to store the text file into the scan file path, the pre-processing method further includes: providing a rendering context for the shader, pre-compiling the text file; and capturing a compiled binary code set from a GPU to form the binary file.

18. The pre-treatment method of claim 11, wherein, Further including: reading the text file and the binary file from the scan file path; storing the text file and the binary file into a file cache, wherein the file cache is used to record a cache entry list; and using a shader description to look up the cache entry list.

19. The pre-treatment method of claim 11, wherein, The steps of generating the text file, the binary file, and the descriptor are used in a debugging mode or a capturing mode in an early development stage of the application program, and are executed when the application program is activated for the first time or when a page of the application program is used for the first time and causes a graphics drawing change.

20. The pre-treatment method of claim 11, wherein, The steps of storing the text file and the binary file into the scan file path are executed when the application program is installed to a local device, or are executed when the application program is activated for the first time.

Images