Image processing method and related device and medium program
By pre-configuring and compiling shader code files and Vulkan pipeline construction files, the problem of long startup time for high-performance image applications is solved, and image processing efficiency and user experience are improved.
Patent Information
- Application Number
- CN202410039895.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-10
- Publication Date
- 2025-07-18
AI Technical Summary
The startup time of high-performance image applications is long, which affects the user experience.
Preset the shader code file and compile it to generate the GPU-operable shader machine code, and combine it with the Vulkan pipeline to optimize the graphics processing process.
It shortens the startup time of image applications, improves image processing efficiency, and improves user experience.
Smart Images
Figure CN120335878A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and particularly to an image processing method, related devices, and media programs. Background Art
[0002] Currently, many high-performance image applications are installed in electronic devices. High-performance image applications refer to applications that require efficient rendering, such as game applications, 3D simulation applications, augmented reality applications, virtual reality applications, etc. Currently, many high-performance image applications have the problem of long startup times, which greatly affects the user experience. Summary of the Invention
[0003] Embodiments of this application provide an image processing method, related devices, and media programs, etc., aiming to solve the problem of relatively long startup times of image applications.
[0004] In a first aspect of the embodiments of this application, an image processing method is provided, which can be specifically executed by an electronic device. The image processing method may include: after detecting a first startup instruction for starting a first image application, starting the first image application in response to the first startup instruction; loading a preset first shader code file; compiling the first shader code file to obtain first shader machine code that can be run by a GPU (graphics processing unit); the first image application uses the first shader machine code that can be run by the GPU during the image processing process.
[0005] Among them, compiling to obtain the first shader machine code can be performed in the initialization stage of the application startup.
[0006] It can be seen that in the above example scheme, by presetting the first shader code file, it is possible to obtain the first shader code file more quickly during the startup process of the image application, which is beneficial to improving the efficiency of obtaining the first shader machine code that can be run by the GPU, and thus beneficial to shortening the startup duration of the image application.
[0007] In some possible embodiments, when the electronic device starts the first image application for the first time in response to a first start instruction, a pre-set first shader code file is loaded; the first shader code file is compiled to obtain first shader machine code that can be run by the GPU, and the first shader machine code is serialized to obtain a serialized file of the first shader machine code, and the serialized file of the first shader machine code is stored in the file system. After detecting a second start instruction for restarting the first image application, the first image application is restarted in response to the second start instruction; the serialized file of the first shader machine code is read from the file system into the cache, and the serialized file of the first shader machine code is deserialized to obtain first shader machine code that can be run by the GPU, wherein the restarted first image application uses the first shader machine code that can be run by the GPU during the image processing process.
[0008] Among some possible embodiments, the method further includes: when the graphics library interface selected by the first image application is the Vulkan interface, a pre-set first Vulkan pipeline construction file is loaded; the first Vulkan pipeline construction file is used to generate first Vulkan pipeline cache data; a first Vulkan pipeline is constructed based on the first Vulkan pipeline cache data; the first image application uses the first Vulkan pipeline during the image processing process.
[0009] It can be seen that in the above-exemplified solution, for the scenario where the graphics library interface is the Vulkan interface, a pre-set Vulkan pipeline construction file is further introduced, thereby improving the efficiency of the Vulkan pipeline cache data, and thus being beneficial to further shortening the startup duration of the image application. Among them, the creation of the Vulkan pipeline can be carried out in the initialization stage of the application startup.
[0010] In some possible embodiments, when the electronic device starts the first image application for the first time, if the graphics library interface selected by the first image application is the Vulkan interface, load the pre-set first Vulkan pipeline construction file; generate first Vulkan pipeline cache data using the first Vulkan pipeline construction file; construct a first Vulkan pipeline based on the first Vulkan pipeline cache data; and the first Vulkan pipeline cache data can be serialized to obtain a serialized file of the first Vulkan pipeline cache data, and the serialized file of the first Vulkan pipeline cache data is stored in the file system. After detecting a second start instruction for restarting the first image application, restart the first image application in response to the second start instruction; read the serialized file of the first Vulkan pipeline cache data from the file system into the cache, deserialize the serialized file of the first Vulkan pipeline cache data to obtain the first Vulkan pipeline cache data, construct a first Vulkan pipeline based on the obtained first Vulkan pipeline cache data, and the restarted first image application uses this first Vulkan pipeline during the image processing process.
[0011] In some possible embodiments, the constructing the first Vulkan pipeline based on the first Vulkan pipeline cache data includes: using the first Vulkan pipeline construction file, calling the Vulkan pipeline construction function, and constructing a first Vulkan pipeline based on the first Vulkan pipeline cache data.
[0012] In some possible embodiments, after starting the first image application in response to the first start instruction, it further includes: detecting the graphics library interface supported by the operating system; when it is detected that the operating system supports the Embedded Open Graphics Library (OpenGL ES) interface but does not support the Vulkan interface, the graphics library interface selected by the first image application is the OpenGL ES interface; when it is detected that the operating system supports both the OpenGL ES interface and the Vulkan interface, the graphics library interface selected by the first image application is the Vulkan interface.
[0013] In some possible embodiments, the first Vulkan pipeline construction file includes: pipeline state data and pipeline configuration data required to generate the first Vulkan pipeline cache data.
[0014] In some possible implementation manners, the pipeline configuration data includes one or more of the following configuration data: rendering pass configuration data, resource binding configuration data. Among them, the pipeline state data includes one or more of the following state data: input assembly state data, rasterization state data, color blending state data, depth and stencil test state data, viewport and clipping state data, multisampling state data.
[0015] In some possible implementation manners, the first shader code file is a first intermediate-level shader language code file, and the first intermediate-level shader language code file is obtained by converting a first high-level shader language code file.
[0016] In a second aspect, an embodiment of the present application provides an electronic device, which includes one or more processors and one or more memories; the one or more memories are coupled to the one or more processors, and the one or more memories store computer instructions; when the one or more processors execute the computer instructions, the electronic device executes the method of any one of the above.
[0017] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which includes computer instructions, and when the computer instructions run, the method of any one of the above is executed.
[0018] In a fourth aspect, an embodiment of the present application provides a chip system, which includes a processor and a communication interface; the processor is used to call and run a computer program stored in a storage medium, and execute the method of any one of the above.
[0019] In a fifth aspect, the present application provides an electronic device, including: one or more functional modules. The one or more functional modules are used to execute the method in any possible implementation manner of any one of the above aspects.
[0020] In a sixth aspect, an embodiment of the present application provides a computer program product, wherein when the computer program product runs on a computer, the computer executes the method in any possible implementation manner of any one of the above aspects.
[0021] In addition, for the technical effects brought by the technical solutions of the second aspect to the sixth aspect, reference may be made to the descriptions related to the methods designed in each part of the above method section, and details are not described herein again. Description of the Drawings
[0022] To more clearly illustrate the technical solutions of the embodiments of the present application, the accompanying drawings in the embodiments are briefly introduced below. It should be understood that the following accompanying drawings only show some embodiments of the present application and should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related accompanying drawings can also be obtained based on these drawings.
[0023] Figure 1 A schematic diagram of a process for an electronic device provided in an embodiment of the present application to perform image rendering according to a rendering instruction stream of an application;
[0024] Figure 2A A schematic diagram of the main game scene provided in an embodiment of the present application;
[0025] Figure 2B For Figure 2A A schematic diagram after a game character moves in the main game scene;
[0026] Figure 3 A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application;
[0027] Figure 4 A schematic diagram of the flow of a method for image processing provided in an embodiment of the present application;
[0028] Figure 5 A schematic diagram of the flow of another method for image processing provided in an embodiment of the present application;
[0029] Figure 6 A schematic diagram of the structure of a chip system provided in an embodiment of the present application. Detailed implementation manners
[0030] In this article, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements that are not explicitly listed, or further includes elements inherent in such a process, method, article or device. Without more limitations, an element defined by the statement "including one" does not exclude the existence of another identical element in the process, method, article or device including that element.
[0031] In the present application, "and / or" describes the association relationship of associated objects and indicates that three relationships may exist. For example, A and / or B may represent: A exists alone, both A and B exist simultaneously, and B exists alone. A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of the present application are used to distinguish similar objects, rather than to describe a specific order or sequence.
[0032] The relevant methods provided by the embodiments of the present application can be executed by an electronic device. The electronic device can be a terminal device, which can also be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The terminal device can be a portable mobile device such as a smart phone, computer, smart TV, wearable device, personal digital assistant, augmented reality (AR) / virtual reality (VR) device, media player, etc. The electronic device can also be a vehicle-mounted device, an Internet of Things device, or other devices capable of performing image rendering processing. The electronic device can be a device running the Android system, IOS system, windows system, or other operating systems. Among them, the embodiments of the present application do not specifically limit the type of the electronic device and the operating system run by the electronic device.
[0033] Please refer to Figure 1 for an exemplary introduction to the process of image rendering by the electronic device according to the rendering instruction stream of the application. The electronic device 100 may include a Central Processing Unit (CPU) 101, a graphics processor 102, a memory 103, and a display component 104. The Central Processing Unit 101 is used to run the application program 111 and the operating system 112. The operating system 112 can provide a graphics library and an image synthesis module. The image synthesis module is used for the synthesis of two-dimensional or three-dimensional images. The application program 111 includes game applications, video applications (video players), etc. The rendering pipeline in the graphics processor 102 is a series of operations performed by the graphics processor 102 during the process of rendering graphics and images, and the rendering pipeline includes, but is not limited to, the following operations: vertex processing, primitive processing, rasterization processing, and fragment processing, etc. The memory 103 includes one or more of internal memory (i.e., memory), cache, Video Memory, and external memory.
[0034] Among them, when the application 111 (the application can be abbreviated as an app or APP) needs to render an image, it can issue a rendering instruction stream. Correspondingly, the central processing unit 101 calls the application programming interface (API) of the graphics library according to the rendering instruction stream issued by the application 111. The API of the graphics library can be abbreviated as the graphics library interface to facilitate instructing the graphics processing unit 102 to perform corresponding rendering operations. The graphics library generates an instruction stream recognizable by the rendering pipeline according to the called application programming interface. The graphics processing unit 102 receives the instruction stream sent by the graphics library and performs rendering through the rendering pipeline. The rendering result after the graphics processing unit 102 performs rendering can be stored in the memory 103 (such as video memory) in the electronic device 100. The image composition module of the operating system 112 can obtain the rendering result from the memory 103, and then synthesize the rendering result and display it on the display component 104 (such as a display screen).
[0035] More and more applications, such as game applications or video applications, etc., need to display images with rich scenes on an electronic device. These images are usually obtained by the electronic device based on model rendering. Taking the application as a game application as an example, the main scene of the game application displayed on the display component is as Figure 2A shown. The main scene can include various models, such as character models, rock models, grass models, etc. Among them, in order to provide images with exquisite picture quality, the image is usually rendered at a relatively high shading rate. The number of pixels shaded simultaneously corresponding to a relatively high shading rate is usually less than the number of pixels shaded simultaneously corresponding to a relatively low shading rate. Figure 2B For Figure 2A the situation after the game character moves in the game main scene.
[0036] The software system of the above-mentioned electronic device can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture, etc. Below, taking the Android system with a layered architecture as an example in the embodiments of the present application, the software structure of the electronic device will be exemplarily described. Of course, in other operating systems, as long as the functions implemented by each functional module are similar to those in the embodiments of the present application, the related solutions in the embodiments of the present application can also be implemented.
[0037] Please refer to Figure 3 for an exemplary introduction to the software structure of the electronic device provided in the embodiments of the present application.
[0038] The layered architecture divides the software into several layers. Among them, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely the application layer, the application framework layer, the system library, and the hardware layer.
[0039] The application layer may include a series of application packages. The application packages may include application programs such as game applications and video applications that need to display pictures or videos to users by rendering images.
[0040] The application framework layer can provide application programming interfaces and programming frameworks for the application programs in the application layer. The application framework layer includes some predefined functions.
[0041] Among them, the application framework layer includes a window manager, a content provider, a view system, a resource manager, an activity manager, an input manager, etc. The window manager provides a window management service (Window Manager Service, WMS). The window management service can be used for window management, window animation management, surface management, and as a transfer station for the input system. The content provider is used to store and obtain data and make this data accessible to application programs. The data may include videos, images, audio, etc. The view system includes visual controls, such as controls for displaying text and controls for displaying pictures.
[0042] Among them, the view system can be used to build application programs. The display interface can be composed of one or more views. For example, the display interface including a short message notification icon can include a view for displaying text and a view for displaying pictures. The resource manager provides various resources for application programs, such as localized strings, icons, pictures, layout files, video files, and so on. The activity manager can provide an activity management service (Activity Manager Service, AMS) for managing the life cycles of various application programs and the navigation back function. AMS can be used for the startup, switching, scheduling of system components (such as activities, services, content providers, broadcast receivers, etc.) and the management and scheduling of application processes. Among them, the input manager can provide an input management service (Input Manager Service, IMS). IMS can be used for managing the input of the system, such as touch screen input, key input, sensor input, etc. IMS retrieves events from the input device nodes and distributes the events to appropriate windows through interaction with WMS.
[0043] In the embodiments of the present application, one or more functional modules can be set in the application framework layer, and these one or more functional modules can be used to implement related methods.
[0044] The system library may include multiple functional modules. For example, a graphics library, a surface manager, etc.
[0045] The graphics library is also known as the drawing library. The graphics library is used to define application programming interfaces across programming languages and platforms, which contain many functions for processing graphics (images). Taking the Open Graphics Library (OpenGL) as an example, the APIs defined by OpenGL include interfaces for implementing corresponding functions. For example, interfaces for drawing two-dimensional or three-dimensional images, which include drawing functions such as the drawing function glDrawElements(). Another example is the interface for presenting the images drawn by the drawing function to the display interface (display component). Among them, this interface includes a presentation function, specifically, the function eglSwapBuffers(), and the embodiments of this application will not list them one by one here.
[0046] Among them, the functions in OpenGL can be called through instructions. For example, the drawing function can be called through the instructions in the rendering instruction stream to draw two-dimensional or three-dimensional images. The instructions in the rendering instruction stream are instructions written by developers according to the functions in the graphics library during game application development, and are used to call the graphics library interfaces corresponding to the instructions. The graphics library interfaces include, but are not limited to: interfaces of the OpenGL for embedded system (OpenGL ES), Vulkan interface (a cross-platform drawing application interface), or the khronos platform graphics interface, etc.
[0047] Among them, the surface manager is used to manage the display subsystem and provides the fusion of two-dimensional layers and three-dimensional layers for multiple applications.
[0048] Among them, the hardware layer may include a processor (such as a central processing unit, a graphics processing unit (GPU), etc.), a component with a storage function (such as a memory), and a component with a display function (such as a display component). The relevant content of the central processing unit, the graphics processing unit, and the memory can be referred to above Figure 1 . Among them, the central processing unit and the graphics processing unit in the electronic device can be located on the same chip or can be separate independent chips.
[0049] In some embodiments, the central processing unit can be used to control each module in the application framework layer to implement its respective related functions, while the graphics processing unit can be used to call the graphics library (such as OpenGL ES) interface to perform corresponding rendering processing, etc.
[0050] Some applications require multi-platform compatibility, that is, the application is suitable for running on multiple devices, and each device may support different graphics library APIs (such as graphics library APIs like OpenGL ES or Vulkan). Currently, many high-performance image applications have the problem of long startup time, which greatly affects the user experience. Therefore, some embodiments of this application aim to solve the problem of long application startup duration.
[0051] In a first aspect of the embodiments of this application, an image processing method is provided, which can be specifically executed by an electronic device. The image processing method may include: after detecting a first startup instruction for starting a first image application, starting the first image application in response to the first startup instruction; loading a pre-set first shader code file; compiling the first shader code file to obtain first shader machine code that can be run by a GPU (Graphics Processing Unit); the first image application uses the first shader machine code that can be run by the GPU during the image processing process.
[0052] It can be seen that in the above-exemplified solution, by pre-setting the first shader code file, it is possible to obtain the first shader code file more quickly during the startup process of the image application, which is beneficial to improving the efficiency of obtaining the first shader machine code that can be run by the GPU, and thus beneficial to shortening the startup duration of the image application.
[0053] Next, an example description will be given of the support situation of the operating system for the OpenGL ES interface and the Vulkan interface.
[0054] See Figure 4 , Figure 4 is a schematic flowchart of an image processing method provided by the embodiments of this application. In this embodiment, it is taken as an example that the operating system supports the Embedded Open Graphics Library OpenGL ES interface but does not support the Vulkan interface. An image processing method may include:
[0055] 401. The supply side obtains the first shader code file.
[0056] Among them, the supply side can be a program development server or a cloud file server, etc.
[0057] When the supply side can be a program development server, then the supply side can obtain the first shader code file by, for example, running the first image application for trial.
[0058] Among them, the first shader code file can conform to the SKSL specification or other specifications, and the shader code file that conforms to the SKSL specification can be called an SKSL shader code file.
[0059] The supply side can further analyze, classify, and manage the storage of shader code files. For example, it can parse the shader code files to understand the structure of the shader code files and extract key functional metrics, such as analyzing the functional characteristics of the shader code files, such as based on function names, parameter types, and frequently occurring code patterns, and recording the uses of the shader code files. Classification of shader code files: classify them into different categories based on the functional characteristics of the shader code files. For example, classify the shader code files for processing lighting effects into one category and the shader code files for processing animation effects into another category. Classify the shader code files according to different graphics rendering requirements. A management system can also be established to organize and store these classified shader code files so as to provide corresponding and matching shader code files for different electronic devices according to requirements.
[0060] 402. The supply side provides a first shader code file to the electronic device.
[0061] Among them, after obtaining the first shader code file, the electronic device can store the first shader code file in the file system.
[0062] In some possible implementation manners, the first shader code file obtained by the electronic device may be a first shader intermediate language code file. The supply side can first obtain the first shader high-level language code file and then convert the first shader high-level language code file into the first shader intermediate language code file. The electronic device stores the first shader intermediate language code file in the file system.
[0063] Of course, the first shader code file obtained by the electronic device may also be a first shader high-level language code file. The electronic device converts the first shader high-level language code file into the first shader intermediate language code file and then stores the first shader intermediate language code file in the file system.
[0064] 403. After detecting a first start instruction for starting a first image application, the electronic device starts the first image application in response to the first start instruction. Detect the graphics library interface supported by the operating system. When it is detected that the operating system supports the Embedded Open Graphics Library (OpenGL ES) interface but does not support the Vulkan interface, the graphics library interface selected by the first image application is the OpenGL ES interface.
[0065] 404. The electronic device loads the first shader code file pre-placed in the file system; compiles the first shader code file to obtain the first shader machine code that can be run by the GPU; the first image application uses the first shader machine code that can be run by the GPU during the image processing process.
[0066] 405. The electronic device serializes the compiled first shader machine code to obtain a serialized file of the first shader machine code, and stores the serialized file of the first shader machine code in the file system.
[0067] The electronic device can generate a unique identifier to distinguish serialized files of different shader machine codes. In the specific implementation process, relatively efficient serialization technology can be used to obtain the serialized file of the shader machine code in a format storable by the file system, and an appropriate compression algorithm can also be adopted for file compression. In addition, the file system can intelligently select the best file saving path according to different operating systems and hardware platforms to ensure fast reading and writing. Certain implementation security measures, such as encryption and access control, can also be set to protect the serialized file of the shader machine code from unauthorized access or tampering.
[0068] 406. After detecting the second start instruction for restarting the first image application, the electronic device restarts the first image application in response to the second start instruction.
[0069] The electronic device reads the serialized file of the first shader machine code from the file system into the cache, deserializes the serialized file of the first shader machine code to obtain the first shader machine code that can be run by the GPU, and the restarted first image application uses the first shader machine code that can be run by the GPU during the image processing process.
[0070] It can be seen that in the above-exemplified solution, by presetting the first shader code file, it is possible to obtain the first shader code file more quickly during the startup process of the image application, which is beneficial to improving the efficiency of obtaining the first shader machine code that can be run by the GPU, and thus beneficial to shortening the startup duration of the image application.
[0071] See Figure 5 , Figure 5 is a schematic flowchart of an image processing method provided by an embodiment of the present application. In this embodiment, it is assumed that the operating system supports the Embedded Open Graphics Library OpenGL ES interface and the Vulkan interface. An image processing method may include:
[0072] 501. The supply side obtains the first shader code file and the first Vulkan pipeline construction file.
[0073] Among them, the supply side can be a program development server or a cloud file server, etc.
[0074] When the supply side can be a program development server, then the supply side can obtain the first shader code file and the first Vulkan pipeline construction file by, for example, running the first image application for trial.
[0075] Among them, after obtaining the first shader code file, the electronic device may store the first shader code file in the file system.
[0076] In some possible implementation manners, the first shader code file obtained by the electronic device may be a first intermediate-level shader language code file. The supply side may first obtain the first high-level shader language code file, and then convert the first high-level shader language code file into the first intermediate-level shader language code file. The electronic device stores the first intermediate-level shader language code file in the file system.
[0077] Of course, the first shader code file obtained by the electronic device may also be a first high-level shader language code file. The electronic device converts the first high-level shader language code file into the first intermediate-level shader language code file, and then stores the first intermediate-level shader language code file in the file system.
[0078] In some possible implementation manners, the first Vulkan pipeline construction file includes: pipeline state data and pipeline configuration data required to generate the first Vulkan pipeline cache data.
[0079] In some possible implementation manners, the pipeline configuration data includes one or more of the following configuration data: render pass configuration data, resource binding configuration data. Among them, the pipeline state data includes one or more of the following state data: input assembly state data, rasterization state data, color blending state data, depth and stencil test state data, viewport and clipping state data, multisampling state data.
[0080] The supply side may further analyze, classify, and manage the storage of the shader code file. For example, the shader code file can be parsed to understand the structure of the shader code file, and key function metrics can be extracted, such as analyzing the functional characteristics of the shader code file, such as based on function names, parameter types, and frequently occurring code patterns, and recording the usage of the shader code file. Classification of shader code files: For example, classifying shader code files based on their functional characteristics. For example, classifying shader code files that handle lighting effects into one category and shader code files that handle animation effects into another category. Classify the shader code files according to different graphics rendering requirements. A management system can also be established to organize and store these classified shader code files, so as to provide corresponding matching shader code files for different electronic devices according to requirements.
[0081] 502. The supply side provides the first shader code file and the first Vulkan pipeline construction file to the electronic device.
[0082] Among them, after obtaining the first shader code file and the first Vulkan pipeline construction file, the electronic device stores the first shader code file and the first Vulkan pipeline construction file in the file system.
[0083] 503. After detecting the first start instruction for starting the first image application, the electronic device starts the first image application to respond to the first start instruction. The graphic library interface supported by the operating system is detected. When it is detected that the operating system supports the embedded Open Graphics Library (OpenGL ES) interface and the Vulkan interface, the graphic library interface selected by the first image application is the Vulkan interface.
[0084] 504. The electronic device loads the first shader code file pre-placed in the file system; compiles the first shader code file to obtain the first shader machine code that can be run by the GPU; the first image application uses the first shader machine code that can be run by the GPU during the image processing process.
[0085] The electronic device loads the pre-set first Vulkan pipeline construction file; generates the first Vulkan pipeline cache data using the first Vulkan pipeline construction file; constructs the first Vulkan pipeline based on the first Vulkan pipeline cache data; the first image application uses the first Vulkan pipeline during the image processing process.
[0086] In the process of constructing the first Vulkan pipeline based on the first Vulkan pipeline cache data, the electronic device can first prepare the basic data for pipeline creation: for example, load and create shader modules, which can include various types of shaders such as vertex, fragment, and geometry shaders, and compile GLSL or HLSL shader code into SPIR-V format. Define the pipeline layout, which specifies the uniforms and push constants that the shader will use; create a render pass, which defines the frame buffer attachments, the number of samples, and how to process the content of the rendering operation. Configure the pipeline state, for example, including the input assembly state: define how to read vertices from the vertex buffer and how to combine them into primitives. Rasterization state: set the parameters of the rasterization stage, such as polygon mode (fill, wireframe, etc.), culling mode, and face orientation. Color blending state: define the color blending and alpha blending operations. Depth and stencil test state: if used, configure the depth test and stencil test operations. Viewport and scissor state: set the viewport size and scissor area. Multisampling state: determine the multisampling settings, which are meaningful for anti-aliasing. Then create the Vulkan pipeline (Pipeline): you can use the previously defined settings and call the pipeline creation function of Vulkan (such as the function vkCreateGraphicsPipelines) to create the pipeline.
[0087] 505. The electronic device serializes the compiled first shader machine code to obtain a serialized file of the first shader machine code, and stores the serialized file of the first shader machine code in the file system. The electronic device can serialize the first Vulkan pipeline cache data to obtain a serialized file of the first Vulkan pipeline cache data, and store the serialized file of the first Vulkan pipeline cache data in the file system.
[0088] The electronic device can generate a unique identifier to distinguish different serialized files of shader machine code. In the specific implementation process, a relatively efficient serialization technology can be used to obtain a serialized file of shader machine code in a format that can be stored in the file system, and an appropriate compression algorithm can also be used for file compression. In addition, the file system can intelligently select the best file saving path according to different operating systems and hardware platforms to ensure fast reading and writing. Certain security measures can also be set, such as encryption and access control, to protect the serialized file of shader machine code from unauthorized access or tampering.
[0089] The electronic device can generate a unique identifier to distinguish serialized files of different pipeline cache data. In the specific implementation process, a relatively efficient serialization technology can be used to obtain the serialized file of the pipeline cache data in a storage format that can be stored by the file system, and an appropriate compression algorithm can also be used for file compression. In addition, the file system can intelligently select the best file saving path according to different operating systems and hardware platforms to ensure fast reading and writing. Certain implementation security measures, such as encryption and access control, can also be set to protect the serialized file of the pipeline cache data from unauthorized access or tampering.
[0090] 506. After detecting the second start instruction for restarting the first image application, the electronic device restarts the first image application in response to the second start instruction.
[0091] 507. The electronic device reads the serialized file of the first shader machine code from the file system into the cache, deserializes the serialized file of the first shader machine code to obtain the first shader machine code that can be run by the GPU, and the restarted first image application uses the first shader machine code that can be run by the GPU during the image processing process. The serialized file of the first Vulkan pipeline cache data is read from the file system into the cache, the serialized file of the first Vulkan pipeline cache data is deserialized to obtain the first Vulkan pipeline cache data, and the first Vulkan pipeline is constructed based on the obtained first Vulkan pipeline cache data. The restarted first image application uses this first Vulkan pipeline during the image processing process.
[0092] In the solution of the embodiments of the present application, by presetting shader code files, pipeline creation files, etc., the startup time of the application is significantly shortened, especially for graphics-intensive applications. The need for just-in-time compilation (JIT) during runtime is reduced, which is beneficial to reducing the CPU load and improving the overall performance. A solution that supports both the OpenGL ES interface and the Vulkan interface can be provided, optimizing the cross-platform development process and simplifying the work of writing and optimizing code for different graphics APIs. The pre-compilation and pre-creation mechanisms make resource management more flexible and efficient, and it is easy to adapt to different hardware and software configurations. Because the application responds and loads faster, and at the same time provides smoother graphics rendering performance, the user experience is improved. By reducing the load of runtime compilation, especially on mobile devices, it helps to reduce energy consumption and extend battery life.
[0093] The solutions of the embodiments of the present application can be applied to various APPs to improve relevant performance. For example, in mobile game applications, it is beneficial to provide a smoother game application experience on mobile devices, especially when dealing with complex 3D graphics and animations. Another example is desktop and console games, which is beneficial to optimize graphics rendering on desktops and game consoles, especially suitable for games with high resolution and high frame rate. Another example is virtual reality (VR) applications and augmented reality (AR) applications, which provide higher rendering performance in VR and AR applications, reduce latency, and enhance the immersive experience. Another example is graphic design and rendering applications, which improve the performance of professional graphic design and rendering applications and increase work efficiency. Another example is scientific visualization applications, which are beneficial to provide faster rendering and response times when processing a large amount of data in scientific research and data visualization applications. Simulation and training system applications: In various simulator and training system applications, it is beneficial to provide more realistic visual effects and response speeds.
[0094] It should be noted that all relevant contents of the steps involved in the above method embodiments can be cited in the function descriptions of the corresponding functional modules, and will not be elaborated here.
[0095] The embodiments of the present application also provide a computer program product. When the computer program product runs on a computer, the computer is enabled to execute the above related steps to implement the methods in the above method embodiments.
[0096] The embodiments of the present application also provide a computer storage medium, including computer instructions. When the computer instructions run on an electronic device, the electronic device is enabled to execute the method as in the above embodiments.
[0097] Figure 6 A schematic diagram of the composition of a chip system 600 is shown. The chip system 600 may include: a processor 601 and a communication interface 602, which are used to support related devices to implement the functions involved in the above embodiments.
[0098] In a possible design, the chip system may further include a memory for storing necessary program instructions and data of the electronic device. The chip system may be composed of chips or may include chips and other discrete devices. It should be noted that in some implementation manners of the present application, the communication interface 602 may also be referred to as an interface circuit.
[0099] Among them, the electronic device, computer storage medium, computer program product or chip system provided by the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be elaborated here.
[0100] Among them, through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be allocated to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.
[0101] In several embodiments provided in this application, it should be understood that the disclosed device and method can also be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the module or unit is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in electrical, mechanical or other forms.
[0102] The unit described as a separated component may or may not be physically separated. The component displayed as a unit may be a physical unit or multiple physical units, that is, it can be located in one place, or it can be distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0103] In addition, each functional unit in various embodiments of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.
[0104] If the above integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. The software product is stored in a storage medium and can include several instructions to enable a device (such as a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods in various embodiments of this application. Among them, the aforementioned storage medium can include: various media that can store program codes such as Read-Only Memory (ROM), USB flash drive, mobile hard disk, magnetic disk, optical disc, Random Access Memory (RAM), etc.
[0105] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be covered by the protection scope of the present application.
Claims
1. An image processing method, characterized in that, Including: After detecting a first start instruction for starting a first image application, start the first image application to respond to the first start instruction; Load a preset first shader code file; Compile the first shader code file to obtain first shader machine code that can be run by a GPU; The first image application uses the first shader machine code that can be run by the GPU during the image processing process.
2. The method according to claim 1, characterized in that The method further includes: when the graphics library interface selected by the first image application is the Vulkan interface, load a preset first Vulkan pipeline construction file; generate first Vulkan pipeline cache data using the first Vulkan pipeline construction file; construct a first Vulkan pipeline based on the first Vulkan pipeline cache data; the first image application uses the first Vulkan pipeline during the image processing process.
3. The method according to claim 2, wherein The constructing the first Vulkan pipeline based on the first Vulkan pipeline cache data includes: using the first Vulkan pipeline construction file, calling a Vulkan pipeline construction function, and constructing the first Vulkan pipeline based on the first Vulkan pipeline cache data.
4. The method according to claim 2 or 3, characterized in that, After starting the first image application to respond to the first start instruction, it further includes: Detect the graphics library interface supported by the operating system; When it is detected that the operating system supports the Embedded Open Graphics Library (OpenGL ES) interface but does not support the Vulkan interface, the graphics library interface selected by the first image application is the OpenGL ES interface; When it is detected that the operating system supports the OpenGL ES interface and the Vulkan interface, the graphics library interface selected by the first image application is the Vulkan interface.
5. The method according to any one of claims 2 to 4, wherein The first Vulkan pipeline construction file includes: pipeline state data and pipeline configuration data required for generating the first Vulkan pipeline cache data.
6. The method according to claim 5, wherein The pipeline configuration data includes one or more of the following configuration data: render pass configuration data, resource binding configuration data; Wherein, The pipeline state data includes one or more of the following state data: input assembly state data, rasterization state data, color blending state data, depth and stencil test state data, viewport and clipping state data, multisampling state data.
7. The method according to any one of claims 1 to 6, characterized in that, The first shader code file is a first shader intermediate language code file, wherein the first shader intermediate language code file is obtained by converting a first shader high-level language code file.
8. An electronic device, characterized in that, The electronic device includes one or more processors and one or more memories; the one or more memories are coupled to the one or more processors, and the one or more memories store computer instructions; when the one or more processors execute the computer instructions, the electronic device executes the method according to any one of claims 1 to 7.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that, when run, perform the method according to any one of claims 1 to 7.
10. A chip system, characterized in that, The chip system includes a processor and a communication interface; the processor is configured to call and run a computer program stored in the storage medium to perform the method according to any one of claims 1 to 7.