Interface display method and electronic device

Abstract

Provided in the embodiments of the present application are an interface display method and an electronic device. The electronic device comprises a hardware composer, an image processor and a display screen. The interface display method comprises: before layer drawing and rendering, generating expected layer information; performing pre-validation on the basis of the expected layer information and specification parameters, so as to obtain a pre-validation result; on the basis of the pre-validation result, performing layer drawing and rendering; and performing layer overlay compositing processing, and displaying on the display screen an image obtained after the processing. The embodiments of the present application can reduce the number of additional uses of the image processor, thus improving layer processing efficiency, reducing the processing load of image processors as well as memory and power consumption overheads, and improving performance.

Description

Interface display method and electronic device

[0001] The present application claims priority to the Chinese patent application No. 202411395384.1, filed on September 30, 2024, entitled "Interface display method and electronic device", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of terminal, and in particular to an interface display method and an electronic device. BACKGROUND

[0003] Graphical rendering and display is a process of presenting user operations and application information after algorithm processing on a physical device. The rendering result changes with user operations and application processing logic. In electronic devices, both application and system rendering services have rendering capabilities, which inevitably produce multiple layers. In the case of multiple layers needing to be displayed on the display screen, the multiple layers rendered need to be subjected to superimposition processing by a dedicated hardware composer (HWC) to finally present to the user a physical screen.

[0004] However, the hardware composer needs to verify the to-be-displayed layer information when performing superimposition processing, identify and mark which layers can finally participate in superimposition display, and which layers cannot meet the hardware specification requirements. The layers that cannot participate in the final superimposition display are re-rendered using an image processor, which introduces the overhead of memory, performance and power consumption. SUMMARY

[0005] The embodiments of the present application provide an interface display method and an electronic device, which can improve the layer composition efficiency and reduce the overhead of memory, performance and power consumption.

[0006] In a first aspect, an embodiment of the present application provides an interface display method, the method is applied to an electronic device, the electronic device comprises a hardware compositor, an image processor and a display screen, the method comprises: obtaining a trigger event, the trigger event is used to instruct the electronic device to display a first image through the display screen; in response to the trigger event, generating drawing information, and generating expected layer information based on the drawing information, the expected layer information comprises a plurality of expected layers; pre-verifying the plurality of expected layers based on a specification parameter of the electronic device to obtain a pre-verification result, the pre-verification result comprises instruction information: the instruction information is used to instruct part or all of the plurality of expected layers to be synthesized by the hardware compositor, or to be pre-processed and then synthesized by the hardware compositor, or to be synthesized by the image processor; generating a rendering instruction based on the pre-verification result, and rendering to obtain a superimposed layer set based on the rendering instruction; synthesizing the first image based on the superimposed layer set, and displaying the first image on the display screen.

[0007] Compared with the prior art of directly superimposing the generated layers, the method provided by the embodiment of the present application increases the pre-verification process, and pre-verifies the expected information of the layers to be generated before the layers are generated. The electronic device can obtain a pre-verification result through the pre-verification process, and perform rendering and drawing in advance according to the pre-verification result according to the constraints of the hardware compositor, such as performing multi-layer merging and video layer scaling in advance, so that the layers finally received by the hardware compositor can meet the constraints as much as possible, the number of times of triggering the re-drawing of the layers is reduced, and the memory, performance and power consumption of the rendering and display process are saved. The electronic device can obtain the operation of the user and the system instruction to generate the drawing information, and generate the expected layer information based on the drawing information. The expected layer information enables the electronic device to verify whether the expected layers meet the processing requirements of the hardware compositor based on the expected layer information, without verifying the generated layers and then verifying the generated layers based on the generated layers, so that the superposition verification before superposition can be performed more quickly, and the rendering and display time is saved.

[0008] In a possible implementation of the first aspect, the electronic device further includes a system rendering service, the drawing information includes first drawing information, and the expected layer information includes first expected layer information; and the generating the drawing information and the generating the expected layer information based on the drawing information includes: generating the first drawing information by the system rendering service and generating the first expected layer information based on the first drawing information. By implementing the embodiment of the application, the system rendering service can receive the to-be-drawn information, the user processing information, and the video and game layers from the application, generate the first drawing information in combination with the system rendering service's own dynamic effect processing on the layers, and generate the first expected layer information based on the first drawing information to implement the generation of the first expected layer information by the system rendering service for pre-checking.

[0009] In a possible implementation of the first aspect, the electronic device further includes a pre-checking interface, the pre-checking result includes first pre-checking result, and the pre-checking the plurality of expected layers based on the specification parameter of the electronic device includes: calling the pre-checking interface by the system rendering service, sending the first expected layer information to the hardware compositor, and pre-checking, by the hardware compositor, the plurality of expected layers included in the first expected layer information based on the specification parameter of the electronic device to obtain the first pre-checking result. In the embodiment of the application, the system rendering service and the hardware compositor can complete the pre-checking through cross-process interaction by deploying and calling the pre-checking interface, and the pre-checking process is mainly completed by the hardware compositor. In addition, the pre-checking interface can be independently deployed in the system layer or in the hardware compositor, and can meet the requirements of specific scenarios to complete the pre-checking to obtain the pre-checking result.

[0010] In a possible implementation of the first aspect, the electronic device runs a first application; the drawing information further includes second drawing information, and the expected layer information further includes second expected layer information; and the generating the drawing information and the generating the expected layer information based on the drawing information further includes: generating the second drawing information by the first application and generating the second expected layer information based on the second drawing information. By implementing the embodiment of the application, the first application can respond to the user operation according to the business logic of the first application, such as the up-down sliding logic response of the screen interface when the screen interface is up-down slid, or the starting of the video or music playing when the play button is clicked. At this time, the trigger event includes the user operation, the first application generates the second drawing information based on the user operation and the business logic of the first application, and generates the second expected layer information based on the second drawing information to implement the generation of the second expected layer information by the first application for pre-checking.

[0011] In a possible implementation manner of the first aspect, the electronic device further includes a pre-check interface, and the pre-check result includes a first pre-check result and a second pre-check result; the pre-checking of the plurality of expected layers based on the specification parameter of the electronic device includes: calling the pre-check interface by the system rendering service, and sending first expected layer information to the hardware compositor; calling the pre-check interface by the first application, and sending second expected layer information to the hardware compositor; and pre-checking, by the hardware compositor, a plurality of expected layers included in the first expected layer information and the second expected layer information based on the specification parameter of the electronic device, to obtain the first pre-check result and the second pre-check result. By implementing the embodiment of the application, the hardware compositor can acquire the first expected layer information sent by the system rendering service and the second expected layer information sent by the first application through the pre-check interface, cross-process interaction is implemented, conflicts with other types of data transmission are avoided, stability and reliability are improved, and the hardware compositor can be caused to acquire the expected layer information from the system rendering service and the first application in parallel, so that the energy efficiency advantage of the hardware compositor is more fully utilized, and the calculation efficiency and system performance are improved.

[0012] In a possible implementation manner of the first aspect, the drawing information includes one or more of expected rendering instructions, and a description of a graphic resource, a scene layout, and any rendering-related parameter; and the expected rendering instructions include one or more of clearing a buffer, setting a view and a projection matrix, binding vertex data and a texture, calling a shader program, and drawing a primitive. In the embodiment of the application, when the system rendering service needs to perform operations such as drawing a primitive, the system rendering service can generate the first expected layer information according to these needs, and when the first application needs to perform operations such as setting a view and a projection matrix, binding vertex data and a texture, calling a shader program, and drawing a primitive, the first application can also generate the corresponding second expected layer information according to these needs, so that the size, resolution, transparency value, layer position, texture operation, or specified renderer and other information of the plurality of layers to be generated can be as completely and clearly provided as possible in the expected layer information, and the hardware compositor can be caused to more accurately determine whether the layers to be generated meet the constraints of the hardware compositor based on the information, to obtain an optimal pre-check result.

[0013] In a possible implementation manner of the first aspect, the method further includes: calling, by the hardware compositor, the pre-check interface, and sending the first pre-check result to the system rendering service. In the embodiment of the application, the hardware compositor sends the pre-check result to the system rendering service, to guide the system rendering service to perform layer rendering, so that the generated layers can meet the constraints of the hardware compositor as much as possible, thereby reducing the number of times of triggering redrawing.

[0014] In a possible implementation form of the first aspect, the method further includes: calling, by the hardware compositor, the pre-check interface, sending the first pre-check result to the system rendering service, and sending the second pre-check result to the first application. In the embodiment of the present application, the hardware compositor can send the first pre-check result to the system rendering service and the second pre-check result to the first application in parallel by calling the pre-check interface, to guide the system rendering service and the first application to perform rendering and drawing respectively, thereby improving the computing efficiency and system performance.

[0015] In a possible implementation form of the first aspect, the electronic device further includes a pre-check component deployed based on a specification parameter of the electronic device, and the pre-check result includes a first pre-check result; and the pre-checking the plurality of expected layers based on the specification parameter of the electronic device includes: calling, by the system rendering service, the pre-check component, obtaining the specification parameter of the electronic device, and pre-checking the plurality of expected layers included in the first expected layer information based on the specification parameter of the electronic device to obtain the first pre-check result. In the embodiment of the present application, the pre-check interface and the pre-check component can be deployed in the electronic device at the same time, to realize the selection of calling the pre-check component or the pre-check interface to complete the pre-check according to the requirement; in addition, the pre-check component can be deployed in different modes: the pre-check component can be deployed as part of the hardware compositor or independently in the system, and the pre-check component calling form can be cross-process calling or direct calling in the same process, to meet the needs of different scenarios and architecture forms and to pre-check more flexibly.

[0016] In a possible implementation form of the first aspect, the electronic device further includes a pre-check component deployed based on a specification parameter of the electronic device, and the pre-check result includes a first pre-check result and a second pre-check result; and the pre-checking the plurality of expected layers based on the specification parameter of the electronic device includes: calling, by the system rendering service, the pre-check component, obtaining the specification parameter of the electronic device, and pre-checking the plurality of expected layers included in the first expected layer information based on the specification parameter of the electronic device to obtain the first pre-check result; and calling, by the first application, the pre-check component, obtaining the specification parameter of the electronic device, and pre-checking the plurality of expected layers included in the second expected layer information based on the specification parameter of the electronic device to obtain the second pre-check result. In the embodiment of the present application, the first application and the system rendering service can call the pre-check component in parallel, to realize the pre-check in the first application process and the system rendering service process respectively, without calling the pre-check interface by the first application and the system rendering service respectively, to meet the needs of different scenarios.

[0017] In a possible implementation form of the first aspect, the electronic device further comprises a system rendering service, and the rendering instruction comprises a first rendering instruction; and the generating the rendering instruction based on the pre-check result comprises: the electronic device generating the first rendering instruction based on the first pre-check result by using the system rendering service. By implementing the embodiment of the application, the electronic device can generate the rendering instruction based on the pre-check result by using the system rendering service, and the generated rendering instruction is used to instruct the graphic processor to perform the drawing and rendering of the layer, so that the layer rendered according to the rendering instruction is more consistent with the constraint of the hardware compositor, thereby reducing the number of times of triggering the redrawing.

[0018] In a possible implementation form of the first aspect, the electronic device further comprises a system rendering service, and the electronic device runs a first application, and the rendering instruction comprises a first rendering instruction and a second rendering instruction; and the generating the rendering instruction based on the pre-check result comprises: the electronic device generating the first rendering instruction based on the first pre-check result by using the system rendering service; and the electronic device generating the second rendering instruction based on the second pre-check result by using the first application. By implementing the embodiment of the application, the electronic device can generate the rendering instruction based on the respective pre-check result by using the system rendering service and the first application, and the generated rendering instruction is used to instruct the graphic processor to perform the drawing and rendering of the layer in the respective process, so that the rendering requirement of different rendering architectures can be met; and meanwhile, the layer rendered according to the rendering instruction is more consistent with the constraint of the hardware compositor, thereby reducing the number of times of triggering the redrawing.

[0019] In a possible implementation manner of the first aspect, before the first image is obtained by performing composition processing on the superimposed layer set, the method further includes: sending the superimposed layer set to the system rendering service by the image processor; sending the superimposed layer set to the hardware compositor by the system rendering service; performing superimposition verification on the superimposed layer set by the hardware compositor based on the specification parameter of the electronic device, and the superimposed layer set, specifically including: determining one or more layers in the superimposed layer set that cannot pass the superimposition processing of the hardware compositor as a third layer set by the hardware compositor, and sending the third layer set to the system rendering service; performing superimposition processing on the layers in the superimposed layer set that can pass the superimposition processing of the hardware compositor by the hardware compositor, and caching the layers obtained by the superimposition processing as a cache layer set; and performing re-rendering based on the third layer set by the system rendering service to obtain a re-drawing layer set, and sending the re-drawing layer set to the hardware compositor. By implementing the embodiment of the present application, the superimposed layer set rendered by the image processor can be more consistent with the constraints of the hardware compositor, and when the superimposed layer set is sent to the hardware compositor for superimposition processing, the number of layers in the superimposed layer set that cannot pass the superimposition processing of the hardware compositor can be reduced, the number of times of triggering re-drawing can be reduced, and the efficiency of superimposition display can be improved.

[0020] In a second aspect, the embodiment of the present application provides an electronic device, including: a hardware compositor, an image processor, a display, one or more processors and one or more memories; the one or more memories are coupled with the one or more processors, and the memory is used to store computer program codes, the computer program codes include computer instructions, when the one or more processors execute the computer instructions, the electronic device performs the interface display method in any possible implementation manner of any one of the aspects.

[0021] In a third aspect, the present application provides a computer storage medium, the computer storage medium stores a computer program, and the computer program is executed by a processor to implement the interface display method in any one of the aspects of the first aspect.

[0022] In a fourth aspect, the embodiment of the present application provides a computer program product, the computer program product includes instructions, when the computer program product is executed by a computer, the computer can execute the image display processing method in any one of the aspects of the first aspect.

[0023] In a fifth aspect, the present application provides an interface display processing apparatus having a function of implementing any of the above image display processing methods. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background art, the drawings needed to be used in the embodiments of the present application or the background art will be described below.

[0025] FIG. 1 is a structural schematic diagram of an electronic device according to an embodiment of the present application.

[0026] FIGS. 2A-2E are system architecture schematic diagrams of a group of electronic devices 100 according to an embodiment of the present application.

[0027] FIG. 3 is a schematic flowchart of an interface display method 300 according to an embodiment of the present application.

[0028] FIGS. 4A and 4B are interface schematic diagrams of a group of electronic devices according to an embodiment of the present application.

[0029] FIG. 5 is a schematic diagram of layer drawing and synthesis according to an embodiment of the present application.

[0030] FIG. 6 is an application scenario schematic diagram of an interface display method according to an embodiment of the present application.

[0031] FIG. 7 is a method flowchart of an interface display according to an embodiment of the present application.

[0032] FIG. 8 is a pre-verification flowchart according to an embodiment of the present application.

[0033] FIG. 9 is another method flowchart of an interface display according to an embodiment of the present application.

[0034] FIG. 10 is another pre-verification flowchart according to an embodiment of the present application.

[0035] FIG. 11 is a redrawing flowchart according to an embodiment of the present application.

[0036] FIG. 12 is an application scenario schematic diagram of another interface display method according to an embodiment of the present application. DETAILED DESCRIPTION

[0037] The present application provides an interface display method and an electronic device.

[0038] The method can be applied to an electronic device. The electronic device is a smart terminal device, and embodiments of the present application do not limit the specific type of the smart terminal device. For example, the electronic device can be a mobile phone, and can also include a tablet computer, a desktop computer, a desktop computer, a laptop, a handheld computer, a notebook computer, a wearable device (such as a smart watch, a smart bracelet, etc.), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a car machine, a game machine, and the like.

[0039] In the embodiments of the present application, the terms such as "first", "second", and the like are used to distinguish the same items or similar items with basically the same functions and effects. For example, the first drawing information and the second drawing information are used to distinguish different drawing information, and do not limit the order. Those skilled in the art can understand that the terms such as "first", "second", and the like do not limit the quantity and execution order, and the terms such as "first", "second", and the like do not necessarily mean different.

[0040] The terms used in the following embodiments of the present application are only used to describe specific embodiments, and are not used as a limitation of the present application. As used in the specification and the appended claims of the present application, the singular forms "a", "an", and "the" are intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present application means any or all possible combinations of one or more listed items.

[0041] FIG. 1 shows a structural schematic diagram of an electronic device 100 provided by an embodiment of the present application.

[0042] The electronic device 100 can include a processor 101, a memory 102, a wireless communication module 103, a mobile communication module 104, an antenna 103A, an antenna 104A, a power switch 105, a sensor module 106, a focusing motor 107, a camera 108, a display screen 109, and the like. The sensor module 106 can include a gyroscope sensor 106A, an acceleration sensor 106B, an ambient light sensor 106C, an image sensor 106D, a distance sensor 106E, and the like. The wireless communication module 103 can include a WLAN communication module, a Bluetooth communication module, and the like. The above-mentioned multiple parts can transmit data through a bus.

[0043] The processor 101 can include one or more processing units, for example: the processor 101 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc.

[0044] Among them, GPU, also known as display core, visual processor, display chip, is a kind of microprocessor specially executing image and graphics related operation work, responsible for executing complex mathematical and geometric calculation to render image, video and other graphics content. GPU can perform well in processing a large number of parallel computing tasks through its highly parallel architecture and a large number of operation units, especially in graphics rendering. This design enables GPU to provide tens or even hundreds of times the performance of CPU in floating-point operations, parallel computing and other parts of the calculation. GPU can be an independent device, or can be integrated in the processor 101.

[0045] The processor 101 can also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 101 is a cache memory. The memory can save instructions or data that the processor 101 has just used or repeatedly uses. If the processor 101 needs to use the instructions or data again, it can be directly called from the memory. Avoiding repeated access reduces the waiting time of the processor 101, thus improving the efficiency of the system.

[0046] In some embodiments, the processor 101 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0047] The memory 102 can be used to store computer executable program codes, which can include instructions. The processor 101 performs various functional applications and data processing of the electronic device 100 by running the instructions stored in the memory 102. The memory 102 can include a program storage area and a data storage area. In a specific implementation, the memory 102 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices.

[0048] The wireless communication function of the electronic device 100 can be implemented through the antenna 103A, the antenna 104A, the mobile communication module 104, the wireless communication module 103, the modem processor, and the baseband processor, etc.

[0049] The antenna 103A and the antenna 104A can be used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas.

[0050] The mobile communication module 104 can provide a solution including 2G / 3G / 4G / 5G wireless communication applied to the electronic device 100.

[0051] The wireless communication module 103 can provide solutions for wireless communication including wireless local area networks (WLAN), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc. applied on the electronic device 100.

[0052] The gyro sensor 106A can be used to determine the motion posture of the electronic device 100.

[0053] The acceleration sensor 106B can detect the magnitude of acceleration of the electronic device 100 in various directions (typically three axes).

[0054] The ambient light sensor 106C can sense the intensity of ambient light and automatically adjust the screen brightness.

[0055] The image sensor 106D can capture optical images (including visible light, infrared light, etc.) and convert them into electrical signals for processing, display, or storage.

[0056] The distance sensor 106E can measure the distance between an object and the sensor.

[0057] The electronic device 100 can implement display functions through a GPU, a display screen 109, and an AP, etc. The GPU is a microprocessor for image processing, connected to the display screen 109 and the AP. The processor 101 can include one or more GPUs that execute program instructions to generate or change display information.

[0058] The display screen 109 is configured to display images, videos, and the like. The display screen 109 includes a display panel. The display panel can be a liquid crystal display (LCD). The display panel can also be manufactured by using an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flex light-emitting diode (FLED), a miniled, a microled, a micro-oled, a quantum dot light emitting diodes (QLED), and the like. In some embodiments, the electronic device can include one or N display screens 109, where N is a positive integer greater than 1.

[0059] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In some other embodiments of the present application, the electronic device 100 can include more or fewer components than those shown, or combine certain components, or split certain components, or different arrangement of components. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0060] FIGS. 2A-2E are schematic diagrams of system architectures of a group of electronic devices 100 according to embodiments of the present application.

[0061] As shown in FIG. 2A, the layered architecture divides the system into several layers, each of which has a clear role and division of labor. The layers communicate with each other through a software interface. In some embodiments, the system is divided into five layers, from top to bottom, the application layer, the framework layer, the system library, the hardware abstraction layer, the kernel layer, and the hardware layer.

[0062] The application layer can include a series of application packages. As shown in FIG. 2A, in the embodiments of the present application, the application packages can include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, and the like.

[0063] The application framework layer provides application programming interfaces (APIs) and programming frameworks for application programs of the application layer. The application framework layer includes some pre-defined functions. As shown in FIG. 2A, the application framework layer can include a window manager, an activity manager service (AMS), a content provider, a view system, a resource manager, a notification manager, and the like.

[0064] The window manager is used to manage window programs. The window manager can acquire a display screen size, determine whether there is a status bar, lock a screen, touch a screen, drag a screen, intercept a screen, and the like.

[0065] The activity manager is used to be responsible for managing the start, state, life cycle, and the like of activities of an application. The activity is an application component, which can provide an interface for a user to interact with an electronic device through the interface to complete a task.

[0066] The content provider is used to store and acquire data, and make the data accessible to application programs. The data can include videos, images, audios, dialed and received calls, browsing history and bookmarks, phone books, and the like.

[0067] The view system includes visual controls, such as a control for displaying text, a control for displaying pictures, and the like. The view system can be used to build an application program. A display interface can be composed of one or more views. For example, a display interface including a short message notification icon can include a view for displaying text and a view for displaying pictures.

[0068] The resource manager provides various resources for application programs, such as localized strings, icons, pictures, layout files, video files, and the like.

[0069] The notification manager enables an application program to display notification information in a status bar, which can be used to convey a message of the notification type, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify a download completion, a message reminder, and the like. The notification manager can also be a notification in the form of a chart or a scroll bar text appearing in the top status bar of the system, such as a notification of an application program running in the background, and can also be a notification in the form of a dialog interface appearing on the screen. For example, a text information is prompted in the status bar, a prompt sound is emitted, the electronic device vibrates, an indicator light flashes, and the like.

[0070] The system library can include a plurality of functional modules or components. For example, a system rendering service, media libraries, a surface manager, and the like. In embodiments of the present application, a pre-verification component (not labeled in FIG. 2A) can also be included, which can be deployed according to hardware parameters of the hardware compositor and can be invoked by an application program or the system rendering service.

[0071] Optionally, in addition to being independently deployed in the system library, the pre-verification component can also be deployed in the hardware compositor as part of the hardware compositor.

[0072] The system rendering service is a system-level rendering service responsible for receiving rendering instructions and data from an application program, scheduling and managing rendering tasks, and outputting rendering results to a display screen. The system rendering service invokes the hardware compositor or GPU to compose display layers.

[0073] The system rendering service can invoke a three-dimensional graphics processing library and a two-dimensional graphics engine, among others. The three-dimensional graphics processing library can be an Open Graphics Library (OpenGL), which is used to implement image rendering, among others. The two-dimensional graphics engine is a drawing engine for two-dimensional drawing. OpenGL refers to a professional graphics programming interface that defines a programming interface specification that is cross-programming language and cross-platform. It is used for three-dimensional images (two-dimensional images are also available) and is a powerful, easy-to-use underlying graphics library. OpenGL ES is a subset of the OpenGL three-dimensional graphics API designed for mobile phones, game consoles, and other embedded devices.

[0074] In addition, the surface manager is used to manage the display subsystem and provides fusion of two-dimensional and three-dimensional layers for a plurality of application programs. The media library supports playback and recording of a variety of commonly used audio, video formats, and static image files, among others. The media library can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, and the like.

[0075] The hardware abstraction layer (HAL) includes a hardware compositor and a graphics memory allocator (Gralloc), among others. In embodiments of the present application, the hardware abstraction layer can also include a pre-verification interface (not labeled in FIG. 2A), which can be invoked by the system rendering service, an application program, or the hardware compositor to implement data transmission and cross-process interaction.

[0076] Optionally, in addition to being deployed in the hardware abstraction layer, the pre-verification interface can also be deployed in the hardware compositor.

[0077] The hardware compositor belongs to a special image processing device, which is used to compose the graphic layers (such as application interfaces, windows, animations, videos, etc.) from different application programs and systems into a single image. Since the hardware compositor is generally constructed based on customized hardware units, the customized hardware includes but is not limited to GPU, display controller and other hardware components related to graphic processing. The hardware compositor provides efficient graphic composition capability for the upper layer through interaction with the hardware device, and can support complex graphic operations and efficient image composition. The hardware compositor has obvious energy efficiency advantage compared with GPU.

[0078] The kernel layer is a layer between hardware and software. The kernel layer can include drivers of various hardware. For example, a display driver.

[0079] The hardware layer is the most basic layer in a computer system or an embedded system, which directly involves the existence and operation of physical hardware devices. The hardware layer is the basis for software to run, including all physically touchable, visible computer components, and those key components that are invisible, such as integrated circuits, circuit boards, etc. The hardware layer can include CPU, GPU, display screen, sensor, etc.

[0080] The graphic processing unit (GPU) is mainly used for processing graphic operations, which is the core component of the commonly said "graphics card".

[0081] The CPU can execute task instructions of the window manager, the system rendering service or the application program.

[0082] The possible deployment manners of the pre-check interface and the pre-check component are described below in combination with FIGS. 2B-2E.

[0083] As shown in FIG. 2B, the pre-check component can be deployed in the system library, and the pre-check interface can be deployed in the hardware abstraction layer at this time. As shown in FIG. 2C, the pre-check component can be deployed in the system library, and the pre-check interface can be deployed in the hardware compositor at this time. As shown in FIG. 2D, the pre-check component can be deployed in the hardware compositor, and the pre-check interface can be deployed in the hardware abstraction layer at this time. As shown in FIG. 2E, the pre-check component and the pre-check interface can be deployed in the hardware compositor at the same time.

[0084] It should be noted that the software architecture shown in FIGS. 2A-2E and the deployment manners of the pre-check interface and the pre-check component are only illustratively described, and do not constitute a specific limitation on the electronic device.

[0085] In order to better understand the present application, the following introduces the terms or terms involved in the embodiments of the present application:

[0086] (1) System Rendering Service: The system-level rendering service is a crucial part of the system, responsible for converting graphical information, user interactions, and animation effects from multiple applications into a user-visible interface. This service usually contains multiple components that work together to ensure proper rendering and efficient display of graphical content. For example, in Android, it can include the SurfaceFlinger, and in iOS, it can include the Render Server.

[0087] (2) Graphics Processing Unit (GPU): A GPU is a specialized processor designed for image processing. The core component of a computer system commonly referred to as a "graphics card" is a GPU.

[0088] (3) Color Space: Color space, also known as color space or color model, is a mathematical model used to represent colors. It is the basis for quantifying, qualifying, and quantifying colors, describing various properties of colors through specific parameters and rules. Color space can be used to describe and define colors in a generally accepted manner under certain standards. There are many types of color spaces, each with its own specific application scenarios and advantages. Common color space types include:

[0089] ① RGB Color Space: RGB stands for Red, Green, and Blue. In the RGB color space, colors are generated by combining different intensities of red, green, and blue.

[0090] ② YUV Color Space: YUV color space separates the representation of brightness (Luminance) as Y and chrominance (Chrominance) as U and V, achieving efficient encoding of image colors. Y represents the gray value or brightness information, while U and V represent the chrominance information, describing the color hue and saturation.

[0091] (4) First Application: The first application mentioned in the embodiments of the present application includes third-party applications and system self-contained applications supported by electronic devices. Third-party applications include but are not limited to music playing applications, shopping applications, short video playing applications, social chat applications, etc. System self-contained applications include but are not limited to calls, short messages, cameras, calendars, galleries, WLAN, Bluetooth, etc.

[0092] (5) User Interface (UI): is the medium interface between the application program or operating system and the user for interaction and information exchange, which realizes the conversion between the internal form of information and the form that the user can accept. The user interface is the source code written by specific computer languages such as Java, extensible markup language (XML), etc. The interface source code is parsed, rendered on the electronic device, and finally presented as content that the user can recognize. The commonly used form of user interface is graphic user interface (GUI), which refers to the user interface related to computer operation displayed in a graphical manner. It can be a visual interface element such as text, icon, button, menu, tab, text box, dialog box, status bar, navigation bar, Widget, etc. displayed in the display screen of the electronic device.

[0093] The implementation of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0094] FIG. 3 is a schematic flowchart of an interface display method 300 provided by an embodiment of the present application. The method can be applied to the electronic device 100 shown in FIG. 1, or can be applied to other electronic devices. The embodiments of the present application do not make any limitation here.

[0095] As shown in FIG. 3, the method 300 shown in FIG. 3 can include S301 to S305. The steps in the method 300 will be described in detail below with reference to FIG. 3.

[0096] S301: Obtain a trigger event.

[0097] Specifically, the electronic device obtains a trigger event, which is used to instruct the electronic device to display a first image through the display screen. The trigger event can be an input of the user, such as a click, a pull-down, a pull-up, a long press, a drag, etc. input action on the touch screen using a finger, a click, a pull-down, a drag, etc. input action through a mouse, etc. or a preset logic or external event.

[0098] For example, FIGS. 4A and 4B are interface schematic diagrams of a group of electronic devices provided by an embodiment of the present application. As shown in FIG. 4A, a gallery application is running in the electronic device, and the current interface displays the interface of the picture thumbnail after the gallery is opened. In the gallery application, the user clicks the thumbnail of a picture in the current interface, at which time the click input of the user is the trigger event. The electronic device should display the picture viewing interface shown in FIG. 4B according to the response logic of the gallery application. At this time, the picture viewing interface displayed on the display screen of the electronic device is the first image.

[0099] S302: Generate drawing information, and generate expected layer information based on the drawing information.

[0100] Specifically, the electronic device generates rendering information in response to a trigger event, and generates expected layer information based on the rendering information. The rendering information can include one or more of expected rendering instructions, and descriptions of graphical resources, scene layout, and any rendering-related parameters. The instructions and information contained in the rendering information can be used to obtain an image displayed on the display screen, i.e., a first image, after a certain rendering composition process in response to the trigger event. The expected rendering instructions refer to rendering instructions including information for pre-checking, and are not used to indicate rendering actions. Alternatively, the expected rendering instructions can include one or more of setting a view and a projection matrix, binding vertex data and textures, calling a shader program, and drawing a primitive. The electronic device generates expected layer information based on the rendering information, which can provide as much information as possible about the size, resolution, transparency value, layer position, texture operation, or specified renderer of the multiple layers to be generated in the expected layer information, so that the hardware compositor can more accurately determine whether the layers to be generated meet the constraints of the hardware compositor, i.e., pre-checking, based on this information.

[0101] The expected layer information can include multiple expected layers, and specifically can include:

[0102] (1) Layer content: visual content contained in each layer, such as text, images, graphics, etc.

[0103] (2) Layer attributes: attribute information of each layer, such as position, size, transparency, rotation angle, scaling ratio, etc.

[0104] (3) Drawing instructions: rendering instructions related to each layer, such as drawing path, fill mode, etc.

[0105] (4) Superimposition relationship between layers: the upper and lower relationship between layers, i.e., which layer should be occluded by which layer.

[0106] (5) Animation and interaction information: if a layer contains animation or interactive elements, relevant animation frame information, interactive trigger conditions, etc. also need to be transmitted.

[0107] (6) Synchronization and timing information: when multiple layer contents need to be displayed synchronously, display timestamps or frame numbers of layer contents, etc. need to be transmitted.

[0108] (7) Video format and codec information: format, resolution, frame rate, codec method, etc. of the video.

[0109] S303: Pre-checking based on the specification parameters and the expected layer information to obtain a pre-checking result.

[0110] Specifically, the electronic device performs pre-checking on the expected layers that need to be rendered in the expected layer information based on its own specification parameters before layer rendering, to obtain a pre-checking result, so as to instruct the electronic device to render the layers, so that the generated layers are as consistent as possible with the hardware constraints.

[0111] The specification parameters of the electronic device refer to a series of technical parameters and limitation conditions followed by the electronic device when processing multi-layer superposition, including the capability of the hardware compositor. Optionally, the specification parameters include but are not limited to screen resolution, refresh rate, supported layer format, maximum number of layers, maximum layer size, graphics format and attribute support, GPU model, video memory size, supported technology (such as DirectX, OpenGL version), display memory, GPU, and interface type and speed between other hardware components (such as display, storage device).

[0112] Specifically, the electronic device can traverse all the layers contained in the expected layer information, perform pre-checking based on the specification parameters, mark and give the required processing for the layers that cannot be superimposed and synthesized by the hardware compositor, and collect the marks and processing to generate a pre-checking result.

[0113] The pre-checking result includes: indication information, layers in the expected layer information that cannot be superimposed and synthesized by the hardware compositor, processing required for the layers that cannot be superimposed and synthesized by the hardware compositor, and layers that can be superimposed and synthesized by the hardware compositor.

[0114] The indication information is used to indicate that part or all of the expected layers are processed by the hardware compositor, or are pre-processed and then processed by the hardware compositor, or are processed by the image processor.

[0115] Specifically, the layers that can be superimposed and synthesized by the hardware compositor include two cases: the layers can be directly superimposed and synthesized by the hardware compositor, and the layers need to be pre-processed and then synthesized by the hardware compositor.

[0116] Exemplarily, the pre-processing can include: format conversion (converting the graphics data of the layers into a format supported by the hardware compositor), transparency processing (the hardware compositor may have limited support for layer transparency, and the transparency needs to be adjusted to adapt to the hardware compositor, which can include transparency merging or separation operation), resolution adjustment, color space conversion (different color spaces such as RGB, YUV, etc. can affect the processing efficiency of the hardware compositor), layer simplification (for complex layers, part of the details and unnecessary features can be removed), etc.

[0117] S304: Perform layer rendering based on the pre-checking result.

[0118] Specifically, the electronic device generates rendering instructions based on the generated pre-check result, and performs rendering based on the rendering instructions to obtain the superimposed layer set.

[0119] Optionally, when generating the rendering instructions based on the pre-check result, the electronic device can modify the drawing instructions included in the expected layer information generated in S302 based on the pre-check result; or the electronic device can ignore the drawing instructions included in the expected layer information generated in S302, and regenerate the rendering instructions based on the pre-check result to ensure the accuracy and consistency of the rendering result.

[0120] Specifically, the rendering instructions are used to instruct the electronic device to perform drawing rendering of the layers in response to the trigger event. The rendering instructions include, but are not limited to, clearing the buffer, setting the view and projection matrix, binding vertex data and texture, calling the shader program, drawing the primitive drawing command (such as drawing points, lines, triangles, etc.), texture operation (loading texture image and sampling), and blending operation (controlling the blending mode between different objects).

[0121] The electronic device can perform drawing rendering of the layers based on the rendering instructions to obtain the superimposed layer set, which includes a plurality of layers. Through certain superimposition processing, the complete interface image displayed on the display screen can be obtained.

[0122] S305: Synthesizing the superimposed layers to obtain a first image, and displaying the first image on the display screen.

[0123] Specifically, the electronic device performs synthesis processing based on the superimposed layer set obtained by rendering. The synthesis processing process is mainly completed by a hardware synthesizer. The superimposed layer set generated by pre-checking contains layers that can better meet the constraints of the hardware synthesizer, and the synthesis processing can be directly completed by the hardware synthesizer to obtain the first image as much as possible, reducing the use of GPU and fully exerting the energy efficiency advantage of the hardware synthesizer.

[0124] Finally, the first image is displayed on the display screen, where the first image refers to the complete interface image displayed on the display screen and finally displayed to the user through the display screen of the electronic device.

[0125] FIG. 5 is a schematic diagram of layer drawing and synthesis according to an embodiment of the present application. As shown in FIG. 5, taking the original Android launcher (Launcher) as an example, the interface drawing can include drawing of four layers, namely, the status bar, the navigation bar, the application interface, and the launcher icon layer. Layer 1, layer 2, layer 3, and layer 4 constitute a superimposed layer set, and the first image is obtained through synthesis processing. The first image can be displayed by the display screen.

[0126] The application interface can be provided by an application service, the launcher icon layer can be drawn by a Launcher application, and the layers can be completed by a system UI thread of the user interface.

[0127] FIG. 6 is a schematic diagram of an application scenario of an interface display method according to an embodiment of the present application.

[0128] The application scenario of the method provided by the embodiments of the present application mainly involves an interaction process between an application program and a physical display screen. Specifically, it is a process of information transmission and processing from software to hardware, and the ultimate goal is to present the content required by the user on the physical display screen.

[0129] Specifically, referring to FIG. 6, first, the user operates the application program, and the electronic device generates or inputs information to be displayed, including various layer data and instructions. These layer data and instructions are basic elements for constituting the final display content, and the display page can be updated after processing by the electronic device to respond to the user operation;

[0130] Further, the information is transmitted to the system service layer, especially the system rendering service. The system rendering service is a key link responsible for converting the layer data and instructions provided by the application program into a format that can be understood and processed by hardware. In this step, the system will synthesize, optimize, and process the layer data according to the preset rendering algorithm and rules to ensure that the final display effect meets the expectation;

[0131] Further, the processed layer data and instructions are then sent to the hardware compositor. The hardware compositor is a key component connecting software and hardware, which is responsible for further converting the data and instructions from the system service layer into instructions that can be directly executed by hardware. It ensures the compatibility and accuracy of the data and instructions;

[0132] Finally, the data and instructions processed by the hardware compositor are converted into to-be-displayed content and sent to the physical display screen for display. In this process, the physical display screen controls the light emission and color of the pixels according to the received data and instructions, thereby presenting the visual effect expected by the user on the screen.

[0133] The application scenario of the method provided by the embodiments of the present application widely exists in various scenarios that need to interact with the physical display screen through the application program, including but not limited to smart phones, tablet computers, notebook computers, televisions, large screens, projectors, smart watches, vehicle display screens, and other devices with screen display. By applying the method provided by the embodiments of the present application, the energy efficiency advantage of using the hardware compositor can be more reasonably realized, the rendering overhead of the whole process is reduced, and a more smooth and realistic visual experience is provided for the user.

[0134] The interface display method provided by the embodiment of the present application will be introduced in detail below in combination with FIG. 7. FIG. 7 is a flowchart of a method of interface display provided by an embodiment of the present application. The method can be applied to the electronic device 100 shown in FIG. 1, or can be applied to other electronic devices, which are not limited herein by the embodiment of the present application. The electronic device can include various devices and components in FIG. 2A, and the description of the various devices and components can refer to the related description of the above-mentioned FIGS. 2A-2E and the related description of the terms and names involved in the embodiment of the present application, which will not be repeated here.

[0135] As shown in FIG. 7, the method of interface display provided by the embodiment of the present application can include but is not limited to the following steps:

[0136] S701: The first application sends rendering input data to the system rendering service.

[0137] Specifically, first, the first application obtains rendering input data by triggering an event. The specific process of obtaining the triggering event can refer to the related description of S301 above, which will not be repeated here.

[0138] The rendering input data refers to a collection of various types of information generated by each application process for graphics rendering and user interface updating. The rendering input data includes but is not limited to the following:

[0139] (1) Layout data:

[0140] ① Element position and size: specifies the position and size of each element (such as buttons, text boxes, pictures, etc.) on the interface;

[0141] ② Layout structure: defines the hierarchical relationship and arrangement method between elements, such as grid layout, stacking layout, etc.

[0142] (2) Visual style data: including color, font, margin, border, shadow, etc., which determines the appearance and style of the interface elements.

[0143] (3) Interaction behavior data:

[0144] ① Callback function: registers the processing function of user interaction events (such as click, slide, long press, etc.), which defines how the application program should respond when the event occurs;

[0145] ② Animation and transition effect: specifies the animation or transition effect that the element should show when the state changes, such as fade-in and fade-out, sliding, etc.

[0146] (4) Content data:

[0147] ① Text content: the text information displayed on the interface, such as title, description, prompt, etc.

[0148] ② Image and media resources: including pictures, videos, audio and other multimedia resources, which need to be displayed to users on the interface.

[0149] Secondly, the first application sends the rendering input data to the system-level rendering service through the API provided by the system. Before sending, the first application can encapsulate the rendering input data into a specific format.

[0150] S702: The system rendering service generates first drawing information, and generates first expected layer information based on the first drawing information.

[0151] Specifically, the system rendering service generates the first drawing information based on the obtained rendering input data and its own rendering data. The system rendering service's own rendering data includes but is not limited to the content contained in the rendering input data.

[0152] The first drawing information includes one or more of expected rendering instructions, descriptions of graphic resources, scene layout, and any rendering-related parameters; the expected rendering instructions include one or more of clearing the buffer, setting the view and projection matrix, binding vertex data and texture, calling the shader program, and drawing the primitive.

[0153] Secondly, the system rendering service generates first expected layer information based on the generated first and drawing information. The first expected layer information includes multiple layers that need to be rendered and drawn by the system rendering service and the system, and the specific content that the expected layer information can include can refer to the related description of the expected layer information in S302, which will not be repeated here.

[0154] S703: The system rendering service sends the first expected layer information to the hardware compositor.

[0155] Specifically, as shown in FIGS. 2A-2E, the electronic device defines a pre-check interface, which is an interface for realizing data interaction between the system rendering service or the application program and the hardware compositor based on a system cross-process interaction mechanism such as Hardware Interface Definition Language (HIDL), and the data includes the first expected layer information, the second expected layer information, the first pre-check result and the second pre-check result. The system rendering service calls the pre-check interface to send the first expected layer information to the hardware compositor.

[0156] Optionally, the pre-check interface can be defined independently in the system layer or other positions of the software architecture, or can be defined in the hardware compositor.

[0157] Exemplarily, first, the system rendering service loads a library file containing a pre-check interface, and then configures parameters of the pre-check interface according to specific requirements of the hardware compositor and the system, such as a communication protocol, a data format, a security setting, and the like; then, the system rendering service encapsulates first expected layer information according to requirements of the pre-check interface; and the system rendering service constructs a request containing the encapsulated first expected layer information based on a communication mechanism (such as IPC, network communication, and the like), and sends the constructed request to the hardware compositor.

[0158] In the cross-process communication process, the first expected layer information can be serialized into a transmittable format, such as a protocol buffer (Protocol Buffers), a parcel (Parcel), and the like, and deserialized at a receiving end to restore an original data structure.

[0159] S704: The hardware compositor performs pre-checking on the expected layer based on the specification parameters of the electronic device, to obtain a first pre-checking result.

[0160] Specifically, the electronic device can define a pre-checking interface based on a system HIDL or the like system cross-process interaction mechanism. First, the hardware compositor calls the pre-checking interface to receive the first expected layer information delivered from the system rendering service.

[0161] Exemplarily, the hardware compositor first establishes a communication connection with the system rendering service; second, the hardware compositor sends a request to the system rendering service through the pre-checking interface, to inform the system rendering service to prepare to send the first expected layer information; the system rendering service responds to the request, and sends the first expected layer information to the hardware compositor through the pre-checking interface; and the hardware compositor parses the data and stores it into an internal data structure after receiving the data.

[0162] Optionally, if asynchronous message passing is used, the hardware compositor registers a callback function to receive the data; and if synchronous calling is used, the hardware compositor directly waits for the pre-checking interface method to return the first expected layer information.

[0163] Specifically, after receiving the first expected layer information, the hardware compositor performs pre-checking based on the first expected layer information. The pre-checking result includes results of whether each expected layer in the plurality of expected layers is processed by the hardware compositor or by the image processor, which can be referred to the related description of the pre-checking result in S303, and will not be described herein.

[0164] Exemplarily, the hardware compositor queries the specification parameters of the current hardware, including but not limited to screen resolution, refresh rate, supported layer format, maximum number of layers, maximum layer size, GPU model, GPU memory size, supported technology (such as DirectX, OpenGL version), display memory, interface type and speed between GPU and other hardware components (such as display, storage device), which can refer to the related description of the specification parameters of the electronic device itself in S303.

[0165] Specifically, the hardware compositor determines whether each layer in the first expected layer information can be processed by the hardware compositor for superimposed composition, and if not, marks the layer that needs GPU further processing and how to process; the hardware compositor can also check whether the dependency relationship between the layers is correct, for example, some layers may depend on the rendering result of other layers, and if the dependency relationship is incorrect or missing, it may cause incorrect rendering result.

[0166] Further, the hardware compositor traverses the layers included in the first expected layer information, collects the marks in the pre-checking process and the layers that cannot be processed by the hardware compositor for superimposed composition and the required processing to generate the first pre-checking result. The content of the first pre-checking result can refer to the related description of the pre-checking result in S303, which is not repeated here. By deploying and calling the pre-checking interface, the system rendering service and the hardware compositor can complete the pre-checking through cross-process interaction, and the pre-checking process is mainly completed by the hardware compositor. In addition, the pre-checking interface can be independently deployed in the system layer, and can also be deployed in the hardware compositor, which can meet the needs of specific scenarios to complete the pre-checking to obtain the pre-checking result.

[0167] S705: The hardware compositor calls the pre-checking interface and sends the first pre-checking result to the system rendering service.

[0168] Specifically, the hardware compositor initiates cross-process interaction to the system rendering service through the HIDL client agent based on the first pre-checking result, and sends the prepared first pre-checking result.

[0169] S706: The system rendering service generates the first rendering instruction based on the first pre-checking result.

[0170] The system rendering service generates the first rendering instruction based on the first pre-checking result after obtaining the first pre-checking result through the pre-checking interface.

[0171] Specifically, the system rendering service can cache S702 the first expected layer information generated by the system rendering service, the first expected layer information including expected rendering instructions, which can instruct the rendering of layers to generate a first image in response to a trigger event, and the first pre-check result can include modifications to the properties of some layers in the first expected layer. The system rendering service can modify the rendering instructions in the cached first expected layer information based on the first pre-check result without the need to regenerate, so that only part of the instructions need to be adjusted without the need to recalculate the entire rendering process, which can reduce the waste of computing resources.

[0172] In addition, the system rendering service can ignore or discard the drawing instructions included in the first expected layer information and regenerate the rendering instructions based on the first pre-check result, which can ensure the accuracy and consistency of the rendering result.

[0173] The first rendering instructions include, but are not limited to, clearing a buffer, setting a view and projection matrix, binding vertex data and textures, calling a shader program, drawing primitive drawing commands (such as drawing points, lines, triangles, etc.), texture operations (loading a texture image and sampling it), and blending operations (controlling the blending mode between different objects).

[0174] S707: The system rendering service sends the first rendering instructions to the GPU.

[0175] Specifically, the first rendering instructions can be stored in an instruction buffer. The buffer is a shared area between the CPU and the GPU, used to temporarily store rendering instructions to be executed, and the GPU reads the rendering instructions from the instruction buffer.

[0176] For example, the first rendering instructions can be transmitted in a push mode, in which the system rendering service directly writes the first rendering instructions into the registers or a specific memory area of the GPU, and the GPU reads and executes the instructions from the area. In addition, the first rendering instructions can be transmitted in a request (push) mode, in which the system rendering service allocates a block of memory as a storage area for the first rendering instructions when the system is initialized, and when rendering is needed, the system rendering service writes the first rendering instructions into the block of memory and notifies the GPU to read and execute.

[0177] S708: The GPU performs rendering based on the first rendering instructions to obtain a set of superimposed layers.

[0178] The GPU performs rendering based on the obtained first rendering instructions to obtain a set of superimposed layers, which includes a plurality of layers.

[0179] Specifically, the GPU reads the first rendering instruction from the instruction buffer and parses it. During the parsing process, the GPU identifies different rendering operations, such as vertex processing, rasterization, fragment processing, etc. According to the parsed instruction, the GPU performs the corresponding rendering operation to obtain the overlay layer set.

[0180] Illustratively, the GPU can store the rendering obtained overlay layer set in a frame buffer. The frame buffer is a temporary storage area for storing image data after rendering is completed.

[0181] S709: The GPU sends the overlay layer set to the system rendering service.

[0182] Optionally, the electronic device can use a video memory sharing mechanism to realize data transmission between the GPU and the system rendering service. Among them, the overlay layer set sent by the GPU to the system rendering service includes the data of the plurality of layers included in the overlay layer set rendered by the GPU in S708.

[0183] S710: The system rendering service sends the overlay layer set to the hardware compositor.

[0184] Specifically, the system rendering service can perform data transmission to the hardware compositor through a specific interface. The interface can be a system-level API, or a hardware abstraction layer (HAL) based interface. Through the interface, the system rendering service can directly transmit layer data to the hardware compositor without additional data conversion or copying process.

[0185] S711: The hardware compositor performs overlay verification on the overlay layer set based on the specification parameters of the electronic device, and determines whether it contains layers that cannot be processed by the hardware compositor. If it contains, execute S712; if it does not contain, execute S714.

[0186] Specifically, the hardware and compositor performs overlay verification on each layer included in the overlay layer set based on the specification parameters of the electronic device. The specification parameters of the electronic device can refer to the related description of the specification parameters of the hardware in S704, which will not be repeated here.

[0187] S712: The hardware compositor determines the third layer set and sends the third layer set to the system rendering service.

[0188] Specifically, the hardware compositor obtains the overlay layer set and traverses each layer in the overlay layer set to determine the third layer set by collecting all layers in the overlay layer set that cannot be processed by the hardware compositor.

[0189] S713: The system rendering service triggers a redraw process to obtain a redraw layer set and sends the redraw layer set to the hardware compositor.

[0190] Specifically, the system rendering service performs a redraw based on the third layer set to obtain a redrawn layer set.

[0191] S714: The hardware compositor performs superimposition processing to obtain the first image.

[0192] The hardware compositor performs superimposition processing based on the obtained layers to obtain the first image. The first image can be referred to the related description in S305, which is not repeated here.

[0193] The specific process of S712-S714 can be referred to the related description of FIG. 11, which is not repeated here.

[0194] S715: The hardware compositor sends the first image to the display screen.

[0195] Specifically, before the hardware compositor sends the first image to the display screen, the hardware compositor can further process and optimize the first image to improve the display effect, including but not limited to color correction, brightness adjustment, contrast enhancement, etc. Secondly, the hardware compositor establishes a connection with the display screen through a specific interface, and the type of the interface depends on the specific type and specifications of the display screen. Finally, the hardware compositor sends the first image to the display screen through the specific interface.

[0196] If the data format of the first image does not match the requirements of the display screen, the hardware compositor can perform necessary data format conversion.

[0197] S716: The display screen displays the first image on the display screen.

[0198] Specifically, first, the display screen parses and verifies the first image after obtaining the first image. Secondly, the processing unit inside the display screen can further process the first image, including but not limited to scaling, rotating, cropping, etc. Finally, the display screen displays the processed first image on the display screen for the user to view.

[0199] In the steps included in the above FIG. 7, the interface drawing, interface rendering, interface composition, and display sending processes can be triggered by a vertical synchronization (VSYNC) signal. The triggering period of the VSYNC signal is the refresh rate of the screen.

[0200] The pre-checking process in the interface display method shown in FIG. 7 is described as follows. The first application sends rendering input data to the system rendering service, that is, the first application sends various information generated by the first application process for graphic rendering and user interface update to the system rendering service, the system rendering service collects and generates expected layer information, and interacts with the hardware compositor to perform pre-checking, so as to realize pre-checking before layer rendering and display.

[0201] For the pre-checking process in the interface display method shown in FIG. 7, another pre-checking process suitable for a separate rendering architecture is provided in an embodiment of the present application. Referring to FIG. 8, FIG. 8 is a pre-checking process diagram provided by an embodiment of the present application. As shown in FIG. 8, the pre-checking process provided by an embodiment of the present application can include the following steps:

[0202] S801: The first application sends rendering input data to the system rendering service.

[0203] Optionally, when the first application selects to send layer data that needs to be rendered to the system rendering service to uniformly perform rendering in the system rendering service process, the first application can send rendering input data to the system rendering service.

[0204] S802: The system rendering service generates first drawing information, and generates first expected layer information based on the first drawing information.

[0205] S803: The system rendering service sends the first expected layer information to the hardware compositor.

[0206] Specifically, the description of S801-S803 can refer to the related description of S701-S703, which is not repeated here.

[0207] S804: The first application generates second drawing information, and generates second expected layer information based on the second drawing information.

[0208] Optionally, when the first application needs to perform rendering of some layers in the first application process, the first application can generate second drawing information based on user operation or response logic of the first application.

[0209] The second drawing information includes one or more of expected rendering instructions, description of graphic resources, scene layout, and any rendering-related parameters; the expected rendering instructions include one or more of clearing a buffer, setting a view and a projection matrix, binding vertex data and a texture, calling a shader program, and drawing a primitive.

[0210] Specifically, the first application generates second expected layer information based on the second drawing information for pre-checking. The second expected layer information includes a plurality of layers that the first application needs to render drawing, and the second expected layer information can also include layer information of the plurality of layers and superimposed synthesis order between the plurality of layers.

[0211] S805: The first application sends the second expected layer information to the hardware compositor.

[0212] Specifically, a pre-checking interface is defined in the electronic device, which can refer to the related description of the pre-checking interface in S703, which will not be repeated here. The first application processes the second expected layer information, calls the pre-checking interface, and sends the processed second expected layer information to the hardware compositor. The specific process can refer to the related description of the system rendering service calling the pre-checking interface to send the first expected layer information in S703, which will not be repeated here.

[0213] The processing of the second expected layer information by the first application involves serialization of data, and the deserialization of data at the hardware compositor end.

[0214] Specifically, S801-S805 above can be executed in parallel. It can be understood that the numerical designations S801-S805 are only used to distinguish different steps, and do not limit the execution order.

[0215] S806: The hardware compositor pre-checks the expected layer based on the specification parameters of the electronic device to obtain a pre-checking result.

[0216] Specifically, the hardware compositor pre-checks the first expected layer information and the second expected layer information obtained based on the specification parameters of the electronic device. First, the electronic device queries the specification parameters of the current hardware, which can refer to the description of querying the specification parameters of the current hardware in S704, which will not be repeated here. The hardware compositor pre-checks the first expected layer information to obtain a first pre-checking result, and pre-checks the second expected layer information to obtain a second pre-checking result.

[0217] Exemplarily, the pre-checking interface can be defined in the electronic device based on a system HIDL or other system cross-process interaction mechanism. The hardware compositor first calls the pre-checking interface to receive the first expected layer information transmitted from the system rendering service and the second expected layer information from the first application.

[0218] The pre-checking process of the first expected layer information by the hardware compositor can refer to the related description of S704 in the above FIG. 7, which will not be repeated here.

[0219] Specifically, the pre-checking process of the hardware compositor on the second expected layer information can include: the hardware compositor first establishes a communication connection with the first application; secondly, the hardware compositor sends a request to the first application through the pre-checking interface; the first application responds to the request and sends the second expected layer information to the hardware compositor through the pre-checking interface; after receiving the data, the hardware compositor parses and stores it in the internal data structure; then, the hardware compositor pre-checks the second expected layer information based on the queried hardware specification parameters; finally, the hardware compositor traverses all the layers included in the second expected layer information that participate in the pre-checking to determine the second pre-checking result. The content of the second pre-checking result can refer to the related description of the pre-checking result in S303, which will not be repeated here.

[0220] The specific process in which the hardware compositor checks the second expected layer information based on the queried hardware specification parameters can refer to the related description of the pre-checking process of the first expected layer information in S704 of FIG. 7, which will not be repeated here.

[0221] S807: The hardware compositor sends the first pre-checking result to the system rendering service through the pre-checking interface.

[0222] S808: The system rendering service generates the first rendering instruction based on the first pre-checking result.

[0223] S809: The system rendering service sends the first rendering instruction to the GPU.

[0224] Specifically, S807-S809 can refer to the related description of S705-S707 in FIG. 7, which will not be repeated here.

[0225] S810: The hardware compositor sends the second pre-checking result to the first application through the pre-checking interface.

[0226] Specifically, the pre-checking interface can be defined in the electronic device based on a system HIDL or other system cross-process interaction mechanism. The hardware compositor can initiate a cross-process call to the pre-checking interface through the HIDL client agent provided by the system, and pass the prepared second pre-checking result to indicate the first application to draw and render the layer in the process.

[0227] In the cross-process communication process, the second expected layer information can also be serialized into a transmittable format such as Protocol Buffers, Parcel, etc., and deserialized at the receiving end to restore the original data structure.

[0228] S811: The first application generates the second rendering instruction based on the second pre-checking result.

[0229] Specifically, the first application obtains the second pre-check result through the pre-check interface, and generates the second rendering instruction based on the second pre-check result within the application process. The first application can construct a user interface (UI) element through its graphical interface framework, such as the view system (View) of Android or the user interface framework (UIKit) of iOS; then, the UI element is converted into a call request of a graphics API, such as OpenGL ES, Vulkan, etc., to generate the second rendering instruction, which describes the properties (such as position, color, texture, etc.) and rendering state (such as lighting, shadow, blending mode, etc.) of a graphics object, and is used to instruct the GPU to render a layer to respond to the trigger event; the first application encapsulates and passes down the second rendering instruction.

[0230] Exemplarily, generating the second rendering instruction can include that the first application can cache the second expected layer information generated by the first application S804, the second expected layer information including expected rendering instructions that can instruct the rendering layer to generate the first image in response to the trigger event, and the second pre-check result can include modifications to the properties of some layers in the second expected layer; the first application can modify the rendering instructions in the cached second expected layer information based on the second pre-check result without the need to regenerate, so that only part of the instructions need to be adjusted without the need to recalculate the entire rendering process, which can reduce the waste of computing resources.

[0231] In addition, the first application can ignore or discard the rendering instructions included in the second expected layer information, and regenerate the rendering instructions based on the second pre-check result, which can ensure the accuracy and consistency of the rendering result.

[0232] The second rendering instruction includes but is not limited to clearing the buffer, setting the view and projection matrix, binding the vertex data and texture, calling the shader program, drawing the primitive drawing command (such as drawing points, lines, triangles, etc.), texture operation (loading the texture image and sampling it), and blending operation (controlling the blending mode between different objects).

[0233] S812: The first application sends the second rendering instruction to the GPU.

[0234] Specifically, the first application can submit the second rendering instruction to the command buffer, and the driver program decodes and optimizes the second rendering instruction, and sends it to the GPU.

[0235] Exemplarily, the second rendering instruction delivery can adopt a push mode, in which the first application writes the second rendering instruction into a register or a specific memory area of the GPU, and the GPU reads and executes the instruction from the area; in addition, the second rendering instruction delivery can adopt a request (push) mode, in which the system rendering service allocates a piece of memory as a storage area of the second rendering instruction when the system is initialized, writes the second rendering instruction into the piece of memory when rendering is needed, and notifies the GPU to read and execute.

[0236] S813: The GPU performs rendering based on the rendering instruction to obtain the superimposed layer set.

[0237] Specifically, the GPU can adopt a parallel processing mechanism to process the first rendering instruction from the system rendering service and the second rendering instruction from the first application.

[0238] Exemplarily, when the first application selects to send the layer data that needs to be rendered to the system rendering service, and the rendering is uniformly performed in the system rendering service process, the GPU can perform rendering based on the first rendering instruction sent by the system rendering service to obtain the superimposed layer set; when the first application needs to perform rendering of some layers in the process itself, the GPU can also receive the second rendering instruction sent by the first application, and can receive the first rendering instruction sent by the system rendering service in parallel, and perform rendering based on the first rendering instruction and the second rendering instruction to obtain the superimposed layer set.

[0239] Specifically, the GPU storing the superimposed layer set can refer to the related description of S708 in FIG. 7 above, which is not repeated here.

[0240] S814: The GPU sends the superimposed layer set to the system rendering service.

[0241] Specifically, the sending of the superimposed layer set can refer to the related description of S709 in FIG. 7 above, which is not repeated here.

[0242] As shown in FIGS. 7 and 8 above, the electronic device can realize interface display through a pre-check interface, and at this time, the pre-check mechanism runs in the hardware compositor.

[0243] The electronic device further includes a pre-check component. Optionally, the electronic device can also complete interface display by calling the pre-check component.

[0244] Optionally, the pre-check component can be independently deployed in a system library, or can be deployed in the hardware compositor as a part of the hardware compositor. The process in which the electronic device completes pre-check by calling the pre-check component is explained in combination with FIGS. 9 and 10.

[0245] It should be noted that the pre-check component and the hardware compositor are not shown in the same block in FIG. 9 and FIG. 10, which does not mean that the pre-check component must be deployed independently outside the hardware compositor, but only illustrates the need for the process.

[0246] First, a method suitable for a unified rendering architecture and realizing interface display by calling a pre-check component is introduced in combination with FIG. 9. FIG. 9 is a flowchart of another method of interface display provided by an embodiment of the present application. As shown in FIG. 9, the method of interface display provided by the embodiment of the present application can further include the following steps:

[0247] S901: The first application sends rendering input data to the system rendering service.

[0248] Specifically, the sending of the rendering input data can refer to the related description of S701, which is not repeated here.

[0249] S902: The system rendering service generates first drawing information, and generates first expected layer information based on the first drawing information.

[0250] Specifically, it can refer to the related description of S702, which is not repeated here.

[0251] S903: The system rendering service calls a pre-check component.

[0252] Specifically, as shown in FIG. 2A-2E, the electronic device can deploy a pre-check component based on hardware parameters, which can include a related service object (SO) and a public component (software, configuration table configuration file, etc.). Wherein, the hardware parameters can include the resolution, refresh rate and color depth of the display screen, the processing capability of the hardware compositor, the processing capability of the GPU, the memory bandwidth, the supported API and characteristics, the available memory and storage speed of the system, etc.

[0253] Wherein, SO refers to a software object encapsulating specific service logic, aiming to provide reusable, service-oriented solutions to support various links in the rendering process. The SO can be responsible for receiving layer information from the system rendering service, and performing pre-checking according to the hardware specifications, which can identify which layers can be processed by the hardware compositor and which need additional processing.

[0254] Specifically, the SO is first registered in the system, the system rendering service reads the registration information, loads the SO into the memory and initializes it; then, the system rendering service calls the method or interface provided by the SO.

[0255] Further, the electronic device can also set a common component including a plurality of software components and configuration resources, which can include a configuration table configuration file that can contain configuration information related to hardware specifications, rendering performance, layer processing strategies, and the like. The system rendering service loads these configuration files at startup to obtain necessary hardware parameters and pre-check rules.

[0256] Specifically, the system rendering service loads the common component into memory and configures it, and calls its functions through the interface provided by the common component for pre-checking.

[0257] The system rendering service can select to call the SO or the common component according to requirements.

[0258] S904: The system rendering service pre-checks the expected layer based on the specification parameters of the electronic device to obtain a first pre-check result.

[0259] Specifically, the system rendering service pre-checks the layers included in the generated first expected layer information according to the hardware parameters and pre-check rules obtained by calling the pre-check component: the system rendering service traverses the layers included in the first expected layer information to determine whether each layer can be superimposed and synthesized by the hardware compositor, and if not, determines how to process in combination with the processing capabilities of the GPU and the constraints of the hardware specifications (mainly the display screen); the system rendering service can also check whether the dependency relationship between the layers is correct, for example, some layers can depend on the rendering results of other layers, and if the dependency relationship is incorrect or missing, it can cause incorrect rendering results.

[0260] At this time, the pre-check mechanism runs in the system rendering service process, and the first pre-check result can be cached by the system rendering service.

[0261] The first pre-check result includes the results of whether each expected layer in the plurality of expected layers included in the first expected layer information is synthesized by the hardware compositor or by the image processor, which can be referred to the related description of the pre-check result in S303 and will not be described here.

[0262] By implementing the embodiments of the present application, the pre-check interface and the pre-check component can be deployed in the electronic device at the same time, and the pre-check component or the pre-check interface can be selected and called according to requirements to complete pre-checking; in addition, the electronic device can deploy the pre-check component in different modes: the pre-check component can be part of the hardware compositor or independently deployed in the system, and according to the deployment mode of the pre-check component, the pre-check component calling form can be cross-process calling or direct calling in the process, to meet the needs of different scenarios and architecture forms, and to more flexibly perform pre-checking.

[0263] S905: The system rendering service generates first rendering instructions based on the first pre-check result.

[0264] Specifically, the system rendering service generates the first rendering instructions based on the first pre-check result obtained through the pre-check. The generation of the first rendering instructions can refer to the related description in S706, and details are not described herein.

[0265] S906: The system rendering service sends the first rendering instructions to the GPU.

[0266] S907: The GPU performs rendering based on the first rendering instructions to obtain an overlay layer set.

[0267] S908: The GPU sends the overlay layer set to the system rendering service.

[0268] S909: The system rendering service sends the overlay layer set to the hardware compositor.

[0269] S910: The hardware compositor performs overlay checking on the overlay layer set based on the specification parameters of the electronic device to determine whether the overlay layer set contains a layer that cannot pass the overlay processing of the hardware compositor. If yes, S911 is performed; if not, S913 is performed.

[0270] S911: The hardware compositor determines a third layer set and sends the third layer set to the system rendering service.

[0271] S912: The system rendering service triggers a repainting process to obtain a repainted layer set and sends the repainted layer set to the hardware compositor.

[0272] S913: The hardware compositor performs overlay processing to obtain a first image.

[0273] S914: The hardware compositor sends the first image to the display screen.

[0274] S915: The display screen displays the first image on the display screen.

[0275] Specifically, the specific process of steps S906-S915 can refer to the related description of steps S707-S716 in FIG. 7, and details are not described herein.

[0276] The above FIG. 9 shows a process of calling a pre-check component to complete interface display of an electronic device. Steps S901-S908 are a pre-check process of calling a pre-check component to complete interface display of an electronic device. Specifically, the first application sends rendering input data generated by a self-process to the system rendering service for graphic rendering and user interface update. The system rendering service performs pre-check in the system rendering service process according to the rendering input data and drawing information generated by the self-process.

[0277] For the pre-check process in the method flow diagram of FIG. 9 for displaying the interface of the pre-check component, the embodiment of the present application provides another pre-check process applicable to the separated rendering architecture. Referring to FIG. 10, FIG. 10 is another pre-check process diagram provided by the embodiment of the present application. As shown in FIG. 10, the another pre-check process provided by the embodiment of the present application can include the following steps:

[0278] S1001: The first application sends the rendering input data to the system rendering service.

[0279] Optionally, the first application can send the drawing information to the system rendering service according to the requirement, and the system rendering service performs the subsequent pre-check and drawing rendering.

[0280] S1002: The system rendering service generates the first drawing information, and generates the first expected layer information based on the first drawing information.

[0281] S1003: The system rendering service calls the pre-check component.

[0282] S1004: The system rendering service pre-checks the expected layer based on the specification parameter of the electronic device, and obtains the first pre-check result.

[0283] Specifically, the specific process of the above steps S1001-S1004 can refer to the related description of steps S901-S904 in FIG. 9, which will not be repeated here.

[0284] S1005: The first application generates the second drawing information, and generates the second expected layer information based on the second drawing information.

[0285] Optionally, the first application can also generate the second expected layer information in the process itself according to the requirement, and perform the subsequent pre-check and drawing rendering. The specific process of the first application generating the second drawing information and the second expected layer information can refer to the related description in S804, which will not be repeated here.

[0286] S1006: The first application calls the pre-check component.

[0287] Specifically, the electronic device can deploy the pre-check component based on the hardware parameter, and the pre-check component includes the related SO or public component (software, configuration table configuration file, etc.). The specific description of the hardware parameter, SO and public component can refer to the related description of S903, which will not be repeated here.

[0288] Specifically, the SO is registered to the service registration center, and the first application can establish a connection with the SO by querying the service registration center and calling the service provided by the SO to perform the pre-check.

[0289] Further, the first application can also read the public component, initialize the public component according to the configuration information in the public component, and then call the API of the public component for pre-checking.

[0290] The first application can select to call the SO or the public component according to requirements.

[0291] S1007: The first application pre-checks the expected layer based on the specification parameter of the electronic device, and obtains a second pre-checking result.

[0292] Specifically, the first application pre-checks the layer included in the generated second expected layer information based on the hardware parameter obtained by calling the pre-checking component and the pre-checking rule: specifically, the first application pre-checks the layer included in the generated second expected layer information based on the hardware parameter obtained by calling the pre-checking component and the pre-checking rule: the first application traverses the layer included in the second expected layer information, determines whether each layer can be superimposed and synthesized by the hardware synthesizer, and if not, determines how to process in combination with the processing capability of the GPU and the constraint of the hardware specification (mainly the display screen); the system rendering service can also check whether the dependency relationship between the layers is correct, for example, some layers can depend on the rendering result of other layers, and if the dependency relationship is incorrect or missing, the rendering result can be incorrect.

[0293] At this time, the pre-checking mechanism runs in the first application process, and the second pre-checking result can be cached by the first application. The content of the second pre-checking result can refer to the related description of the pre-checking result in S303, and will not be repeated here.

[0294] S1008: The system rendering service generates first rendering instructions based on the first pre-checking result.

[0295] S1009: The system rendering service sends the first rendering instructions to the GPU.

[0296] Specifically, the specific process of S1008-S1009 can refer to the related description of S706-S707, and will not be repeated here.

[0297] S1010: The first application generates second rendering instructions based on the second pre-checking result.

[0298] Specifically, after the first application generates the second pre-checking result in the process itself, the first application generates the second rendering instructions based on the second pre-checking result obtained by pre-checking. The process of generating the second rendering instructions can refer to the specific description in S811, and will not be repeated here.

[0299] S1011: The first application sends the second rendering instructions to the GPU.

[0300] Specifically, the process can refer to the specific description of S812, which is not repeated here.

[0301] S1012: The GPU performs rendering based on the rendering instruction to obtain the superimposed layer set.

[0302] Specifically, the GPU can use a parallel processing mechanism to process the second rendering instruction from the first application and the first rendering instruction from the system rendering service through the interface. For details, please refer to the related description in S813, which is not repeated here.

[0303] S1013: The GPU sends the superimposed layer set to the system rendering service.

[0304] Specifically, the sending of the superimposed layer set can refer to the related description of S709 in FIG. 7, which is not repeated here.

[0305] In the above steps, S1002 and S1005 can be executed in parallel. It can be understood that the numerical designations S1001-S1005 are only used to distinguish different steps, and do not limit the execution order.

[0306] In the processes shown in FIGS. 9 and 10, the electronic device performs pre-checking by calling a pre-checking component, which is deployed in a system library. At this time, the pre-checking mechanism runs in the system rendering service or the system rendering service and the first application, without the need for cross-process interaction. Alternatively, the pre-checking component can also be deployed in a hardware compositor to be called by the system rendering service and the first application, to meet the needs of different production and architecture forms, and to more flexibly perform pre-checking. For details of the deployment form, please refer to the related description of FIGS. 2B-2E.

[0307] Next, in combination with FIG. 11, a process of performing superposition verification and triggering redrawing before superimposing the generated layers before display is described.

[0308] The hardware compositor performs superposition verification on the superimposed layer set based on the specification parameters, determines whether the superimposed layer set contains layers that cannot be processed by the hardware compositor, and if not, the hardware compositor performs superposition processing on the superimposed layer set to obtain a first image. If it contains, the redrawing process is triggered.

[0309] As shown in FIG. 11, FIG. 11 is a redrawing process schematic diagram provided by an embodiment of the present application. The redrawing process can include the following steps:

[0310] S1101: The hardware compositor determines a third layer set based on the superimposed layer set, and caches a cache layer set in the superimposed layer set.

[0311] Specifically, the hardware compositor traverses all the layers included in the overlay layer set, determines all the layers in the overlay layer set that cannot be processed by the hardware compositor for overlay as the third layer set, and determines all the layers in the overlay layer set that can be processed by the hardware compositor for overlay as the cache layer set, and performs caching.

[0312] In the method, the hardware compositor can perform overlay processing on the layers in the cache layer set first and then cache: the hardware compositor can perform overlay processing on the layers that meet the hardware specification first, and then cache the result of the overlay processing. This processing can reduce subsequent repeated calculation, and directly read the synthesized image from the cache without the need for further overlay operation.

[0313] In addition, the hardware compositor can also cache the layers in the cache layer set first and then perform overlay processing: the hardware compositor can cache the layers that meet the hardware specification first, and then perform overlay processing when display is required. This processing can more flexibly handle changes in layers, and if a layer changes, only the layer needs to be redrawn and cached, without the need to reprocess other layers.

[0314] Specifically, the hardware compositor selects whether to perform overlay processing first or caching first according to specific requirements.

[0315] In the method, the third layer set further includes an overlay check result, which is used to guide the system rendering service on how to further process the layers to meet the requirements of screen display. The overlay check result can include reasons why each layer in the third layer set cannot be processed by the hardware compositor for overlay, adjustment suggestions for each layer in the third layer set, and details of hardware specification limitations of the electronic device.

[0316] S1102: The hardware compositor sends the third layer set to the system rendering service.

[0317] Specifically, the hardware compositor can send the cached third layer set to the system rendering service.

[0318] S1103: The system rendering service generates third rendering instructions based on the third layer set.

[0319] Specifically, the system rendering service can determine which layers need further processing through the third layer set, and determine how the layers need to be processed through the overlay check result included in the third layer set, and generate the third rendering instructions according to the information.

[0320] The third rendering instruction is used to instruct the GPU to perform re-drawing rendering on the layers included in the third layer set. The third rendering instruction can include a re-drawing layer list (specifying which layers need to be re-drawn), re-drawing parameters (such as new layer size, format, resolution, color depth, etc.), and a layer merging strategy (if the hardware compositor suggests merging multiple layers to reduce processing burden, the layer merging strategy should be included, such as which layers should be merged, the order of merging, the properties of the merged layers, etc.).

[0321] S1104: The system rendering service sends the third rendering instruction to the GPU.

[0322] Specifically, the process of sending the third rendering instruction can refer to the description of the system rendering service sending the first rendering instruction to the GPU in S707, which will not be repeated here.

[0323] S1105: The GPU performs rendering based on the third layer set according to the third rendering instruction, to obtain a re-drawn layer set.

[0324] Specifically, the GPU redraws each layer that needs to be redrawn according to the third rendering instruction. If the layer merging strategy is included in the third rendering instruction, the GPU can merge multiple layers into one layer according to the specified merging order and properties, to reduce the processing burden of the hardware compositor, to obtain the re-drawn layer set.

[0325] S1106: The GPU sends the re-drawn layer set to the system rendering service.

[0326] Specifically, the GPU can cache the re-drawn layer set obtained by rendering, and send the cached re-drawn image set to the system rendering service.

[0327] S1107: The system rendering service sends the re-drawn layer set to the hardware compositor.

[0328] Specifically, the system rendering service can instruct the hardware compositor to perform composition processing based on the re-drawn layer by sending the re-drawn layer set to the hardware compositor.

[0329] S1108: The hardware compositor performs superimposition processing based on the cached layer set and the re-drawn layer set, to obtain a first image.

[0330] Specifically, when the hardware compositor adopts the processing mode of first performing superimposition processing on the layers in the cached layer set and then caching in S1101, the hardware compositor will perform superimposition synthesis based on the cached image synthesized by the superimposition of the cached layer set and the re-drawn layer set, to obtain the first image. When the hardware compositor adopts the processing mode of first caching the layers in the above-mentioned cached layer set and then superimposing in S1101, the hardware compositor will perform superimposition synthesis based on the cached layers that have not been subjected to superimposition processing and the re-drawn layer set, to obtain the first image.

[0331] S1109: The hardware compositor sends the first image to the display screen.

[0332] S1110: The display screen displays the first image on the display screen.

[0333] Specifically, the specific process of S1109-S1110 can refer to the related description of S715-S716, which is not described here.

[0334] FIG. 12 is a schematic diagram of an application scenario of another interface display method provided by an embodiment of the present application.

[0335] As shown in FIG. 12, when the display screen of the electronic device displays a current interface, the electronic device acquires a trigger event, which can be an input (such as clicking, swiping down, dragging, etc.) of a user. In response to the trigger event, the electronic device updates the interface displayed on the display screen to a first image.

[0336] During the rendering generation process of the first image, the electronic device needs to generate a plurality of layers first. These layers can generate the first image through certain superimposition processing. The superimposition processing is usually completed by a specific hardware compositor. The plurality of layers need to be superimposed and verified before being processed by the hardware compositor. The layers that cannot be processed by the hardware compositor need to be re-rendered.

[0337] The method provided by the embodiment of the present application adds a pre-verification process before the superimposition processing. The layers generated based on the pre-verification result obtained by the pre-verification process can be more consistent with the constraints of the hardware compositor, and the number of re-rendered layers can be reduced.

[0338] Specifically, as shown in FIG. 12, the method provided by the embodiment of the present application involves the following processes: the electronic device generates expected layer information based on the trigger event, the expected layer information including data of a plurality of expected layers and other expected layer information; the electronic device performs pre-verification based on the hardware specification parameters and the expected layer information to obtain a pre-verification result, the pre-verification result including layers that do not meet the hardware constraints and modification suggestions, and layers that meet the hardware constraints; the electronic device performs rendering based on the pre-verification result to obtain a superimposed layer set, wherein the layers included in the superimposed layer set can be determined as cache layers and re-rendered layers; superimposition processing based on the superimposed layer set can obtain the first image; the display screen displays the first image, and thus the electronic device completes the response to the trigger event, and the display screen updates the mask to the first image.

[0339] It should be understood that each step in the above method embodiment can be completed by the integrated logic circuit of hardware in the processor or the instruction in the form of software. The method steps provided by the embodiments of the present application can be directly embodied as hardware processor execution completion, or executed by a combination of hardware and software modules in the processor.

[0340] The application also provides an electronic device, which can include a memory and a processor. The memory can be used to store a computer program, and the processor can be used to invoke the computer program in the memory to enable the electronic device to perform the method performed by the electronic device in any one of the above embodiments.

[0341] The application also provides a computer program product, which includes a computer program (also referred to as code or instructions). When the computer program is executed, the computer performs the method performed by the electronic device in any one of the above embodiments.

[0342] The application also provides a computer-readable storage medium, which stores a computer program (also referred to as code or instructions). When the computer program is executed, the computer performs the method performed by the electronic device in any one of the above embodiments.

[0343] The embodiments of the application can be combined in any manner to achieve different technical effects.

[0344] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0345] Those skilled in the art can understand that all or part of the processes in the above-mentioned method embodiments can be implemented by a computer program instructing relevant hardware to complete, and the program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The aforementioned storage medium includes ROM, random access memory (RAM), magnetic disk or optical disk, and various storage media that can store program codes.

[0346] In summary, the above only describes the embodiments of the technical scheme of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of the present application shall be included in the protection scope of the present application.

Claims

1. An interface display method characterized by, The method is applied to an electronic device, the electronic device comprising a hardware compositor, an image processor and a display screen; the method comprising: obtaining a trigger event, the trigger event being used to instruct the electronic device to display a first image through the display screen; in response to the trigger event, generating drawing information and generating expected layer information based on the drawing information, the expected layer information comprising a plurality of expected layers; based on a specification parameter of the electronic device, pre-verifying the plurality of expected layers to obtain a pre-verification result, the pre-verification result comprising indication information, the indication information being used to indicate that part or all of the plurality of expected layers are to be subjected to a composition process by the hardware compositor, or subjected to a composition process by the hardware compositor after being pre-processed, or subjected to a composition process by the image processor; generating a rendering instruction based on the pre-verification result and performing rendering based on the rendering instruction to obtain a superimposed layer set; based on the superimposed layer set, performing a composition process to obtain the first image and displaying the first image on the display screen.

2. The method of claim 1, wherein, The electronic device further comprises a system rendering service, the drawing information comprises first drawing information, and the expected layer information comprises first expected layer information; the generating drawing information and generating expected layer information based on the drawing information comprises: generating the first drawing information through the system rendering service and generating the first expected layer information based on the first drawing information.

3. The method according to claim 1 or 2, characterized in that, The electronic device further comprises a pre-verification interface, and the pre-verification result comprises a first pre-verification result; the pre-verifying the plurality of expected layers based on the specification parameter of the electronic device comprises: calling the pre-verification interface through the system rendering service, sending the first expected layer information to the hardware compositor; based on the specification parameter of the electronic device, pre-verifying a plurality of expected layers included in the first expected layer information through the hardware compositor to obtain a first pre-verification result.

4. The method of claim 3, wherein, The method further comprises: sending the first pre-verification result to the system rendering service through the pre-verification interface by the hardware compositor.

5. The method according to any one of claims 1 to 3, characterized in that, A first application is running on the electronic device; the drawing information further comprises second drawing information, and the expected layer information further comprises second expected layer information; the generating drawing information and generating expected layer information based on the drawing information further comprises: generating the second drawing information through the first application and generating second expected layer information based on the second drawing information.

6. The method of claim 3, wherein, The electronic device further comprises a pre-verification interface, and the pre-verification result comprises a first pre-verification result and a second pre-verification result; the pre-verifying the plurality of expected layers based on the specification parameter of the electronic device comprises: calling the pre-verification interface through the system rendering service, sending the first expected layer information to the hardware compositor; calling the pre-verification interface through the first application, sending the second expected layer information to the hardware compositor; The hardware synthesizer performs pre-checking on the plurality of expected layers included in the first expected layer information and the second expected layer information respectively based on the specification parameters of the electronic device, to obtain the first pre-checking result and the second pre-checking result.

7. The method of claim 5, wherein, The method further includes: the hardware synthesizer sending the first pre-checking result to the system rendering service and the second pre-checking result to the first application through the pre-checking interface.

8. The method of any one of claims 1, 2, 4, 6, 7, wherein, The drawing information includes one or more of expected rendering instructions, and descriptions of graphic resources, scene layout, and any rendering-related parameters; the expected rendering instructions include one or more of drawing commands, texture operations, and blending operations.

9. The method of claim 2, wherein, The electronic device further includes a pre-checking component deployed based on the specification parameters of the electronic device, and the pre-checking result includes a first pre-checking result; The pre-checking on the plurality of expected layers based on the specification parameters of the electronic device includes: The system rendering service calls the pre-checking component to obtain the specification parameters of the electronic device, and performs pre-checking on the plurality of expected layers included in the first expected layer information based on the specification parameters of the electronic device, to obtain the first pre-checking result.

10. The method of claim 2, wherein, The electronic device further includes a pre-checking component deployed based on the specification parameters of the electronic device, and the pre-checking result includes a first pre-checking result and a second pre-checking result; The pre-checking on the plurality of expected layers based on the specification parameters of the electronic device includes: The system rendering service calls the pre-checking component to obtain the specification parameters of the electronic device, and performs pre-checking on the plurality of expected layers included in the first expected layer information based on the specification parameters of the electronic device, to obtain the first pre-checking result; The first application calls the pre-checking component to obtain the specification parameters of the electronic device, and performs pre-checking on the plurality of expected layers included in the second expected layer information based on the specification parameters of the electronic device, to obtain the second pre-checking result.

11. The method of claim 1, wherein, The electronic device further includes a system rendering service, and the rendering instructions include a first rendering instruction; the generating of the rendering instructions based on the pre-checking result includes: The electronic device generates the first rendering instruction based on the first pre-checking result through the system rendering service.

12. The method of claim 1, wherein, The electronic device further includes a system rendering service, and the first application is running on the electronic device, and the rendering instructions include a first rendering instruction and a second rendering instruction; the generating of the rendering instructions based on the pre-checking result includes: The electronic device generates the first rendering instruction based on the first pre-checking result through the system rendering service; The electronic device generates the second rendering instruction based on the second pre-checking result through the first application.

13. The method of claim 1, wherein, The preprocessing includes one or more of format conversion, transparency processing, resolution adjustment, color space conversion, and layer simplification.

14. An electronic device, comprising: The preprocessing includes one or more of format conversion, transparency processing, resolution adjustment, color space conversion, and layer simplification. A hardware compositor, an image processor, a display, one or more processors and one or more memories; the one or more memories are coupled with the one or more processors, the memories are configured to store computer program codes, the computer program codes comprise computer instructions, when the one or more processors execute the computer instructions, the electronic device is caused to perform: obtaining a trigger event, the trigger event is used to instruct the electronic device to display a first image through a display screen; in response to the trigger event, generating drawing information, and generating expected layer information based on the drawing information, the expected layer information comprises a plurality of expected layers; based on a specification parameter of the electronic device, pre-verifying the plurality of expected layers to obtain a pre-verification result, the pre-verification result comprises indication information, the indication information is used to indicate that part or all of the plurality of expected layers are subjected to a composition process by the hardware compositor, or are subjected to a composition process by the hardware compositor after being pre-processed, or are subjected to a composition process by the image processor; generating rendering instructions based on the pre-verification result, and performing rendering based on the rendering instructions to obtain a superimposed layer set; based on the superimposed layer set, performing a composition process to obtain the first image, and displaying the first image on the display screen.

15. A computer storage medium, comprising, The computer storage medium stores a computer program, and the computer program is executed by a processor to implement the method in any one of claims 1-13.

16. A computer program product, characterised in that, The computer program product comprises instructions, and when the computer program product is executed by a computer, the computer is caused to perform the method in any one of claims 1-13.

Images