Interface display method and electronic equipment

Abstract

The embodiment of the invention provides an interface display method and electronic equipment, the electronic equipment comprises a hardware synthesizer, an image processor and a display screen, the interface display method comprises the following steps: before layer drawing and rendering, generating expected layer information, performing pre-verification based on the expected layer information and specification parameters to obtain a pre-verification result, and displaying the pre-verification result on the hardware synthesizer. And drawing and rendering the layers according to the pre-verification result, carrying out superposition synthesis processing on the layers, and displaying an image obtained after processing on a display screen. According to the embodiment of the invention, the frequency of additionally using the image processor can be reduced, the layer processing efficiency is improved, the processing load, memory and power consumption overhead of the image processor is reduced, and the performance is improved.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an interface display method and an electronic device. Background Technology

[0002] Graphics rendering and display is the process of processing user operations and application information through algorithms and then presenting it on a physical device. The rendering result changes according to changes in user operations and application processing logic. In electronic devices, both applications and system rendering services have rendering capabilities, inevitably generating multiple layers. When multiple layers need to be displayed on a screen, the rendered layers need to be overlaid by a dedicated hardware compositor (HWC) before being presented on the physical screen visible to the user.

[0003] However, when the hardware compositor is processing the overlay, it needs to first verify the information of the layers to be displayed, identify and mark which layers can ultimately participate in the overlay and display, and which cannot meet the hardware specifications. The layers that cannot participate in the final overlay and display are then redrawn and rendered using the image processor, which introduces the overhead of memory, performance and power consumption. Summary of the Invention

[0004] This application provides an interface display method and an electronic device that can improve layer composition efficiency and reduce memory, performance, and power consumption overhead.

[0005] In a first aspect, embodiments of this application provide an interface display method applied to an electronic device, the electronic device including a hardware compositor, an image processor, and a display screen. The method includes: acquiring a trigger event, the trigger event being used to instruct the electronic device to display a first image through the display screen; responding to the trigger event, generating drawing information, and generating expected layer information based on the drawing information, the expected layer information including multiple expected layers; performing pre-verification on the multiple expected layers based on the specifications of the electronic device, obtaining a pre-verification result, the pre-verification result including indication information: the indication information being used to indicate that some or all of the multiple expected layers are to be composited by the hardware compositor, or pre-processed and then composited by the hardware compositor, or composited by the image processor; generating a rendering instruction based on the pre-verification result, and rendering based on the rendering instruction to obtain an overlay layer set; performing compositing based on the overlay layer set to obtain the first image, and displaying the first image on the display screen.

[0006] The method provided in this application, compared to the prior art of directly overlaying layers after generation, adds a pre-verification process. Before generating layers, a pre-verification is performed based on the expected information of the layers to be generated. The electronic device can obtain the pre-verification result through the pre-verification process and render and draw in advance according to the constraints of the back-end hardware compositor based on the pre-verification result, such as pre-merging multiple layers and scaling video layers. This ensures that the layers finally received by the hardware compositor can meet the constraints as much as possible, reducing the number of times layers are redrawn, thereby reducing the memory, performance, and power consumption overhead of the rendering and display process. The electronic device can generate drawing information by obtaining user operations and system instructions, and generate expected layer information based on the drawing information. This expected layer information allows the electronic device to verify whether the expected layer meets the processing requirements of the hardware compositor, without having to actually render and generate the layer and then perform verification based on the generated layer. This allows for faster overlay verification before overlaying and saves rendering and display time.

[0007] In one possible implementation of the first aspect, the electronic device further includes a system rendering service, the drawing information includes first drawing information, and the expected layer information includes first expected layer information; generating the drawing information and generating the expected layer information based on the drawing information includes: generating the first drawing information through the system rendering service, and generating the first expected layer information based on the first drawing information. Implementing embodiments of this application, the system rendering service can receive drawing information, user processing information, and video / game layers from the application, and combine this with the system rendering service's own animation processing of the layers to generate first drawing information, and generate first expected layer information based on the first drawing information, thereby achieving the generation of first expected layer information through the system rendering service for pre-verification.

[0008] In one possible implementation of the first aspect, the electronic device further includes a pre-verification interface, the pre-verification result includes a first pre-verification result, and the pre-verification of the plurality of expected layers based on the specifications of the electronic device includes: calling the pre-verification interface through the system rendering service to send the first expected layer information to the hardware compositor; and the hardware compositor performing pre-verification on the plurality of expected layers included in the first expected layer information based on the specifications of the electronic device to obtain the first pre-verification result. In this embodiment, by deploying and calling the pre-verification interface, the system rendering service and the hardware compositor can complete the pre-verification through cross-process interaction, in which case the pre-verification process is mainly completed by the hardware compositor. In addition, the pre-verification interface can be deployed independently at the system layer or in the hardware compositor, which can meet the needs of specific scenarios to complete the pre-verification and obtain the pre-verification result.

[0009] In one possible implementation of the first aspect, the electronic device runs a first application; the drawing information further includes second drawing information, and the expected layer information further includes second expected layer information; the step of generating drawing information and generating expected layer information based on the drawing information further includes: generating the second drawing information through the first application, and generating the second expected layer information based on the second drawing information. Implementing the embodiments of this application, the first application can respond to user operations according to its own business logic. For example, when swiping up or down, a vertical swiping logic response should appear on the screen interface, or when the user clicks the play button, video or music playback should start. In this case, the triggering event includes the user operation. The first application generates the second drawing information based on the user operation and its own business logic, and generates the second expected layer information based on the second drawing information, thereby enabling the generation of the second expected layer information through the first application for pre-verification.

[0010] In one possible implementation of the first aspect, the electronic device further includes a pre-verification interface, and the pre-verification result includes a first pre-verification result and a second pre-verification result. The pre-verification of the plurality of expected layers based on the specifications of the electronic device includes: calling the pre-verification interface through the system rendering service to send first expected layer information to the hardware compositor; calling the pre-verification interface through the first application to send second expected layer information to the hardware compositor; and the hardware compositor performing pre-verification on the plurality of expected layers included in the first and second expected layer information based on the specifications of the electronic device, respectively, to obtain the first pre-verification result and the second pre-verification result. By implementing the embodiments of this application, the hardware compositor can obtain the first expected layer information sent by the system rendering service and the second expected layer information sent by the first application through the pre-verification interface, achieving cross-process interaction, avoiding conflicts with other types of data transmission, and improving stability and reliability. Furthermore, it enables the hardware compositor to acquire expected layer information from the system rendering service and the first application in parallel, making fuller use of the energy efficiency advantages of the hardware compositor and improving computational efficiency and system performance.

[0011] In one possible implementation of the first aspect, the drawing information includes one or more of the following: expected rendering instructions, a description of graphics resources, scene layout, and any rendering-related parameters; the expected rendering instructions include one or more of the following: clearing the buffer, setting the view and projection matrix, binding vertex data and textures, calling the shader program, and drawing primitives. In this embodiment, when the system rendering service needs to perform operations such as drawing primitives, it can generate first expected layer information based on these requirements. When the first application needs to perform operations such as setting the view and projection matrix, binding vertex data and textures, calling the shader program, and drawing primitives, the first application can also generate corresponding second expected layer information based on these requirements. This allows the expected layer information to provide as complete and clear as possible information such as the size, resolution, transparency value, layer position, texture operation, or specified renderer of the multiple layers to be generated. This enables the hardware compositor to more accurately determine whether the layers to be generated meet the constraints of the hardware compositor based on this information, and obtain the optimal pre-verification result.

[0012] In one possible implementation of the first aspect, the method further includes: calling the pre-verification interface through the hardware compositor to send the first pre-verification result to the system rendering service. In this embodiment, the hardware compositor sending the pre-verification result to the system rendering service guides the system rendering service in layer rendering, which can ensure that the generated layers meet the constraints of the hardware compositor as much as possible, thereby reducing the number of redraws triggered.

[0013] In one possible implementation of the first aspect, the method further includes: calling the pre-verification interface through the hardware compositor to send the first pre-verification result to the system rendering service and the second pre-verification result to the first application. In this embodiment, the hardware compositor can send the first pre-verification result to the system rendering service and the second pre-verification result to the first application in parallel by calling the pre-verification interface, thereby guiding the system rendering service and the first application to perform rendering and drawing, improving computational efficiency and system performance.

[0014] In one possible implementation of the first aspect, the electronic device further includes a pre-verification component, which is deployed based on the specifications of the electronic device, and the pre-verification result includes a first pre-verification result. The pre-verification of the plurality of expected layers based on the specifications of the electronic device includes: calling the pre-verification component through the system rendering service to obtain the specifications of the electronic device, and performing pre-verification on the plurality of expected layers included in the first expected layer information based on the specifications of the electronic device to obtain the first pre-verification result. By implementing the embodiments of this application, a pre-verification interface and a pre-verification component can be deployed simultaneously in the electronic device, enabling the selection of either the pre-verification component or the pre-verification interface to complete the pre-verification as needed. Furthermore, the electronic device can deploy the pre-verification component in different modes: the pre-verification component can be part of a hardware synthesizer or deployed independently in the system. Depending on the deployment mode of the pre-verification component, the calling form of the pre-verification component can be a cross-process call or a direct call within the same process, to meet the needs of different scenarios and architectures, and to perform pre-verification more flexibly.

[0015] In one possible implementation of the first aspect, the electronic device further includes a pre-verification component deployed based on the specifications of the electronic device, and the pre-verification result includes a first pre-verification result and a second pre-verification result. The pre-verification of the plurality of expected layers based on the specifications of the electronic device includes: calling the pre-verification component through the system rendering service to obtain the specifications of the electronic device, and pre-verifying the plurality of expected layers included in the first expected layer information based on the specifications of the electronic device to obtain the first pre-verification result; and calling the pre-verification component through the first application to obtain the specifications of the electronic device, and pre-verifying the plurality of expected layers included in the second expected layer information based on the specifications of the electronic device to obtain the second pre-verification result. By implementing the embodiments of this application, the first application and the system rendering service can call the pre-verification component in parallel, enabling pre-verification to be performed separately in the first application process and the system rendering service process, without needing to call the pre-verification interface separately through the first application and the system rendering service, thus meeting the needs of different scenarios.

[0016] In one possible implementation of the first aspect, the electronic device further includes a system rendering service, and the rendering instruction includes a first rendering instruction; generating the rendering instruction based on the pre-verification result includes: the electronic device generating the first rendering instruction based on the first pre-verification result through the system rendering service. By implementing the embodiments of this application, the electronic device can generate rendering instructions based on the pre-verification result through the system rendering service. The generated rendering instructions are used to instruct the graphics processor to draw and render layers, making the layers rendered according to these instructions more consistent with the constraints of the hardware compositor, thereby reducing the number of redraws triggered.

[0017] In one possible implementation of the first aspect, the electronic device further includes a system rendering service, a first application running on the electronic device, and the rendering instructions including a first rendering instruction and a second rendering instruction; generating the rendering instructions based on the pre-verification result includes: the electronic device generating the first rendering instruction based on the first pre-verification result through the system rendering service; and the electronic device generating the second rendering instruction based on the second pre-verification result through the first application. By implementing the embodiments of this application, the electronic device can generate rendering instructions based on the respective pre-verification results through the system rendering service and the first application, which are used to instruct the graphics processor to draw or render layers in their respective processes, thus meeting the rendering requirements of different rendering architectures; at the same time, it can make the layers rendered according to this rendering instruction more consistent with the constraints of the hardware compositor, thereby reducing the number of times redrawing is triggered.

[0018] In one possible implementation of the first aspect, after generating rendering instructions based on the pre-verification result and rendering based on the rendering instructions to obtain an overlay layer set, and before performing compositing processing based on the overlay layer set to obtain the first image, the method further includes: sending the overlay layer set to the system rendering service through the image processor; sending the overlay layer set to the hardware compositor through the system rendering service; and performing overlay verification by the hardware compositor based on the specifications of the electronic device and the overlay layer set, specifically including: if the overlay layer set contains one or more layers that cannot be overlaid by the hardware compositor, the hardware compositor determines the one or more layers that cannot be overlaid by the hardware compositor in the overlay layer set as a third layer set, and sends the third layer set to the system rendering service; the hardware compositor performs overlay processing on the layers in the overlay layer set that can be overlaid by the hardware compositor, and caches the layers obtained by the overlay processing as a cached layer set; and the system rendering service performs redrawing rendering based on the third layer set to obtain a redrawn layer set, and sends it to the hardware compositor. By implementing the embodiments of this application, based on the pre-verification process, the overlay layer set rendered by the image processor can better conform to the constraints of the hardware compositor. When sent to the hardware compositor for overlay processing, the number of layers in the overlay layer that cannot be overlaid by the hardware compositor can be reduced, the number of times redraw is triggered can be reduced, and the efficiency of overlay display can be improved.

[0019] Secondly, embodiments of this application provide an electronic device, including: a hardware synthesizer, an image processor, a display, one or more processors, and one or more memories; the one or more memories are coupled to the one or more processors, and the memories are used to store computer program code, the computer program code including computer instructions, which, when the one or more processors execute the computer instructions, cause the electronic device to perform the interface display method in any possible implementation of any of the above aspects.

[0020] Thirdly, this application provides a computer storage medium storing a computer program that, when executed by a processor, implements the interface display method flow described in any one of the first aspects above.

[0021] Fourthly, embodiments of this application provide a computer program product including instructions that, when executed by a computer, enable the computer to perform the image display processing method flow described in any of the first aspects above.

[0022] Fifthly, this application provides an interface display processing apparatus that has the function of implementing any of the above-described image display processing methods. This function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described functions. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0024] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0025] Figures 2A-2E This is a schematic diagram of the system architecture of a group of electronic devices 100 provided in the embodiments of this application.

[0026] Figure 3 This is a schematic flowchart of an interface display method 300 provided in an embodiment of this application.

[0027] Figure 4A and Figure 4B This is a schematic diagram of the interface of a set of electronic devices provided in the embodiments of this application.

[0028] Figure 5 This is a schematic diagram of layer drawing and compositing disclosed in an embodiment of this application.

[0029] Figure 6 This is a schematic diagram illustrating an application scenario of an interface display method provided in an embodiment of this application.

[0030] Figure 7 This is a schematic flowchart of a method for displaying an interface provided in an embodiment of this application.

[0031] Figure 8 This is a schematic diagram of a pre-verification process provided in an embodiment of this application.

[0032] Figure 9 This is a schematic diagram of another interface display method provided in an embodiment of this application.

[0033] Figure 10 This is a schematic diagram of another pre-verification process provided in an embodiment of this application.

[0034] Figure 11 This is a schematic diagram of a redrawing process provided in an embodiment of this application.

[0035] Figure 12 This is a schematic diagram illustrating an application scenario of another interface display method provided in this application embodiment. Detailed Implementation

[0036] This application provides a method for displaying an interface and an electronic device.

[0037] This method can be applied to electronic devices. The electronic device is a smart terminal device, and this application does not limit the specific type of smart terminal device. For example, the electronic device can be a mobile phone, and may also include tablet computers, desktop computers, laptop computers, handheld computers, wearable devices (such as smartwatches, smart bracelets, etc.), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, in-vehicle systems, game consoles, etc.

[0038] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with essentially the same function and purpose. For example, "first drawing information" and "second drawing information" are used to distinguish different drawing information, but do not limit their order. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that terms such as "first" and "second" do not necessarily imply that they are different.

[0039] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “some,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to any or all possible combinations that include one or more of the listed items.

[0040] Figure 1 A schematic diagram of the structure of the electronic device 100 provided in an embodiment of this application is shown.

[0041] Electronic device 100 may include a processor 101, a memory 102, a wireless communication module 103, a mobile communication module 104, an antenna 103A, an antenna 104A, a power switch 105, a sensor module 106, a focusing motor 107, a camera 108, a display screen 109, etc. The sensor module 106 may include a gyroscope sensor 106A, an accelerometer sensor 106B, an ambient light sensor 106C, an image sensor 106D, a proximity sensor 106E, etc. The wireless communication module 103 may include a WLAN communication module, a Bluetooth communication module, etc. All of the above components can transmit data via a bus.

[0042] Processor 101 may include one or more processing units, such as application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc.

[0043] The GPU, or Graphics Processing Unit, also known as a display core, visual processor, or display chip, is a miniature processor specifically designed to perform image and graphics-related computations. It is responsible for performing complex mathematical and geometric calculations to render images, videos, and other graphical content. Through its highly parallel architecture and numerous computing units, the GPU excels at handling large-scale parallel computing tasks, especially in graphics rendering. This design allows the GPU to provide tens or even hundreds of times the performance of the CPU in areas such as floating-point operations and parallel computing. The GPU can be a standalone device or integrated into the processor 101.

[0044] The processor 101 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 101 is a cache memory. This memory can store instructions or data that the processor 101 has just used or that are used repeatedly. If the processor 101 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 101, and thus improves the efficiency of the system.

[0045] In some embodiments, the processor 101 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0046] Memory 102 can be used to store computer executable program code, which may include instructions. Processor 101 executes various functional applications and data processing of electronic device 100 by running the instructions stored in memory 102. Memory 102 may include a program storage area and a data storage area. In specific implementations, memory 102 may include high-speed random access memory, and may also include non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices.

[0047] The wireless communication function of the electronic device 100 can be implemented through antenna 103A, antenna 104A, mobile communication module 104, wireless communication module 103, modem processor, and baseband processor.

[0048] Antennas 103A and 104A can be used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization.

[0049] The mobile communication module 104 can provide wireless communication solutions, including 2G / 3G / 4G / 5G, for use on electronic devices 100.

[0050] The wireless communication module 103 can provide solutions for wireless communication applications on electronic devices 100, including wireless local area networks (WLAN), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR).

[0051] The gyroscope sensor 106A can be used to determine the motion attitude of the electronic device 100.

[0052] Accelerometer 106B can detect the magnitude of acceleration of electronic device 100 in various directions (generally three axes).

[0053] The 106C ambient light sensor can detect the intensity of ambient light and automatically adjust the screen brightness.

[0054] The image sensor 106D can capture optical images (including visible light, infrared light, etc.) and convert them into electrical signals for processing, display, or storage.

[0055] The 106E distance sensor can measure the distance between an object and the sensor.

[0056] Electronic device 100 can implement display functions through GPU, display screen 109, and AP, etc. GPU is a microprocessor for image processing, connected to display screen 109 and AP. Processor 101 may include one or more GPUs, which execute program instructions to generate or change display information.

[0057] The display screen 109 is used to display images, videos, etc. The display screen 109 includes a display panel. The display panel can be a liquid crystal display (LCD). The display panel can also be manufactured using organic light-emitting diodes (OLEDs), active-matrix organic light-emitting diodes (AMOLEDs), flexible light-emitting diodes (FLEDs), miniled, microled, micro-OLEDs, quantum dot light-emitting diodes (QLEDs), etc. In some embodiments, the electronic device may include one or N displays 109, where N is a positive integer greater than 1.

[0058] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0059] Figures 2A-2E This is a schematic diagram of the system architecture of a group of electronic devices 100 provided in the embodiments of this application.

[0060] like Figure 2A As shown, the layered architecture divides the system into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the system is divided into five layers, from top to bottom: the application layer, the application framework layer, the system library, the hardware abstraction layer, the kernel layer, and the hardware layer.

[0061] The application layer can include a series of application packages. For example... Figure 2A As shown in the embodiments of this application, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0062] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example... Figure 2AAs shown, the application framework layer may include a window manager, an activity manager service (AMS), a content provider, a view system, a resource manager, a notification manager, etc.

[0063] The window manager is used to manage windowed applications. It can obtain the screen size, determine if a status bar is present, lock the screen, allow screen touch, drag the screen, and capture the screen, among other things.

[0064] The Activity Manager is responsible for managing the launch, state, lifecycle, and other activity-related aspects of an application. An activity is an application component that provides a user interface (UI) through which users interact with the electronic device to complete a task.

[0065] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0066] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0067] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0068] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog-style notifications on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0069] The system library may include multiple functional modules or components. For example, a system rendering service, media libraries, and a surface manager. In this embodiment, it may also include a pre-verification component. Figure 2A (Not marked in the text) This pre-verification component is deployed based on the hardware parameters of the hardware compositor and can be called by the application or system rendering service.

[0070] Alternatively, the pre-verification component can be deployed independently in the system library or as part of the hardware synthesizer.

[0071] The system rendering service is a system-level rendering service responsible for receiving rendering instructions and data from applications, scheduling and managing rendering tasks, and outputting rendering results to the display screen. The system rendering service calls the hardware compositor or GPU to composite display layers.

[0072] The system rendering service can call 3D graphics processing libraries and 2D graphics engines. The 3D graphics processing library can be an Open Graphics Library (OpenGL), used for image rendering. The 2D graphics engine is the drawing engine for 2D graphics. OpenGL refers to a professional graphics programming interface that defines a cross-programming language, cross-platform programming interface specification. It is used for 3D graphics (and also 2D), and is a powerful and easy-to-use low-level graphics library. OpenGL ES is a subset of the OpenGL 3D graphics API, designed for embedded devices such as mobile phones and game consoles.

[0073] In addition, the Surface Manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications; the Media Library supports playback and recording of various common audio and video formats, as well as still image files. The Media Library supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0074] The hardware abstraction layer (HAL) includes a hardware synthesizer and a graphics memory allocator (Gralloc), among others. In this embodiment, the hardware abstraction layer may further include a pre-verification interface (…). Figure 2A (Not marked in the text) This pre-verification interface can be called by the system rendering service, application, or hardware compositor to achieve data transmission and cross-process interaction.

[0075] Alternatively, the pre-verification interface can be deployed not only in the hardware abstraction layer but also in the hardware synthesizer.

[0076] Hardware compositors are specialized image processing devices used to combine graphics layers (such as application interfaces, windows, animations, and videos) from different applications and systems into a single image. Because they are typically built on customized hardware units, these customized hardware units include, but are not limited to, GPUs, display controllers, and other graphics processing-related hardware components. Hardware compositors provide efficient graphics compositing capabilities to upper layers through interaction with hardware devices, supporting complex graphics operations and efficient image compositing. Compared to GPUs, hardware compositors offer significant energy efficiency advantages.

[0077] The kernel layer is the layer between hardware and software. The kernel layer can include drivers for various hardware components, such as display drivers.

[0078] The hardware layer is the most fundamental layer in a computer system or embedded system, directly involving the existence and operation of physical hardware devices. The hardware layer is the foundation upon which software can run, including all physically tangible and visible computer components, as well as those invisible but equally crucial components such as integrated circuits and circuit boards. The hardware layer can include the CPU, GPU, display, sensors, etc.

[0079] The Graphics Processing Unit (GPU) is the core component of what is commonly referred to as a "graphics card" and is primarily used for graphics processing.

[0080] The CPU can execute task instructions from the window manager, system rendering services, or applications.

[0081] The following is combined Figures 2B-2E This describes the possible deployment methods for the pre-verification interface and pre-verification components.

[0082] like Figure 2B As shown, the pre-verification component can be deployed in the system library, in which case the pre-verification interface can be deployed in the hardware abstraction layer; for example... Figure 2C As shown, the pre-verification component can be deployed in the system library, and the pre-verification interface can be deployed in the hardware synthesizer; for example... Figure 2D As shown, the pre-verification component can be deployed in the hardware synthesizer, in which case the pre-verification interface can be deployed in the hardware abstraction layer; for example... Figure 2E As shown, the pre-verification component and the pre-verification interface can be deployed simultaneously in the hardware synthesizer.

[0083] It should be noted that, Figures 2A-2E The software architecture, pre-verification interface, and deployment method of the pre-verification components shown are merely illustrative examples and do not constitute a specific limitation on the electronic device.

[0084] To better understand this application, the following describes the nouns or terms used in the embodiments of this application:

[0085] (1) System Rendering Service: The system-level rendering service is a crucial part of the system, responsible for converting graphical information, user interactions, and animation effects from multiple applications into a user-visible interface. This service typically includes multiple components that work together to ensure the correct rendering and efficient display of graphical content. For example, in Android, this may include a layer manager (SurfaceFlinger), and in Apple's operating system (iOS), it may include a rendering server (RenderServer).

[0086] (2) A graphics processing unit (GPU) is a processor specifically designed for image processing. In computer systems, the core component commonly referred to as a "graphics card" is the GPU.

[0087] (3) Color Space: A color space, also known as a color model, is a mathematical model used to represent color. It forms the basis for the quantification, qualitative and quantitative measurement of color, describing various attributes of color through specific parameters and rules. A color space can be used to describe and define color in a generally acceptable manner under specific standards. There are various types of color spaces, each with its specific application scenarios and advantages. Common color space types include:

[0088] ① RGB color space: RGB represents the three primary colors: red, green, and blue. In the RGB color space, colors are produced by combining different intensities of these three colors.

[0089] ②YUV Color Space: The YUV color space separates luminance (Y) and chrominance (U and V) to achieve efficient color encoding of images. Y represents grayscale value or luminance information, while U and V represent chrominance information, used to describe the hue and saturation of a color.

[0090] (4) First Application: The first application mentioned in this application embodiment includes third-party applications supported by the electronic device and system-built-in applications. Among them, third-party applications include, but are not limited to, music playback applications, shopping applications, short video playback applications, social chat applications, etc.; system-built-in applications include, but are not limited to, call, SMS, camera, calendar, gallery, WLAN, Bluetooth, etc.

[0091] (5) User Interface (UI): This is the medium through which applications or operating systems interact and exchange information with users. It converts the internal form of information into a form that users can understand. The user interface is source code written in specific computer languages ​​such as Java or Extensible Markup Language (XML). This source code is parsed and rendered on electronic devices, ultimately presenting content that users can recognize. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be visible interface elements such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets displayed on the screen of an electronic device.

[0092] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0093] Figure 3 This is a schematic flowchart illustrating an interface display method 300 provided in an embodiment of this application. This method can be applied to... Figure 1 The electronic device 100 shown can also be used in other electronic devices, and this application embodiment does not limit it.

[0094] like Figure 3 As shown, Figure 3 The method 300 shown may include S301 to S305. The following is in conjunction with… Figure 3 Detail each step in method 300.

[0095] S301: Get the trigger event.

[0096] Specifically, the electronic device acquires a trigger event, which instructs the electronic device to display a first image on the screen. The trigger event can be user input, such as clicking, pulling down, swiping up, long-pressing, or dragging on a touchscreen with a finger, or clicking, sliding down, or dragging with a mouse, or it can be a preset logical or external event.

[0097] For example, Figure 4A and Figure 4B This is a schematic diagram of the interface of a set of electronic devices provided in an embodiment of this application. For example... Figure 4A As shown, an electronic device is running a gallery application. The current interface displays thumbnails of images after opening the gallery. In the gallery application, when a user clicks on a thumbnail of an image on the current interface, this click becomes the trigger event. According to the gallery application's response logic, the electronic device should then display... Figure 4B The image viewing interface shown is the first image displayed on the screen of the electronic device.

[0098] S302: Generate drawing information and generate expected layer information based on the drawing information.

[0099] Specifically, in response to a trigger event, the electronic device generates drawing information and, based on this drawing information, generates expected layer information. This drawing information may include expected rendering instructions, a description of graphics resources, scene layout, and one or more of any rendering-related parameters. The instructions and information contained in this drawing information, after certain rendering compositing processing, can be used to respond to the trigger event and obtain an image displayed on the screen, i.e., the first image. The expected rendering instructions refer to rendering instructions included in this information used for pre-verification and are not used to instruct rendering actions. Optionally, the expected rendering instructions may include one or more of setting view and projection matrices, binding vertex data and textures, calling shader programs, and drawing primitives. The electronic device generates expected layer information based on the drawing information, and can provide as complete and clear as possible information as possible in this expected layer information regarding the size, resolution, transparency values, layer positions, texture operations, or specified renderers of the multiple layers to be generated, so that the hardware compositor can more accurately determine whether the layers to be generated meet the constraints of the hardware compositor, i.e., perform pre-verification.

[0100] The expected layer information may include multiple expected layers, specifically including:

[0101] (1) Layer content: The visual content contained in each layer, such as text, images, graphics, etc.;

[0102] (2) Layer properties: The property information of each layer, such as position, size, transparency, rotation angle, scaling ratio, etc.

[0103] (3) Drawing instructions: Rendering instructions related to each layer, such as drawing path, fill mode, etc.;

[0104] (4) Layer stacking relationship: the vertical relationship between layers, i.e., which layer should be occluded by which layer;

[0105] (5) Animation and interaction information: If the layer contains animation or interactive elements, it is also necessary to pass the relevant animation frame information, interaction trigger conditions, etc.

[0106] (6) Synchronization and timing information: When multiple layers of content need to be displayed synchronously, information such as the display timestamp or frame number of the layer content needs to be transmitted.

[0107] (7) Video format and encoding / decoding information: video format, resolution, frame rate, encoding / decoding method, etc.

[0108] S303: Perform pre-verification based on specification parameters and expected layer information to obtain pre-verification results.

[0109] Specifically, based on its own specifications, the electronic device pre-verifies the expected layers that need to be drawn and rendered in the expected layer information before drawing and rendering the layers. The pre-verification result is used to instruct the electronic device to draw and render the layers, so that the generated layers conform to the hardware constraints as much as possible.

[0110] The specifications of the electronic device itself refer to a series of technical parameters and constraints that the electronic device follows when processing multi-layer overlay, including the capabilities of the hardware compositor. Optionally, the specifications include, but are not limited to, screen resolution, refresh rate, supported layer formats, maximum number of layers, maximum layer size, supported graphics formats and attributes, GPU model, video memory size, supported technologies (such as DirectX, OpenGL version), display memory, and the interface type and speed between the GPU and other hardware components (such as monitor, storage device).

[0111] Specifically, the electronic device can traverse all layers contained in the expected layer information, perform pre-verification based on specification parameters, mark layers that cannot be overlaid and composited by the hardware compositor, provide the necessary processing, and collect the marking and processing to generate pre-verification results.

[0112] The pre-verification results include: indication information, layers in the expected layer information that cannot be composited by the hardware compositor, the processing required for the layers that cannot be composited by the hardware compositor, and layers that can be composited by the hardware compositor.

[0113] The indication information is used to indicate whether some or all of the multiple expected layers are composited by the hardware compositor, or composited by the hardware compositor after preprocessing, or composited by the image processor.

[0114] Specifically, the ability to combine using a hardware synthesizer includes two scenarios: direct combination using a hardware synthesizer, and combination that requires some preprocessing before being combined using a hardware synthesizer.

[0115] For example, preprocessing may include: format conversion (converting the graphic data of the layer into a format supported by the hardware compositor), transparency processing (the hardware compositor may have limited support for layer transparency, so the transparency needs to be adjusted to suit the hardware compositor, which may include transparency merging or separation operations), resolution adjustment, color space conversion (different color spaces such as RGB, YUV, etc. may affect the processing efficiency of the hardware compositor), layer simplification (for complex layers, some details and unnecessary features can be removed), etc.

[0116] S304: Perform layer drawing and rendering based on pre-verification results.

[0117] Specifically, the electronic device generates rendering instructions based on the generated pre-verification results, and performs rendering based on the rendering instructions to obtain an overlay layer set.

[0118] Optionally, when generating rendering instructions based on the pre-verification results, the electronic device may modify the drawing instructions included in the expected layer information generated in S302 based on the pre-verification results; the electronic device may also ignore the drawing instructions included in the expected layer information generated in S302 and regenerate the rendering instructions based on the pre-verification results to ensure the accuracy and consistency of the rendering results.

[0119] Specifically, this rendering instruction is used to instruct the electronic device to draw and render layers in response to a triggering event. This rendering instruction includes, but is not limited to, clearing the buffer, setting the view and projection matrix, binding vertex data and textures, calling the shader program, drawing primitive drawing commands (such as drawing points, lines, triangles, etc.), texture operations (loading texture images and sampling them), and blending operations (controlling the blending method between different objects).

[0120] Electronic devices can draw and render layers based on this rendering instruction to obtain an overlay layer set. This overlay layer set includes multiple layers, and through certain overlay processing, an image of the complete interface can be displayed on the screen.

[0121] S305: Perform composite processing on the overlay layers to obtain the first image and display it on the screen.

[0122] Specifically, the electronic device performs compositing based on the rendered overlay layer set, a process primarily handled by a hardware compositor. The pre-verified overlay layer set contains layers that better conform to the constraints of the hardware compositor, allowing the first image to be obtained directly through the hardware compositor as much as possible, reducing GPU usage and fully leveraging the energy efficiency advantages of the hardware compositor.

[0123] Finally, a first image is displayed on the screen, where the first image refers to the image of the complete interface that is ultimately shown to the user on the screen of the electronic device.

[0124] Figure 5 This is a schematic diagram illustrating layer drawing and compositing as disclosed in an embodiment of this application. For example... Figure 5 As shown, taking the original Android launcher as an example, the interface rendering can include the rendering of four layers: the status bar, the navigation bar, the application interface, and the launcher icon layer. Layer 1, Layer 2, Layer 3, and Layer 4 constitute an overlay layer set, which is then composited to obtain the first image, which can be displayed on the screen.

[0125] The application interface can be provided by an application service, the launcher icon layer can be drawn by the Launcher application, and the aforementioned layers can be completed by the system UI thread of the user interface.

[0126] Figure 6 This is a schematic diagram illustrating an application scenario of an interface display method provided in an embodiment of this application.

[0127] The application scenarios of the method provided in this application mainly involve the interaction process between an application and a physical display screen. Specifically, it is a process of information transmission and processing from software to hardware, with the ultimate goal of presenting the content required by the user on the physical display screen.

[0128] Specifically, see Figure 6 First, the user interacts with the application, and the electronic device generates or inputs the information to be displayed, including various layer data and instructions. This layer data and instructions are the basic elements that constitute the final displayed content. After processing by the electronic device, the displayed page can be updated to respond to the user's actions.

[0129] Furthermore, this information is passed to the system service layer, particularly the system rendering service. The system rendering service is a crucial step responsible for converting the layer data and instructions provided by the application into a format that can be understood and processed by the hardware. In this step, the system performs compositing, optimization, and other processing on the layer data according to preset rendering algorithms and rules to ensure that the final display effect meets expectations.

[0130] Furthermore, the processed layer data and instructions are then sent to the hardware synthesizer. The hardware synthesizer is a key component connecting software and hardware; it is responsible for further converting the data and instructions from the system service layer into instructions that the hardware can directly execute. This ensures the compatibility and accuracy of the data and instructions.

[0131] Finally, the data and instructions processed by the hardware synthesizer are converted into content to be displayed and sent to the physical display screen. During this process, the physical display screen controls the illumination and color of the pixels based on the received data and instructions, thereby presenting the visual effect desired by the user on the screen.

[0132] The methods provided in this application have wide applications in various scenarios where interaction between an application and a physical display screen is required, including but not limited to devices with screen displays such as smartphones, tablets, laptops, televisions, large screens, projectors, smartwatches, and in-vehicle displays. By applying the methods provided in this application, the energy efficiency advantages of hardware compositors can be utilized more rationally, rendering overhead throughout the process can be reduced, and users can enjoy a smoother and more realistic visual experience.

[0133] The following is combined Figure 7 This application provides a complete description of the interface display method provided in its embodiments. Figure 7 This is a schematic flowchart illustrating a method for displaying an interface according to an embodiment of this application. This method can be applied to... Figure 1 The electronic device 100 shown can also be used in other electronic devices, and this application embodiment does not limit its application. The electronic device may include... Figure 2A For descriptions of the various devices and components, please refer to the above. Figures 2A-2E The relevant descriptions of the nouns or terms involved in the embodiments of this application are not repeated here.

[0134] like Figure 7 As shown, the interface display method provided in this application embodiment may include, but is not limited to, the following steps:

[0135] S701: The first application sends rendering input data to the system rendering service.

[0136] Specifically, firstly, the first application obtains rendering input data by triggering an event. The specific process of obtaining the triggering event can be referred to in the relevant description of S301 above, and will not be repeated here.

[0137] Rendering input data refers to the collection of various types of information generated by each application process for graphics rendering and user interface updates. Rendering input data includes, but is not limited to, the following:

[0138] (1) Layout data:

[0139] ① Element position and size: Specifies the position and size of various elements on the interface (such as buttons, text boxes, images, etc.);

[0140] ② Layout structure: Defines the hierarchical relationship and arrangement of elements, such as grid layout, stacked layout, etc.

[0141] (2) Visual style data: including color, font, margin, border, shadow, etc., which determine the appearance and style of interface elements.

[0142] (3) Interaction behavior data:

[0143] ① Event listeners: These register handler functions for user interaction events (such as clicks, swipes, long presses, etc.). These functions define how the application should respond when the event occurs.

[0144] ② Animation and transition effects: Specifies the animation or transition effects that should be displayed when the element changes state, such as fade in / out, slide, etc.

[0145] (4) Content data:

[0146] ①Text content: Text information displayed on the interface, such as title, description, prompts, etc.;

[0147] ② Image and media resources: including multimedia resources such as pictures, videos, and audio, which need to be displayed to users on the interface.

[0148] Secondly, the first application sends the rendering input data to the system-level rendering service through the API provided by the system. Before sending, the first application can encapsulate the rendering input data into a specific format.

[0149] S702: The system rendering service generates the first drawing information and generates the first expected layer information based on the first drawing information.

[0150] Specifically, the system rendering service generates first drawing information based on the acquired rendering input data and its own rendering data. The system rendering service's own rendering data includes, but is not limited to, the content contained in the rendering input data.

[0151] The first rendering information includes one or more of the following: expected rendering instructions, description of graphics resources, scene layout, and any rendering-related parameters; the expected rendering instructions include one or more of the following: clearing the buffer, setting the view and projection matrix, binding vertex data and textures, calling the shader program, and drawing primitives.

[0152] Secondly, the system rendering service generates first expected layer information based on the generated first and drawing information. This first expected layer information includes multiple layers that the first application needs to render using the system rendering service, as well as layers that the system needs to render. The specific content that the expected layer information may include can be found in the relevant description of expected layer information in S302, and will not be elaborated upon here.

[0153] S703: The system rendering service sends the first expected layer information to the hardware compositor.

[0154] Specifically, such as Figures 2A-2E As shown, the electronic device defines a pre-verification interface. This pre-verification interface is an interface built based on system cross-process interaction mechanisms such as Hardware Interface Definition Language (HIDL) to enable data interaction between the system rendering service or application and the hardware compositor. This data includes first expected layer information, second expected layer information, first pre-verification result, and second pre-verification result. The system rendering service calls this pre-verification interface to send the first expected layer information to the hardware compositor.

[0155] Optionally, the pre-verification interface can be defined independently at the system layer or other locations in the software architecture, or it can be defined in the hardware synthesizer.

[0156] For example, firstly, the system rendering service loads the library file containing the pre-verification interface, and then configures the parameters of the pre-verification interface, such as communication protocol, data format, security settings, etc., according to the specific requirements of the hardware compositor and the system. Then, the system rendering service encapsulates the first expected layer information according to the requirements of the pre-verification interface. The system rendering service constructs a request for data containing the encapsulated first expected layer information based on a communication mechanism (such as IPC, network communication, etc.) and sends the constructed request to the hardware compositor.

[0157] During inter-process communication, the first expected layer information can be serialized into a transmittable format, such as protocol buffers or parcels, and then deserialized at the receiving end to restore the original data structure.

[0158] S704: The hardware synthesizer pre-verifies the expected layer based on the specifications of the electronic device to obtain the first pre-verification result.

[0159] Specifically, electronic devices can define a pre-verification interface based on system cross-process interaction mechanisms such as HIDL. First, the hardware compositor calls the pre-verification interface to receive the first expected layer information passed from the system rendering service.

[0160] For example, the hardware compositor first establishes a communication connection with the system rendering service; second, the hardware compositor sends a request to the system rendering service through the pre-verification interface, informing the system rendering service that it is ready to send the first expected layer information; the system rendering service responds to the request and sends the first expected layer information to the hardware compositor through the pre-verification interface; after receiving the data, the hardware compositor parses it and stores it in its internal data structure.

[0161] Optionally, if asynchronous message passing is used, the hardware synthesizer registers a callback function or listener to receive data; if synchronous calls are used, the hardware synthesizer simply waits for the pre-validation interface method to return the first expected layer information.

[0162] Specifically, after receiving the first expected layer information, the hardware compositor performs pre-verification based on the first expected layer information. The pre-verification result includes the result of compositing each of the multiple expected layers by the hardware compositor or by the image processor. For details, please refer to the relevant description of the pre-verification result in S303, which will not be repeated here.

[0163] For example, the hardware synthesizer queries the specifications of the current hardware, including but not limited to screen resolution, refresh rate, supported layer formats, maximum number of layers, maximum layer size, GPU model, video memory size, supported technologies (such as DirectX, OpenGL version), display memory, interface type and speed between the GPU and other hardware components (such as display, storage device), for details please refer to the relevant description of the specifications of the electronic device itself in S303.

[0164] Specifically, the hardware compositor determines whether each layer in the first expected layer information can be overlaid and composited by the hardware compositor. If not, it marks that the layer needs further processing by the GPU and how it needs to be processed. The hardware compositor can also check whether the dependencies between layers are correct. For example, some layers may depend on the rendering results of other layers. If the dependencies are incorrect or missing, the rendering results may be incorrect.

[0165] Furthermore, the hardware compositor traverses the layers included in the first expected layer information, collects the markers from the pre-verification process and the layers that cannot be overlaid and composited by the hardware compositor, as well as the necessary processing to generate the first pre-verification result. The content of the first pre-verification result can be found in the relevant description of the pre-verification result in S303, and will not be repeated here. Implementing this embodiment of the application allows the system rendering service and the hardware compositor to complete pre-verification through cross-process interaction by deploying and calling the pre-verification interface. In this case, the pre-verification process is mainly completed by the hardware compositor. In addition, this pre-verification interface can be deployed independently at the system layer or in the hardware compositor, which can meet the needs of specific scenarios to complete pre-verification and obtain the pre-verification result.

[0166] S705: The hardware compositor calls the pre-verification interface to send the first pre-verification result to the system rendering service.

[0167] Specifically, based on the first pre-verification result, the hardware compositor initiates a cross-process interaction with the system rendering service through the HIDL client agent and sends the prepared first pre-verification result.

[0168] S706: The system rendering service generates the first rendering instruction based on the first pre-verification result.

[0169] After obtaining the first pre-verification result through the pre-verification interface, the system rendering service generates the first rendering instruction based on the first pre-verification result.

[0170] Specifically, the system rendering service can cache the first expected layer information generated by the system rendering service in S702. The first expected layer information includes expected rendering instructions, which can instruct the rendering layer to generate a first image in response to the trigger event. The first pre-verification result can include modifications to the attributes of some layers in the first expected layer. The system rendering service can modify the rendering instructions in the cached first expected layer information based on the first pre-verification result without regenerating them. In this way, only some instructions need to be adjusted, without recalculating the entire rendering process, which can reduce the waste of computing resources.

[0171] In addition, the system rendering service can ignore or discard the drawing instructions included in the first expected layer information and regenerate the rendering instructions based on the first pre-verification result, which can ensure the accuracy and consistency of the rendering results.

[0172] The first rendering instruction includes, but is not limited to, clearing the buffer, setting the view and projection matrix, binding vertex data and textures, calling the shader program, drawing primitive drawing commands (such as drawing points, lines, triangles, etc.), texture operations (loading texture images and sampling them), and blending operations (controlling the blending method between different objects).

[0173] S707: The system rendering service sends the first rendering instruction to the GPU.

[0174] Specifically, the first rendering instruction can be stored in an instruction buffer. This buffer is a shared area between the CPU and GPU used to temporarily store rendering instructions to be executed, from which the GPU reads the rendering instructions.

[0175] For example, the first rendering instruction can be delivered in push mode. In this mode, the system rendering service directly writes the first rendering instruction into the GPU's registers or a specific memory area, and the GPU reads and executes the instruction from that area. Alternatively, the first rendering instruction can be delivered in push mode. In this mode, the system rendering service allocates a memory block as a storage area for the first rendering instruction during system initialization. When rendering is required, the system rendering service writes the first rendering instruction into this memory block and notifies the GPU to read and execute it.

[0176] S708: The GPU renders based on the first rendering instruction to obtain an overlay layer set.

[0177] The GPU renders the obtained first rendering instruction to obtain an overlay layer set, which includes multiple layers.

[0178] Specifically, the GPU reads the first rendering instruction from the instruction buffer and parses it. During parsing, the GPU identifies different rendering operations, such as vertex processing, rasterization, and fragment processing. Based on the parsed instructions, the GPU executes the corresponding rendering operations to obtain an overlay layer set.

[0179] For example, the GPU can store the rendered overlay of layers in a frame buffer. This frame buffer is a temporary storage area used to store the rendered image data.

[0180] S709: The GPU sends an overlay layer set to the system rendering service.

[0181] Optionally, the electronic device can employ a shared video memory mechanism to achieve data transfer between the GPU and the system rendering service. The overlay layer set sent by the GPU to the system rendering service includes data from multiple layers within the overlay layer set rendered by the GPU in S708.

[0182] S710: The system rendering service sends overlay layer sets to the hardware compositor.

[0183] Specifically, the system rendering service can transfer data to the hardware compositor through a specific interface. This interface can be a system-level API or an interface based on the Hardware Abstraction Layer (HAL). Through this interface, the system rendering service can directly pass layer data to the hardware compositor without requiring additional data conversion or copying processes.

[0184] S711: The hardware synthesizer performs an overlay verification on the overlay layer set based on the specifications of the electronic device, determining whether it contains layers that cannot be processed by the hardware synthesizer. If it contains layers, proceed to S712; otherwise, proceed to S714.

[0185] Specifically, the hardware and synthesizer perform overlay verification on each layer in the overlay layer set based on the specifications of the electronic devices. The specifications of the electronic devices can be found in the relevant description of the hardware specifications in S704, and will not be elaborated upon here.

[0186] S712: The hardware compositor determines the third layer set and sends the third layer set to the system rendering service.

[0187] Specifically, the hardware compositor acquires an overlay layer set and iterates through each layer in the overlay layer set. It then determines a third layer set by collecting all layers in the overlay layer set that cannot be overlaid by the hardware compositor.

[0188] S713: The system rendering service triggers a redraw process to obtain a redraw layer set and sends the redraw layer set to the hardware compositor.

[0189] Specifically, the system rendering service redraws based on the third layer set to obtain the redraw layer set.

[0190] S714: The hardware synthesizer performs overlay processing to obtain the first image.

[0191] The hardware synthesizer performs overlay processing on the acquired layers to obtain the first image. The first image can be referred to in the relevant description in S305, and will not be repeated here.

[0192] The specific processes described in S712-S714 above can be found by referring to... Figure 11 The relevant descriptions will not be repeated here.

[0193] S715: The hardware synthesizer sends the first image to the display screen.

[0194] Specifically, before the hardware compositor sends the first image to the display screen, the hardware compositor can further process and optimize the first image to improve the display effect, including but not limited to color correction, brightness adjustment, and contrast enhancement; secondly, the hardware compositor establishes a connection with the display screen through a specific interface, the type of which depends on the specific type and specifications of the display screen; finally, the hardware compositor sends the first image to the display screen through the specific interface.

[0195] If the data format of the first image does not match the requirements of the display screen, the hardware synthesizer can perform the necessary data format conversion.

[0196] S716: The display shows the first image on the screen.

[0197] Specifically, firstly, the display screen parses and verifies the first image after acquiring it; secondly, the processing unit inside the display screen can further process the first image, including but not limited to scaling, rotating, and cropping; finally, the display screen displays the processed first image on the screen for the user to view.

[0198] The above Figure 7 The included steps—interface drawing, interface rendering, interface composition, and display—can all be triggered by the vertical synchronization (VSYNC) signal. The trigger period of the VSYNC signal is the screen refresh rate.

[0199] The above Figure 7 S701 to S709 illustrate the pre-verification process in the interface display method provided in the embodiments of this application. Specifically, the first application sends rendering input data to the system rendering service, which means the first application sends various types of information generated by its own process for graphics rendering and user interface updates to the system rendering service. The system rendering service collects the drawing information and generates expected layer information, and interacts with the hardware compositor to perform pre-verification, thus achieving pre-verification before layer rendering and display.

[0200] Regarding the above Figure 7 The pre-verification process shown in the schematic diagram of the interface display method is illustrated in this application embodiment, which provides another pre-verification process suitable for a separate rendering architecture. (See also...) Figure 8 , Figure 8 This is a schematic diagram of a pre-verification process provided in an embodiment of this application. For example... Figure 8 As shown, a pre-verification process provided in this application embodiment may include the following steps:

[0201] S801: The first application sends rendering input data to the system rendering service.

[0202] Optionally, when the first application chooses to send the layer data that needs to be drawn and rendered to the system rendering service for unified rendering within the system rendering service process, the first application can send rendering input data to the system rendering service.

[0203] S802: The system rendering service generates the first drawing information and generates the first expected layer information based on the first drawing information.

[0204] S803: The system rendering service sends the first expected layer information to the hardware compositor.

[0205] Specifically, the descriptions of S801 to S803 above can be found in the relevant descriptions of S701 to S703 above, and will not be repeated here.

[0206] S804: The first application generates second drawing information and generates second expected layer information based on the second drawing information.

[0207] Optionally, when the first application needs to draw and render certain layers within its own process, the first application can generate second drawing information based on user operations or the first application's own response logic.

[0208] The second rendering information includes one or more of the following: expected rendering instructions, description of graphics resources, scene layout, and any rendering-related parameters; the expected rendering instructions include one or more of the following: clearing the buffer, setting the view and projection matrix, binding vertex data and textures, calling the shader program, and drawing primitives.

[0209] Specifically, the first application generates second expected layer information based on the second rendering information for pre-verification. The second expected layer information includes multiple layers that the first application needs to render, and may also include layer information for these multiple layers and the stacking order between them.

[0210] S805: The first application sends the second expected layer information to the hardware synthesizer.

[0211] Specifically, a pre-verification interface is defined in the electronic device. This pre-verification interface can be referenced in the relevant description of the pre-verification interface in S703, and will not be elaborated upon here. The first application processes the second expected layer information and calls the pre-verification interface to send the processed second expected layer information to the hardware compositor. The specific process can be referenced in the relevant description of the system rendering service calling the pre-verification interface to send the first expected layer information in S703, and will not be elaborated upon here.

[0212] The first application processes the second expected layer information, which involves data serialization, while the hardware synthesizer involves data deserialization.

[0213] Specifically, S801 to S805 can be executed in parallel. It is understood that the numerical labels S801 to S805 are only used to distinguish different steps and do not limit the execution order.

[0214] S806: The hardware synthesizer pre-validates the expected layer based on the specifications of the electronic device and obtains the pre-validation result.

[0215] Specifically, the hardware synthesizer pre-validates the acquired first and second expected layer information based on the specifications of the electronic device. First, the electronic device queries the current hardware specifications; details can be found in S704 regarding the query of current hardware specifications, which will not be elaborated upon here. The hardware synthesizer pre-validates the first expected layer information to obtain a first pre-validation result, and pre-validates the second expected layer information to obtain a second pre-validation result.

[0216] For example, an electronic device can define a pre-verification interface based on system cross-process interaction mechanisms such as system HIDL. The hardware compositor first calls the pre-verification interface to receive the first expected layer information transmitted from the system rendering service and the second expected layer information from the first application.

[0217] The pre-verification process for the first expected layer information by the hardware synthesizer can be referenced above. Figure 7 The relevant descriptions of S704 will not be repeated here.

[0218] Specifically, the pre-verification process of the hardware synthesizer for the second expected layer information may include: the hardware synthesizer first establishes a communication connection with the first application; second, the hardware synthesizer sends a request to the first application through the pre-verification interface; the first application responds to the request and sends the second expected layer information to the hardware synthesizer through the pre-verification interface; after receiving the data, the hardware synthesizer parses it and stores it in its internal data structure; then, the hardware synthesizer performs pre-verification on the second expected layer information based on the queried hardware specification parameters; finally, the hardware synthesizer iterates through all the layers included in the second expected layer information that participated in the pre-verification to determine the second pre-verification result. The content of the second pre-verification result can be referred to the relevant description of the pre-verification result in S303, and will not be repeated here.

[0219] The specific process by which the hardware synthesizer verifies the second expected layer information based on the queried hardware specification parameters can be found above. Figure 7 The pre-verification process for the first expected layer information in S704 will not be elaborated here.

[0220] S807: The hardware compositor sends the first pre-verification result to the system rendering service through the pre-verification interface.

[0221] S808: The system rendering service generates the first rendering instruction based on the first pre-verification result.

[0222] S809: The system rendering service sends the first rendering instruction to the GPU.

[0223] Specifically, the above S807-S809 can be referred to Figure 7The relevant descriptions of S705-S707 will not be repeated here.

[0224] S810: The hardware synthesizer sends the second pre-verification result to the first application through the pre-verification interface.

[0225] Specifically, electronic devices can define a pre-verification interface based on system cross-process interaction mechanisms such as HIDL. The hardware compositor can initiate a cross-process call to the pre-verification interface through the HIDL client agent provided by the system based on the second pre-verification result, and pass the prepared second pre-verification result to instruct the first application to draw the rendering layer in the process.

[0226] During inter-process communication, the second expected layer information can also be serialized into a transmittable format, such as protocol buffers or parcels, and deserialized at the receiving end to restore the original data structure.

[0227] S811: The first application generates a second rendering instruction based on the second pre-verification result.

[0228] Specifically, the first application obtains the second pre-verification result through a pre-verification interface and generates a second rendering instruction based on the second pre-verification result within the application process. The first application can construct user interface (UI) elements through its graphical interface framework, such as Android's View system or iOS's UIKit; then, the UI elements are converted into graphics API call requests, such as OpenGL ES, Vulkan graphics API, etc., to generate the second rendering instruction. The second rendering instruction describes the attributes of the graphics object (such as position, color, texture, etc.) and the rendering state (such as lighting, shadow, blending mode, etc.), and is used to instruct the GPU to render and generate layers in response to the triggering event; the first application encapsulates the second rendering instruction and passes it down.

[0229] For example, generating a second rendering instruction may include: a first application may cache second expected layer information generated by the first application in S804, the second expected layer information including expected rendering instructions, which may instruct the rendering layer to generate a first image in response to a triggering event; the second pre-verification result may include modifications to the properties of certain layers in the second expected layer; the first application may modify the rendering instructions in the cached second expected layer information based on the second pre-verification result without regenerating them, thus only needing to adjust some instructions without recalculating the entire rendering process, which can reduce the waste of computing resources;

[0230] In addition, the first application can ignore or discard the drawing instructions included in the second expected layer information and regenerate the rendering instructions based on the second pre-verification result, which can ensure the accuracy and consistency of the rendering results.

[0231] The second rendering instruction includes, but is not limited to, clearing the buffer, setting the view and projection matrix, binding vertex data and textures, calling the shader program, drawing primitive drawing commands (such as drawing points, lines, triangles, etc.), texture operations (loading texture images and sampling them), and blending operations (controlling the blending method between different objects).

[0232] S812: The first application sends the second rendering instruction to the GPU.

[0233] Specifically, the first application can submit the second rendering instruction to the command buffer, where the driver performs decoding and optimization, and then sends it to the GPU.

[0234] For example, the second rendering instruction can be delivered in push mode, in which the first application can write the second rendering instruction into the GPU's registers or a specific memory area, and the GPU reads and executes the instruction from that area; or, the second rendering instruction can be delivered in request mode, in which the system rendering service allocates a block of memory as a storage area for the second rendering instruction during system initialization, and when rendering is required, the system rendering service writes the second rendering instruction into this memory and notifies the GPU to read and execute it.

[0235] S813: The GPU renders based on rendering instructions to obtain an overlay of layers.

[0236] Specifically, the GPU can use a parallel processing mechanism to process the first rendering instruction from the system rendering service and the second rendering instruction from the first application.

[0237] For example, when the first application chooses to send the layer data that needs to be drawn and rendered to the system rendering service, and the rendering is performed uniformly within the system rendering service process, the GPU can render based on the first rendering instruction sent by the system rendering service to obtain an overlay layer set; when the first application needs to draw and render certain layers within its own process, the GPU can also receive the second rendering instruction sent by the first application, and can simultaneously receive the first rendering instruction sent by the system rendering service, and render based on the first rendering instruction and the second rendering instruction to obtain an overlay layer set.

[0238] Specifically, the GPU storage overlay layer set can be found in the above description. Figure 7 The relevant descriptions of S708 will not be repeated here.

[0239] S814: The GPU sends an overlay layer set to the system rendering service.

[0240] Specifically, sending overlay layer sets can be referenced above. Figure 7 The relevant descriptions of S709 will not be repeated here.

[0241] As mentioned above Figure 7 and Figure 8 As shown, electronic devices can display an interface through a pre-verification interface, at which point the pre-verification mechanism runs in the hardware synthesizer.

[0242] The electronic device also includes a pre-verification component. Optionally, the electronic device can also complete the interface display by calling the pre-verification component.

[0243] Optionally, the pre-verification component can be deployed independently in the system library or as part of the hardware synthesizer, in conjunction with... Figure 9 and Figure 10 This describes the process by which electronic devices complete pre-verification by calling the pre-verification component.

[0244] It should be noted that, Figure 9 and Figure 10 The fact that the pre-validation component and the hardware synthesizer are not shown in the same box does not mean that the pre-validation component is necessarily deployed independently outside the hardware synthesizer; it is just to illustrate the need of the process.

[0245] First, combined Figure 9 This paper introduces a method for displaying the interface by calling a pre-validation component, which is applicable to a unified rendering architecture. Figure 9 This is a schematic diagram of another interface display method provided in an embodiment of this application. For example... Figure 9 As shown, the interface display method provided in this application embodiment may further include the following steps:

[0246] S901: The first application sends rendering input data to the system rendering service.

[0247] Specifically, the sending of rendering input data can be found in the relevant description of S701, and will not be elaborated here.

[0248] S902: The system rendering service generates the first drawing information and generates the first expected layer information based on the first drawing information.

[0249] For details, please refer to the relevant description in S702, which will not be repeated here.

[0250] S903: The system rendering service calls the pre-verification component.

[0251] Specifically, such as Figures 2A-2EAs shown, electronic devices can deploy pre-verification components based on hardware parameters. These pre-verification components can include relevant service objects (SOs) and common components (software, configuration files, etc.). The hardware parameters can include the display resolution, refresh rate, and color depth; the processing power of the hardware compositor; the processing power of the GPU; memory bandwidth; supported APIs and features; and the system's available memory and storage speed.

[0252] Here, SO refers to a software object that encapsulates specific service logic, aiming to provide reusable, service-oriented solutions to support various stages of the rendering process. This SO can be responsible for receiving layer information from the system rendering service and performing pre-verification based on hardware specifications, identifying which layers can be processed by the hardware compositor and which require additional processing.

[0253] Specifically, the SO is first registered in the system. The system rendering service reads the registration information, loads the SO into memory, and initializes it. Then, the system rendering service calls the methods or interfaces provided by the SO.

[0254] In addition, electronic devices can also be configured with a common component that includes multiple software components and configuration resources. This common component can include configuration table configuration files, which may contain configuration information related to hardware specifications, rendering performance, layer processing strategies, etc. The system rendering service loads these configuration files at startup to obtain the necessary hardware parameters and pre-verification rules.

[0255] Specifically, the system rendering service loads the public component into memory and configures it, and then calls its functions through the interface provided by the public component for pre-verification.

[0256] The system rendering service can choose to call SO or public components as needed.

[0257] S904: The system rendering service performs a pre-verification of the expected layer based on the specifications of the electronic device and obtains the first pre-verification result.

[0258] Specifically, the system rendering service pre-verifies the layers included in the generated first expected layer information based on the hardware parameters and pre-verification rules obtained by calling the pre-verification component. The system rendering service traverses the layers included in the first expected layer information to determine whether each layer can be overlaid and composited by the hardware compositor. If not, it determines what kind of processing is required by combining the processing power of the GPU and the constraints of the hardware specifications (mainly referring to the display screen). The system rendering service can also check whether the dependencies between layers are correct. For example, some layers may depend on the rendering results of other layers. If the dependencies are incorrect or missing, the rendering results may be incorrect.

[0259] At this time, the pre-verification mechanism runs in the system rendering service process, and the first pre-verification result can be cached by the system rendering service.

[0260] The first pre-verification result includes the result of compositing each of the multiple expected layers contained in the first expected layer information by a hardware compositor or by an image processor. For details, please refer to the relevant description of the pre-verification result in S303, which will not be repeated here.

[0261] By implementing the embodiments of this application, a pre-verification interface and a pre-verification component can be deployed simultaneously in an electronic device, enabling the selection of either the pre-verification component or the pre-verification interface to complete pre-verification as needed. Furthermore, the electronic device can deploy the pre-verification component in different modes: the pre-verification component can be deployed as part of a hardware synthesizer or independently in the system. Depending on the deployment mode of the pre-verification component, the calling form of the pre-verification component can be a cross-process call or a direct call within the current process, in order to meet the needs of different scenarios and architectures and to perform pre-verification more flexibly.

[0262] S905: The system rendering service generates the first rendering instruction based on the first pre-verification result.

[0263] Specifically, the system rendering service generates a first rendering instruction based on the first pre-verification result obtained from the pre-verification. For details on generating the first rendering instruction, please refer to the relevant description in S706, which will not be repeated here.

[0264] S906: The system rendering service sends the first rendering instruction to the GPU.

[0265] S907: The GPU renders based on the first rendering instruction to obtain an overlay layer set.

[0266] S908: The GPU sends the overlay layer set to the system rendering service.

[0267] S909: The system rendering service sends the overlay layer set to the hardware compositor.

[0268] S910: The hardware synthesizer performs an overlay verification on the overlay layer set based on the specifications of the electronic device, determining whether it contains layers that cannot be processed by the hardware synthesizer. If it does, proceed to S911; otherwise, proceed to S913.

[0269] S911: The hardware compositor determines the third layer set and sends the third layer set to the system rendering service.

[0270] S912: The system rendering service triggers a redraw process to obtain a redraw layer set and sends the redraw layer set to the hardware compositor.

[0271] S913: The hardware synthesizer performs overlay processing to obtain the first image.

[0272] S914: The hardware synthesizer sends the first image to the display screen.

[0273] S915: The display shows the first image on the screen.

[0274] Specifically, the detailed process of steps S906-S915 above can be found in [reference needed]. Figure 7 The relevant descriptions of steps S707-S716 are not repeated here.

[0275] The above Figure 9 The process of an electronic device calling a pre-verification component to complete the interface display is demonstrated. Steps S901-S908 are the pre-verification process of the electronic device calling the pre-verification component to complete the interface display. Among them, the first application sends rendering input data generated by its own process for graphics rendering and user interface updates to the system rendering service. The system rendering service performs pre-verification in its process based on the rendering input data and the drawing information generated by its own process.

[0276] In response to the above Figure 9 The illustrated pre-verification process shown in the flowchart of the method for calling the pre-verification component to complete the interface display is provided in this application embodiment, which also provides another pre-verification process suitable for a separate rendering architecture. See also... Figure 10 , Figure 10 This is a schematic diagram of another pre-verification process provided in an embodiment of this application. For example... Figure 10 As shown, another pre-verification process provided in this application embodiment may include the following steps:

[0277] S1001: The first application sends rendering input data to the system rendering service.

[0278] Optionally, the first application can send the drawing information to the system rendering service as needed, and the system rendering service can then perform subsequent pre-verification and drawing rendering.

[0279] S1002: The system rendering service generates the first drawing information and generates the first expected layer information based on the first drawing information.

[0280] S1003: The system rendering service calls the pre-verification component.

[0281] S1004: The system rendering service performs a pre-verification of the expected layer based on the specifications of the electronic device and obtains the first pre-verification result.

[0282] Specifically, the detailed process of steps S1001-S1004 above can be found in [reference needed]. Figure 9 The relevant descriptions of steps S901-S904 are not repeated here.

[0283] S1005: The first application generates second drawing information and generates second expected layer information based on the second drawing information.

[0284] Optionally, the first application can also generate second expected layer information in its own process as needed and perform subsequent pre-verification and rendering. The specific process of the first application generating the second rendering information and the second expected layer information can be found in the relevant description in S804, and will not be elaborated here.

[0285] S1006: The first application calls the pre-verification component.

[0286] Specifically, electronic devices can deploy pre-verification components based on hardware parameters. These pre-verification components include relevant SOs or common components (software, configuration files, etc.). For detailed descriptions of the hardware parameters, SOs, and common components, please refer to the relevant descriptions in S903; they will not be repeated here.

[0287] Specifically, the SO is registered with the service registry. The first application can establish a connection with the SO and call the services provided by the SO by querying the service registry for pre-verification.

[0288] In addition, the first application can also read the above-mentioned public components, initialize the public components according to the configuration information in the public components, and then call the API of the public components for pre-verification.

[0289] The first application can choose to call SO or public components according to its needs.

[0290] S1007: The first application performs a pre-verification of the expected layer based on the specifications of the electronic device, and obtains a second pre-verification result.

[0291] Specifically, the first application pre-verifies the layers included in the generated second expected layer information based on the hardware parameters and pre-verification rules obtained by calling the pre-verification component. Specifically, the first application pre-verifies the layers included in the generated second expected layer information according to the hardware parameters and pre-verification rules obtained by calling the pre-verification component. The first application iterates through the layers included in the second expected layer information to determine whether each layer can be overlaid and composited by the hardware compositor. If not, it determines the required processing based on the GPU's processing power and hardware specifications (mainly referring to the display screen). The system rendering service can also check whether the dependencies between layers are correct. For example, some layers may depend on the rendering results of other layers. If the dependencies are incorrect or missing, the rendering results may be incorrect.

[0292] At this point, the pre-verification mechanism runs in the first application process, and the second pre-verification result can be cached by the first application. The content of the second pre-verification result can be found in the relevant description of the pre-verification result in S303, and will not be repeated here.

[0293] S1008: The system rendering service generates the first rendering instruction based on the first pre-verification result.

[0294] S1009: The system rendering service sends the first rendering instruction to the GPU.

[0295] Specifically, the detailed processes of S1008-S1009 can be referred to the relevant descriptions of S706-S707, and will not be repeated here.

[0296] S1010: The first application generates a second rendering instruction based on the second pre-verification result.

[0297] Specifically, after generating the second pre-verification result within its own process, the first application generates the second rendering instruction based on the second pre-verification result obtained from the pre-verification. The process of generating the second rendering instruction can be referred to the specific description in S811, and will not be repeated here.

[0298] S1011: The first application sends the second rendering instruction to the GPU.

[0299] For details, please refer to the specific description in S812, which will not be repeated here.

[0300] S1012: The GPU renders based on rendering instructions to obtain an overlay layer set.

[0301] Specifically, the GPU can use a parallel processing mechanism through the interface to process the second rendering instruction from the first application and the first rendering instruction from the system rendering service. For details, please refer to the relevant description in S813, which will not be repeated here.

[0302] S1013: The GPU sends an overlay layer set to the system rendering service.

[0303] Specifically, for sending overlay layer sets, please refer to... Figure 7 The relevant descriptions of S709 will not be repeated here.

[0304] In the above steps, S1002 and S1005 can be executed in parallel. It is understood that the numerical labels S1001 to S1005 are only used to distinguish different steps and do not restrict the execution order.

[0305] The above Figure 9 and Figure 10In the process demonstrated, the electronic device performs pre-verification by calling a pre-verification component. This component is deployed in the system library, and the pre-verification mechanism runs within the system rendering service or between the system rendering service and the first application, without requiring cross-process interaction. Optionally, the pre-verification component can also be deployed in the hardware compositor for joint invocation by the system rendering service and the first application to meet the needs of different product architectures and provide more flexible pre-verification. For specific deployment methods, please refer to [reference needed]. Figures 2B-2E Related explanations.

[0306] The following is combined Figure 11 This indicates that the process of performing overlay verification and triggering redrawing is performed after the pre-verification process and before the generated layers are overlaid, before the display is sent.

[0307] The hardware compositor performs overlay verification on the overlay layer set based on specification parameters to determine whether it contains layers that cannot be overlaid by the hardware compositor. If it does not contain any layers, the hardware compositor overlays the overlay layer set to obtain the first image; if it does contain any layers, it triggers a redraw process.

[0308] like Figure 11 As shown, Figure 11 This is a schematic diagram of a redrawing process provided in an embodiment of this application. The redrawing process may include the following steps:

[0309] S1101: The hardware synthesizer determines the third layer set based on the overlay layer set and caches the cached layer set in the overlay layer set.

[0310] Specifically, the hardware compositor traverses all layers contained in the overlay layer set, determines all layers in the overlay layer set that cannot be overlaid by the hardware compositor as the third layer set, and determines all layers in the overlay layer set that can be overlaid by the hardware compositor as the cache layer set, and caches them.

[0311] The hardware compositor can first overlay layers from the cached layer set before caching them: it can first overlay layers that meet hardware specifications, and then cache the overlay result. This reduces subsequent redundant calculations, allowing the composited image to be read directly from the cache without needing to perform overlay operations again.

[0312] Furthermore, the hardware compositor can also cache the layers in the cached layer set before overlaying them: the hardware compositor can cache the layers that meet the hardware specifications separately, and then overlay them when they need to be displayed. This approach allows for more flexible handling of layer changes; if a layer changes, only that layer needs to be redrawn and cached, without having to reprocess other layers.

[0313] Specifically, the hardware synthesizer chooses whether to perform the stacking process first or the buffering process first, depending on the specific requirements.

[0314] The third layer set also includes overlay verification results, which are used to guide the system rendering service on how to further process the layers to meet the requirements of screen display. The overlay verification results may include the reasons why each layer in the third layer set cannot be overlaid by the hardware compositor, adjustment suggestions for each layer in the third layer set, and details of the hardware specification limitations of the electronic device.

[0315] S1102: The hardware compositor sends the third layer set to the system rendering service.

[0316] Specifically, the hardware compositor can send the cached third layer set to the system rendering service.

[0317] S1103: The system rendering service generates a third rendering instruction based on the third layer set.

[0318] Specifically, the system rendering service can determine which layers need further processing through the third layer set, determine what kind of processing the layers need through the overlay verification results included in the third layer set, and generate a third rendering instruction based on this information.

[0319] The third rendering instruction is used to instruct the GPU to redraw and render the layers contained in the third layer set. The third rendering instruction may include a list of redraw layers (specifying which layers need to be redrawn), redraw parameters (such as new layer size, format, resolution, color depth, etc.), and layer merging strategy (if the hardware compositor recommends merging multiple layers to reduce processing burden, then the layer merging strategy should be included, such as which layers should be merged, the merging order, and the properties of the merged layers).

[0320] S1104: The system rendering service sends a third rendering instruction to the GPU.

[0321] Specifically, the process of sending the third rendering instruction can be referred to in the description of the system rendering service sending the first rendering instruction to the GPU in S707, and will not be repeated here.

[0322] S1105: The GPU renders based on the third layer set according to the third rendering instruction, and obtains the redraw layer set.

[0323] Specifically, the GPU redraws each layer that needs to be redrawn according to the third rendering instruction. If the third rendering instruction includes a layer merging strategy, the GPU can merge multiple layers into one layer according to the specified merging order and attributes to reduce the processing burden of the hardware compositor and obtain a redrawn layer set.

[0324] S1106: The GPU sends a redraw layer set to the system rendering service.

[0325] Specifically, the GPU can cache the redrawn image set obtained from rendering and send the cached redrawn image set to the system rendering service.

[0326] S1107: The system rendering service sends a redraw layer set to the hardware compositor.

[0327] Specifically, the system rendering service can send instructions to the hardware compositor to redraw the layer set, and the hardware compositor can then perform compositing based on the redrawn layers.

[0328] S1108: The hardware synthesizer performs overlay processing based on the cached layer set and the redrawn layer set to obtain the first image.

[0329] Specifically, when the hardware compositor adopts the processing method in S1101 of first overlaying the layers in the cached layer set and then caching them, the hardware compositor overlays the image based on the cached layer set and the redrawn layer set to obtain the first image; when the hardware compositor adopts the processing method in S1101 of first caching the layers in the above-mentioned cached layer set and then overlaying them, the hardware compositor overlays the layers based on the cached layers that have not been overlaid and the redrawn layer set to obtain the first image.

[0330] S1109: The hardware synthesizer sends the first image to the display screen.

[0331] S1110: The display screen shows the first image on the display screen.

[0332] Specifically, the detailed processes of S1109-S1110 can be found in the relevant descriptions of S715-S716, which will not be repeated here.

[0333] Figure 12 This is a schematic diagram illustrating an application scenario of another interface display method provided in this application embodiment.

[0334] like Figure 12 As shown, when the display screen of an electronic device shows the current interface, the electronic device acquires a trigger event, which can be user input (such as clicking, swiping, dragging, etc.). In response to the trigger event, the electronic device updates the interface displayed on the display screen to the first image.

[0335] During the rendering and generation of the first image, the electronic device needs to generate multiple layers. These layers are then overlaid to generate the first image. This overlay process is typically performed by a specific hardware compositor. Before overlay processing, the multiple layers need to undergo overlay verification. Layers that cannot be processed by the hardware compositor need to be redrawn and rendered.

[0336] The method provided in this application adds a pre-verification process before the overlay process. The layer generated based on the pre-verification result obtained from the pre-verification process can better conform to the constraints of the hardware compositor and reduce the number of times the rendering layer is redrawn.

[0337] Specifically, such as Figure 12 As shown, the method provided in this application includes the following process: an electronic device generates expected layer information based on a trigger event, the expected layer information containing data of multiple expected layers and other expected layer information; the electronic device performs pre-verification based on hardware specification parameters and expected layer information to obtain a pre-verification result, the pre-verification result including layers that do not conform to hardware constraints and modification suggestions, and layers that conform to hardware constraints; the electronic device renders based on the pre-verification result to obtain an overlay layer set, wherein the layers included in the overlay layer set can be determined as cached layers and redrawn layers; an overlay processing is performed based on the overlay layer set to obtain a first image; the display screen displays the first image, at which point the electronic device completes its response to the trigger event, and the display screen updates the displayed mask to the first image.

[0338] It should be understood that each step in the above method embodiments can be completed by the integrated logic circuits in the processor hardware or by instructions in software form. The method steps provided in conjunction with the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0339] This application also provides an electronic device, which may include a memory and a processor. The memory may be used to store a computer program; the processor may be used to invoke the computer program in the memory, causing the electronic device to execute the method executed by the electronic device in any of the above embodiments.

[0340] This application also provides a computer program product comprising: a computer program (also referred to as code or instructions) that, when run, causes a computer to perform the method executed by the electronic device in any of the above embodiments.

[0341] This application also provides a computer-readable storage medium storing a computer program (also referred to as code or instructions). When the computer program is run, it causes the computer to perform the method executed by the electronic device in any of the above embodiments.

[0342] The various embodiments of this application can be combined arbitrarily to achieve different technical effects.

[0343] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0344] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0345] In summary, the above description is merely an embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made based on the disclosure of this application should be included within the scope of protection of this application.

Claims

1. An interface display method characterized by, The method is applied to an electronic device, the electronic device comprising a hardware compositor, an image processor and a display screen; the method comprising: obtaining a trigger event, the trigger event being used to instruct the electronic device to display a first image through the display screen; in response to the trigger event, generating drawing information and generating expected layer information based on the drawing information, the expected layer information comprising a plurality of expected layers; based on a specification parameter of the electronic device, pre-verifying the plurality of expected layers to obtain a pre-verification result, the pre-verification result comprising indication information, the indication information being used to indicate that part or all of the plurality of expected layers are to be subjected to a composition process by the hardware compositor, or subjected to a composition process by the hardware compositor after being pre-processed, or subjected to a composition process by the image processor; generating a rendering instruction based on the pre-verification result and performing rendering based on the rendering instruction to obtain a superimposed layer set; based on the superimposed layer set, performing a composition process to obtain the first image and displaying the first image on the display screen.

2. The method of claim 1, wherein, The electronic device further comprises a system rendering service, the drawing information comprises first drawing information, and the expected layer information comprises first expected layer information; the generating of the drawing information and the generating of the expected layer information based on the drawing information comprises: generating the first drawing information by the system rendering service and generating the first expected layer information based on the first drawing information.

3. The method according to claim 1 or 2, characterized in that, The electronic device further comprises a pre-verification interface, and the pre-verification result comprises a first pre-verification result; the pre-verifying of the plurality of expected layers based on the specification parameter of the electronic device comprises: calling the pre-verification interface by the system rendering service, sending the first expected layer information to the hardware compositor; based on the specification parameter of the electronic device, pre-verifying, by the hardware compositor, a plurality of expected layers included in the first expected layer information to obtain a first pre-verification result.

4. The method of claim 3, wherein, The method further comprises: sending, by the hardware compositor, the first pre-verification result to the system rendering service through the pre-verification interface.

5. The method according to any one of claims 1 to 3, characterized in that, A first application is running on the electronic device; the drawing information further comprises second drawing information, and the expected layer information further comprises second expected layer information; the generating of the drawing information and the generating of the expected layer information based on the drawing information further comprises: generating the second drawing information by the first application and generating second expected layer information based on the second drawing information.

6. The method of claim 3, wherein, The electronic device further comprises a pre-verification interface, and the pre-verification result comprises a first pre-verification result and a second pre-verification result; the pre-verifying of the plurality of expected layers based on the specification parameter of the electronic device comprises: calling the pre-verification interface by the system rendering service, sending the first expected layer information to the hardware compositor; calling the pre-verification interface by the first application, sending the second expected layer information to the hardware compositor; The hardware synthesizer performs pre-checking on the plurality of expected layers included in the first expected layer information and the second expected layer information respectively based on the specification parameters of the electronic device, to obtain the first pre-checking result and the second pre-checking result.

7. The method of claim 5, wherein, The method further includes: the hardware synthesizer sending the first pre-checking result to the system rendering service and the second pre-checking result to the first application through the pre-checking interface.

8. The method of any one of claims 1, 2, 4, 6, 7, wherein, The drawing information includes one or more of expected rendering instructions, and descriptions of graphic resources, scene layout, and any rendering-related parameters; the expected rendering instructions include one or more of drawing commands, texture operations, and blending operations.

9. The method of claim 2, wherein, The electronic device further includes a pre-checking component deployed based on the specification parameters of the electronic device, and the pre-checking result includes a first pre-checking result; The pre-checking on the plurality of expected layers based on the specification parameters of the electronic device includes: The system rendering service calls the pre-checking component to obtain the specification parameters of the electronic device, and performs pre-checking on the plurality of expected layers included in the first expected layer information based on the specification parameters of the electronic device, to obtain the first pre-checking result.

10. The method of claim 2, wherein, The electronic device further includes a pre-checking component deployed based on the specification parameters of the electronic device, and the pre-checking result includes a first pre-checking result and a second pre-checking result; The pre-checking on the plurality of expected layers based on the specification parameters of the electronic device includes: The system rendering service calls the pre-checking component to obtain the specification parameters of the electronic device, and performs pre-checking on the plurality of expected layers included in the first expected layer information based on the specification parameters of the electronic device, to obtain the first pre-checking result; The first application calls the pre-checking component to obtain the specification parameters of the electronic device, and performs pre-checking on the plurality of expected layers included in the second expected layer information based on the specification parameters of the electronic device, to obtain the second pre-checking result.

11. The method of claim 1, wherein, The electronic device further includes a system rendering service, and the rendering instructions include a first rendering instruction; the generating of the rendering instructions based on the pre-checking result includes: The electronic device generates the first rendering instruction based on the first pre-checking result through the system rendering service.

12. The method of claim 1, wherein, The electronic device further includes a system rendering service, and the first application is running on the electronic device, and the rendering instructions include a first rendering instruction and a second rendering instruction; the generating of the rendering instructions based on the pre-checking result includes: The electronic device generates the first rendering instruction based on the first pre-checking result through the system rendering service; The electronic device generates the second rendering instruction based on the second pre-checking result through the first application.

13. The method of claim 1, wherein, The preprocessing includes one or more of format conversion, transparency processing, resolution adjustment, color space conversion, and layer simplification.

14. An electronic device, comprising: The preprocessing includes one or more of format conversion, transparency processing, resolution adjustment, color space conversion, and layer simplification. A hardware compositor, an image processor, a display, one or more processors and one or more memories; the one or more memories are coupled with the one or more processors, the memories are configured to store computer program codes, the computer program codes include computer instructions, when the one or more processors execute the computer instructions, the electronic device is caused to perform: obtaining a trigger event, the trigger event is used to instruct the electronic device to display a first image through a display screen; in response to the trigger event, generating drawing information, and generating expected layer information based on the drawing information, the expected layer information includes a plurality of expected layers; based on the specification parameters of the electronic device, pre-verifying the plurality of expected layers to obtain a pre-verification result, the pre-verification result includes indication information: the indication information is used to indicate that part or all of the plurality of expected layers are subjected to a composition process by the hardware compositor, or are subjected to a composition process by the hardware compositor after being pre-processed, or are subjected to a composition process by the image processor; based on the pre-verification result, generating a rendering instruction, and performing rendering based on the rendering instruction to obtain a superimposed layer set; based on the superimposed layer set, performing a composition process to obtain the first image, and displaying the first image on the display screen.

15. A computer storage medium, comprising: The computer storage medium stores a computer program, and the computer program is executed by a processor to implement the method in any one of claims 1-13.

16. A computer program product, characterised in that, The computer program product includes instructions, when the computer program product is executed by a computer, the computer is caused to perform the method in any one of claims 1-13.

Images