Multi-window display method and electronic device

Abstract

The application discloses a multi-window display method and an electronic device, relates to the technical field of display, and can provide pixel-level identification of blocked areas, avoids unnecessary drawing rendering tasks from causing processing load and processing resource waste of the electronic device. In the application, when multiple windows having a blocking relationship are displayed, pixel-level blocked pixel identification can be realized, only unblocked pixels are drawn and rendered, blocked pixels are not drawn and rendered, unnecessary drawing rendering tasks are eliminated to the maximum extent without affecting the display effect of the interface, processing load and processing resource waste caused by unnecessary drawing rendering tasks of the electronic device are avoided, problems such as display interface freezing and frame dropping caused by excessively high processing load are solved, and the use experience of users is improved.

Description

Technical Field

[0001] This application relates to the field of display technology, and in particular to multi-window display methods and electronic devices. Background Technology

[0002] Currently, in the interface display scenarios of personal computers (PCs) and other electronic devices, multi-window display is quite common to facilitate users' daily activities such as work and study. For example, electronic devices can support users opening multiple windows. Figure 1 As shown, multiple independent windows. When an electronic device displays multiple independent windows, such as... Figure 1 As shown, windows may be stacked in an overlapping manner, causing some or all areas of some windows to be invisible to the user.

[0003] However, when displaying an interface with multiple windows, electronic devices render and draw each window separately before displaying it. This results in unnecessary rendering and drawing of many invisible window areas. Especially when the user has many windows open, this excessive unnecessary rendering can overload the graphics processing unit (GPU) and other related processing modules, even exceeding their processing capabilities. This can cause display stuttering, frame drops, and other problems, negatively impacting the user experience. Summary of the Invention

[0004] This application provides a multi-window display method and electronic device, which can provide pixel-level identification of occluded areas and avoid unnecessary drawing and rendering tasks that would burden the electronic device with processing load and waste processing resources.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] In a first aspect, a multi-window display method is provided, which may include: in response to a first user operation that triggers the display of a first interface, an electronic device displays the first interface, wherein the number of rendered pixels of the first interface is a first rendered pixel count, the first interface includes a first window and a second window, and the first window and the second window have no occlusion relationship; in response to a second user operation that changes the relative position of the first window and the second window in the first interface, the electronic device displays a second interface, wherein the number of rendered pixels of the second interface is a second rendered pixel count, the first window in the second interface occludes a part of the second window, and the difference between the second rendered pixel count and the first rendered pixel count is linearly related to the number of pixels corresponding to the occluded area.

[0007] The solution provided in the first aspect above enables pixel-level identification of occluded pixels when an electronic device displays multiple windows with occlusion relationships. It allows for rendering only the unoccluded pixels and excludes the occluded pixels, thereby minimizing unnecessary rendering tasks without affecting the interface display. This avoids unnecessary rendering tasks from causing processing load and wasting processing resources on the electronic device, solving problems such as display interface stuttering and frame drops caused by excessive processing load, and improving the user experience.

[0008] It should be noted that in the second interface displayed by the electronic device, the occluded area in the second window is not visible to the user. When the electronic device renders the second window, it does not render the pixels corresponding to the occluded area. Therefore, the difference between the second rendered pixel count of the second interface and the first rendered pixel count of the first interface is linearly related to the pixel count corresponding to the occluded area.

[0009] For example, the larger the number of pixels corresponding to the occluded area, the greater the difference between the number of pixels rendered in the second rendering and the number of pixels rendered in the first rendering; the smaller the number of pixels corresponding to the occluded area, the smaller the difference between the number of pixels rendered in the second rendering and the number of pixels rendered in the first rendering.

[0010] As a possible example, the first user action is the action of the user opening a second window when the electronic device displays an interface including a first window (such as referred to as the fourth interface).

[0011] As a possible example, the first user action is the action of the user opening the first window when the electronic device displays an interface that includes a second window (such as referred to as the fifth interface).

[0012] As a possible example, the first user operation is when the electronic device displays an interface including a first window and a second window (such as referred to as the sixth interface), the user adjusts the stacking order of the first window and the second window, and / or adjusts one or more of the position, size, display attributes, etc. of the first window or the second window.

[0013] As a possible example, the first user action is the user closing or minimizing the fourth window when the electronic device displays an interface including a first window, a second window, and a fourth window (such as referred to as the seventh interface).

[0014] As a possible example, the second user action is an operation to adjust the position and / or size of the first window or the second window. The change in the position and / or size of the first window or the second window will change the relative position of the first window and the second window, causing the first window to obscure part of the second window.

[0015] As a possible example, an electronic device may first obtain the first rendered pixel count of the first interface and then display the first interface, or it may first display the first interface and then obtain the first rendered pixel count of the first interface, or it may display the first interface at the same time as obtaining the first rendered pixel count of the first interface.

[0016] As one possible implementation, before the electronic device displays the second interface, the method further includes: in response to a second user operation, the electronic device determines the pixels corresponding to the occluded area; the electronic device draws and renders the first window and the second window, and while rendering the second window, the electronic device abandons drawing and rendering the pixels corresponding to the occluded area. In this way, the electronic device can effectively eliminate unnecessary drawing and rendering tasks, avoiding the processing load and resource waste caused by unnecessary drawing and rendering tasks.

[0017] As one possible implementation, the electronic device determines the pixels corresponding to the occluded area by: the electronic device determining the hierarchical sorting values ​​of the first window and the second window according to the drawing order of the first window and the second window, and determining the first window as the occluder; the electronic device generating a drawing instruction set for the second interface, the drawing instruction set including drawing instructions for each pixel of the first window and the second window, each pixel's drawing instruction including the hierarchical sorting value of the window where the corresponding pixel is located; the electronic device constructing a template buffer according to the first window, the template buffer including a first region and a second region, the first region being the pixel region corresponding to the first window, the template value of the pixels in the first region being the hierarchical sorting value of the first window, the second region having no correspondence with the window, the template value of the pixels in the second region being 0; the electronic device comparing the hierarchical sorting value in the drawing instruction with the template value of the corresponding pixel in the template buffer for each pixel in the second window to determine whether the pixel is occluded. In this way, it can be compatible with the existing drawing and rendering process of electronic devices. By implementing logical improvements, such as generating drawing instructions with hierarchical sorting values ​​when generating drawing instructions, using occlusion bodies to construct template buffers, and comparing the hierarchical sorting values ​​in the drawing instructions with the template values ​​in the buffers during drawing and rendering, it can determine whether a pixel is occluded, thereby providing pixel-level inter-window occlusion area recognition functionality.

[0018] As one possible implementation, the electronic device determines the hierarchy sort values ​​of the first and second windows based on their drawing order, and identifies the second window as an occluder. This includes: the electronic device determines, based on the drawing order of the first and second windows, that the hierarchy sort value of the first window (drawn later) is a first value, and the hierarchy sort value of the second window (drawn earlier) is a second value, where the first value is greater than the second value, and identifies the first window (drawn later) as an occluder. In this way, the hierarchy sort values ​​of multiple windows and the occluder can be determined in an orderly, efficient, and accurate manner.

[0019] As one possible implementation, the electronic device compares the pixel value in the drawing instruction with the corresponding template value in the template buffer for each pixel in the second window to determine whether the pixel is occluded. This includes: if the level sort value in the drawing instruction is less than the corresponding template value in the template buffer, the electronic device determines that the pixel is occluded; if the level sort value in the drawing instruction is greater than or equal to the corresponding template value in the template buffer, the electronic device determines that the pixel is not occluded. This allows for efficient and accurate determination of whether a pixel is occluded.

[0020] As one possible implementation, the above method is applied to an electronic device, which includes a graphics system. The graphics system determines the hierarchical sorting values ​​of the first and second windows based on their drawing order, identifies the first window as an occlusion, generates a drawing instruction set for the second interface, and constructs a template buffer based on the first window. In this way, the electronic device can be compatible with existing graphics systems, relying on the graphics system's ability to aggregate information from each window to determine the occluded area at the pixel level.

[0021] As one possible implementation, the electronic device includes a graphics rendering pipeline. This pipeline compares the hierarchical sort value in the drawing instruction with the corresponding template value in the template buffer for each pixel in the second window to determine whether the pixel is occluded. In this way, the electronic device can be compatible with existing graphics rendering pipelines and processing flows, enabling the determination of occluded areas at the pixel level.

[0022] As one possible implementation, the graphics rendering pipeline compares the hierarchical sort value in the drawing instruction with the corresponding stencil value in the stencil buffer for each pixel in the second window to determine whether the pixel is occluded. This includes: after rasterizing the pixels in the second window, the graphics rendering pipeline performs stencil detection, where stencil detection includes: determining whether a pixel is occluded by comparing the hierarchical sort value in the drawing instruction with the corresponding stencil value in the stencil buffer. In this way, the electronic device can be compatible with existing graphics rendering pipelines, fully utilizing the processing flow reserved in the graphics rendering pipeline to achieve pixel-level determination of occluded areas.

[0023] As one possible implementation, the electronic device abandons rendering the pixels corresponding to the occluded area. This includes: the electronic device abandoning fragment shading and one or more other rendering processes on the pixels corresponding to the occluded area through the graphics rendering pipeline. In this way, it can be compatible with the existing processing flow of the graphics rendering pipeline, eliminating pixels that do not need to be rendered before fragment shading, minimizing the number of pixels involved in fragment shading, and avoiding unnecessary processing load and waste of processing resources on the electronic device caused by unnecessary fragment shading and other rendering tasks.

[0024] As one possible implementation, the first and second interfaces also include a third window. The third window in the first interface has no occlusion relationship with either the first or second window. The third window in the second interface occludes the second window, and the third window is occluded by the first window. The method further includes: the electronic device determining the hierarchical sorting value of the third window based on its drawing order; and the electronic device comparing the hierarchical sorting value in the drawing instruction with the corresponding template value in the template buffer for each pixel in the third window to determine whether the pixel is occluded. This allows for flexible selection of occluders used to construct the template buffer, considering both benefits and processing costs. While achieving greater culling benefits, it avoids excessive resource consumption, long processing time, and high processing costs associated with template testing and pixel culling due to excessive template data, thus improving rendering efficiency.

[0025] As one possible implementation, the above method further includes: the electronic device determining the third window as an occluder; when constructing the template buffer, the electronic device constructs a third region in the template buffer based on the third window, the third region being the pixel region corresponding to the third window that is not occluded by the first window, and the template value of the pixels in the third region being the hierarchical sort value of the third window. In this way, the construction of the template buffer can fully consider each occluder, maximizing the elimination of pixels that do not need to be drawn and rendered, and avoiding unnecessary drawing and rendering tasks that would burden the electronic device with processing load and waste processing resources.

[0026] In a second aspect, an electronic device is provided, comprising: a display screen for displaying an interface; a memory for storing computer program instructions; and a processor for executing the computer program instructions to support the electronic device in implementing the methods as described in any possible implementation of the first aspect.

[0027] Thirdly, a computer-readable storage medium is provided that stores computer program instructions that, when executed by a processor, implement the method as described in any possible implementation of the first aspect.

[0028] Fourthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to implement the method as described in any possible implementation of the first aspect.

[0029] Fifthly, a chip system is provided, comprising processing circuitry and a storage medium storing computer program instructions; when executed by the processor, the computer program instructions implement the method as described in any possible implementation of the first aspect. The chip system may be composed of chips or may include chips and other discrete devices. Attached Figure Description

[0030] Figure 1 A multi-window display interface embodiment provided in this application is shown in the figure;

[0031] Figure 2 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0032] Figure 3 A schematic diagram of the software structure of an electronic device provided in an embodiment of this application;

[0033] Figure 4A A flowchart illustrating a multi-window display process is provided in this application embodiment;

[0034] Figure 4B A flowchart of a multi-window display method provided in an embodiment of this application;

[0035] Figure 5A Example diagram of a first type of second interface provided in the embodiments of this application;

[0036] Figure 5B This is an example diagram of a second type of second interface provided in an embodiment of this application;

[0037] Figure 6 This application provides an example diagram of an implementation architecture.

[0038] Figure 7 This is a schematic diagram of an interface drawing and rendering process provided in an embodiment of this application;

[0039] Figure 8A A schematic diagram of a template buffer provided in an embodiment of this application;

[0040] Figure 8B This is a schematic diagram of the first template testing and pixel removal process provided in the embodiments of this application;

[0041] Figure 8C This is a schematic diagram of the second template testing and pixel removal process provided in the embodiments of this application;

[0042] Figure 8D This is a schematic diagram of the third template testing and pixel removal process provided in the embodiments of this application;

[0043] Figure 8E This is a schematic diagram of the fourth template testing and pixel removal process provided in the embodiments of this application;

[0044] Figure 8F This is a schematic diagram of the fifth template testing and pixel removal process provided in the embodiments of this application;

[0045] Figure 9 This is a schematic diagram of a multi-window drawing and rendering process provided in an embodiment of this application;

[0046] Figure 10A This is a schematic diagram of another template buffer provided in an embodiment of this application;

[0047] Figure 10B This is a schematic diagram of the sixth template testing and pixel removal process provided in the embodiments of this application;

[0048] Figure 10C This is a schematic diagram of another multi-window drawing and rendering process provided in an embodiment of this application;

[0049] Figure 11 A virtual functional structure block diagram of an electronic device provided in this application embodiment;

[0050] Figure 12 This is a hardware structure block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0052] Hereinafter, the terms "first," "second," etc., are used only to distinguish different descriptive objects and do not limit the position, order, priority, quantity, or content of the described objects. The use of ordinal numbers and other prefixes used to distinguish descriptive objects in the embodiments of this application does not constitute a limitation on the described objects. The description of the described objects is given in the claims or the context of the embodiments, and should not constitute an unnecessary limitation due to the use of such prefixes.

[0053] Furthermore, in the embodiments of this application, "connection" can be a direct connection or an indirect connection; in addition, it can refer to an electrical connection or a communication connection; for example, the connection of two electrical components A and B can refer to A and B being directly connected, or it can refer to A and B being indirectly connected through other electrical components or connection media, or it can refer to A and B being indirectly connected through other communication devices or communication media, as long as it enables communication between A and B.

[0054] To avoid unnecessary rendering tasks and resource waste on electronic devices when displaying multiple staggered windows, this application provides a multi-window display method. This method provides a graphics system-level window occlusion area recognition mechanism. By comprehensively analyzing the stacking of multiple windows, it achieves pixel-level occlusion area recognition. Therefore, when rendering windows, the rendering tasks of pixels corresponding to occluded areas are abandoned. Without affecting the interface display effect, it avoids unnecessary rendering tasks and resource waste on electronic devices, solves the problem of insufficient processing power caused by unnecessary rendering, and thus improves the user experience by preventing display interface lag and frame drops.

[0055] As an example, an electronic device can display a first interface in response to a first user operation that triggers the display of the first interface. The first interface has a first number of rendered pixels, and the first window and the second window in the first interface do not obscure each other. When the user performs a second user operation that changes the relative position of the first window and the second window in the first interface, the electronic device can display a second interface including the first window and the second window in response to the second user operation. The second interface has a second number of rendered pixels, and the first window in the second interface obscures a portion of the second window.

[0056] The difference between the second and first rendered pixel counts is linearly related to the number of pixels corresponding to the occluded area. For example, the larger the number of pixels corresponding to the occluded area, the larger the difference between the second and first rendered pixel counts; the smaller the number of pixels corresponding to the occluded area, the smaller the difference between the second and first rendered pixel counts.

[0057] As an example, the electronic device described in this application embodiment may include, but is not limited to, any device with window display functionality, such as a smartphone, netbook, tablet computer, smart drawing tablet, handwriting tablet, smartwatch, smart bracelet, phone watch, smart glasses, smart camera, PDA, in-vehicle computer, personal computer (PC), personal digital assistant (PDA), portable multimedia player (PMP), augmented reality (AR) / virtual reality (VR) device, motion-sensing game console in human-computer interaction scenarios, or in-vehicle infotainment system, etc. Alternatively, the electronic device may also be other types or functions of devices, which are not limited in this application.

[0058] For example, please refer to Figure 2 , Figure 2 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application is shown.

[0059] like Figure 2 As shown, the electronic device includes a processor 210, a memory (including an external memory interface 220 and an internal memory 221), a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headphone jack 270D, a sensor module 280, buttons 290, a motor 291, indicator lights 292, a camera 293, a display screen 294, etc.

[0060] The processor 210 includes one or more processing units. For example, the processor 210 may include an application processor (AP), a microcontroller unit (MCU), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. For example, the application processor may include a graphics processor and a digital signal processor, and the microcontroller unit may include a graphics processor.

[0061] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0062] The processor 210 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. This memory can store instructions or data that the processor 210 has just used or that are used repeatedly. If the processor 210 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0063] In some embodiments of this application, when an electronic device is about to display a second interface including multiple windows, the processor 210 can determine the vertical stacking of the multiple windows according to the drawing order of the multiple windows, and determine the pixels corresponding to the occluded area according to the vertical stacking of the multiple windows and the pixel position of each window. When drawing and rendering multiple window layers, the processor 210 can abandon the drawing and rendering of the pixels corresponding to the occluded area.

[0064] For example, taking a second interface including a first window and a second window, where the first window occludes all or part of the area of ​​the second window, the processor 210 can determine the vertical stacking of the first window and the second window based on the drawing order of the first window and the second window. Based on the vertical stacking of the windows and the position of each pixel in each window, the processor can determine the pixels corresponding to the occluded area in the second window. When drawing and rendering the second window, the processor can abandon the drawing and rendering of the pixels corresponding to the occluded area.

[0065] For example, taking a second interface that includes a first window, a second window, and a third window, where the first window occludes all or part of the area of ​​the second window, the third window occludes all or part of the area of ​​the second window, and the first window occludes all or part of the area of ​​the third window, the processor 210 can determine the vertical stacking of the first, second, and third windows based on the drawing order of the first window, the second window, and the third window. Based on the vertical stacking of the windows and the position of each pixel in each window, the processor can determine the pixels corresponding to the occluded areas in the second and third windows. When drawing and rendering the second and third windows, the processor can abandon the drawing and rendering of the pixels corresponding to the occluded areas.

[0066] As an example, the processor 210 can sort multiple windows hierarchically based on their stacking arrangement on the second interface, and generate drawing instructions based on the hierarchical sorting values ​​of each window, such as generating a drawing instruction set. This drawing instruction set includes drawing instructions for each pixel within each window, and each pixel's drawing instruction includes the hierarchical sorting value of the window containing that pixel. When drawing and rendering multiple window layers according to the drawing instruction set, the processor determines whether a pixel is occluded based on the hierarchical sorting value in the drawing instructions for each pixel within the multiple windows. If it is determined that a pixel is occluded, the drawing and rendering of that pixel is abandoned; if it is determined that a pixel is not occluded, the drawing and rendering of that pixel proceeds normally.

[0067] The wireless communication function of the electronic device can be realized through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem processor and baseband processor, etc.

[0068] Antennas 1 and 2 are used to transmit and receive electromagnetic wave signals. The wireless communication module 260 can provide solutions for wireless communication applications in electronic devices, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), and intrabody communication (IBC). The wireless communication module 260 can be one or more devices integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 210. The wireless communication module 260 can also receive signals to be transmitted from processor 210, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0069] In some embodiments of this application, antenna 1 of the electronic device is coupled to mobile communication module 250, and antenna 2 is coupled to wireless communication module 260, enabling the electronic device to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, IR technology, and / or IBC technology, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0070] In some embodiments of this application, the electronic device can realize communication functions through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, etc.

[0071] In some embodiments of this application, the electronic device can communicate with one or more application servers through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, etc., to obtain application interface data in the window of the first interface, application interface data in the window of the second interface, etc.

[0072] Display screen 294 includes a display panel. Exemplarily, the display panel may be a low-temperature poly-silicon (LTPS) display, a low-temperature polycrystalline oxide (LTPO) display, a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED) display, a flexible light-emitting diode (FLED), a Miniled LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc.

[0073] In this embodiment of the application, the electronic device may include one or more displays 294.

[0074] In some embodiments of this application, the electronic device can implement display functions through a GPU, display screen 294, application processor, microcontroller unit, etc., such as displaying an interface including multiple windows (e.g., a first interface or a second interface). The GPU is a microprocessor for image processing, connected to the display screen 294, application processor, and microcontroller unit. The GPU is used to perform mathematical and geometric calculations, and for drawing, rendering, or compositing graphics. The processor 210 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0075] In some embodiments of this application, the GPU can be used to perform vertex processing, clipping, rasterization, template testing and pixel culling, fragment shading and transparency blending of geometric data of multiple windows on an interface (such as a first interface or a second interface) through a graphics rendering pipeline (such as a GPU-Pipeline) to complete the rendering of pixels corresponding to the unoccluded areas in multiple windows.

[0076] Electronic devices can implement shooting functions through an ISP, a camera 293, a video codec, a GPU, a display 294, and an application processor. The ISP is used to process data fed back by the camera 293, and the camera 293 is used to capture images, such as pictures, videos, or GIFs. In some embodiments, one or N cameras 293 may be included, where N is a positive integer greater than 1.

[0077] The sensor module 280 includes one or more sensors such as a touch sensor 280A and a pressure sensor 280B.

[0078] In some embodiments of this application, one or more sensors, such as touch sensor 280A and pressure sensor 280B, can be used to detect operations for triggering the display of the first interface (such as a first user operation), operations for changing the relative position of the first window and the second window in the first interface (such as a second user operation), etc.

[0079] The external memory interface 220 can be used to connect external memory cards, such as Micro SD cards, to expand the storage capacity of electronic devices. The external memory card communicates with the processor 210 through the external memory interface 220 to perform data storage.

[0080] Internal memory 221 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM). The RAM can be directly read and written by the processor 210 and can be used to store executable programs (e.g., machine instructions) of the operating system or other running programs, as well as user and application data. The NVM can also store executable programs and user and application data, and can be pre-loaded into the RAM for direct read and write operations by the processor 210.

[0081] In some embodiments of this application, the external storage card and / or internal memory 221 can be used to store interface data, including the drawing order of multiple windows in the interface, geometric data (such as window geometric data and graphic geometric data, etc.).

[0082] In some embodiments of this application, the external storage card and / or internal memory 221 can be used to store template data of occluders in multiple windows, template buffer data, etc.

[0083] In some embodiments of this application, the external storage card and / or internal memory 221 can be used to store a set of drawing instructions for an interface including multiple windows.

[0084] about Figure 2 The descriptions of the charging management module 240, power management module 241, battery 242, audio module 270, speaker 270A, receiver 270B, motor 291, indicator light 292, etc. shown can be found in conventional technology and will not be elaborated here.

[0085] This application Figure 2The illustrated structure does not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0086] The electronic device provided in this application embodiment can run an operating system (OS), which can be various operating systems used in the industry, such as those based on OpenHarmony. TM Developed operating systems, such as Harmony TM OS; or other operating systems such as Android TM iOS TM Mobile operating system; it can also be various open-source operating systems or their derivatives, such as Linux. TM Operating systems (OS) include other embedded operating systems; they can also be future new operating systems, such as artificial intelligence (AI) operating systems. An operating system is a set of interconnected system software programs that manage and control the operation of electronic devices, utilize and run hardware and software resources, and provide public services to organize user interaction. In electronic devices, the operating system connects downwards to the physical hardware layer and upwards to provide a runtime environment for application software.

[0087] An operating system typically includes a kernel layer, a middleware layer, and an application layer. The application layer includes applications, which can include system applications and third-party applications. The middleware layer includes a suite of software providing various services to application developers, or frameworks providing services such as databases, multimedia, and graphics, or capabilities such as distributed scheduling and system scaling. For example, the middleware layer may include a framework layer and / or a system service layer. The framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The system service layer includes the core capabilities of the system, providing services to applications through the framework layer. The kernel layer is the layer between hardware and software. The kernel layer may include hardware drivers and the operating system kernel. In addition to providing hardware drivers, the kernel layer also supports functions such as memory management and system process management.

[0088] The electronic devices we use in our daily lives come in various types and forms, and are applied in a wide range of scenarios. Therefore, based on the different forms and functions of electronic devices, different application scenarios, and different user needs, the operating systems used in these devices may also differ. The basic functions implemented by the electronic device provided in this application can be implemented using a general-purpose operating system or a dedicated operating system. To more clearly illustrate the implementation of the embodiments of this application under a specific operating system, the following shows Harmony... TM Based on the architecture of the operating system, those skilled in the art can deduce the implementation of the embodiments of this application under other specific operating systems, such as Android. TM Implementation under operating systems, etc.

[0089] As an example, please refer to Figure 3 , Figure 3 This illustration shows a schematic diagram of the software structure of an electronic device according to an embodiment of this application. For example... Figure 3 As shown, the software architecture of an electronic device can be divided into several layers. In some embodiments, from bottom to top, these layers are: kernel layer, system service layer, framework layer, and application layer. Layers communicate with each other through software interfaces. System functions can be tailored, added, or combined at the subsystem granularity in different device deployment scenarios, and each subsystem can also be tailored, added, or combined at the functional granularity.

[0090] As an example, the kernel layer may include the kernel abstract layer (KAL), kernel subsystems, and driver subsystems.

[0091] The kernel abstraction layer provides basic kernel capabilities to upper layers by shielding the differences between multiple kernels, including but not limited to process / thread management, memory management, file system, network management, and peripheral device management. The kernel subsystem supports the selection of appropriate operating system kernels for different resource-constrained devices. The driver subsystem provides unified peripheral access capabilities and a framework for driver development and management, such as providing unified display drivers, camera drivers, audio drivers, Bluetooth drivers, sensor drivers, and storage drivers.

[0092] In this embodiment, the kernel layer can provide an interface for calling the graphics rendering pipeline, such as the GPU-Pipeline. For example, the kernel layer calls the graphics rendering pipeline through the interface, and the graphics rendering pipeline can perform drawing and rendering according to the call from the kernel layer, including rasterization of geometric data. After rasterization, template testing and pixel culling are performed to identify occluded areas in multiple windows at the pixel level. After the identification of occluded areas at the pixel level is completed, subsequent drawing and rendering processing of the pixels corresponding to the occluded areas is abandoned, such as abandoning one or more processing such as fragment shading and alpha blending of the pixels corresponding to the occluded areas.

[0093] In some embodiments of this application, prior to rasterizing the geometric data, the graphics rendering pipeline (such as a GPU-Pipeline) may also perform one or more of the following processing on the geometric data: vertex processing, clipping.

[0094] As an example, the system service layer includes the core set of system capabilities used to provide services to the application layer, such as providing various basic services. This layer includes multiple sets of subsystems, such as, but not limited to, one or more sets of system basic capability subsystems, basic software service subsystems, enhanced software service subsystems, and hardware service subsystems.

[0095] The system comprises several subsystems, including: a basic capability subsystem set providing foundational capabilities for the operation, scheduling, and migration of distributed applications across multiple devices; a multi-language runtime environment (MLU) including distributed soft bus, distributed data management, distributed task scheduling, and Ark multi-language runtime; and a multi-modal input subsystem, graphics system, security subsystem, and AI business subsystem. A basic software service subsystem set provides common, general-purpose software services; including event notification subsystem, telephone service subsystem, and multimedia subsystem. An enhanced software service subsystem set provides differentiated, enhanced software services for different devices; including dedicated business subsystems for smart screens, wearables, and the Internet of Things (IoT). A hardware service subsystem set provides hardware services; including location service subsystem, identity and access management (IAM) subsystem, wearable-specific hardware service subsystem, biometric recognition, and IoT-specific hardware service subsystem.

[0096] The framework layer provides APIs and programming frameworks for applications in the application layer. The framework layer includes the ArkUI framework, the user application framework, and the Ability framework. The ArkUI framework provides a complete infrastructure for UI development of system applications, including UI functionalities such as components, layouts, animations, and interactive events, as well as a real-time interface preview tool. The Ability framework is a lightweight application that schedules and manages the operation and lifecycle of Abilities. Different devices may run different operating systems, and therefore support different APIs. APIs are a series of open capabilities provided to support operating system application development. APIs can be set in the framework layer or independently of the framework layer.

[0097] In some embodiments of this application, a graphics system supported by a system service layer and a framework layer is used to handle graphics display-related tasks.

[0098] As an example, a graphics system can be used to communicate with applications in the application layer via APIs to provide services such as graphics drawing, rendering, and compositing for unobstructed areas in the application's window, generating display data. For instance, the graphics system can use a rendering engine to convert data from unobstructed areas into pixel data on the screen, so that display-related modules can send this pixel data for display.

[0099] In some embodiments of this application, the graphics system can be used to comprehensively analyze the stacking of multiple windows when an electronic device is displaying an interface comprising multiple windows, to perform hierarchical sorting of the multiple windows, for example, from bottom to top, and to generate a drawing instruction set that takes into account the occlusion between multiple windows. The system then performs drawing and rendering of the multiple windows according to the drawing instruction set. The kernel layer can call the graphics rendering pipeline through an interface. When drawing and rendering multiple windows, the graphics rendering pipeline can identify occluded areas at the pixel level. After identifying the occluded areas at the pixel level, it performs subsequent drawing and rendering processing on the pixels corresponding to the unoccluded areas, and discards the drawing and rendering of the pixels corresponding to the occluded areas.

[0100] For example, taking a first interface consisting of a first window and a second window, where the first and second windows do not obstruct each other, the graphics system can comprehensively analyze the stacking of the first and second windows, perform hierarchical sorting of the first and second windows (e.g., from bottom to top), and generate a drawing instruction set that considers the occlusion between windows. The system then performs drawing and rendering of the first and second windows based on this instruction set. During the drawing and rendering of the first and second windows, the graphics rendering pipeline can identify occluded areas at the pixel level, determine that there are no occluded areas in either window, and perform full drawing and rendering of all pixels in both windows.

[0101] For example, taking a second interface consisting of a first window and a second window, where the first window occludes all or part of the second window, the graphics system can comprehensively analyze the stacking of the first and second windows, perform hierarchical sorting (e.g., from bottom to top), and generate a drawing instruction set that considers the occlusion between windows. The system then renders the first and second windows according to this instruction set. During the rendering of the first and second windows, the graphics rendering pipeline can identify occluded areas at the pixel level. For instance, it can identify pixels corresponding to occluded areas and pixels corresponding to unoccluded areas in the second window, determine that there are no occluded areas in the first window, and then render the pixels in the first window and the pixels corresponding to the unoccluded areas in the second window, while discarding the rendering of pixels corresponding to occluded areas in the second window.

[0102] In some embodiments of this application, the graphics rendering pipeline can also be used to consider the screen characteristics of the electronic device, such as resolution, refresh rate, or one or more other characteristics, when performing graphics rendering, so as to ensure that the generated interface display data can be correctly displayed on the screen and to obtain a display effect and display performance that better matches the screen characteristics.

[0103] The application layer includes a series of applications, such as Figure 3 The examples shown include video, map, music, and game applications. Of course, applications in the application layer can also include any other native applications (such as memos, contacts, camera, gallery, calendar, etc. integrated into the operating system) or functions, and / or other third-party applications (such as shopping, instant messaging, etc., downloaded and installed by the user through an app store). This application's embodiments are not limited to these.

[0104] In some embodiments of this application, the multiple windows displayed by the electronic device may include the window of at least one application in the application layer (hereinafter referred to as "application"), including the windows of native applications and / or third-party applications.

[0105] This application Figure 3The illustrated structure does not constitute a specific limitation on the software architecture of an electronic device. In other embodiments of this application, the various levels of the electronic device may include more or fewer subsystems than illustrated, or combine some subsystems, or split some subsystems, or have different subsystem layouts.

[0106] The multi-window display method provided in the embodiments of this application will be described in detail below.

[0107] In some embodiments of this application, the electronic device can comprehensively analyze the stacking of multiple windows to be displayed, sort the multiple windows hierarchically, generate drawing instructions based on the hierarchical sorting results, and when drawing and rendering multiple windows according to the drawing instructions, identify the occluded areas in the multiple windows at the pixel level and abandon drawing and rendering the pixels corresponding to the occluded areas.

[0108] As an example, please refer to Figure 4A , Figure 4A A flowchart illustrating a multi-window display process according to an embodiment of this application is shown. This method can be applied to electronic devices. Figure 4A As shown, the implementation process of this method includes S401A-S402A:

[0109] S401A, in response to a first user operation, a first interface is displayed. The number of rendered pixels of the first interface is the first number of rendered pixels. The first user operation is an operation that triggers the display of the first interface. The first interface includes a first window and a second window. The first window and the second window do not overlap.

[0110] As a possible example, the first user action is the action of the user opening a second window when the electronic device displays an interface including a first window (such as referred to as the fourth interface).

[0111] As a possible example, the first user action is the action of the user opening the first window when the electronic device displays an interface that includes a second window (such as referred to as the fifth interface).

[0112] As a possible example, the first user operation is when the electronic device displays an interface including a first window and a second window (such as referred to as the sixth interface), the user adjusts the stacking order of the first window and the second window, and / or adjusts one or more of the position, size, display attributes, etc. of the first window or the second window.

[0113] As a possible example, the first user action is the user closing or minimizing the fourth window when the electronic device displays an interface including a first window, a second window, and a fourth window (such as referred to as the seventh interface).

[0114] The embodiments of this application do not limit the operation scenarios, interface conditions, and operation methods of the first user operation. For example, the operation methods of the first user operation may include, but are not limited to, touch operation, voice command, gesture operation, body movement, etc. Touch operation may include, but are not limited to, click operation (such as single click, double click, etc.), long press operation, preset swipe operation, etc., depending on the specific situation of device structure, function, settings, etc.

[0115] S402A, in response to a second user operation, a second interface is displayed. The number of rendered pixels of the second interface is the second number of rendered pixels. The second user operation is an operation to change the relative position of the first window and the second window in the first interface. The first window in the second interface occludes part of the second window. The difference between the second number of rendered pixels and the first number of rendered pixels is linearly related to the number of pixels corresponding to the occluded area.

[0116] As an example, the difference between the second number of rendered pixels and the first number of rendered pixels is linearly related to the number of pixels corresponding to the occluded area. For example, the larger the number of pixels corresponding to the occluded area, the larger the difference between the second number of rendered pixels and the first number of rendered pixels; the smaller the number of pixels corresponding to the occluded area, the smaller the difference between the second number of rendered pixels and the first number of rendered pixels.

[0117] In some embodiments, the first number of rendered pixels minus the second number of rendered pixels equals the number of pixels in the occluded area.

[0118] In some embodiments, the value of (first rendered pixels - second rendered pixels - number of pixels in the occluded area) / number of pixels in the occluded area is less than a preset ratio.

[0119] As a possible example, the second user action is an operation to adjust the position and / or size of the first window or the second window. The change in the position and / or size of the first window or the second window will change the relative position of the first window and the second window, causing the first window to obscure part of the second window.

[0120] This application does not limit the operation scenarios and methods of the second user operation. For example, the operation methods of the second user operation may include, but are not limited to, touch operation, voice command, gesture operation, body movement, etc., depending on the specific circumstances such as device structure, function, and settings.

[0121] In the second interface displayed by the electronic device, the occluded area in the second window is not visible to the user. When the electronic device renders the second window, it does not render the pixels corresponding to the occluded area. Therefore, the difference between the second rendered pixel count of the second interface and the first rendered pixel count of the first interface is linearly related to the pixel count corresponding to the occluded area.

[0122] The embodiments of this application do not limit the order in which the electronic device obtains the first rendered pixel count of the first interface and displays the first interface. For example, the electronic device may obtain the first rendered pixel count of the first interface first and then display the first interface, or it may display the first interface first and then obtain the first rendered pixel count of the first interface, or it may display the first interface at the same time as obtaining the first rendered pixel count of the first interface.

[0123] Figure 4A As an example of an interface display process, in some embodiments, the electronic device may also, in response to a third user operation that changes the relative position of the first window and the second window in the second interface, acquire a third rendered pixel count of the third interface and display a third interface including the first window and the second window, wherein the first window in the third interface occludes a portion of the second window. The difference between the third rendered pixel count and the second rendered pixel count is linearly related to the change in the number of pixels in the occluded area.

[0124] In this embodiment, upon receiving a first user operation, the electronic device, in response to the first user operation, can comprehensively analyze the stacking of the first window and the second window in the first interface, sort the first window and the second window hierarchically, and generate a drawing instruction set that considers the occlusion between windows. The electronic device then performs drawing and rendering of the first window and the second window according to the drawing instruction set. During the drawing and rendering of the first window and the second window, the electronic device can identify occluded areas at the pixel level, determine that there are no occluded areas in the first window and the second window, and perform full drawing and rendering processing on the pixels of the first window and the pixels of the second window.

[0125] When a user performs a second user operation to change the relative position of the first and second windows in the first interface, in response to this operation, the electronic device can comprehensively analyze the stacking of the first and second windows in the second interface, sort the first and second windows hierarchically, and generate a drawing instruction set that takes into account the occlusion between windows. The electronic device then performs drawing and rendering of the first and second windows according to this instruction set. During the drawing and rendering of the first and second windows, the electronic device can identify occluded areas at the pixel level, such as identifying pixels corresponding to occluded areas and unoccluded areas in the second window. If it determines that there are no occluded areas in the first window, it performs drawing and rendering processing on the pixels of the first window and the pixels corresponding to the unoccluded areas in the second window, while discarding the drawing and rendering of pixels corresponding to occluded areas in the second window.

[0126] As one possible implementation, in response to a second user operation, the electronic device determines the pixels corresponding to the occluded area and renders a first window and a second window. When rendering the second window, the pixels corresponding to the unoccluded area are rendered, and the rendering of the pixels corresponding to the occluded area is abandoned.

[0127] As an example, the electronic device determines the pixels corresponding to the occluded area by: determining the hierarchical sort values ​​of the first and second windows based on their drawing order, and identifying the first window as the occluder; generating a drawing instruction set for the second interface, including drawing instructions for each pixel of the first and second windows, with each pixel's drawing instruction including the hierarchical sort value of the window containing the corresponding pixel; constructing a template buffer based on the first window as the occluder, the template buffer including a first region and a second region, the first region being the pixel region corresponding to the first window, with the template value of the pixels in the first region being the hierarchical sort value of the first window, and the second region having no correspondence with the window, with the template value of the pixels in the second region being 0; and comparing the hierarchical sort value in the drawing instructions with the template value of the corresponding pixel in the template buffer for each pixel in the second window to determine whether the pixel is occluded. In this way, the pixels corresponding to the occluded area can be determined.

[0128] As an example, the electronic device determines the hierarchical sorting value of the first window and the hierarchical sorting value of the second window based on the drawing order of the first window and the second window, and determines the first window as an occluder, including: the electronic device determines the hierarchical sorting value of the first window (drawn later) as a first value and the hierarchical sorting value of the second window (drawn earlier) as a second value based on the order of the first window and the second window, wherein the first value is greater than the second value, and determines the first window (drawn later) as an occluder.

[0129] For example, the first value is 2, the second value is 1; or the first value is 3, the second value is 1 or 2; or the first value is 4, the second value is 1, 2 or 3; or the first value is 5, the second value is 1, 2, 3 or 4, and so on.

[0130] As an example, when determining whether a pixel is occluded, the electronic device can determine that the pixel is occluded if the hierarchical sort value in the pixel's drawing instruction is less than the template value of the corresponding pixel in the template buffer; and determine that the pixel is not occluded if the hierarchical sort value in the pixel's drawing instruction is greater than or equal to the template value of the corresponding pixel in the template buffer.

[0131] For example, suppose the layer sort value in the drawing instruction of each pixel in the second window is 1, the template value of the pixel corresponding to the first region in the template buffer is 2, and the template value of the pixel corresponding to the second region in the template buffer is 0. The electronic device can compare the layer sort value in the drawing instruction of each pixel in the second window with the template value of the corresponding pixel in the template buffer pixel by pixel, and determine that the pixel with the layer sort value of 1 in the drawing instruction and the template value of 0 in the template buffer is not occluded, and determine that the pixel with the layer sort value of 1 in the drawing instruction and the template value of 2 in the template buffer is occluded.

[0132] Figure 4A Taking the example that both the first interface and the second interface include a first window and a second window, in some embodiments, the first interface may also include one or more other windows that do not have an obstruction relationship with the first window and the second window, such as a third window. In response to the second user operation, the second interface also includes a third window, in which the third window in the second interface obscures the second window and the third window is obscured by the first window.

[0133] Suppose a third window in the second interface occludes the second window, and the third window is occluded by the first window. The template buffer is constructed based on the template data of the first window. The electronic device can compare the hierarchical sort value in the drawing instruction with the template value of the corresponding pixel in the template buffer for each pixel in the second window to determine whether the pixel is occluded. In this way, the pixel corresponding to the occluded area in the second window can be determined. Similarly, the electronic device can determine the hierarchical sort value of the third window based on the drawing order of the third window, and compare the hierarchical sort value in the drawing instruction with the template value of the corresponding pixel in the template buffer for each pixel in the third window to determine whether the pixel is occluded. In this way, the pixel corresponding to the occluded area in the third window can be determined.

[0134] Alternatively, suppose a third window in the second interface occludes the second window, and the third window is also occluded by the first window. The template buffer is constructed based on the template data of the first and third windows. For example, a third region is constructed in the template buffer based on the third window. The third region is the pixel region corresponding to the third window that is not occluded by the first window, and the template value of the pixels in the third region is the hierarchical sort value of the third window. The electronic device can compare the hierarchical sort value in the drawing instruction with the template value of the corresponding pixel in the template buffer for each pixel in the second window to determine whether the pixel is occluded. In this way, the pixels corresponding to the occluded area in the second window can be determined. Furthermore, the electronic device can determine the hierarchical sort value of the third window based on its drawing order, and compare the hierarchical sort value in the drawing instruction with the template value of the corresponding pixel in the template buffer for each pixel in the third window to determine whether the pixel is occluded. In this way, the pixels corresponding to the occluded area in the third window can be determined.

[0135] As an example, please refer to Figure 4B , Figure 4B A flowchart illustrating a multi-window display method with occlusion relationships provided in an embodiment of this application is shown. Figure 4B As shown, the implementation process of this method includes S401B-S404B:

[0136] S401B, in response to a second user operation, the electronic device acquires data of a second interface to be displayed, the second interface including multiple windows, and the multiple windows including an obscured area.

[0137] As an example, the multiple windows include a first window and a second window, with the first window obscuring all or part of the area of ​​the second window. Optionally, the multiple windows may also include at least one window (such as a third window), which may or may not obscure the first or second window, and this is not limited.

[0138] As an example, multiple windows may include one or more occluded areas.

[0139] As an example, taking a second interface that includes M windows (M being an integer greater than 1), K1 windows (K1 being a positive integer less than M) act as occluders, occluding all or part of the area of ​​at least one other window.

[0140] In some examples, the K1 windows do not obstruct each other. For instance, windows K2, ..., M act as occluders, obscuring all or part of the area of ​​windows 1, ..., K2-1, where M - K2 + 1 = K1. For an example, please refer to... Figure 5A and Figure 5B , Figure 5A and Figure 5B Taking M=5 and K1=4 as an example, two example diagrams of the second interface provided in the embodiments of this application are shown. Figure 5A As shown, windows 2, 3, 4, and 5 act as occluders, each occluding different areas of window 1. Window 1 includes multiple occluded areas, meaning the second interface includes multiple occluded areas. In this example, windows 2, 3, 4, and 5 act as occluders, occluding parts of other windows, but none of them are completely occluded.

[0141] As an example, the first window in the second interface might include Figure 5A or Figure 5B The second window in the second interface may include at least one of windows 2, 3, 4, and 5. Figure 5A or Figure 5B At least one of window 1, window 2, window 3 and window 4 in the window.

[0142] In some examples, at least two windows within the K1 windows are occluded. For instance, windows K2, ..., and window M act as occluders, obscuring all or part of the area of ​​windows 1, ..., and K2-1, and one or more windows, including window M, within the K1 windows act as occluders, obscuring all or part of the area of ​​one or more other windows within the K1 windows. Figure 5B As shown, window 5 obscures windows 1, 2, 3, and 4; window 4 obscures windows 1, 2, and 3; window 3 obscures windows 1 and 2; and window 2 obscures window 1. Window 1, 2, 3, and 4 all include obscured areas, meaning the second interface includes multiple obscured areas. In this example, windows 2, 3, and 4 not only obscure parts of other windows but are also partially obscured themselves.

[0143] Figure 5A and Figure 5B As examples of two possible second interfaces, the second interface may also include... Figure 5A and Figure 5B More or fewer windows, and multiple windows on a second interface may also have different... Figure 5A and Figure 5B The occlusion relationship between windows shown is not specifically limited in this embodiment.

[0144] In some embodiments of this application, the data of the second interface includes the drawing order of multiple windows in the second interface, which indicates the order of the layers corresponding to the windows in the drawing queue. For example, an electronic device can draw and render multiple windows on the second interface in an order from bottom to top.

[0145] Assuming the rendering mechanism of an electronic device is from lower to upper layers, Figure 5A and Figure 5B Taking the second interface as an example, the drawing order of windows 1, 2, 3, 4 and 5 may be "1", "2", "3", "4" and "5" respectively, indicating that the layer of window 1 is drawn first, then the layer of window 2 is drawn, and so on, with the layer of window 5 being drawn last.

[0146] In this embodiment, the drawing order of multiple windows can be used to determine the stacking order between them. The stacking order can reflect the occlusion between the multiple windows, such as whether there is an occlusion relationship.

[0147] As a possible example, Figure 4BThe second user operation shown is when the electronic device displays an interface comprising M windows, the user adjusts the position and / or size of at least one window among the M windows. The change in the position and / or size of at least one window among the M windows will change the relative position of the M windows, resulting in the inclusion of an obscured area in the M windows.

[0148] S402B: The electronic device determines the pixels corresponding to the occluded area based on the data from the second interface.

[0149] As one possible implementation, the data of the second interface includes the drawing order of multiple windows. The electronic device can analyze the drawing order of multiple windows, determine the vertical stacking of the windows, and sort the windows hierarchically according to the vertical stacking, generating a drawing instruction set for the second interface. The drawing instruction set includes drawing instructions for each pixel in multiple windows, and each pixel's drawing instruction includes the hierarchical sorting value of the window to which the corresponding pixel belongs. This drawing instruction set takes into account the occlusion situation between multiple windows, providing support for subsequent pixel-level identification of occluded areas.

[0150] As one possible implementation, the electronic device can determine the occluders in multiple windows based on their stacking order. A template buffer is constructed based on the windows that serve as occluders. The template buffer includes multiple regions, including pixel regions corresponding to the occluders and pixel regions that do not correspond to the occluders. The template value of the pixels in the pixel regions corresponding to the occluders is the hierarchical sorting value of the occluders, while the template value of the pixels in the pixel regions that do not correspond to the occluders is a default value, such as 0.

[0151] As an example, when constructing a template buffer, an electronic device can cache window data of windows identified as occluders as template data in the template buffer and mark template values ​​for the template data according to the window hierarchy sorting value as the occluder.

[0152] As one possible implementation, the electronic device can compare the layer sorting value in the drawing instruction with the template value of the corresponding pixel in the template buffer on a window-by-window (e.g., in order from bottom to top) and pixel-by-pixel basis to determine whether the pixel is occluded.

[0153] As an example, an electronic device can hierarchically sort multiple windows from bottom to top.

[0154] by Figure 5A or Figure 5BTaking the second interface shown as an example, suppose the electronic device sorts 5 windows from bottom to top. The sorting values ​​(such as StencilRef) of window 1, window 2, window 3, window 4 and window 5 may be "1", "2", "3", "4" and "5" respectively, indicating that window 1 is at the bottom, and window 2, window 3, window 4 and window 5 are stacked on top of window 1 in sequence. For example, window 2 is stacked on top of window 1, window 3 is stacked on top of window 2, window 4 is stacked on top of window 3, and window 5 is stacked on top of window 4.

[0155] S403B: When drawing and rendering multiple windows, the electronic device abandons drawing and rendering pixels corresponding to occluded areas.

[0156] As one possible implementation, the electronic device can draw and render pixels in multiple windows according to a set of drawing instructions. When drawing and rendering pixels in multiple windows, the electronic device can determine whether a pixel is occluded window-by-window (e.g., in order from bottom to top) and pixel-by-pixel, based on the layer sort value in the drawing instruction for each pixel and the template value of the corresponding pixel in the template buffer. If it is determined that the pixel is occluded, the drawing and rendering of that pixel is abandoned; if it is determined that the pixel is not occluded, the drawing and rendering of that pixel proceeds normally.

[0157] In some embodiments, the data of the second interface includes geometric data of multiple windows. The geometric data includes window geometric data and graphic geometric data. The window geometric data includes window position information, size information, etc., and the graphic geometric data includes one or more attributes such as vertex position, color, and texture coordinates of the graphics in the window.

[0158] As one possible implementation, an electronic device can perform one or more processing operations such as vertex processing, clipping, and rasterization on the pixels of multiple windows according to the drawing instruction set. Then, it can determine which pixels need to be drawn and which do not through template testing and pixel culling, and remove the corresponding drawing instructions for the pixels that do not need to be drawn and rendered.

[0159] As one possible implementation, the electronic device can normally draw and render unoccluded pixels, such as performing fragment shading, opacity blending, or one or more other drawing and rendering processes on unoccluded pixels, while abandoning the drawing and rendering of occluded pixels, and also abandoning the performance of fragment shading, opacity blending, or other other drawing and rendering processes on occluded pixels. Vertex processing is used to convert the window's geometric data into fragment data, such as pixel data. For example, the electronic device can transform and calculate one or more attributes in the geometric data, such as vertex position, color, and texture coordinates.

[0160] Clipping is used to remove vertices and primitives in geometric data that are not within the view frustum (such as the spatial range visible to the camera). For example, an electronic device can check the coordinates of each vertex or primitive according to a clipping algorithm; if a vertex or primitive is completely outside the view frustum, it is discarded; if a vertex or primitive is partially within the view frustum, it is clipped, retaining only the portion inside the view frustum.

[0161] Rasterization is used to convert cropped vertices and primitives into pixels on the screen. For example, an electronic device can traverse the pixel region covered by the cropped vertices and primitives according to a rasterization algorithm, calculate the position and attributes (such as color) of each pixel, and assign corresponding attributes to these pixels.

[0162] Template testing and pixel culling are used to determine which pixels need to be drawn and which do not. Drawing instructions for pixels that do not need to be drawn are culled. For example, an electronic device can determine whether a pixel is occluded by comparing the hierarchical sort value in the pixel's drawing instruction with the template value of the corresponding pixel in the template buffer, thereby determining whether the pixel needs to be drawn and rendered.

[0163] Fragment shaders are used to perform color calculations on pixels obtained after template testing and pixel culling. For example, an electronic device can calculate the color value of each pixel based on a fragment shader through one or more processes such as texture sampling, lighting calculation, and color blending, where texture sampling is used to obtain color values ​​from textures and lighting calculation is used to simulate lighting effects.

[0164] Transparency blending is used to blend semi-transparent or transparent objects with an already rendered background to create a realistic visual effect.

[0165] S404B: The electronic device displays a second interface, which includes multiple windows.

[0166] In the second interface displayed by the electronic device, the occluded areas in the second window are not visible to the user. When the electronic device renders the second window, it does not render the pixels corresponding to the occluded areas. Based on the multi-window display method shown in Figure 4, the electronic device has the ability to summarize and comprehensively analyze the stacking relationship of multiple windows. Using this ability, it obtains the vertical stacking status of multiple windows and identifies occluded areas at the pixel level based on the vertical stacking status. For example, it completes the identification and rendering of pixels corresponding to occluded areas before fragment shading (such as texture and shading). Without affecting the interface display effect, it minimizes unnecessary rendering tasks, avoiding the processing load and resource waste caused by unnecessary rendering tasks, solving problems such as display interface stuttering and frame drops caused by excessive processing load, and improving the user experience.

[0167] In some embodiments, the multi-window display method provided in this application can be implemented based on a graphics system, a graphics engine, and a graphics rendering pipeline. For example, the graphics system can comprehensively analyze the stacking of multiple windows, perform hierarchical sorting of the windows (e.g., from bottom to top), generate drawing instructions based on the hierarchical sorting result, and transmit them to the graphics engine. The graphics engine can translate the drawing instructions and transmit the translated drawing instructions to the graphics rendering pipeline. The graphics rendering pipeline can, based on the drawing instructions and the hierarchical sorting values ​​corresponding to the multiple windows, identify occluded areas in the multiple windows at the pixel level, and discard the rendering of pixels corresponding to occluded areas when rendering the windows.

[0168] For example, Harmony TM Taking an OS as an example, the graphics system and graphics engine may be supported by the system service layer and the framework layer.

[0169] As an example, please refer to Figure 6 , Figure 6 A diagram illustrating an implementation architecture provided in an embodiment of this application is shown.

[0170] like Figure 6 As shown, the system used to implement the multi-window display method includes a graphics system 610, a graphics engine 620, and a graphics rendering pipeline 630.

[0171] The graphics system 610, when displaying a second interface comprising multiple windows, comprehensively analyzes the stacking of these windows, performs hierarchical sorting (e.g., from bottom to top), generates drawing instructions that take into account occlusion between windows, and transmits these instructions to the graphics engine 620. The graphics engine 620 translates these drawing instructions, converting high-level instructions into low-level instructions that the graphics rendering pipeline 630 can recognize and process, and then transmits the translated drawing instructions to the graphics rendering pipeline 630.

[0172] As an example, the graphics system 610 can determine the hierarchical sorting value of the first window and the second window based on the drawing order of the first window and the second window in the second interface, determine the first window as an occluder, generate a drawing instruction set for the second interface, and construct a template buffer based on the first window as an occluder.

[0173] For example, the graphics system 610 can be Figure 3 The graphics system shown, graphics system 610 can be composed of... Figure 3 The system service layer and framework layer shown provide support.

[0174] A graphics rendering pipeline 630 (such as a GPU-Pipeline) is used to render multiple windows according to rendering instructions. In this embodiment, the graphics rendering pipeline 630 can rasterize the geometric data of multiple windows, and after rasterization, perform template testing and pixel culling to identify occluded areas in multiple windows at the pixel level. After completing the identification of occluded areas at the pixel level, subsequent rendering processing is performed on the pixels corresponding to the unoccluded areas, while rendering the pixels corresponding to the occluded areas is abandoned. For example, the graphics rendering pipeline 630 abandoning rendering the pixels corresponding to the occluded areas includes: the graphics rendering pipeline 630 abandoning fragment shading and one or more other rendering processes on the pixels corresponding to the occluded areas.

[0175] As an example, the graphics rendering pipeline 630 can compare the hierarchical sort value in the drawing instruction with the template value of the corresponding pixel in the template buffer for each pixel in the second window of the second interface to determine whether the pixel is occluded. In this way, the pixel corresponding to the occluded area in the second interface can be determined.

[0176] For example, the graphics rendering pipeline 630 can rasterize the pixels in the second window of the second interface and then perform template detection. The template detection includes: determining whether a pixel is occluded by comparing the hierarchical sort value in the drawing instruction with the template value of the corresponding pixel in the template buffer pixel by pixel.

[0177] As an example, please refer to Figure 7, Figure 7 by Figure 6 Taking the implementation architecture shown as an example, a schematic diagram of an interface rendering process provided by an embodiment of this application is illustrated.

[0178] like Figure 7 As shown, assuming the second interface to be displayed includes 5 windows, the graphics system 610 can obtain the node information of the corresponding nodes (RenderNode) of each of the 5 windows. The node information includes the node identifiers, drawing order, and geometric data of the graphics in the window. In some embodiments, the node information may also include, but is not limited to, one or more of the following: hierarchical relationship data, material data, transformation data, and rendering state data. For example, the node identifiers corresponding to the 5 windows are Node1, Node2, Node3, Node4, and Node5, respectively.

[0179] Among them, node identifier is used to uniquely identify the window; drawing order is used to indicate the order of the window in the drawing queue; geometric data is used to represent one or more data such as vertex position, color, and texture coordinates of the graphic represented by the node. Geometric data can be used to define the shape and appearance of the graphic; hierarchy data is used to represent the relative position and transformation inheritance relationship between graphics; material data is used to represent the surface properties of the graphic, such as one or more such as color, gloss, transparency, and reflectivity. Material data can be used to determine how the graphic interacts with light and to determine the visual effect presented on the screen; transformation data is used to represent the position, rotation, and scaling of the node in three-dimensional space. Transformation data can be used to place the graphic in the correct position and make necessary posture adjustments. When drawing and rendering the window, its geometric data can be converted from model space to world space according to the transformation data before subsequent processing; rendering state data is used to indicate one or more such as whether shadows are enabled, whether anti-aliasing is performed, and the rendering mode used. Rendering state data can be used to determine the rendering method and effect of the node to improve drawing and rendering efficiency while ensuring the quality of the display interface.

[0180] In some embodiments of this application, the graphics system 610 has the ability to summarize and comprehensively analyze the stacking relationship of multiple windows at the graphics system level. The graphics system 610 can rely on this ability to comprehensively analyze the node information of five windows in the second interface data, such as analyzing the drawing order of multiple nodes, determining the vertical stacking of multiple windows, performing hierarchical sorting of the five nodes, for example, sorting from the bottom layer to the top layer, recording the hierarchical sorting value of each of the five nodes, and generating drawing instructions that take into account the occlusion between multiple windows based on the hierarchical sorting value of each of the five nodes.

[0181] As an example, the drawing instructions are a set of instructions that adjust the drawing order and drawing parameters based on the hierarchical order of the five nodes and after taking into account the occlusion between the windows corresponding to the nodes.

[0182] When generating drawing instructions, as one possible implementation, the graphics system 610 can generate drawing instructions, such as the DrawCall instruction, corresponding to the five windows, based on the hierarchical sorting values ​​of the five nodes.

[0183] In some embodiments, the graphics system 610 can also determine whether the windows corresponding to the five nodes are occluders based on their respective hierarchical sorting values, and construct a template buffer based on the determined occluders. For example, the graphics system 610 can cache the window data of the window determined to be an occluder as template data in the template buffer, and mark the template data with template values ​​(such as Stencil values), which are determined based on the hierarchical sorting values ​​of the occluders.

[0184] The second interface includes Figure 5B Taking the five windows shown as an example, the graphics system 610 determines whether the windows corresponding to the five nodes are occluders based on their respective hierarchical sorting values ​​"1", "2", "3", "4" and "5". Finally, it determines that nodes 2, 3, 4 and 5, with hierarchical sorting values ​​of "2", "3", "4" and "5" respectively, are occluders. Then, the graphics system 610 constructs a template buffer based on the determined occluders. For example, the graphics system 610 can cache the window data corresponding to nodes 2, 3, 4 and 5 as template data in the template buffer and mark the template data with template values ​​(such as Stencil values).

[0185] As an example, please refer to Figure 8A , Figure 8A The second interface includes Figure 5B The five windows shown are an example to illustrate a template buffer provided in an embodiment of this application.

[0186] like Figure 8A As shown, the graphics system 610 sequentially marks template values ​​(such as Stencil values) at the corresponding pixels of the template according to the hierarchical sorting values ​​corresponding to nodes 2, 3, 4, and 5. For example, the template value of the window pixel corresponding to node 2 is marked as... Figure 8A As shown in Figure 2, the template value of the window pixel corresponding to node 3 is marked as... Figure 8A As shown in Figure 3, the template value of the window pixel corresponding to node 4 is marked as... Figure 8A As shown in Figure 4, the template value of the window pixel corresponding to node 5 is marked as... Figure 8AFigure 5 shows the completion of the template buffer construction. In some embodiments, as shown in Figure 4, the graphics system 610 can mark the template value of a non-occluded volume, such as the window pixel corresponding to node 1, as... Figure 8A The template value shown is 0; and the template value of other blank pixels is marked as 0. Figure 8A As shown in Figure 0, blank pixels refer to pixels that do not correspond to a window.

[0187] As an example, the first area in the template buffer might include Figure 8A The template value is a region containing one or more values ​​from 5, 4, 3, and 2. The second region in the template buffer may include... Figure 8A The area where the template value is 0.

[0188] As an example, the first area in the template buffer might include Figure 8A The area with a template value of 5, and the second area in the template buffer may include... Figure 8A The region with a template value of 0, and the third region in the template buffer may include... Figure 8A The area with template values ​​of one or more of 4, 3, and 2.

[0189] like Figure 7 As shown, after generating drawing instructions, the graphics system 610 can send the drawing instructions to the graphics engine 620.

[0190] like Figure 7 As shown, the graphics engine 620 can translate drawing instructions, converting high-level drawing instructions into low-level instructions that the graphics rendering pipeline 630 can recognize and process, and then transmitting the translated drawing instructions to the graphics rendering pipeline 630. This translation of drawing instructions improves compatibility between modules.

[0191] As an example, the translation of drawing instructions may include, but is not limited to, one or more of the following translations: basic graphic element translation, graphic attribute translation, typesetting and rendering instruction translation, image transformation instruction translation, etc.

[0192] like Figure 7 As shown, the graphics rendering pipeline 630 can perform vertex processing, clipping, rasterization, template testing and pixel culling, fragment shading and alpha blending of geometric data from multiple windows.

[0193] Vertex processing (e.g., vertex processing) is used to convert geometric data from multiple windows into fragment data, such as pixel data. For example, the graphics rendering pipeline 630 can transform and calculate one or more attributes in the geometric data, such as vertex position, color, and texture coordinates. For example, the transformation methods may include, but are not limited to, one or more of model transformation, view transformation, and projection transformation, where model transformation is used to convert vertices from model space to world space, view transformation is used to convert vertices in world space to camera space, and projection transformation is used to project fixed points in camera space onto a two-dimensional plane.

[0194] Clipping (e.g., clip / project / cull) is used to remove vertices and primitives from the geometric data that are not within the view frustum (e.g., the space visible to the camera). For example, the graphics rendering pipeline 630 can check the coordinates of each vertex or primitive according to a clipping algorithm. If a vertex or primitive is completely outside the view frustum, it is discarded; if a vertex or primitive is partially within the view frustum, it is clipped, retaining only the portion inside the view frustum.

[0195] Rasterization is used to convert clipped vertices and primitives into pixels on the screen. For example, the graphics rendering pipeline 630 can traverse the pixel region covered by the clipped vertices and primitives according to a rasterization algorithm, calculate the position and attributes (such as color) of each pixel, and assign corresponding attributes to these pixels.

[0196] Template testing and pixel culling (such as ZS culling) are used to determine which pixels need to be drawn and which do not. Drawing instructions for pixels that do not need to be drawn and rendered are culled. For example, the graphics rendering pipeline 630 can determine whether a pixel is occluded by comparing the hierarchical sort value in the pixel's drawing instruction with the template value of the corresponding pixel in the template buffer, thereby determining whether the pixel needs to be drawn and rendered.

[0197] Fragment shading (such as texture & shading) is used to perform shading calculations on the pixels obtained after template testing and pixel culling. For example, the graphics rendering pipeline 630 can calculate the color value of each pixel based on the fragment shader through one or more processes such as texture sampling, lighting calculation, and color blending, where texture sampling is used to obtain color values ​​from textures and lighting calculation is used to simulate lighting effects.

[0198] Alpha blending is used to blend semi-transparent or transparent objects with a rendered background to produce a realistic visual effect. For example, the graphics rendering pipeline 630 can calculate the display color value of a pixel based on one or more blending algorithms such as additive blending and multiplicative blending, according to the pixel's alpha value and the background pixel's color value. For instance, the alpha value can range from 0 to 100, where 0 represents complete transparency and 100 represents complete opacity.

[0199] Figure 7 As an example of a possible multi-window display interface rendering process, in practical applications, the hardware and / or software relied upon for each step in the multi-window display interface rendering process are not limited, nor are the specific implementation details of each step limited. For example, in practical applications, the graphics rendering pipeline may include... Figure 7 The number of steps shown may be more or less.

[0200] pass Figure 7 The multi-window display rendering process shown can be implemented without adding any new hardware to the electronic device. It leverages the existing processing flow of the graphics system, graphics engine, and graphics rendering pipeline, providing graphics system-level occlusion region recognition functionality. For example, the graphics system's aggregation and comprehensive analysis capabilities are used to determine the stacking of multiple windows, perform hierarchical sorting, and generate drawing instructions that consider occlusion based on the sorting results. The graphics engine's instruction translation capabilities are used to translate the drawing instructions into low-level instructions that the graphics rendering pipeline can recognize and process. Furthermore, the graphics rendering pipeline's template testing and pixel culling capabilities are used to identify pixel-level occluded areas. For instance, before fragment shading (such as texture and shading), the identification and rendering of pixels corresponding to occluded areas are completed. This minimizes unnecessary rendering tasks without affecting the display effect, avoiding unnecessary processing load and resource waste on the electronic device, resolving issues such as display stuttering and frame drops caused by excessive processing load, and improving the user experience.

[0201] As an example, please refer to Figures 8B-8F , Figures 8B-8F The second interface includes Figure 5B Taking the five windows shown as an example, this illustrates a schematic diagram of the template testing and pixel removal process provided in the embodiment of this application.

[0202] After the graphics system 610 caches the template data of the window identified as an occluder into the template buffer and marks the template data with template values, the distribution of template values ​​for each pixel in the template buffer may be as follows: Figure 8AFor example, the template value of each pixel in window 1 is 0, the template value of each pixel in window 2 is 2, the template value of each pixel in window 3 is 3, the template value of each pixel in window 4 is 4, and the template value of each pixel in window 5 is 5. When drawing and rendering the second interface, the graphics rendering pipeline 630 can, according to the drawing instructions and in order from bottom to top, traverse the pixels of window 1, window 2, window 3, window 4, and window 5 to determine whether drawing and rendering are required. Pixels that need to be drawn and rendered are drawn normally, while pixels that do not need to be drawn and rendered are abandoned. For example, fragment shading and one or more other drawing and rendering processes may be abandoned for pixels that do not need to be drawn and rendered. These other drawing and rendering processes may include transparency blending.

[0203] When determining whether a pixel needs to be drawn and rendered, the graphics rendering pipeline 630 compares the hierarchical sort value in the pixel's drawing instruction with the corresponding pixel's template value in the template buffer. If the hierarchical sort value in the pixel's drawing instruction is less than the corresponding pixel's template value in the template buffer, the pixel is determined to be occluded by the occluder and does not need to be drawn and rendered; if the hierarchical sort value in the pixel's drawing instruction is greater than or equal to the corresponding pixel's template value in the template buffer, the pixel is determined not to be occluded by the occluder and needs to be drawn and rendered.

[0204] For example, such as Figure 8B As shown, when rendering each pixel in window 1, the graphics rendering pipeline 630 can compare the hierarchical sort value "1" in the pixel's drawing instruction with the corresponding pixel's template value in the template buffer. It determines that pixels with a template value of 0 corresponding to hierarchical sort value "1" need to be drawn, while pixels with template values ​​of 2, 3, 4, or 5 corresponding to hierarchical sort value "1" do not need to be drawn. Figure 9 As shown in 901, the graphics rendering pipeline 630 can normally draw the pixels that need to be drawn and rendered in the rendering window 1, and abandon drawing the pixels that do not need to be drawn and rendered in the rendering window 1.

[0205] Similarly, such as Figure 8C The graphics rendering pipeline 630 can determine the pixels in window 2 that need to be rendered and those that do not need to be rendered using a method similar to that used in window 1. Figure 9 As shown in 902, the graphics rendering pipeline 630 can normally draw the pixels that need to be rendered in rendering window 2, and abandon the drawing of pixels that do not need to be rendered in rendering window 2. Figure 8D As shown, the graphics rendering pipeline 630 can determine the pixels in window 3 that need to be rendered and the pixels that do not need to be rendered in a similar way to window 1, such as... Figure 9As shown in 903, the graphics rendering pipeline 630 can normally draw the pixels that need to be rendered in rendering window 3, and abandon the drawing of pixels that do not need to be rendered in rendering window 3. Figure 8E As shown, the graphics rendering pipeline 630 can determine the pixels in window 4 that need to be rendered and the pixels that do not need to be rendered in a similar way to window 1, such as... Figure 9 As shown in 904, the graphics rendering pipeline 630 can normally draw the pixels that need to be drawn and rendered in the rendering window 4, and abandon drawing the pixels that do not need to be drawn and rendered in the rendering window 4.

[0206] like Figure 8F As shown, when rendering each pixel in window 5, the graphics rendering pipeline 630 can compare the hierarchical sort value "5" in the pixel's drawing instruction with the template value of the corresponding pixel in the template buffer to determine which pixels with template values ​​of 0, 2, 3, 4, and 5 corresponding to the hierarchical sort value "5" need to be drawn. Figure 9 As shown in 905, the graphics rendering pipeline 630 can draw the pixels in the rendering window 5 normally.

[0207] Figure 8A The process of constructing a template buffer is illustrated by an example of an electronic device caching window data of all windows identified as occluders as template data in a template buffer and marking the template data with template values. In some embodiments, the electronic device may also cache window data of only some windows identified as occluders as template data in the template buffer to avoid excessive resource consumption, long processing time, and high processing costs in the template testing and pixel culling processes due to excessive template data.

[0208] As one possible implementation, the electronic device can select the Top N windows with the largest hierarchical sort values ​​based on the windows identified as occluders. The window data of these N windows can then be cached as template data in a template buffer, and template values ​​can be assigned to the template data. Here, N may be a value preset by the developer, or it may be determined by the electronic device based on the number of windows and / or occluders on the second interface; there is no limitation. For example, N may include, but is not limited to, values ​​such as 2, 3, 4, and 5.

[0209] For example, taking an occluder comprising windows 2, 3, 4, and 5, with hierarchical sort values ​​of "2", "3", "4", and "5" respectively, and N=3, the electronic device can select and cache the window data of windows 3, 4, and 5 as template data in a template buffer based on the hierarchical sort value of each window and N, and mark the template data with template values. For example, the graphics system 610 sequentially marks template values ​​(such as Stencil values) at the corresponding pixels of the template according to the hierarchical sort values ​​corresponding to nodes 3, 4, and 5. For example, the template value of the window pixel corresponding to node 3 is marked as... Figure 10A As shown in Figure 3, the template value of the window pixel corresponding to node 4 is marked as... Figure 10A As shown in Figure 4, the template value of the window pixel corresponding to node 5 is marked as... Figure 10A As shown in Figure 5, the template values ​​of other pixels are marked as... Figure 10A As shown in Figure 0.

[0210] As an example, the first area in the template buffer might include Figure 10A The template value is a region containing one or more values ​​of 5, 4, and 3. The second region in the template buffer may include... Figure 10A The area where the template value is 0.

[0211] As an example, the first area in the template buffer might include Figure 10A The area with a template value of 5, and the second area in the template buffer may include... Figure 10A The region with a template value of 0, and the third region in the template buffer may include... Figure 10A The area with template values ​​of 4 and 3 or more.

[0212] and Figures 8B-8F The process shown is similar, such as Figure 10B As shown, the graphics rendering pipeline can compare the hierarchical sort value "1" in the pixel's draw instruction with the corresponding stencil value in the stencil buffer. It determines that pixels with a stencil value of 0 corresponding to hierarchical sort value "1" need to be drawn, while pixels with stencil values ​​of 3, 4, or 5 corresponding to hierarchical sort value "1" do not need to be drawn. Similarly, with... Figures 8C-8FSimilarly, the graphics rendering pipeline can compare the hierarchical sort value "2" in the pixel's draw instruction with the corresponding pixel's stencil value in the stencil buffer. It determines that pixels with a stencil value of 0 corresponding to hierarchical sort value "2" need to be drawn, while pixels with stencil values ​​of 3, 4, or 5 do not need to be drawn. It also compares the hierarchical sort value "3" in the pixel's draw instruction with the corresponding pixel's stencil value in the stencil buffer. This determines that pixels with a stencil value of 0 or 3 corresponding to hierarchical sort value "3" need to be drawn, while pixels with stencil values ​​of 3 do not need to be drawn. Pixels with a template value of 4 or 5 do not need to be drawn or rendered; the layer sort value "4" in the pixel drawing instruction is compared with the template value of the corresponding pixel in the template buffer to determine that pixels with a template value of 0, 3, or 4 corresponding to the layer sort value "4" need to be drawn, while pixels with a template value of 5 corresponding to the layer sort value "4" do not need to be drawn or rendered; and the layer sort value "5" in the pixel drawing instruction is compared with the template value of the corresponding pixel in the template buffer to determine that pixels with a template value of 0, 3, 4, or 5 corresponding to the layer sort value "5" need to be drawn. And, as... Figure 10C As shown in 1001-1005, the graphics rendering pipeline can sequentially and normally draw the pixels that need to be drawn and rendered in rendering windows 1, 2, 3, 4 and 5, and abandon drawing the pixels that do not need to be drawn and rendered.

[0213] exist Figure 10C In the example shown, compared to Figure 9 The example shown only renders pixels within the window that are occluded by window 2 but not by windows 3, 4, and 5. This number is relatively small compared to the total number of pixels culled. By selecting to cache some of the occluded window data as template data in the template buffer, a larger culling benefit (e.g., a culling rate of 40-60%) can be achieved while avoiding excessive resource consumption, processing time, and processing costs associated with template testing and pixel culling due to excessive template data, thus improving rendering efficiency.

[0214] As an example, when displaying a test interface, the multi-window display method described in this application, which abandons drawing and rendering pixels corresponding to the occluded area, is compared with the conventional multi-window display method that fully draws pixels in multiple windows. The number of pixels required to draw and render the test interface is as follows: The conventional method that fully draws pixels requires 37,995,458 pixels to display the test interface; the solution of this application, which abandons drawing and rendering pixels corresponding to the occluded area, requires 20,064,616 pixels to display the test interface. Compared with the conventional method, the solution of this application renders 17,930,842 fewer pixels, a reduction of 47.2%, and achieves a significant benefit from pixel removal.

[0215] It should be understood that the various solutions in the embodiments of this application can be used in a reasonable combination, and the explanations or descriptions of the various terms appearing in the embodiments can be referenced or explained to each other in the various embodiments, without limitation.

[0216] It should also be understood that, in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0217] It is understood that, in order to achieve the functions of any of the above embodiments, the electronic device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0218] This application embodiment can divide electronic devices into functional modules. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0219] As an example, please refer to Figure 11 , Figure 11 A virtual functional structure block diagram of an electronic device provided in an embodiment of this application is shown. For example... Figure 11 As shown, the electronic device may include a processing unit 1110, a storage unit 1120, and a display unit 1130.

[0220] The processing unit 1110 is used to support the electronic device in responding to the second user operation, determining the pixels corresponding to the occluded area in the second interface, abandoning the rendering of the pixels corresponding to the occluded area when rendering the second window, and / or other processes related to the implementation of this application.

[0221] As an example, the processing unit 1110 is used to support the electronic device in determining the hierarchical sorting value of the first window and the hierarchical sorting value of the second window according to the drawing order of the first window and the drawing order of the second window, determining that the first window is an occlusion, generating a drawing instruction set for the second interface, constructing a template buffer according to the first window, and comparing the hierarchical sorting value in the drawing instruction with the template value of the corresponding pixel in the template buffer for each pixel in the second window to determine whether the pixel is occluded.

[0222] As an example, the processing unit 1110 includes a graphics system that has the ability to summarize and comprehensively analyze the stacking relationship of multiple windows. The processing unit 1110 can use the graphics system to determine the hierarchical sorting value of the first window and the hierarchical sorting value of the second window according to the drawing order of the multiple windows, and determine the first window as an occluder.

[0223] As an example, the processing unit 1110 includes a graphics rendering pipeline. The processing unit 1110 can rasterize the pixels in the second window through the graphics rendering pipeline and then perform template detection. The template detection includes: determining whether a pixel is occluded by comparing the hierarchical sort value in the drawing instruction with the template value of the corresponding pixel in the template buffer pixel by pixel.

[0224] The storage unit 1120 is used to support the electronic device in storing data, such as interface data, template data of occluders in multiple windows of the interface, interface drawing instructions, data of the interface template buffer, and / or other data related to the implementation of this application.

[0225] The display unit 1130 is used to support the electronic device in displaying an interface, such as displaying a first interface including multiple windows, displaying a second interface including multiple windows, and / or other interfaces related to the implementation of this application.

[0226] Figure 11 As an example of a virtual functional structure for an electronic device, this application Figure 11 The illustrated structure does not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer units than illustrated, or combine some units, or split some units, or have different unit arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware. For example, in some examples, the electronic device may also include a communication unit.

[0227] As an example, please refer to Figure 12 , Figure 12 A hardware structure block diagram of an electronic device according to an embodiment of this application is shown. Figure 12As shown, the electronic device may include a processor 1201, a memory 1202, and communication components, such as a communication component including a communication line 1203 and at least one communication interface 1204. Figure 12 (This is merely an example illustrating the concept of including a single communication interface).

[0228] The processor 1201 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application.

[0229] In a specific implementation, as one example, processor 1201 may include one or more CPUs, such as CPU0 and CPU1.

[0230] Communication line 1203 may include a path for transmitting information between the aforementioned components.

[0231] Communication interface 1204 uses any transceiver-like device for communication with other devices, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc.

[0232] Memory 1202 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory 1203 may exist independently and be connected to processor 1201 via communication line 1203. Memory 1203 may also be integrated with processor 1201.

[0233] The memory 1203 stores computer program instructions for executing the scheme of this application. The memory 1203 can store instructions for implementing two modular functions: send instructions and link instructions, and their execution is controlled by the processor 1201. The processor 1201 executes the computer program instructions stored in the memory 1203 to implement the method provided in the following embodiments of this application.

[0234] Optionally, the computer program instructions in the embodiments of this application may also be referred to as application code, and the embodiments of this application do not specifically limit this.

[0235] It should also be understood that the various modules in an electronic device can be implemented in software and / or hardware, without specific limitations. In other words, an electronic device is presented in the form of functional modules. Here, "module" can refer to application-specific integrated circuits (ASICs), circuits, processors and memories that execute one or more software or firmware programs, integrated logic circuits, and / or other devices that can provide the above functions.

[0236] In an alternative approach, when data transmission is implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are implemented. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disk (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0237] The steps of the methods or algorithms described in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, compact disc read-only memory (CD-ROM), or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the ASIC can reside in an electronic device. Of course, the processor and storage medium can also exist as discrete components.

[0238] Through the above description of the implementation methods, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

Claims

1. A multi-window display method, characterized in that, The method includes: In response to a first user operation, the first interface is displayed; wherein, the number of rendered pixels of the first interface is the first number of rendered pixels; the first user operation is an operation that triggers the display of the first interface, the first interface includes a first window and a second window, and the first window and the second window have no occlusion relationship; In response to a second user operation, the second interface is displayed; wherein the number of rendered pixels of the second interface is a second number of rendered pixels; the second user operation is an operation to change the relative position of the first window and the second window in the first interface; the first window in the second interface occludes a part of the second window; the difference between the second number of rendered pixels and the first number of rendered pixels is linearly related to the number of pixels corresponding to the occluded area.

2. The method according to claim 1, characterized in that, Before displaying the second interface, the method further includes: In response to the second user operation, determine the pixels corresponding to the occluded area; Draw and render the first window and the second window. When rendering the second window, discard the drawing and rendering of the pixels corresponding to the occluded area.

3. The method according to claim 2, characterized in that, Determining the pixels corresponding to the occluded area includes: The layer sorting values ​​of the first window and the second window are determined according to the drawing order of the first window and the drawing order of the second window, and the first window is determined to be an occluder. A drawing instruction set for the second interface is generated. The drawing instruction set includes drawing instructions for each pixel of the first window and drawing instructions for each pixel of the second window. The drawing instruction for each pixel includes the hierarchical sorting value of the window where the corresponding pixel is located. A template buffer is constructed based on the first window. The template buffer includes a first region and a second region. The first region is the pixel region corresponding to the first window. The template value of the pixels in the first region is the hierarchical sorting value of the first window. The second region has no correspondence with the window. The template value of the pixels in the second region is 0. For each pixel in the second window, the hierarchical sorting value in the drawing instruction is compared with the template value of the corresponding pixel in the template buffer to determine whether the pixel is occluded.

4. The method according to claim 3, characterized in that, For each pixel in the second window, the pixel value in the drawing instruction is compared pixel by pixel with the template value of the corresponding pixel in the template buffer to determine whether the pixel is occluded, including: When the hierarchical sort value in the drawing instruction is less than the template value of the corresponding pixel in the template buffer, it is determined that the pixel is occluded. When the hierarchical sort value in the drawing instruction is greater than or equal to the template value of the corresponding pixel in the template buffer, it is determined that the pixel is not occluded.

5. The method according to any one of claims 2-4, characterized in that, The invention is applied to an electronic device, which includes a graphics system. The graphics system determines the hierarchical sorting value of the first window and the hierarchical sorting value of the second window based on the drawing order of the first window and the drawing order of the second window, determines the first window as an occluder, generates a drawing instruction set for the second interface, and constructs a template buffer based on the first window.

6. The method according to any one of claims 2-5, characterized in that, The electronic device includes a graphics rendering pipeline. The step of comparing the hierarchical sort value in the drawing instruction with the template value of the corresponding pixel in the template buffer for each pixel in the second window to determine whether the pixel is occluded includes: The graphics rendering pipeline compares the hierarchical sort value in the drawing instruction with the template value of the corresponding pixel in the template buffer for each pixel in the second window to determine whether the pixel is occluded.

7. The method according to claim 6, characterized in that, The graphics rendering pipeline compares the hierarchical sort value in the drawing instruction with the template value of the corresponding pixel in the template buffer for each pixel in the second window to determine whether the pixel is occluded, including: The graphics rendering pipeline rasterizes the pixels in the second window and then performs template detection. The template detection includes: comparing the hierarchical sort value in the drawing instruction with the template value of the corresponding pixel in the template buffer to determine whether the pixel is occluded.

8. The method according to claim 6 or 7, characterized in that, The step of abandoning the rendering of pixels corresponding to the occluded area includes: The graphics rendering pipeline abandons fragment shading and one or more other drawing and rendering processes for the pixels corresponding to the occluded area.

9. The method according to any one of claims 3-8, characterized in that, The first interface and the second interface also include a third window. The third window in the first interface does not obstruct either the first window or the second window. The third window in the second interface obstructs the second window and is also obstructed by the first window. The method further includes: The hierarchy sorting value of the third window is determined according to the drawing order of the third window; For each pixel in the third window, the hierarchical sorting value in the drawing instruction is compared with the template value of the corresponding pixel in the template buffer to determine whether the pixel is occluded.

10. The method according to claim 9, characterized in that, The method further includes: The third window is identified as an obstruction. When constructing the template buffer, a third region is constructed in the template buffer according to the third window. The third region is the pixel region corresponding to the third window that is not obscured by the first window, and the template value of the pixels in the third region is the hierarchical sorting value of the third window.

11. An electronic device, characterized in that, The electronic device includes: A display screen is used to display the interface. Memory is used to store computer program instructions; A processor for executing the computer program instructions to support the electronic device in implementing the method as described in any one of claims 1-10.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processing circuit, implement the method as described in any one of claims 1-10.

13. A computer program product containing instructions, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1-10.

Images