Method, apparatus, device, medium and product for display processing

Abstract

According to embodiments of the present disclosure, a method, apparatus, device, medium and product for display processing are provided. The method comprises: determining a plurality of rendering layers to be rendered on a display screen, each rendering layer corresponding to at least a partial interface region of an application interface to be displayed; in response to a number of the plurality of rendering layers being greater than a threshold number, determining respective priorities of the plurality of rendering layers, the threshold number indicating an upper limit of a number of rendering layers that can be completed rendering within a predetermined period; performing rendering on the plurality of rendering layers within the predetermined period based on the respective priorities of the plurality of rendering layers to obtain a rendering result; and presenting the rendering result in the display screen.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and more particularly to methods, apparatus, devices, and computer-readable storage media for display processing. Background Technology

[0002] With the widespread adoption of smart terminal devices and the rapid development of the mobile internet, modern computing terminals are evolving from traditional single-task modes to multi-task, multi-window parallel processing modes. Especially in extended reality (XR) devices, smartphones, tablets, and other terminal devices, as display screens become larger, users' demands for running multiple applications in parallel and interacting across windows are increasing. The concurrent operation and display of multiple windows and tasks requires more efficient resource management and display processing strategies to ensure a smooth user experience. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for display processing is provided. The method includes: determining a plurality of rendering layers to be rendered on a display screen, each rendering layer corresponding to at least a portion of an interface area in an application interface to be displayed; determining the priority of each of the plurality of rendering layers in response to a number greater than a threshold number, the threshold number indicating an upper limit on the number of rendering layers that can be rendered within a predetermined period; performing rendering on the plurality of rendering layers within the predetermined period based on their respective priorities to obtain a rendering result; and presenting the rendering result on the display screen.

[0004] In a second aspect of this disclosure, an apparatus for display processing is provided. The apparatus includes: a rendering layer determination module configured to determine a plurality of rendering layers to be rendered on a display screen, each rendering layer corresponding to at least a portion of an interface area in an application interface to be displayed; a priority determination module configured to determine the priority of each of the plurality of rendering layers in response to a number greater than a threshold number, the threshold number indicating an upper limit on the number of rendering layers that can be rendered within a predetermined period; a rendering module configured to perform rendering on the plurality of rendering layers within the predetermined period based on the respective priorities of the plurality of rendering layers, to obtain a rendering result; and a rendering module configured to render the rendering result on the display screen.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processing unit, the instructions cause the electronic device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores computer-executable instructions that can be executed by a processor to implement the method of the first aspect.

[0007] In a fifth aspect of this disclosure, a computer-executable instruction product is provided. The computer-executable instruction product is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method of the first aspect.

[0008] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0011] Figure 2 A flowchart illustrating a process for displaying a certain embodiment of the present disclosure is shown;

[0012] Figure 3 A schematic diagram of a system for display processing according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A schematic structural block diagram of an apparatus for display processing according to certain embodiments of the present disclosure is shown; and

[0014] Figure 5 A block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented is shown. Detailed Implementation

[0015] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0016] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0017] In this document, unless explicitly stated otherwise, performing a step in response to A does not mean that the step is performed immediately after A, but may include one or more intermediate steps.

[0018] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0019] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0020] For example, in response to receiving a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information, thereby enabling the user to choose whether to provide personal information to the software or hardware such as electronic devices, applications, servers or storage media that perform the operation of the technical solution disclosed herein, based on the prompt message.

[0021] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, such as a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0022] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0023] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In this example environment 100, an application 120 is installed on a terminal device 110. A user 140 can interact with the application 120 via the terminal device 110 and / or an attached device to the terminal device 110. The application 120 can be a content presentation application (e.g., a video playback application), an online shopping application, or any other suitable application.

[0024] exist Figure 1 In environment 100, if application 120 is active, terminal device 110 can display application interface 150 of application 120. Application interface 150 may include various user interfaces provided by application 120. Application 120 can provide content viewing functionality to view various types of content published in application 120. Through corresponding pages, application 120 can provide various types of online content to user 140. The embodiments of this disclosure do not limit the specific functions of the application or the content presented. The application can be activated by an appropriate triggering method, such as clicking or selecting an application icon. Note that although a single application is shown in the figure, multiple applications may actually be installed and running on terminal device 110.

[0025] In some embodiments, terminal device 110 communicates with server 130 to provide services to application 120. Terminal device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, terminal device 110 can also support any type of user-facing interface (such as "wearable" circuitry). Server 130 can be various types of computing systems / servers capable of providing computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0026] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0027] In traditional terminal graphics rendering and display methods, each application occupies rendering resources independently. An application's user interface may have multiple rendering layers, but this usually does not exceed the threshold that the graphics processing unit (GPU) can handle. Therefore, in the case of a single application display, the GPU has sufficient rendering resources, and all rendering layers can be rendered smoothly within the vertical synchronization (Vsync) time.

[0028] Vsync time (also known as a Vsync cycle) typically refers to the time interval between two screen refreshes on a display device (e.g., a display screen). It directly impacts the rendering system's time budget and task scheduling, especially on devices that support the simultaneous display of multiple application interfaces.

[0029] With the popularization of smart terminal devices and the development of the mobile internet, terminal devices such as XR devices, personal computers (PCs), tablets, and smartphones have evolved from single-task modes to environments that support the parallel operation of multiple applications. For example, XR devices, as a new type of spatial computing platform, support the simultaneous opening of multiple application interfaces in three-dimensional space. Users can launch multiple application interfaces (such as browsers, document editors, calendar management, etc.) on devices with large display screens (such as XR devices), and run mixed reality (MR) applications (such as virtual collaboration, mixed reality meetings, etc.) on this basis. In addition, terminal devices such as PCs, tablets, and smartphones also face similar needs. For example, users may need to display multiple application interfaces such as video players, games, and office software on the screen simultaneously through split-screen or small window methods.

[0030] Furthermore, modern display devices (such as XR devices and high refresh rate monitors) are highly dependent on processor resources. XR devices, in particular, typically require refresh rates of 90Hz, 120Hz, or higher to avoid dizziness and ensure optimal display quality. If each application independently occupies a rendering layer, the processor needs to handle the rendering tasks of multiple layers simultaneously. In this situation, the excessive number of rendering layers requiring screen refresh may prevent the completion of rendering within a single vertical synchronization time. This can lead to frame rate drops, stuttering, screen tearing, and other issues, negatively impacting the user experience.

[0031] To address the aforementioned issues, manual optimization is conventionally used to ensure application display quality and smooth operation. For example, reducing non-critical graphics elements decreases unnecessary rendering layers, or dynamically updated rendering layers reduce the GPU load. However, this approach still requires manual adjustments by developers. If system load changes (e.g., a user suddenly opens multiple application interfaces), this method cannot dynamically adapt to changes in system resource load. Therefore, a system-level display processing solution is needed that can automatically optimize rendering layer allocation in multi-window, multi-application environments to avoid display anomalies caused by processing unit overload.

[0032] In view of the above, according to embodiments of this disclosure, an improved scheme for display processing is proposed. The scheme includes: first, determining multiple rendering layers to be rendered on a display screen, each rendering layer corresponding to at least a portion of the interface area in the application interface to be displayed; if the number of multiple rendering layers is greater than a threshold number, determining the priority of each of the multiple rendering layers; the threshold number indicating an upper limit on the number of rendering layers that can be rendered within a predetermined period; then, performing rendering on the multiple rendering layers within the predetermined period based on their respective priorities to obtain a rendering result; and finally, presenting the rendering result on the display screen.

[0033] Therefore, by intelligently managing and scheduling multiple rendering layers, the allocation of rendering resources is optimized, ensuring smooth system operation in multi-interface and multi-tasking environments. Furthermore, during the rendering process, dynamic adjustments are made based on the priority of rendering layers to ensure that high-priority interfaces receive sufficient rendering resources and that all rendering layers complete rendering within the predetermined period. This approach avoids display anomalies such as stuttering, screen tearing, or glitches, ensuring users experience a smooth and clear display in multi-interface and multi-tasking environments.

[0034] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure. Furthermore, the exemplary embodiments will be described primarily with respect to terminal device 110.

[0035] Figure 2 A flowchart of a display processing procedure 200 according to some embodiments of the present disclosure is shown. Procedure 200 can be implemented at terminal device 110. Reference is made below. Figure 1 Describe the process 200.

[0036] In box 210, terminal device 110 determines multiple rendering layers to be rendered on the display screen. Each rendering layer corresponds to at least a portion of the interface area in the application interface 150 to be displayed.

[0037] A display screen refers to a device or area used to display the content of terminal device 110. The display screen can be integrated into terminal device 110 or be an external device of terminal device 110. The display screen can include a physical display device or a virtual projection display (e.g., for XR devices).

[0038] The rendering layer, also known as the display layer or layer, manages the rendering of multiple application interfaces or interface elements in a layered manner. The rendering layers primarily originate from applications 120 within the terminal device 110; each application interface of 120 can have one or more rendering layers. An application interface is also called an application window or window. In an environment where multiple application interfaces are displayed on the screen, depending on the display position and activity state of the application interfaces, different application interfaces of 120 may occupy different display areas of the screen. After each rendering layer is rendered, its rendering result will correspond to at least a portion of the interface area of ​​the application interface to be displayed, corresponding to a part of the content of application interface 150. The display content corresponding to the rendering layer may be an independent element of an application interface (such as a background, button, or scrolling area), or a window of the entire application interface.

[0039] In some embodiments, different application interfaces typically correspond to different rendering layers. For example, a launcher application itself occupies one rendering layer. After multiple applications are launched (such as video applications, office applications, etc.), each application may create one or more rendering layers. If an application supports multiple interface displays (such as a browser application that can open multiple tabs or child windows), each application interface may also correspond to a different rendering layer.

[0040] In some embodiments, rendering layers may not be strictly divided according to the application; an application interface may also be divided into multiple rendering layers. For example, in a game application interface, the background layer, interaction layers such as the controller, user interface (UI) controls, and game character layer may each correspond to a rendering layer. This means that an application interface can correspond to multiple rendering layers. The specific number of rendering layers to be rendered on the display screen can depend on factors such as the number of applications launched, the complexity of the application interface, and the specific display strategy.

[0041] In some embodiments, the terminal device 110 can determine a set of rendering layers for the application interface to be displayed on the screen. Then, based on the position information of each rendering layer on the screen, the terminal device 110 determines multiple rendering layers within the user's field of view from the set of rendering layers for rendering. That is, for all currently open application interfaces, some rendering layers may not need to be rendered; therefore, only the multiple rendering layers within the user's field of view can be considered for rendering. The terminal device 110 can determine the multiple rendering layers within the user's field of view as the multiple rendering layers to be rendered. The rendering content corresponding to rendering layers whose center point or main part falls outside the user's field of view is not easily perceived or seen by the user. Therefore, they do not need to be rendered and submitted, thereby reducing unnecessary computational and resource load.

[0042] The field of view (FOV) refers to the area that a user can see on a display device (such as the display screen of an XR device, a PC monitor, etc.). For example, in an XR device, the FOV is the optical field of view, which is determined by the device's hardware attributes (such as lenses, optical modules, screen, etc.).

[0043] As an example, terminal device 110 can determine whether a rendering layer is within the field of view (FOV) based on the rendering layer's location information. Each rendering layer has its own coordinate information (such as X / Y / Z coordinates) and size information (such as width and height). This information can be used to determine the position of at least a portion of the application interface corresponding to the rendering layer on the display screen, thereby determining whether these rendering layers belong to the FOV.

[0044] Continue to refer to Figure 2 In box 220, in response to the number of multiple rendering layers to be rendered being greater than a threshold number, terminal device 110 determines the priority of each of the multiple rendering layers.

[0045] In multi-tasking and multi-interface usage scenarios, user 140 may trigger multiple applications or application interfaces. If user 140 does not close unnecessary applications or application interfaces, their corresponding rendering layers may persist on the display screen, causing the number of rendering layers to accumulate. Terminal device 110 needs to process multiple application interfaces and their corresponding rendering layers simultaneously. These rendering layers consume computing resources in the processor's rendering pipeline to render their corresponding display content. If the number of these rendering layers exceeds a threshold, terminal device 110 may be unable to complete the rendering tasks of all rendering layers within a predetermined period.

[0046] In some embodiments, the threshold number (denoted as Layer_Max) indicates the upper limit of the number of rendering layers that can be rendered within a predetermined period. Layer_Max is an upper limit value representing the maximum number of rendering layers that the device can effectively process and complete rendering within a predetermined time range.

[0047] In some embodiments, the predetermined period can be the VSync time of the display screen. VSync refers to the synchronization signal emitted by the display screen after completing one vertical refresh cycle. This signal is used to synchronize the GPU's frame rendering and the display's refresh rate. Each time a vertical refresh is triggered, the display screen begins to draw a new frame of image. The Vsync time (i.e., the vertical refresh cycle) is the time interval between the completion of the display of the current frame and the start of the refresh of the next frame. The VSync time is usually related to the refresh rate of the display screen. The Vsync time determines how many times the display can refresh the image per second. For example, the Vsync time of a display screen with a refresh rate of 60Hz can be 16.67 milliseconds (i.e., 60 refreshes per second).

[0048] In some embodiments, the threshold number is determined based on a predetermined period and the capability information of the processing unit configured to perform rendering. The predetermined period (i.e., Vsync time) determines the available time for each rendering task. The display content corresponding to each application (such as windows, UI, background images, etc.) needs to be rendered through rendering layers and combined into the final image by a compositor. It is generally expected that the rendering of all rendering layers will be completed simultaneously within the predetermined period. If the Vsync period is short (e.g., for high refresh rate displays), the rendering time for each rendering layer will be more limited.

[0049] The processing unit that performs rendering can be a GPU and / or any other processing unit with rendering capabilities. The capability information of this processing unit, also known as its performance, determines how many rendering tasks the terminal device 110 can complete within a predetermined period. If the processing unit has good performance, the predetermined Layer_Max can be higher, indicating that more rendering layers can be rendered within one Vsync cycle. If the processing unit has weak performance, Layer_Max will be lower to avoid exceeding the processing unit's capacity and ensure that the rendering task of each rendering layer is completed within the specified time.

[0050] In some embodiments, the capability information considered for the processing unit may indicate the data transfer rate and / or throughput of the processing unit. The data transfer rate of the processing unit typically refers to the rate of data transfer between the video memory and the processing unit. A higher data transfer rate helps improve the data transfer capability of the processing unit during graphics rendering. Throughput includes metrics such as pixel fill rate and texture fill rate, which determine the generation speed of content per frame. Alternatively or additionally, capability information may also indicate other metrics related to graphics rendering and display, such as shading capability, video memory bandwidth, parallel computing capability, and rendering latency. In some examples, if the data transfer rate and / or throughput of the processing unit is higher, then the value of Layer_Max can be determined to be higher.

[0051] As an example, Figure 3 A schematic diagram of a system 300 for display processing according to some embodiments of the present disclosure is shown. System 300 may be implemented at a terminal device 110.

[0052] refer to Figure 3 The system 300 determines the number of multiple rendering layers to be rendered on the display screen based on the launched spatial application 310.

[0053] Here, spatial application 310 can be different applications or application interfaces. For example, multiple applications may be launched simultaneously on terminal device 110, such as media playback applications, browser applications, game applications, etc., or one application may be launched with multiple interfaces that need to be displayed on the display screen simultaneously. Spatial application 310 can also be different application instances in an XR device. For example, on an XR device, spatial application 310 may represent a virtual screen, an interactive interface, or a UI component. Rendering resources can be requested from application management module 320 corresponding to at least one rendering layer of spatial application 310 to obtain the corresponding rendering results to be presented on the display screen.

[0054] In box 321, the application management module 320 determines whether the number of rendering layers requiring rendering exceeds a threshold. If the number of rendering layers exceeds the threshold, it means the number of rendering layers exceeds the processing unit's processing capacity. This indicates that the processing unit may be unable to render all rendering layers within one Vsync time, potentially leading to delayed screen updates and issues such as stuttering, screen tearing, and glitches. In this case, the application management module 320 can proceed to box 322. In box 322, the application management module 320 determines the priority of each of the multiple rendering layers.

[0055] In an environment where multiple application interfaces are displayed on a screen, depending on the display position and activity level of each interface, different application interfaces may occupy different display areas of the screen. Application interfaces may overlap or obscure each other, and some rendering layers may be obscured or covered by other rendering layers. To optimize rendering efficiency and allocate computing resources rationally, the application management module 320 can determine the priority of each rendering layer, and then determine the rendering strategy for rendering layers of different priorities.

[0056] In some embodiments, the application management module 320 can determine the priority of each of the multiple rendering layers based on the position information of the multiple rendering layers on the display screen. Alternatively or additionally, the application management module 320 can determine the priority of each of the multiple rendering layers based on the active state of the application interface corresponding to each of the multiple rendering layers.

[0057] In some embodiments, the position information of a rendering layer on the display screen refers to the position information of at least a portion of the interface area in the application interface corresponding to each of the multiple rendering layers on the display screen. The position information of a rendering layer can be used to determine the presentation position or display area of ​​at least a portion of the interface area in the application interface corresponding to that rendering layer. The position information may include the size of the application interface and its coordinates on the display screen.

[0058] In some embodiments, the application management module 320 can determine the degree of occlusion of at least a portion of the interface area corresponding to each rendering layer on the display screen based on the position information of each rendering layer among multiple rendering layers. The degree of occlusion refers to the extent to which at least a portion of the interface area corresponding to a rendering layer is occluded on the display screen by at least a portion of the interface area corresponding to other rendering layers. For example, the application management module 320 can calculate the percentage of the interface area that is occluded to obtain a continuous proportional number to represent the degree of occlusion. Alternatively or additionally, the application management module 320 can also represent the degree of occlusion using discrete levels (such as high, medium, low, etc.).

[0059] As an example, the application management module 320 can determine the position of each rendering layer on the display screen based on the coordinates and size information of each rendering layer. Then, based on the relative position of each rendering layer with other rendering layers, it calculates the proportion of the area of ​​the rendering layer that is obscured by other windows on the screen to the total area, thereby determining the degree of obscuration of that rendering layer.

[0060] If a rendering layer has a lower degree of occlusion, it can be considered more important to the user experience and should be rendered first. For example, foreground application interfaces and interactive UI components need to be rendered with high quality. However, if a rendering layer is mostly covered by other application interfaces (high degree of occlusion), rendering that rendering layer completely and independently may lead to a waste of computing resources. The application management module 320 can reduce the priority of these rendering layers, thereby reducing unnecessary rendering calculations.

[0061] In some embodiments, a first rendering layer among a plurality of rendering layers has a first occlusion degree and / or a first active state, a second rendering layer has a second occlusion degree and / or a second active state, and the first occlusion degree is greater than the second occlusion degree, the first active state indicates an activity level lower than the activity level indicated by the second active state, and wherein a first priority of the first rendering layer is determined to be lower than a second priority of the second rendering layer.

[0062] As an example, if a first rendering layer has a first degree of occlusion and a second rendering layer has a second degree of occlusion, and the first degree of occlusion is greater than the second degree of occlusion, the application management module 320 can determine that the first priority of the first rendering layer is lower than the second priority of the second rendering layer. The determination of rendering layer priorities can be deduced similarly.

[0063] As another example, if the first and second rendering layers have the same degree of occlusion (e.g., all rendering layers are fully visible or have the same occlusion percentage), the application management module 320 can further determine their priorities based on their activity states. If the first rendering layer has a first active state, the second rendering layer has a second active state, and the activity level indicated by the first active state is lower than that indicated by the second active state, the application management module 320 can determine that the first priority of the first rendering layer is lower than the second priority of the second rendering layer. The activity state information at least indicates the display order (the order in which the application interface corresponding to the rendering layer is launched or triggered), that is, the most recently opened application interface is considered to be the one most relevant to the current user and should have a higher priority.

[0064] By prioritizing rendering layers, the allocation of rendering resources is effectively optimized, ensuring that critical content is rendered first and improving display processing efficiency. Based on factors such as the visibility, occlusion level, and activity status of rendering layers, important interfaces within the user's current interaction or field of view are rendered first, avoiding low-priority rendering layers consuming excessive computing resources. Furthermore, the prioritization of rendering layers allows for dynamic adjustments to subsequent display processing, thereby improving the device's adaptability to multitasking environments and enabling it to quickly respond to changes in the application interface.

[0065] The above discussion covered determining the priority of the rendering layer. (Return to reference) Figure 2 In frame 230, terminal device 110 performs rendering on multiple rendering layers within a predetermined period based on the priority of each rendering layer, and obtains the rendering result.

[0066] In some embodiments, the terminal device 110 divides the multiple rendering layers into a first group of rendering layers and a second group of rendering layers based on their respective priorities. The first group of rendering layers includes a predetermined number of rendering layers. For example, if the number of rendering layers exceeds the maximum number of rendering layers, the terminal device 110 can divide these rendering layers into different groups according to their priorities and assign different rendering strategies to each group. This ensures that rendering is not exceeded when multiple rendering layers are rendered.

[0067] In some embodiments, the terminal device 110 may determine a predetermined number of rendering layers with higher priority among multiple rendering layers as a first group of rendering layers. For example, the terminal device 110 may determine the first group of rendering layers as the top (Layer_Max-1) rendering layers with the highest priority in the priority sequence. The first group of rendering layers may be rendering layers with higher priority, such as those corresponding to the application interface that the user is interacting with or UI components that need to be displayed in the foreground, and therefore can be allocated more rendering resources.

[0068] In other embodiments, the terminal device 110 identifies the remaining rendering layers among multiple rendering layers as a second group of rendering layers. For example, the terminal device 110 assigns the remaining lower-priority rendering layers to the second group of rendering layers. In some examples, if the first group of rendering layers includes (Layer_Max-1) rendering layers, then the second group of rendering layers includes the remaining rendering layers from all the rendering layers to be rendered. The second group of rendering layers may be partially occluded, lower-priority rendering layers that can have their computational cost reduced through optimization.

[0069] In some embodiments, the terminal device 110 renders the display content corresponding to the first group of rendering layers in a predetermined number of independent rendering layers within a predetermined period, thereby obtaining a first number of first rendering results. Since the number of the first group of rendering layers is fixed, the first number can be equal to the number of the first group of rendering layers, i.e., a predetermined number, such as (Layer_Max-1).

[0070] As an example, return to the reference. Figure 3 In box 323, the application management module 320 can submit a predetermined number (e.g., Layer_Max-1) of the rendering layers in the first group of rendering layers to the compositor 330 for rendering. The display content corresponding to each rendering layer can be rendered independently in the rendering pipeline, ensuring smooth display. For example, the terminal device 110 can allocate corresponding rendering resources (such as independent frame buffers) for each independent rendering layer. Then, by performing a standard rendering process on the display content corresponding to each (Layer_Max-1) rendering layer in each independent rendering layer, corresponding (Layer_Max-1) first rendering results are obtained. These independent rendering results can be cached in the compositor 330 for use in subsequent final image compositing.

[0071] In some embodiments, the terminal device 110 determines a second rendering result by rendering the display content corresponding to the second set of rendering layers through shared rendering layer rendering within the same predetermined period.

[0072] As an example, see reference Figure 3In box 324, instead of allocating independent rendering resources for each of the second number of rendering layers with lower priority, terminal device 110 projects the second number of rendering layers into a shared rendering layer (such as called the EyeBuffer Projection layer) and submits it as a single rendering layer to compositor 330.

[0073] A shared rendering layer can be understood as a base rendering layer. It can be used to merge and host multiple lower-priority rendering layers, reducing the number of independently rendered layers and thus improving rendering efficiency. The shared rendering layer, as the base rendering layer, can always exist, ensuring that even if only one application is running, there is a base layer available for rendering. The corresponding UI area for the shared rendering layer can be the entire display area of ​​the screen, used to display the basic background or merge multiple lower-priority rendering layers.

[0074] In some examples, the display content corresponding to the second set of rendering layers is projected onto the shared rendering layer. For instance, the display content corresponding to the second set of rendering layers is mapped as a texture or map and then projected onto the shared rendering layer. The size and position of this display content within the shared rendering layer remain unchanged; it can still be displayed on the screen, but no longer occupies a separate rendering layer. The display content corresponding to the second set of rendering layers is merged into the shared rendering layer and rendered as a whole, resulting in a second rendering result. This process still involves rendering calculations, but the computational cost is lower than that of independently rendering multiple rendering layers.

[0075] In some embodiments, the rendering result may include image data obtained after processing such as calculation, coloring, and rasterization, for compositing and displaying on the display screen. The compositor 330 may determine the rendering result to be displayed on the display screen based on a first number of first rendering results and second rendering results. For example, the compositor 330 performs layer compositing on the first number of first rendering results and second rendering results, superimposing the rendering results of multiple rendering layers together according to predetermined rules to generate the final rendering result.

[0076] Because the number of rendering layers submitted to the compositor never exceeds the threshold, even if multiple applications or interfaces are launched simultaneously, the terminal device 110 can still complete the rendering and compositing within the vertical refresh time. In this way, smooth display output can be maintained, avoiding problems such as stuttering, screen tearing, and glitches.

[0077] Return to reference Figure 2 In frame 240, terminal device 110 presents the rendering results on the display screen.

[0078] As an example, terminal device 110 can retrieve the final rendered result from the compositor at each vertical refresh time to refresh the screen. Terminal device 110 then replaces the previous frame with the rendered result obtained from the compositor, thereby presenting the new rendered image on the display screen.

[0079] Continue to refer to Figure 3 ,like Figure 3 As shown in box 321, if the application management module 320 determines that the number of multiple rendering layers is less than or equal to the threshold number, it can perform rendering on the display content corresponding to each of the multiple rendering layers in an independent rendering layer to obtain the rendering result.

[0080] For example, the application management module 320 can exclusively submit multiple rendering layers to the compositor 330 for rendering. The application management module 320 can render the display content corresponding to each rendering layer separately in each independent rendering layer and obtain its corresponding rendering result. These rendering results can be cached in the compositor 330 for subsequent final image compositing. The terminal device 110 can obtain the composited rendering result and present it on the display screen.

[0081] In summary, according to the various embodiments of this disclosure, by determining the priority of each rendering layer and optimizing the rendering method of low-priority rendering layers, the number of independent rendering layers can be reduced without limiting the number of open application windows. This effectively reduces the rendering load on the screen. By doing so, problems such as stuttering, screen tearing, and glitches caused by overload rendering can be avoided, thereby improving the user experience in multi-window display environments. Example devices and equipment

[0082] Figure 4 A schematic structural block diagram of an apparatus 400 for display processing according to some embodiments of the present disclosure is shown. The apparatus 400 may be implemented in or included in a terminal device 140, for example. The various modules / components in the apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0083] like Figure 4As shown, the apparatus 400 includes a rendering layer determination module 410, configured to determine a plurality of rendering layers to be rendered on a display screen, each rendering layer corresponding to at least a portion of the interface area in the application interface to be displayed; a priority determination module 420, configured to determine the priority of each of the plurality of rendering layers in response to the number of the plurality of rendering layers being greater than a threshold number, the threshold number indicating an upper limit on the number of rendering layers that can be rendered within a predetermined period; a rendering module 430, configured to perform rendering on the plurality of rendering layers within a predetermined period based on the priority of each of the plurality of rendering layers to obtain a rendering result; and a presentation module 440, configured to present the rendering result on the display screen.

[0084] In some embodiments, the rendering layer determination module 410 is further configured to determine a set of rendering layers of the application interface to be displayed on the display screen; determine multiple rendering layers within the user's field of view on the display screen based on the position information of each rendering layer in the rendering layer set on the display screen; and determine the multiple rendering layers within the user's field of view as multiple rendering layers to be rendered.

[0085] In some embodiments, the priority determination module 420 is further configured to determine the priority of each of the multiple rendering layers based on at least one of the following: the position information of the multiple rendering layers on the display screen, and the active state of the application interface corresponding to each of the multiple rendering layers.

[0086] In some embodiments, the priority determination module 420 is further configured to determine the degree of occlusion of at least a portion of the interface area corresponding to each rendering layer on the display screen based on the position information of each rendering layer in the plurality of rendering layers; and to determine the priority of each of the plurality of rendering layers based on the degree of occlusion and / or the active state of each rendering layer.

[0087] In some embodiments, a first rendering layer among a plurality of rendering layers has a first occlusion degree and / or a first active state, a second rendering layer has a second occlusion degree and / or a second active state, and the first occlusion degree is greater than the second occlusion degree, the first active state indicates an activity level lower than the activity level indicated by the second active state, and wherein a first priority of the first rendering layer is determined to be lower than a second priority of the second rendering layer.

[0088] In some embodiments, the rendering module 430 is further configured to divide the multiple rendering layers into a first group of rendering layers and a second group of rendering layers based on the priority of each of the multiple rendering layers, wherein the first group of rendering layers includes a predetermined number of rendering layers; within a predetermined period, by performing rendering on the display content corresponding to the first group of rendering layers in a predetermined number of independent rendering layers respectively, a first number of first rendering results are obtained, and by performing rendering on the display content corresponding to the second group of rendering layers in a shared rendering layer, a second rendering result is obtained; and based on the first number of first rendering results and second rendering results, a rendering result is determined.

[0089] In some embodiments, the rendering module 430 is further configured to determine a predetermined number of rendering layers with higher priority among the plurality of rendering layers as a first group of rendering layers; and to determine the remaining rendering layers among the plurality of rendering layers as a second group of rendering layers.

[0090] In some embodiments, the rendering module 430 is further configured to determine the rendering result by synthesizing a first number of first rendering results and second rendering results.

[0091] In some embodiments, the rendering module 430 is further configured to project display content corresponding to a second number of second rendering layers onto a shared rendering layer; and to determine a second rendering result by rendering the second display content in the shared rendering layer.

[0092] In some embodiments, the apparatus 400 is further configured to, in response to the number of multiple rendering layers being less than or equal to a threshold number, perform rendering on the display content corresponding to each of the multiple rendering layers in an independent rendering layer to obtain a rendering result; and present the rendering result on a display screen.

[0093] The units and / or modules included in device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units and / or modules in device 400 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.

[0094] It should be understood that one or more steps in the above methods can be performed by suitable electronic devices or combinations of electronic devices. Such electronic devices or combinations of electronic devices may include, for example, […]. Figure 1Terminal device 110 and / or server 130.

[0095] Figure 5 A block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 5 The electronic device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 5 The illustrated electronic device 500 may include or be implemented as Figure 1 The terminal device 110, or Figure 4 Device 400.

[0096] like Figure 5 As shown, electronic device 500 is in the form of a general-purpose electronic device. Components of electronic device 500 may include, but are not limited to, one or more processors or processing units 510, memory 520, storage devices 530, one or more communication units 540, one or more input devices 550, and one or more output devices 550. Processing unit 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.

[0097] Electronic device 500 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 500.

[0098] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 5As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0099] Communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of components of electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0100] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 550 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. External devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0101] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0102] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0103] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0104] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0105] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some, as newer, implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0106] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A display processing method, comprising: Determine multiple rendering layers to be rendered on the display screen, each rendering layer corresponding to at least a portion of the interface area in the application interface to be displayed; In response to the fact that the number of the plurality of rendering layers is greater than a threshold number, the priority of each of the plurality of rendering layers is determined, wherein the threshold number indicates the upper limit of the number of rendering layers that can be rendered within a predetermined period. Rendering is performed on the multiple rendering layers within the predetermined period based on their respective priorities to obtain rendering results; as well as The rendering result is presented on the display screen.

2. The method of claim 1, wherein determining the plurality of rendering layers to be rendered on the display screen comprises: Determine the set of rendering layers for the application interface to be displayed on the display screen; Based on the position information of each rendering layer in the rendering layer set on the display screen, multiple rendering layers within the user's field of view of the display screen are determined from the rendering layer set. as well as The multiple rendering layers within the user's field of view are determined as the multiple rendering layers to be rendered.

3. The method of claim 1, wherein determining the priority of each of the plurality of rendering layers comprises: The priority of each of the plurality of rendering layers is determined based on at least one of the following: The position information of the multiple rendering layers on the display screen. The active status of the application interface corresponding to each of the multiple rendering layers.

4. The method according to claim 3, wherein determining the priority of each of the plurality of rendering layers based on the display position of at least a portion of the application interface corresponding to each of the plurality of rendering layers on the display screen includes: Based on the position information of each of the plurality of rendering layers, the degree of occlusion of at least a portion of the interface area corresponding to each rendering layer on the display screen is determined respectively. as well as The priority of each of the multiple rendering layers is determined based on the degree of occlusion and / or the active state of each rendering layer.

5. The method according to claim 4, wherein the first rendering layer of the plurality of rendering layers has a first occlusion degree and / or a first active state, the second rendering layer has a second occlusion degree and / or a second active state, and the first occlusion degree is greater than the second occlusion degree, the activity level indicated by the first active state is lower than the activity level indicated by the second active state, and The first priority of the first rendering layer is determined to be lower than the second priority of the second rendering layer.

6. The method according to claim 1, wherein rendering is performed on the plurality of rendering layers within the predetermined period based on the respective priorities of the plurality of rendering layers to obtain a rendering result, comprising: Based on the priority of each of the multiple rendering layers, the multiple rendering layers are divided into a first group of rendering layers and a second group of rendering layers, wherein the first group of rendering layers includes a predetermined number of rendering layers. Within the predetermined period, by rendering the display content corresponding to the first group of rendering layers in the predetermined number of independent rendering layers respectively, a first number of first rendering results are obtained, and by rendering the display content corresponding to the second group of rendering layers in the shared rendering layer, a second rendering result is obtained; as well as The rendering result is determined based on the first rendering result and the second rendering result of the first number.

7. The method of claim 6, wherein dividing the plurality of rendering layers into a first group of rendering layers and a second group of rendering layers comprises: The predetermined number of rendering layers with the highest priority among the plurality of rendering layers are determined as the first group of rendering layers; as well as The remaining rendering layers among the plurality of rendering layers are determined as the second group of rendering layers.

8. The method of claim 6, wherein determining the rendering result based on the first number of first rendering results and the second rendering result comprises: The rendering result is determined by synthesizing the first number of first rendering results and the second rendering results.

9. The method according to claim 6, wherein determining the second rendering result by rendering the display content corresponding to the second set of rendering layers through shared rendering layer rendering includes: The display content corresponding to the second set of rendering layers is projected onto the shared rendering layer; as well as The second rendering result is determined by rendering the display content in the shared rendering layer.

10. The method of claim 1, wherein the number of thresholds is determined based on the predetermined period and the capability information of the processing unit configured to perform rendering.

11. The method according to claim 1, further comprising: In response to the number of the plurality of rendering layers being less than or equal to the threshold number, rendering is performed on the display content corresponding to each of the plurality of rendering layers in an independent rendering layer to obtain a rendering result; as well as The rendering result is presented on the display screen.

12. An apparatus for display processing, comprising: The rendering layer determination module is configured to determine multiple rendering layers to be rendered on the display screen, each rendering layer corresponding to at least a portion of the interface area in the application interface to be displayed. The priority determination module is configured to determine the priority of each of the multiple rendering layers in response to the number of the multiple rendering layers being greater than a threshold number, wherein the threshold number indicates the upper limit of the number of rendering layers that can be rendered within a predetermined period. The rendering module is configured to perform rendering on the plurality of rendering layers within a predetermined period based on the priority of each of the plurality of rendering layers, and obtain a rendering result; as well as A rendering module is configured to render the rendering result on the display screen.

13. An electronic device, comprising: At least one processor; as well as At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 11 when executed by the at least one processor.

14. A computer-readable storage medium having stored thereon computer-executable instructions that can be executed by a processor to implement the method according to any one of claims 1 to 11.

15. A computer-executable instruction product, the computer-executable instruction product being tangibly stored in a computer storage medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 11.

Images