Multi-window display processing method, electronic device and storage medium

Abstract

The present application relates to the technical field of window display, and to a multi-window display processing method, an electronic device and a storage medium. In the method, for a plurality of windows displayed simultaneously in a display interface of a large-screen electronic device, some or all of blocked areas in some or all of blocked windows can be obtained, so that when rendering the blocked windows, in the display interface, simple rendering can be performed on the blocked areas and complex rendering can be performed on blocking areas. In this way, the repeated and complex rendering for pixels corresponding to the blocked areas can be reduced, and on the basis of the characteristic that rendering a solid color box consumes less power than rendering a texture, the power consumption of the electronic device can be reduced.

Description

Multi-window display processing method, electronic device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on November 1, 2023, with application number 202311450475.6 and application name “A multi-window display processing method, electronic device and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of window display technology, and in particular to a multi-window display processing method, electronic device, and storage medium. Background Art

[0003] Currently, displaying multiple rounded-corner windows (i.e., windows with four rounded corners) on a display interface is gradually becoming a necessity for large-screen electronic devices (such as tablets, foldable dual-screen mobile phones, etc.).

[0004] In some multi-window display processing methods, the display interface is first drawn by drawing the layers corresponding to multiple rounded-corner windows, and then a hardware compositor is used to composite these multiple layers. However, due to the large number of rounded-corner windows in the display interface, the hardware compositor is under great pressure to perform the synthesis. Therefore, in other multi-window display processing methods, a graphics processor is used to composite multiple layers.

[0005] However, since multiple rounded-corner windows are displayed on the display interface, part or all of the areas of some windows may be blocked by part or all of the areas of other windows. Therefore, when drawing the layers corresponding to the multiple rounded-corner windows, it is necessary to perform complex drawing on the blocked areas of the blocked windows on some pixels of the display interface, and perform complex drawing on the blocked areas of the blocking windows on these pixels. This will cause repeated complex drawing of some pixels, that is, redundant drawing of multiple graphics elements overlapping, which will increase the power consumption of the electronic device.

[0006] Summary of the Invention

[0007] In order to solve the problem that repeated and complex drawing of some pixels in a display interface increases the power consumption of an electronic device, the present application provides a multi-window display processing method, an electronic device and a storage medium.

[0008] In a first aspect, the present application provides a multi-window display processing method for an electronic device, which may include: obtaining a set of windows to be drawn corresponding to a first display interface; the set of windows to be drawn includes a first window to be drawn and a second window to be drawn that partially overlap; obtaining a first area of ​​the first window to be drawn in the set of windows to be drawn, wherein at least a partial area of ​​the second window to be drawn that overlaps with the first area is the second area; obtaining a first drawing instruction for the second area, the first drawing instruction being used to draw a first solid color box in the second area; when drawing the first display interface, skipping the second drawing instruction, and executing the first drawing instruction; the second drawing instruction includes a native drawing instruction for the second area of ​​the second window to be drawn.

[0009] Based on the above scheme, by skipping the native drawing instructions of the occluded area and executing the solid color frame drawing instructions of the occluded area, that is, simply drawing the occluded window, the repeated and complex drawing of pixels corresponding to the occluded area can be reduced, and based on the characteristic that drawing a solid color frame consumes less power than drawing a texture, the power consumption of electronic devices can be reduced.

[0010] It can be understood that the window to be drawn in the set of windows to be drawn can be a rounded-corner window, and the first window to be drawn can be located above the second window to be drawn, that is, the first window to be drawn can be a blocking window, and the second window to be drawn can be a blocked window, the first area can be a blocking area, and the second area can be a blocked area.

[0011] In some optional examples, the first solid color frame should not be understood as a hollow frame, but should be understood as an entire area, and executing the first drawing instruction may be to draw the first area with the same RGB value.

[0012] In some optional instances, the technical solution can be applied to window drawing within a single frame, or to window drawing of adjacent multiple frames. For example, in a scenario where the stacking relationship between windows has not changed, the steps before obtaining the first drawing instruction for the second area can be implemented within the time of the first frame, and obtaining the first drawing instruction for the second area and the subsequent steps can be implemented within the time of the second frame. The time of the first frame and the second frame, for example 8.33ms or 16.67ms, can be used as an implementation cycle of the technical solution. Among them, the first frame draws and displays the corresponding interface, but does not perform operations such as drawing a solid color box. Alternatively, an implementation cycle may also include a third frame, and the third frame may be running the above-mentioned technical solution on the basis of the second frame.

[0013] In some optional examples, after detecting that a stacking relationship between windows has changed, for example, a change in the overlapping area between a first window to be drawn and a second window to be drawn, the above technical solution is executed.

[0014] In some optional instances, the central processing unit may first send the second drawing instruction to the composite display module and then send the first drawing instruction. When the composite display module draws the first display interface, the second drawing instruction may be skipped and the first drawing instruction may be executed.

[0015] In some optional implementations of the first aspect, an area of ​​the second region is smaller than or equal to an area of ​​a region in the second window to be drawn that overlaps with the first region.

[0016] In some optional implementations of the first aspect, the multi-window display processing method further includes reporting the second area to a first drawing thread of a first application corresponding to a second window to be drawn, and the first drawing thread of the first application generates a first drawing instruction based on the second area.

[0017] In some optional embodiments, the composite display module can perform a multi-rectangular overlap determination on the windows to be drawn in the set of windows to be drawn, output the occlusion area and / or occluded area of ​​each window to be drawn, and report this information in the form of occlusion tuple information to the main thread within the application corresponding to each window to be drawn, that is, to the central processing unit. Then, the drawing thread of the application corresponding to each window to be drawn in the central processing unit can generate a drawing instruction, such as a first drawing instruction, based on the occlusion tuple information of each window to be drawn.

[0018] In some optional instances, a window to be drawn may have multiple occlusion areas, that is, part or all of the area of ​​a window to be drawn may occlude part or all of the area of ​​each of multiple windows to be drawn; a window to be drawn may have multiple occluded areas, that is, part or all of the area of ​​a window to be drawn may be occluded by part or all of the area of ​​each of multiple windows to be drawn.

[0019] In some specific implementations, for window 0 to be drawn corresponding to application 0, window 1 to be drawn corresponding to application 1, window 2 to be drawn corresponding to application 2, etc., the occlusion tuple information corresponding to the set of windows to be drawn can be expressed as {<app0,rect0> ,<app1,rect1> ,<app2,rect2> ,...}, where the occlusion tuple information of each window to be drawn can be expressed as<app i,rect i(x,y,w,h)> , where x and y can represent the coordinates of the point corresponding to the upper left corner of the rectangle corresponding to the occluding area or the occluded area, w can represent the width of the rectangle corresponding to the occluding area or the occluded area, and h can represent the height of the rectangle corresponding to the occluding area or the occluded area.

[0020] In some optional implementations of the first aspect, skipping the second drawing instruction includes deleting the second drawing instruction, discarding the second drawing instruction, or not executing the second drawing instruction.

[0021] In some optional implementations of the first aspect, the native drawing instruction is used to draw the texture and / or display special effects of the second area of ​​the second window to be drawn.

[0022] It is understood that the display special effects may include frosted glass effects, transparent effects, anti-aliasing effects, and other special effects.

[0023] In some optional implementations of the first aspect, the first window to be drawn and the second window to be drawn belong to the same application, or the first window to be drawn and the second window to be drawn belong to different applications.

[0024] In some optional examples of the first aspect, the drawing mode of the first drawing instruction is an overlay mode.

[0025] In some optional implementations of the first aspect, the multi-window display processing method further includes: obtaining a third area of ​​a third window to be drawn in the set of windows to be drawn, wherein at least a portion of the area of ​​the third window to be drawn that overlaps with the second area is the third area; obtaining a third drawing instruction for the third area, the third drawing instruction being used to draw a second solid color frame in the third area; when drawing the first display interface, skipping the fourth drawing instruction, and executing the third drawing instruction; the fourth drawing instruction includes a native drawing instruction for the third area of ​​the third window to be drawn.

[0026] Based on the above scheme, by skipping the native drawing instructions of the occluded area and executing the solid color frame drawing instructions of the occluded area, that is, simply drawing the occluded window, the repeated and complex drawing of pixels corresponding to the occluded area can be reduced, and based on the characteristic that drawing a solid color frame consumes less power than drawing a texture, the power consumption of electronic devices can be reduced.

[0027] It can be understood that the third window to be drawn can be located below the second window to be drawn and below the first window to be drawn, that is, the first window to be drawn can be a blocking window, the second window to be drawn can be a blocked window, and the third window to be drawn can also be a blocked window. The first area can be a blocking area, the second area can be a blocked area, and the third area can also be a blocked area.

[0028] In some optional examples, the second solid color frame is not a hollow frame but a whole area, and executing the fifth drawing instruction may be to draw the third area with the same RGB value.

[0029] In some optional instances, the central processing unit may first send the fourth drawing instruction and then the third drawing instruction to the composite display module. When the composite display module draws the first display interface, the fourth drawing instruction may be skipped and the third drawing instruction may be executed.

[0030] In a second aspect, an embodiment of the present application provides a multi-window display processing method for an electronic device, comprising: obtaining a set of windows to be drawn corresponding to a first display interface; the set of windows to be drawn includes a first rounded-corner window to be drawn and a second rounded-corner window to be drawn that partially overlap; obtaining a first part of the first rounded-corner window to be drawn in the set of windows to be drawn, wherein at least a partial area of ​​the first part of the first rounded-corner window to be drawn overlaps with a first area of ​​the second rounded-corner window to be drawn; obtaining a first drawing instruction for the first part of the first rounded-corner window to be drawn, the first drawing instruction including a native drawing instruction for the first part; when drawing the first display interface, skipping the second drawing instruction, and executing the first drawing instruction; the second drawing instruction including a native drawing instruction for the first area of ​​the second rounded-corner window to be drawn.

[0031] In some optional examples of the second aspect, the first portion of the first window with rounded corners to be drawn is a portion corresponding to an inscribed rectangle in the first window with rounded corners to be drawn.

[0032] In some optional examples of the second aspect, skipping the second drawing instruction includes deleting the second drawing instruction, discarding the second drawing instruction, or not executing the second drawing instruction.

[0033] In some optional examples of the second aspect, the native drawing instruction is used to draw the texture and / or display special effects of the first area of ​​the second rounded-corner window to be drawn.

[0034] It is understood that the display special effects may include frosted glass effects, transparent effects, anti-aliasing effects, and other special effects.

[0035] In some optional examples of the first aspect, the drawing mode of the first drawing instruction is an overlay mode.

[0036] In some optional examples of the second aspect, the first window to be drawn with rounded corners and the second window to be drawn with rounded corners belong to the same application, or the first window to be drawn with rounded corners and the second window to be drawn with rounded corners belong to different applications.

[0037] In some optional instances of the second aspect, the multi-window display processing method further includes: obtaining a second area of ​​a third rounded-corner window to be drawn in the set of windows to be drawn, wherein a partial area of ​​the third rounded-corner window to be drawn that overlaps with the first area is the second area; obtaining a third drawing instruction for the second area, the third drawing instruction including a native drawing instruction for the second area; when drawing the first display interface, skipping the third drawing instruction, and executing the first drawing instruction.

[0038] In a third aspect, the present application provides an electronic device comprising: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, which is one of the one or more processors of the electronic device, for executing the multi-window display processing method mentioned in the present application.

[0039] In a fourth aspect, the present application provides a readable storage medium having instructions stored thereon. When the instructions are executed on an electronic device, the electronic device executes the multi-window display processing method mentioned in the present application.

[0040] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising: a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium containing a computer program code for executing a multi-window display processing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] FIG1 is a schematic diagram showing a plurality of rounded-corner windows displayed on a display interface according to some examples of the present application;

[0042] FIG2 is a schematic diagram showing a comparison of usage frequencies of functions of multiple rounded-corner windows in a tablet according to some examples of the present application;

[0043] FIG3 is a schematic diagram showing a plurality of rounded-corner windows superimposed on a display interface according to some examples of the present application;

[0044] FIG4 is a schematic diagram showing a plurality of rounded-corner windows superimposed on a display interface according to some examples of the present application;

[0045] FIG5 is a schematic diagram showing a plurality of rounded-corner windows superimposed on a display interface according to some examples of the present application;

[0046] FIG6 shows a schematic diagram of overlapping multiple rounded-corner windows in a display interface according to some examples of the present application.

[0047] FIG7 is a flow chart showing a multi-window display processing method according to some examples of the present application;

[0048] FIG8 is a flowchart illustrating a multi-window display processing method according to some examples of the present application;

[0049] FIG9 is a schematic diagram showing occlusion tuple information of an occluded area of ​​a to-be-drawn window 5 corresponding to App5 that is occluded by a to-be-drawn window 0 corresponding to App0 according to some examples of the present application;

[0050] FIG10 is a schematic diagram showing a method of drawing a solid color frame on an obscured area of ​​a to-be-drawn window 5 corresponding to App5 according to some examples of the present application;

[0051] FIG11 is a schematic diagram showing a method of drawing a solid color frame on a window to be drawn in which a part or all of the area obscures a part or all of the area of ​​another window to be drawn, according to some examples of the present application;

[0052] FIG12 is a schematic diagram showing a method of clipping an inscribed rectangle of a window to be drawn that partially or completely obscures a part or all of an area of ​​another window to be drawn, according to some examples of the present application;

[0053] FIG13 is a schematic diagram showing a method of dividing a layer corresponding to a blocking window into five parts according to some examples of the present application;

[0054] FIG14 shows a hardware structure of an electronic device according to some examples of the present application. DETAILED DESCRIPTION

[0055] The illustrative embodiments of the present application include, but are not limited to, a multi-window display processing method, an electronic device, and a storage medium.

[0056] It can be understood that the multi-window display processing method mentioned in the embodiment of the present application can be used for any large-screen electronic device that can be implemented, such as a tablet, a foldable dual-screen mobile phone, a laptop computer, a desktop computer or a personal digital assistant.

[0057] It is understood that the multi-window display processing method mentioned in the embodiments of the present application can be applied to scenarios such as the simultaneous display of multiple windows of multiple applications, the simultaneous display of multiple windows of a single application, etc. In some optional instances, the multi-window display method mentioned in the embodiments of the present application can be applied to scenarios where multiple rounded-corner windows are displayed simultaneously.

[0058] In order to clearly illustrate the solutions mentioned in the embodiments of the present application, the terms involved in the embodiments of the present application are first explained.

[0059] Operating system: software that manages the hardware of electronic devices and implements functions such as resource allocation. For example, an operating system may include wait.

[0060] Surface Flinger: A system service within the operating system that is specifically responsible for compositing layers and delivering them to the display subsystem. For example, in the Android operating system, this Surface Flinger service can be called Surface Flinger (SF).

[0061] Display subsystem (DSS): The subsystem in the operating system that is responsible for managing data in the memory and sending it to the monitor for display.

[0062] Hardware composer (HWC): A device that composites the layers received from the interface drawing service.

[0063] Vsyn: A signal generated by the display hardware of a terminal device. This signal is used to notify upper-layer software to perform layer drawing and synthesis based on the signal.

[0064] Forward pixel kill (FPK): A technique used by Mali GPUs on the Arm architecture to reduce overdraw. It performs depth testing on pixels in opaque primitives and removes overdrawn pixels based on the depth.

[0065] The following describes how to display multiple rounded-corner windows on a display interface.

[0066] As shown in Figure 1, the tablet's display interface can display window 1 corresponding to a short video application, window 2 corresponding to a chat application, and window 3 corresponding to a conference application, meeting the needs of various office scenarios, online learning scenarios, and entertainment scenarios. Figure 2 shows a schematic diagram comparing the frequency of use of the function of displaying multiple rounded corner windows on a tablet and a straight-screen mobile phone. As shown in Figure 2, the frequency of use of the parallel horizon function on the tablet is much higher than that on the straight-screen mobile phone.

[0067] The following introduces the multi-window display processing methods mentioned in some embodiments.

[0068] In some multi-window display processing methods, the display interface is first drawn by drawing the layers corresponding to multiple rounded-corner windows, and then a hardware compositor is used to composite these multiple layers. However, due to the large number of rounded-corner windows in the display interface, the hardware compositor is under great pressure to perform the synthesis. Therefore, in other multi-window display processing methods, a graphics processor is used to composite multiple layers.

[0069] However, as shown in FIG3, when multiple rounded-corner windows are displayed on the display interface, some windows may be partially or completely blocked by other windows. Therefore, when drawing the layers corresponding to the multiple rounded-corner windows, it is necessary to perform complex drawing on the blocked area of ​​the blocked window on some pixels of the display interface, and to perform complex drawing on the blocked area of ​​the blocking window on the said part of pixels. For example, in the drawing process of rounded-corner window A and rounded-corner window B in FIG3, it is necessary to perform complex drawing on the blocked area a of the blocked window A, and to perform complex drawing on the blocked area b of the blocking window B. The blocked area a and the blocked area b overlap and are located in the same part of pixels. By repeatedly performing complex drawing on the same part of pixels (such as drawing the texture corresponding to the blocked area a and drawing the texture corresponding to the blocked area b on the same part of pixels), repeated complex drawing of some pixels will be caused, that is, there is redundant drawing when multiple graphics elements overlap, which will increase the power consumption of the electronic device.

[0070] In order to solve the above problems, an embodiment of the present application provides another multi-window display processing method, which can obtain part or all of the obscured areas in some or all of the obscured rounded-corner windows for multiple rounded-corner windows displayed synchronously on the display interface of a large-screen electronic device (such as a tablet, a foldable dual-screen mobile phone, etc.), so that when drawing the obscured rounded-corner window, the obscured area can be simply drawn in the display interface, and the obscured area can be complexly drawn. For example, in the drawing process of rounded-corner window A and rounded-corner window B in Figure 3, the obscured area a of the obscured rounded-corner window A is simply drawn, such as drawing the obscured area a as a solid color frame, and the obscured area b of the obscured rounded-corner window B is complexly drawn, such as drawing the texture of the obscured area b and / or displaying special effects. In this way, the repeated complex drawing of pixels corresponding to the obscured area can be reduced, and based on the characteristic that drawing a solid color frame consumes less power than drawing a texture, the power consumption of the electronic device can be reduced.

[0071] For example, for multiple rounded-corner windows of multiple applications displayed synchronously on the display interface of a large-screen electronic device, such as rounded-corner window 0 corresponding to application 0, rounded-corner window 1 corresponding to application 1, rounded-corner window 2 corresponding to application 2, rounded-corner window 3 corresponding to application 3, rounded-corner window 4 corresponding to application 4, and rounded-corner window 5 corresponding to application 5 in Figure 4, the obscured area A of rounded-corner window 5 obscured by rounded-corner window 0 can be obtained, and when drawing rounded-corner window 5, the obscured area A can be drawn as a solid color box.

[0072] In some specific implementations, a simple drawing instruction corresponding to the obscured area can be generated based on the obscured area of ​​the obscured rounded window. When drawing the display interface, the complex drawing instruction corresponding to the obscured area is skipped and the simple drawing instruction is executed. That is, the central processing unit can generate a solid color frame drawing instruction corresponding to the obscured area A, that is, a simple drawing instruction, and first send the complex drawing instruction corresponding to the obscured area A to the synthesis and display module, and then send the simple drawing instruction corresponding to the obscured area A to the synthesis and display module, so that the synthesis and display module skips the complex drawing instruction corresponding to the obscured area A and executes the simple drawing instruction corresponding to the obscured area A.

[0073] It is understood that for multiple rounded-corner windows displayed simultaneously on the display interface of a large-screen electronic device (such as a tablet or a foldable phone), it is possible to obtain the partially or completely occluded area in the rounded-corner window, so that when the layers corresponding to the multiple rounded-corner windows are synthesized, complex rendering can be performed on the occluded area in the display interface. In this way, the repeated complex rendering of the pixels corresponding to the occluded area can be reduced while ensuring the display effect of the occluded area.

[0074] For example, for multiple rounded-corner windows of multiple applications displayed synchronously on the display interface of a large-screen electronic device, such as rounded-corner window 0 corresponding to application 0, rounded-corner window 1 corresponding to application 1, rounded-corner window 2 corresponding to application 2, rounded-corner window 3 corresponding to application 3, rounded-corner window 4 corresponding to application 4, and rounded-corner window 5 corresponding to application 5 in Figure 5, the occlusion area B of rounded-corner window 1, rounded-corner window 2, rounded-corner window 3, rounded-corner window 4, and rounded-corner window 5 blocked by rounded-corner window 0 can be obtained. When synthesizing rounded-corner windows 0, 1, 2, 3, 4, and 5, the occlusion area B in the display interface can be complexly drawn.

[0075] It is understood that for multiple rounded-corner windows displayed simultaneously on the display interface of a large-screen electronic device (such as a tablet or a foldable dual-screen mobile phone), it is possible to obtain a portion or all of the area within the rounded-corner window that is partially or completely obscured. This allows complex rendering to be performed on the portion or all of the area within the obscured rounded-corner window in the display interface when synthesizing the layers corresponding to the multiple rounded-corner windows. This reduces the need for repeated complex rendering of pixels corresponding to the obscured area while ensuring the display quality of the obscured area.

[0076] For example, for the synchronous display of multiple rounded-corner windows on the display interface of a large-screen electronic device, such as rounded-corner window 0 corresponding to application 0, rounded-corner window 1 corresponding to application 1, rounded-corner window 2 corresponding to application 2, rounded-corner window 3 corresponding to application 3, rounded-corner window 4 corresponding to application 4, and rounded-corner window 5 corresponding to application 5 in Figure 6, part or all of area C (such as including occluded areas and non-occluded areas) in rounded-corner window 0 can be obtained, and when synthesizing rounded-corner windows 0, 1, 2, 3, 4, and 5, complex drawing can be performed on area C in the display interface.

[0077] In some specific implementations, simple drawing instructions can be generated based on the occluded area of ​​the window that occludes the rounded corners, and complex drawing instructions can be generated based on the occluded window. When drawing the display interface, the complex drawing instructions corresponding to the occluded area are skipped, and the simple drawing instructions for the occluded area are executed first, and then the complex instructions for the occluded area are executed. Alternatively, the complex drawing instructions corresponding to the occluded area are skipped, and the complex instructions for the occluded area are executed.

[0078] The following describes the multi-window display processing method mentioned in the embodiment of the present application. FIG7 shows a flow chart of a multi-window display processing method, which can be executed by an electronic device. As shown in FIG7 , the multi-window display processing method may include:

[0079] 701: Obtain a set of windows to be drawn, and determine the occluded area and / or occluded area of ​​each window to be drawn in the set of windows to be drawn.

[0080] It is understood that the set of windows to be drawn may include multiple windows of multiple applications, or multiple windows of a single application. The windows may include windows with rounded corners. After obtaining the set of windows to be drawn, occlusion regions may be identified and occlusion tuple information may be transmitted for each window to be drawn in the set of windows to be drawn. The occlusion tuple information may include the coordinates of the point corresponding to the upper left corner of the rectangle corresponding to the occluding region or the occluded region, the width of the rectangle, and the height of the rectangle.

[0081] In some optional instances, when an electronic device pre-displays multiple windows, the HWC in the electronic device can generate a Vsync signal and upload it to the upper-level software surface flinger. The surface flinger can obtain a set of windows to be drawn, and perform multi-rectangular overlap judgment on the windows to be drawn in the set of windows to be drawn, output the occlusion area and / or occluded area of ​​each window to be drawn, and report it to the main thread in the application corresponding to each window to be drawn in the form of occlusion tuple information along with the Vsync signal, that is, report it to the central processing unit.

[0082] In some optional instances, a window to be drawn may have multiple occlusion areas, that is, part or all of the area of ​​a window to be drawn may occlude part or all of the area of ​​each of multiple windows to be drawn; a window to be drawn may have multiple occluded areas, that is, part or all of the area of ​​a window to be drawn may be occluded by part or all of the area of ​​each of multiple windows to be drawn.

[0083] In some optional examples, the occlusion area and the occluded area of ​​the window to be drawn in the window to be drawn set can be represented in the form of occlusion tuple information. For example, for window 0 to be drawn corresponding to application 0, window 1 to be drawn corresponding to application 1, window 2 to be drawn corresponding to application 2, etc., the occlusion tuple information corresponding to the window to be drawn set can be represented as {<app0,rect0> ,<app1,rect1> ,<app2,rect2> ,...}, the occlusion tuple information of each window to be drawn can be expressed as<app i,rect i(x,y,w,h)> , where x and y can represent the coordinates of the point corresponding to the upper left corner of the rectangle corresponding to the occluding area or the occluded area, w can represent the width of the rectangle corresponding to the occluding area or the occluded area, and h can represent the height of the rectangle corresponding to the occluding area or the occluded area.

[0084] In some optional instances, since a window to be drawn may have multiple occlusion areas and multiple occluded areas, the occlusion information rect i(x, y, w, h) in the occlusion tuple information of each window to be drawn may also include rect i1(x, y, w, h), rect i2(x, y, w, h), ..., where rect i1(x, y, w, h) may represent an occluded area, and rect i2(x, y, w, h) may represent another occluded area. rect i1(x, y, w, h) may also represent an occluded area, and rect i2(x, y, w, h) may represent an occlusion area. The embodiment of the present application does not specifically limit the specific form of the occlusion tuple information.

[0085] 702: Generate drawing instructions corresponding to each window to be drawn based on the blocking area and / or blocked area of ​​part or all of the windows to be drawn.

[0086] It is understood that after the CPU obtains the occlusion tuple information corresponding to the set of windows to be drawn, it can perform occlusion culling within the application. Before the rendering thread performs drawing, that is, before submitting the drawing instruction, it generates a solid-color frame drawing instruction corresponding to the occluded area of ​​each window to be drawn. The solid-color frame drawing instruction corresponding to the occluded area is used to instruct the drawing of a solid-color frame. The solid-color frame is not a hollow frame, but a whole area. Executing the solid-color frame drawing instruction can draw the occluded area with the same RGB value.

[0087] In some specific implementations, after receiving the occlusion tuple information of the window to be drawn, the main thread within the application corresponding to each window to be drawn in the set of windows to be drawn can record the occlusion tuple information and, based on the occlusion tuple information, generate a solid-color frame drawing instruction corresponding to the occluded area of ​​the window to be drawn. The main thread first sends the complex drawing instructions for the occluded area (such as instructions for drawing the texture of the occluded area and / or displaying special effects) to the graphics processing unit (GPU), and then sends the solid-color frame drawing instructions corresponding to the occluded area to the GPU. The display special effects may include frosted glass effects, transparency effects, or anti-aliasing effects.

[0088] In some optional instances, a complex drawing instruction corresponding to the unobstructed window to be drawn may be generated, so that complex information such as texture and / or display effects of the unobstructed window to be drawn may be drawn on the display interface based on the complex drawing instruction.

[0089] In other optional instances, a solid color frame drawing instruction corresponding to the occluded area can be generated based on the occluded area of ​​the window to be drawn, so that a solid color frame can be drawn first on the display interface based on the solid color frame drawing instruction corresponding to the occluded area, and then the occluded window can be drawn to cover the solid color frame.

[0090] In some other optional examples, based on the occluded area of ​​the window to be drawn, a drawing instruction for drawing the maximum inscribed rectangle of the occluded area using the overlay mode and a drawing instruction for drawing the rounded corners of the occluded area using the transparent mode may be generated.

[0091] In some optional instances, for window 1 to be drawn corresponding to application 1, window 2 to be drawn corresponding to application 2, window 3 to be drawn corresponding to application 3, window 4 to be drawn corresponding to application 4, window 5 to be drawn corresponding to application 5, and window 0 to be drawn corresponding to application 0, which are stacked from bottom to top, a solid color frame drawing instruction corresponding to each obscured area of ​​windows 1, 2, 3, 4, and 5 to be drawn can be generated, or a solid color frame drawing instruction corresponding to each obscured area of ​​windows 1, 3, and 5 to be drawn can be generated.

[0092] 703: Based on the drawing instruction corresponding to the window to be drawn, perform simple drawing on the blocked area of ​​the window to be drawn, and perform complex drawing on the blocked area of ​​the window to be drawn.

[0093] It can be understood that after receiving the drawing instructions corresponding to each window to be drawn, the graphics processing unit (GPU) can execute the solid color frame drawing instructions corresponding to the obscured area when drawing a window to be drawn whose partial or complete area is obscured by part or all of the areas of other windows to be drawn, and does not execute the complex drawing instructions corresponding to the obscured area, that is, simply draw the obscured area and draw the obscured area as a solid color frame.

[0094] It can be understood that when synthesizing the layers corresponding to the windows to be drawn in the set of windows to be drawn, complex drawing can be performed on the windows to be drawn whose partial or complete areas obscure partial or complete areas of other windows to be drawn according to the stacking order of the windows to be drawn in the set of windows to be drawn, that is, drawing the texture and / or display effects of partial or complete obstructed areas, partial areas (including obstructed areas and unobstructed areas) or complete areas (including obstructed areas and unobstructed areas) of part or all of the windows to be drawn (that is, the obstructing windows) with obstructed areas.

[0095] The multi-window display processing method mentioned above is introduced below in combination with the architecture of the electronic device. Figure 8 shows an architectural block diagram of an electronic device. As shown in Figure 8, the electronic device may include a hardware hybrid renderer 810 and an interface drawing service 820. The interface drawing service 820 may include a signal receiving module 821, an occlusion judgment module 822, a solid color culling occlusion module 823, a rounded corner splitting culling module 824 and a synthetic display module 825.

[0096] The multi-window display processing method may include:

[0097] When the electronic device pre-displays multiple windows, the hardware hybrid renderer 810 in the electronic device can generate a Vsync signal and upload it to the signal receiving module 821 in the upper-layer software interface drawing service 820 .

[0098] When the signal receiving module 821 receives the Vsync signal, the interface drawing service 820 can obtain a set of windows to be drawn, for example, obtaining window 1 to be drawn corresponding to App1, window 2 to be drawn corresponding to App2, window 3 to be drawn corresponding to App3, etc. The signal receiving module 821 in the interface drawing service 820 can also send a trigger signal to the occlusion determination module 822 in the interface drawing service 820 after receiving the Vsync signal uploaded by the hardware hybrid renderer 810.

[0099] The occlusion judgment module 822 in the interface drawing service 820 can perform multi-rectangular overlap judgment on the windows to be drawn in the set of windows to be drawn after receiving the trigger signal sent by the occlusion judgment module 822, output the occlusion area and / or the occluded area of ​​each window to be drawn, and output the occlusion tuple information {<app1,rect1> ,<app2,rect2> , ...} in the form of Vsync signal to the main thread of the application corresponding to each window to be drawn. For example, the occlusion tuple information of the blocked area of ​​the window 5 to be drawn corresponding to App5 in Figure 9 that is blocked by the window 0 to be drawn corresponding to App0 can be<app5-pid,x,y,w,h> Report to the main thread in App5.

[0100] After receiving the occlusion tuple information corresponding to the window to be drawn, the main thread of the application can record the occlusion tuple information, and before the rendering thread draws the controls / graphic elements of the window to be drawn, that is, before submitting the drawing instruction of the window to be drawn, generate a solid color frame drawing instruction to draw the occluded area of ​​the window to be drawn as a solid color frame. For example, as shown in Figure 10, a solid color frame drawing instruction can be generated to draw the occluded area of ​​the window 5 to be drawn corresponding to App5 as a solid color frame. Then, the complex drawing instruction of the window to be drawn can be submitted to the interface drawing service 820 first, and then the solid color frame drawing instruction of the occluded area (that is, the instruction to draw the occluded area as a solid color frame) can be submitted to the interface drawing service 820.

[0101] The occlusion determination module 822 in the interface drawing service 820 can determine the occlusion area of ​​each window to be drawn in the set of windows to be drawn, and then pass the occlusion area of ​​the window to be drawn to the solid color occlusion culling module 823 in the interface drawing service 820, and pass the occlusion culling drawing instruction corresponding to the occlusion area of ​​the window to be drawn to the solid color occlusion culling module 823 in the interface drawing service 820; or, pass the occlusion culling drawing instruction corresponding to the occlusion area of ​​the window to be drawn to the rounded corner splitting culling module 824. When drawing a window to be drawn whose part or all of the area is partially or completely occluded by part or all of the area of ​​another window to be drawn, the solid color occlusion culling module 823 can draw the occluded area as a solid color frame. When drawing a window to be drawn whose part or all of the area occludes part or all of the area of ​​another window to be drawn, the solid color occlusion culling module 823 can draw a solid color frame in the area corresponding to the occlusion area. When drawing a window to be drawn that partially or completely obscures part or all of the area of ​​another window to be drawn, the rounded corner splitting and culling module 824 can divide the obscured area into multiple regions based on the largest inscribed rectangle of the obscured area, draw the texture corresponding to the largest inscribed rectangle using texture drawing mode, and draw the regions corresponding to the four rounded corners of the obscured area using transparent mode. The layers corresponding to each window to be drawn in the set of windows to be drawn can then be passed to the composite display module 825 in the interface drawing service 820.

[0102] The synthesis and display module 825 in the interface drawing service 820 can synthesize each window to be drawn according to the order in which the layers corresponding to each window to be drawn are stacked, and display the synthesized layers to the display interface.

[0103] As shown in FIG11 , in some optional examples, a solid color frame can be drawn on a window to be drawn that partially or completely obscures part or all of the area of ​​another window to be drawn, and then the layer corresponding to the obstructing window is drawn to cover the solid color frame. That is, a solid color frame is first drawn on the area corresponding to the obstructing area on the lower layer where the layer corresponding to the obstructing window is to be set. The layer corresponding to the obstructing window is then set on the solid color frame corresponding to the obstructing area on the lower layer. Specifically, the solid color frame shown in A in FIG11 can be first drawn on the layer corresponding to app5, and then the layer corresponding to app0 can cover the solid color frame A.

[0104] As shown in Figure 12, in other optional examples, the inscribed rectangle of the window to be drawn that partially or completely obscures part or all of the area of ​​other windows to be drawn can be cropped, for example, the maximum inscribed rectangle is cropped, and the drawing mode of the inscribed rectangle is set to the overlay mode (BlendMode=kSrc), so that when drawing the layer corresponding to the obstructing window, the part corresponding to the maximum inscribed rectangle is textured, and the part corresponding to the rounded corner is transparently drawn.

[0105] For example, as shown in Figure 13, the layer corresponding to the occlusion window can be divided into a first part and a second part, wherein the first part can be the part corresponding to the maximum inscribed rectangle of the occlusion window, such as the shaded area shown as ⑤ in Figure 13, and the second part can be the remaining part of the occlusion window except the part corresponding to the maximum inscribed rectangle, such as the non-shaded area shown as ①, ②, ③, and ④ in Figure 13, wherein the four parts ① to ④ are drawn according to the original mode (because the four rounded corners need to be drawn as transparent parts and FPK is not enabled), and the part ⑤ can be set to the overlay mode (BlendMode=kSrc) for drawing mode, so that FPK can be enabled, and the pixels of the part ⑤ can be texture drawn, that is, the display information of the pixels of the part ⑤ is set to the texture data corresponding to the area ⑤.

[0106] In some optional instances, you can use dumpsys SurfaceFlinger to view the synthesis method of the layers corresponding to the windows to be drawn in the set of windows to be drawn. When there are many layers, most of the layers will adopt the Client synthesis method, that is, they will be synthesized using the GPU, and the layers corresponding to the topmost windows to be drawn are also synthesized using the GPU. Therefore, the layers corresponding to the topmost windows to be drawn can be processed in the manner shown in Figure 13. In this way, the repeated and complex drawing of pixels corresponding to the occluded areas can be reduced, thereby reducing the power consumption of electronic devices.

[0107] In some optional examples, the multi-window display method can be applied to a scenario where windows corresponding to multiple applications are displayed simultaneously.

[0108] For example, in a scenario where multiple windows corresponding to a chat application and a short video application are displayed at the same time, after the user opens the chat application, the window corresponding to the chat application can be displayed on the display interface of the electronic device. After the user opens the short video application, the window corresponding to the short video application can be displayed on the display interface of the electronic device. Among them, the window corresponding to the short video application can partially cover the window corresponding to the chat application.

[0109] The following describes the hardware structure of the electronic device. As shown in FIG14 , the electronic device 1400 may include a processor 1410, an external memory interface 1420, an internal memory 1421, a universal serial bus (USB) interface 1430, a charging management module 1440, a power management module 1441, a battery 1442, antenna 1, antenna 2, a mobile communication module 1450, a wireless communication module 1460, an audio module 1470, a speaker 1470A, a receiver 1470B, a microphone 1470C, an earphone jack 1470D, a sensor module 1480, a button 1490, a motor 1491, an indicator 1492, a camera 1493, a display 1494, and a subscriber identification module (SIM) card interface 1495. The sensor module 1480 may include a pressure sensor 1480A, a gyroscope sensor 1480B, an air pressure sensor 1480C, a magnetic sensor 1480D, an acceleration sensor 1480E, a distance sensor 1480F, a proximity light sensor 1480G, a fingerprint sensor 1480H, a temperature sensor 1480J, a touch sensor 1480K, an ambient light sensor 1480L, a bone conduction sensor 1480M, and the like.

[0110] It is understood that the structures illustrated in the embodiments of the present invention do not constitute specific limitations on the electronic device. In other embodiments of the present application, the electronic device 1400 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0111] The processor 1410 may include one or more processing units, for example: the processor 1410 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices or integrated into one or more processors. For example, the processor 1410 can execute the multi-window display processing method mentioned in the embodiment of the present application.

[0112] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0113] Processor 1410 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in processor 1410 is a cache memory. This memory can store instructions or data that have just been used or are recycled by processor 1410. If processor 1410 needs to use the instruction or data again, it can be directly called from the memory. This avoids repeated accesses, reduces the waiting time of processor 1410, and thus improves system efficiency. For example, the memory can store the multi-window display processing method mentioned in the embodiments of the present application.

[0114] Electronic device 1400 implements display functionality through a GPU, display screen 1494, and an application processor. The GPU is a microprocessor for image processing that connects display screen 1494 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 1410 may include one or more GPUs that execute program instructions to generate or modify display information.

[0115] Display screen 1494 is used to display images, videos, and the like. Display screen 1494 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, the electronic device can include one or N display screens 194, where N is a positive integer greater than one.

[0116] The above describes the hardware structure that an electronic device may have. It is understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, separate certain components, or arrange the components differently. The components shown in the figure can be implemented in hardware, software, or a combination of software and hardware.

[0117] In the accompanying drawings, some structural or method features may be shown in a particular arrangement and / or order. However, it should be understood that such a particular arrangement and / or order may not be required. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of a structural or method feature in a particular figure does not imply that such feature is required in all embodiments, and in some embodiments, such features may not be included or may be combined with other features.

[0118] The various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0119] Program code can be applied to input instructions to perform the functions described herein and generate output information. The output information can be applied to one or more output devices in a known manner. For purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit, or a microprocessor.

[0120] Program code can be implemented with a high-level programming language or an object-oriented programming language to communicate with the processing system. Where necessary, program code can also be implemented in assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0121] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed over a network or through other computer-readable media. Therefore, a machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), including but not limited to floppy disks, optical disks, optical discs, read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Therefore, a machine-readable medium includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

[0122] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems raised by this application. In addition, in order to highlight the innovative part of this application, the above-mentioned device embodiments of this application do not introduce units / modules that are not closely related to solving the technical problems raised by this application. This does not mean that other units / modules do not exist in the above-mentioned device embodiments.

[0123] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "including a" does not exclude the presence of other identical elements in the process, method, article or device that includes the element.

[0124] While the present application has been shown and described with reference to certain embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the present application.

Claims

1. A multi-window display processing method, used in an electronic device, characterized in that: include: Obtain a set of windows to be drawn corresponding to the first display interface; The set of windows to be drawn includes a first window to be drawn and a second window to be drawn which partially overlap; Acquire a first area of ​​a first window to be drawn in the set of windows to be drawn, wherein at least a portion of an area of ​​the second window to be drawn that overlaps with the first area is a second area; Acquire a first drawing instruction for the second area, where the first drawing instruction is used to draw a first solid color frame in the second area; When drawing the first display interface, skip the second drawing instruction and execute the first drawing instruction; the second drawing instruction includes a native drawing instruction of the second area of ​​the second window to be drawn.

2. The method according to claim 1, characterized in that The area of ​​the second region is smaller than or equal to the area of ​​a region in the second window to be drawn that overlaps with the first region.

3. The method according to claim 1, characterized in that Also includes: reporting the second area to a first drawing thread of a first application corresponding to the second window to be drawn, The first drawing thread of the first application generates the first drawing instruction based on the second area.

4. The method according to claim 1, characterized in that: The skipping the second drawing instruction includes deleting the second drawing instruction, discarding the second drawing instruction, or not executing the second drawing instruction.

5. The method according to claim 1, characterized in that The native drawing instruction is used to draw the texture and / or display special effects of the second area of ​​the second window to be drawn.

6. The method according to claim 1, characterized in that The first window to be drawn and the second window to be drawn belong to the same application, or The first window to be drawn and the second window to be drawn belong to different applications.

7. The method according to claim 1, characterized in that The drawing mode of the first drawing instruction is an overlay mode.

8. The method according to claim 1, characterized in that Also includes: Acquire a third area of ​​a third window to be drawn in the set of windows to be drawn, wherein at least a portion of the area of ​​the third window to be drawn that overlaps with the second area is the third area; Acquire a third drawing instruction for the third area, where the third drawing instruction is used to draw a second solid color frame in the third area; When drawing the first display interface, the fourth drawing instruction is skipped and the third drawing instruction is executed; the fourth drawing instruction includes a native drawing instruction of the third area of ​​the third window to be drawn.

9. A multi-window display processing method, used in electronic equipment, characterized in that: include: Obtain a set of windows to be drawn corresponding to the first display interface; The set of windows to be drawn includes a first window to be drawn with rounded corners and a second window to be drawn with rounded corners that partially overlap; Acquire a first part of a first rounded-corner window to be drawn in the set of windows to be drawn, wherein at least a partial area of ​​the first part of the first rounded-corner window to be drawn overlaps with a first area of ​​the second rounded-corner window to be drawn; Obtaining a first drawing instruction for a first part of the first window with rounded corners to be drawn, where the first drawing instruction includes a native drawing instruction for the first part; When drawing the first display interface, skip the second drawing instruction and execute the first drawing instruction; the second drawing instruction includes the native drawing instruction of the first area of ​​the second rounded-corner window to be drawn.

10. The method according to claim 9, characterized in that The first part of the first rounded-corner window to be drawn is a part corresponding to the inscribed rectangle in the first rounded-corner window to be drawn.

11. The method according to claim 9, characterized in that The skipping the second drawing instruction includes deleting the second drawing instruction, discarding the second drawing instruction, or not executing the second drawing instruction.

12. The method according to claim 9, characterized in that The native drawing instruction is used to draw the texture and / or display special effects of the first area of ​​the second rounded-corner window to be drawn.

13. The method according to claim 9, characterized in that The drawing mode of the first drawing instruction is an overlay mode.

14. The method according to claim 9, characterized in that The first window with rounded corners to be drawn and the second window with rounded corners to be drawn belong to the same application, or The first window with rounded corners to be drawn and the second window with rounded corners to be drawn belong to different applications.

15. The method according to claim 9, characterized in that Also includes: Acquire a second area of ​​a third window with rounded corners to be drawn in the window set to be drawn, wherein a partial area of ​​the third window with rounded corners to be drawn that overlaps with the first area is the second area; Acquire a third drawing instruction for the second area, where the third drawing instruction includes a native drawing instruction for the second area; When drawing the first display interface, the third drawing instruction is skipped and the first drawing instruction is executed.

16. An electronic device, characterized in that: include: A memory, used to store instructions executed by one or more processors of the electronic device, and a processor, which is one of the one or more processors of the electronic device, used to execute the multi-window display processing method described in any one of claims 1-15.

17. A readable storage medium, characterized in that: The readable storage medium stores instructions, and when the instructions are executed on an electronic device, the electronic device executes the multi-window display processing method according to any one of claims 1 to 15.

Images