Multi-window display processing method, electronic equipment and storage medium

Abstract

A multi-window display processing method, an electronic device and a storage medium, in the method, for a plurality of windows synchronously displayed on a display interface of a large-screen electronic device, a part or all of shielded areas in the part or all of the shielded windows can be obtained, so that when the shielded windows are drawn, the shielded areas of the plurality of windows can be drawn. The shielded area can be simply drawn in the display interface, and the shielded area can be complexly drawn. Therefore, repeated and complex drawing of the pixels corresponding to the shielded area can be reduced, and the power consumption of the electronic equipment can be reduced based on the characteristic that the power consumption of drawing the pure color frame is smaller than that of drawing the texture.

Description

Technical Field

[0001] The present application relates to the field of window display technology, and in particular to a multi-window display processing method, an electronic device and a storage medium. Background Art

[0002] Currently, displaying multiple rounded-corner windows (i.e., windows with four rounded corners) on a display interface is gradually becoming a rigid demand for large-screen electronic devices (such as tablets, foldable dual-screen mobile phones, etc.).

[0003] In some multi-window display processing methods, in the display interface, the layers corresponding to the multiple rounded-corner windows are first drawn, and then the multiple layers are synthesized by a hardware synthesizer. However, due to the large number of rounded-corner windows in the display interface, the synthesis pressure of the hardware synthesizer is relatively large. Therefore, in other multi-window display processing methods, a graphics processor is used to synthesize the multiple layers.

[0004] However, when multiple rounded-corner windows are displayed on the display interface, part or all of the areas of some windows may be obstructed by part or all of the areas of other windows. Therefore, when drawing the layers corresponding to the multiple rounded-corner windows, it is necessary to perform complex drawing on the obstructed area of ​​the obstructed window on some pixels of the display interface, and to perform complex drawing on the obstructed area of ​​the blocking window on these some pixels. This will cause repeated complex drawing of some pixels, that is, redundant drawing of multiple overlapping graphics elements, which will increase the power consumption of the electronic device. Summary of the invention

[0005] In order to solve the problem that repeated and complex drawing of some pixels in a display interface will increase the power consumption of an electronic device, the present application provides a multi-window display processing method, an electronic device and a storage medium.

[0006] In a first aspect, the present application provides a multi-window display processing method for an electronic device, which may include: obtaining a set of windows to be drawn corresponding to a first display interface; the set of windows to be drawn includes a first window to be drawn and a second window to be drawn that partially overlap; obtaining a first area of ​​the first window to be drawn in the set of windows to be drawn, wherein at least a portion of the area overlapping the first area in the second window to be drawn is the second area; obtaining a first drawing instruction for the second area, the first drawing instruction being used to draw a first solid color box in the second area; when drawing the first display interface, skipping the second drawing instruction, and executing the first drawing instruction; the second drawing instruction includes a native drawing instruction for the second area of ​​the second window to be drawn.

[0007] Based on the above scheme, by skipping the native drawing instructions of the occluded area and executing the solid color frame drawing instructions of the occluded area, that is, simply drawing the occluded window, the repeated and complex drawing of pixels corresponding to the occluded area can be reduced, and based on the characteristic that drawing a solid color frame consumes less power than drawing a texture, the power consumption of the electronic device can be reduced.

[0008] It can be understood that the window to be drawn in the set of windows to be drawn can be a rounded corner window, and the first window to be drawn can be located above the second window to be drawn, that is, the first window to be drawn can be a blocking window, and the second window to be drawn can be a blocked window, the first area can be a blocking area, and the second area can be a blocked area.

[0009] In some optional examples, the first solid color frame should not be understood as a hollow frame, but should be understood as an entire area, and executing the first drawing instruction may be to draw the first area with the same RGB value.

[0010] In some optional instances, the technical solution can be applied to window drawing within a single frame, or to window drawing of adjacent multiple frames. Exemplarily, in a scenario where the stacking relationship between windows has not changed, the steps before obtaining the first drawing instruction for the second area can be implemented within the time of the first frame, and obtaining the first drawing instruction for the second area and the subsequent steps can be implemented within the time of the second frame. The time of the first frame and the second frame, for example 8.33ms or 16.67ms, can be used as an implementation cycle of the technical solution. Among them, the first frame draws and displays the corresponding interface, but does not perform operations such as drawing a solid color box. Alternatively, an implementation cycle may also include a third frame, and the third frame may be running the above-mentioned technical solution on the basis of the second frame.

[0011] In some optional examples, after it is detected that the stacking relationship between windows has changed, for example, the overlapping area between the first window to be drawn and the second window to be drawn has changed, the above technical solution is executed.

[0012] In some optional instances, the central processing unit may first send the second drawing instruction to the composite display module and then send the first drawing instruction. When the composite display module draws the first display interface, the second drawing instruction may be skipped and the first drawing instruction may be executed.

[0013] In some optional implementations of the first aspect, an area of ​​the second region is smaller than or equal to an area of ​​a region in the second window to be drawn that overlaps with the first region.

[0014] In some optional implementations of the first aspect, the multi-window display processing method further includes reporting the second area to a first drawing thread of a first application corresponding to a second window to be drawn, and the first drawing thread of the first application generates a first drawing instruction based on the second area.

[0015] In some optional examples, the synthesis display module can perform multi-rectangular overlap judgment on the windows to be drawn in the set of windows to be drawn, output the occlusion area and / or the occluded area of ​​each window to be drawn, and report it to the main thread in the application corresponding to each window to be drawn, that is, report it to the central processing unit in the form of occlusion tuple information. Then, the drawing thread of the application corresponding to each window to be drawn in the central processing unit can generate a drawing instruction, such as a first drawing instruction, based on the occlusion tuple information of each window to be drawn.

[0016] In some optional instances, a window to be drawn may have multiple occlusion areas, that is, part or all of the area of ​​a window to be drawn may occlude part or all of the area of ​​each of multiple windows to be drawn; a window to be drawn may have multiple occluded areas, that is, part or all of the area of ​​a window to be drawn may be occluded by part or all of the area of ​​each of multiple windows to be drawn.

[0017] In some specific implementations, for window 0 to be drawn corresponding to application 0, window 1 to be drawn corresponding to application 1, window 2 to be drawn corresponding to application 2, etc., the occlusion tuple information corresponding to the window set to be drawn can be expressed as {<app0,rect0> ,<app1,rect1> ,<app2,rect2> , ...}, where the occlusion tuple information of each window to be drawn can be expressed as<app i,rect i(x,y,w,h)> , where x, y can be expressed as the coordinates of the point corresponding to the upper left corner of the rectangle corresponding to the occluding area or the occluded area, w can be expressed as the width of the rectangle corresponding to the occluding area or the occluded area, and h can be expressed as the height of the rectangle corresponding to the occluding area or the occluded area.

[0018] In some optional implementations of the first aspect, skipping the second drawing instruction includes deleting the second drawing instruction, discarding the second drawing instruction, or not executing the second drawing instruction.

[0019] In some optional implementations of the first aspect, the native drawing instruction is used to draw the texture and / or display special effects of the second area of ​​the second window to be drawn.

[0020] It can be understood that the display special effects may include frosted glass special effects, transparent special effects, anti-aliasing special effects, and other special effects.

[0021] In some optional implementations of the first aspect, the first window to be drawn and the second window to be drawn belong to the same application, or the first window to be drawn and the second window to be drawn belong to different applications.

[0022] In some optional examples of the first aspect, the drawing mode of the first drawing instruction is an overlay mode.

[0023] In some optional implementations of the first aspect, the multi-window display processing method also includes: obtaining a third area of ​​a third window to be drawn in the set of windows to be drawn, wherein at least a portion of the area in the third window to be drawn that overlaps with the second area is the third area; obtaining a third drawing instruction for the third area, the third drawing instruction being used to draw a second solid color frame in the third area; when drawing the first display interface, skipping the fourth drawing instruction, and executing the third drawing instruction; the fourth drawing instruction includes a native drawing instruction for the third area of ​​the third window to be drawn.

[0024] Based on the above scheme, by skipping the native drawing instructions of the occluded area and executing the solid color frame drawing instructions of the occluded area, that is, simply drawing the occluded window, the repeated and complex drawing of pixels corresponding to the occluded area can be reduced, and based on the characteristic that drawing a solid color frame consumes less power than drawing a texture, the power consumption of the electronic device can be reduced.

[0025] It can be understood that the third window to be drawn can be located below the second window to be drawn and below the first window to be drawn, that is, the first window to be drawn can be a blocking window, the second window to be drawn can be a blocked window, and the third window to be drawn can also be a blocked window. The first area can be a blocking area, the second area can be a blocked area, and the third area can also be an blocked area.

[0026] In some optional examples, the second solid color frame is not a hollow frame but an integral area, and executing the fifth drawing instruction may be to draw the third area with the same RGB value.

[0027] In some optional instances, the central processing unit may first send the fourth drawing instruction to the composite display module and then send the third drawing instruction. When the composite display module draws the first display interface, the fourth drawing instruction may be skipped and the third drawing instruction may be executed.

[0028] In a second aspect, an embodiment of the present application provides a multi-window display processing method for an electronic device, comprising: obtaining a set of windows to be drawn corresponding to a first display interface; the set of windows to be drawn includes a first rounded-corner window to be drawn and a second rounded-corner window to be drawn that partially overlap; obtaining a first part of the first rounded-corner window to be drawn in the set of windows to be drawn, wherein at least a partial area of ​​the first part of the first rounded-corner window to be drawn overlaps with a first area of ​​the second rounded-corner window to be drawn; obtaining a first drawing instruction for the first part of the first rounded-corner window to be drawn, the first drawing instruction including a native drawing instruction for the first part; when drawing the first display interface, skipping the second drawing instruction, and executing the first drawing instruction; the second drawing instruction includes a native drawing instruction for the first area of ​​the second rounded-corner window to be drawn.

[0029] In some optional examples of the second aspect, the first portion of the first rounded-corner window to be drawn is a portion corresponding to an inscribed rectangle in the first rounded-corner window to be drawn.

[0030] In some optional examples of the second aspect, skipping the second drawing instruction includes deleting the second drawing instruction, discarding the second drawing instruction, or not executing the second drawing instruction.

[0031] In some optional examples of the second aspect, the native drawing instruction is used to draw the texture and / or display special effects of the first area of ​​the second rounded-corner window to be drawn.

[0032] It can be understood that the display special effects may include frosted glass special effects, transparent special effects, anti-aliasing special effects, and other special effects.

[0033] In some optional examples of the first aspect, the drawing mode of the first drawing instruction is an overlay mode.

[0034] In some optional examples of the second aspect, the first window with rounded corners to be drawn and the second window with rounded corners to be drawn belong to the same application, or the first window with rounded corners to be drawn and the second window with rounded corners to be drawn belong to different applications.

[0035] In some optional instances of the second aspect, the multi-window display processing method further includes: obtaining a second area of ​​a third rounded-corner window to be drawn in the set of windows to be drawn, wherein a partial area of ​​the third rounded-corner window to be drawn that overlaps with the first area is the second area; obtaining a third drawing instruction for the second area, the third drawing instruction including a native drawing instruction for the second area; when drawing the first display interface, skipping the third drawing instruction, and executing the first drawing instruction.

[0036] In a third aspect, the present application provides an electronic device, comprising: a memory for storing instructions executed by one or more processors of the electronic device, and a processor, which is one of the one or more processors of the electronic device, for executing the multi-window display processing method mentioned in the present application.

[0037] In a fourth aspect, the present application provides a readable storage medium, on which instructions are stored. When the instructions are executed on an electronic device, the electronic device executes the multi-window display processing method mentioned in the present application.

[0038] In a fifth aspect, an embodiment of the present application provides a computer program product, including: a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium containing a computer program code for executing a multi-window display processing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 According to some examples of the present application, a schematic diagram of displaying multiple rounded-corner windows on a display interface is shown;

[0040] Figure 2 According to some examples of the present application, a schematic diagram showing a comparison of usage frequencies of usage functions of multiple rounded-corner windows in a tablet is shown;

[0041] Figure 3 According to some examples of the present application, a schematic diagram of overlapping multiple rounded-corner windows in a display interface is shown;

[0042] Figure 4 According to some examples of the present application, a schematic diagram of overlapping multiple rounded-corner windows in a display interface is shown;

[0043] Figure 5 According to some examples of the present application, a schematic diagram of overlapping multiple rounded-corner windows in a display interface is shown;

[0044] Figure 6 According to some examples of the present application, a schematic diagram showing a plurality of rounded corner windows superimposed in a display interface is shown.

[0045] Figure 7 According to some examples of the present application, a flowchart of a multi-window display processing method is shown;

[0046] Figure 8 According to some examples of the present application, a flowchart of a multi-window display processing method is shown;

[0047] Fig. 9 According to some examples of the present application, a schematic diagram of occlusion tuple information of an occluded area of ​​a to-be-drawn window 5 corresponding to App5 being occluded by a to-be-drawn window 0 corresponding to App0 is shown;

[0048] Fig.10 According to some examples of the present application, a schematic diagram of drawing a solid color frame on the blocked area of ​​the to-be-drawn window 5 corresponding to App5 is shown;

[0049] Fig.11 According to some examples of the present application, a schematic diagram of drawing a solid color frame on a window to be drawn in which a part or all of the area obscures a part or all of the area of ​​other windows to be drawn is shown;

[0050] Fig.12 According to some examples of the present application, a schematic diagram is shown of cutting out an inscribed rectangle of a window to be drawn whose partial or entire area obscures a partial or entire area of ​​another window to be drawn;

[0051] Fig.13 According to some examples of the present application, a schematic diagram of dividing a layer corresponding to a shielding window into five parts is shown;

[0052] Fig.14 According to some examples of the present application, a hardware structure of an electronic device is shown. DETAILED DESCRIPTION

[0053] The illustrative embodiments of the present application include, but are not limited to, a multi-window display processing method, an electronic device, and a storage medium.

[0054] It can be understood that the multi-window display processing method mentioned in the embodiment of the present application can be used for any large-screen electronic device that can be implemented, such as a tablet, a foldable dual-screen mobile phone, a laptop computer, a desktop computer or a personal digital assistant.

[0055] It can be understood that the multi-window display processing method mentioned in the embodiment of the present application can be applicable to scenarios such as synchronous display of multiple windows of multiple applications, synchronous display of multiple windows of one application, etc. In some optional instances, the multi-window display method mentioned in the embodiment of the present application can be applied to the scenario of synchronous display of multiple rounded corner windows.

[0056] In order to clearly illustrate the scheme mentioned in the embodiments of the present application, the terms involved in the embodiments of the present application are first explained.

[0057] Operating system: software that manages the hardware of electronic devices and implements functions such as resource allocation. For example, an operating system may include wait.

[0058] Interface drawing service: A system service in the operating system, which is specifically used to synthesize layers and transfer and send layers to the display subsystem. For example, in the Android operating system, the interface drawing service can be surfaceflinger, referred to as SF.

[0059] Display subsystem (DSS): The subsystem in the operating system that is responsible for managing data in the memory and sending it to the monitor for display.

[0060] Hardware composer (HWC): A device that composites the layers received from the interface drawing service.

[0061] Vsyn: A signal generated by the display hardware of a terminal device. This signal is used to notify upper-layer software to perform layer drawing and synthesis based on the signal.

[0062] Forward pixel kill (FPK): A technique proposed by Mali GPU under the Arm architecture to reduce overdrawing. It can perform depth testing on pixels in opaque primitives and remove overdrawn pixels based on depth.

[0063] The following is an introduction to displaying multiple rounded-corner windows on a display interface.

[0064] like Figure 1 As shown, window 1 corresponding to the short video application, window 2 corresponding to the chat application, and window 3 corresponding to the conference application can be displayed on the display interface of the tablet, which can meet the needs of various office scenarios, online learning scenarios and entertainment scenarios. Figure 2 A schematic diagram showing a comparison of the usage frequency of a function of displaying multiple rounded corner windows between a tablet and a straight-screen mobile phone is shown. Figure 2 As shown, the frequency of use of the parallel vision function in tablets is much higher than that in straight-screen mobile phones.

[0065] The following is an introduction to the multi-window display processing methods mentioned in some embodiments.

[0066] In some multi-window display processing methods, in the display interface, the layers corresponding to the multiple rounded-corner windows are first drawn, and then the multiple layers are synthesized by a hardware synthesizer. However, due to the large number of rounded-corner windows in the display interface, the synthesis pressure of the hardware synthesizer is relatively large. Therefore, in other multi-window display processing methods, a graphics processor is used to synthesize the multiple layers.

[0067] However, if Figure 3 As shown, when multiple rounded-corner windows are displayed on the display interface, some windows may be partially or completely blocked by other windows. Therefore, when drawing the layers corresponding to the multiple rounded-corner windows, it is necessary to perform complex drawing on the blocked areas of the blocked windows on some pixels of the display interface, and perform complex drawing on the blocked areas of the blocking windows on the some pixels. For example, Figure 3In the drawing process of the rounded window A and the rounded window B, complex drawing needs to be performed on the blocked area a of the blocked window A, and complex drawing needs to be performed on the blocked area b of the blocking window B, and the blocked area a and the blocked area b overlap and are located in the same part of pixels. Repeated complex drawing in the same part of pixels (such as drawing the texture corresponding to the blocked area a and drawing the texture corresponding to the blocked area b in the same part of pixels) will cause repeated complex drawing of some pixels, that is, there is redundant drawing when multiple graphics elements overlap, which will increase the power consumption of the electronic device.

[0068] In order to solve the above problems, the embodiment of the present application provides another multi-window display processing method. For multiple rounded-corner windows synchronously displayed on the display interface of a large-screen electronic device (such as a tablet, a foldable dual-screen mobile phone, etc.), part or all of the blocked areas in the partially or completely blocked rounded-corner windows can be obtained, so that when drawing the blocked rounded-corner windows, the blocked areas can be simply drawn in the display interface, and the blocked areas can be complexly drawn. Figure 3 In the process of drawing the rounded-corner window A and the rounded-corner window B, the blocked area a of the blocked rounded-corner window A is simply drawn, such as drawing the blocked area a as a solid color frame, and the blocked area b of the blocked rounded-corner window B is complexly drawn, such as drawing the texture of the blocked area b and / or displaying special effects. In this way, the repeated complex drawing of pixels corresponding to the blocked area can be reduced, and based on the characteristic that drawing a solid color frame consumes less power than drawing a texture, the power consumption of the electronic device can be reduced.

[0069] For example, for multiple rounded-corner windows of multiple applications displayed synchronously on a display interface of a large-screen electronic device, such as Figure 4 In the example, the rounded corner window 0 corresponds to application 0, the rounded corner window 1 corresponds to application 1, the rounded corner window 2 corresponds to application 2, the rounded corner window 3 corresponds to application 3, the rounded corner window 4 corresponds to application 4, and the rounded corner window 5 corresponds to application 5. The obscured area A of the rounded corner window 5 obscured by the rounded corner window 0 can be obtained. When drawing the rounded corner window 5, the obscured area A can be drawn as a solid color box.

[0070] In some specific implementations, a simple drawing instruction corresponding to the obscured area can be generated based on the obscured area of ​​the obscured rounded window. When drawing the display interface, the complex drawing instruction corresponding to the obscured area is skipped and the simple drawing instruction is executed, that is, the central processing unit can generate a pure color frame drawing instruction corresponding to the obscured area A, that is, a simple drawing instruction, and first send the complex drawing instruction corresponding to the obscured area A to the synthesis sending display module, and then send the simple drawing instruction corresponding to the obscured area A to the synthesis sending display module, so that the synthesis sending display module skips the complex drawing instruction corresponding to the obscured area A and executes the simple drawing instruction corresponding to the obscured area A.

[0071] It can be understood that for multiple rounded-corner windows synchronously displayed on the display interface of a large-screen electronic device (such as a tablet, a foldable phone, etc.), it is possible to obtain a portion or all of the occlusion area in the rounded-corner window that is partially or completely occluded, so that when synthesizing the layers corresponding to the multiple rounded-corner windows, the occlusion area in the display interface can be complexly drawn. In this way, the display effect of the occlusion area can be guaranteed while reducing the repeated complex drawing of the pixels corresponding to the occlusion area.

[0072] For example, for multiple rounded-corner windows of multiple applications displayed synchronously on a display interface of a large-screen electronic device, such as Figure 5 In the example, the rounded corner window 0 corresponds to application 0, the rounded corner window 1 corresponds to application 1, the rounded corner window 2 corresponds to application 2, the rounded corner window 3 corresponds to application 3, the rounded corner window 4 corresponds to application 4, and the rounded corner window 5 corresponds to application 5. The occlusion area B of rounded corner window 1, rounded corner window 2, rounded corner window 3, rounded corner window 4, and rounded corner window 5 that is occluded by rounded corner window 0 can be obtained. When the rounded corner windows 0, 1, 2, 3, 4, and 5 are synthesized, the occlusion area B in the display interface can be complexly drawn.

[0073] It can be understood that for multiple rounded corner windows synchronously displayed on the display interface of a large-screen electronic device (such as a tablet, a foldable dual-screen mobile phone, etc.), it is possible to obtain a part or all of the area in the rounded corner window that is partially or completely blocked, so that when synthesizing the layers corresponding to the multiple rounded corner windows, complex drawing can be performed on the part or all of the area in the blocked rounded corner window in the display interface. In this way, the display effect of the blocked area can be guaranteed while reducing the repeated complex drawing of the pixels corresponding to the blocked area.

[0074] For example, when a plurality of rounded-corner windows are displayed synchronously on a display interface of a large-screen electronic device, such as Figure 6 The rounded corner window 0 corresponding to application 0, the rounded corner window 1 corresponding to application 1, the rounded corner window 2 corresponding to application 2, the rounded corner window 3 corresponding to application 3, the rounded corner window 4 corresponding to application 4 and the rounded corner window 5 corresponding to application 5 can obtain part or all of the area C (such as the occluded area and the non-occluded area) in the rounded corner window 0. When the rounded corner windows 0, 1, 2, 3, 4, and 5 are synthesized, complex drawing can be performed on area C in the display interface.

[0075] In some specific implementations, a simple drawing instruction may be generated based on the occluded area of ​​the occluded rounded corner window, and a complex drawing instruction may be generated based on the occluded rounded corner window. When drawing the display interface, the complex drawing instruction corresponding to the occluded area is skipped, and the simple drawing instruction of the occluded area is executed first, and then the complex instruction of the occluded area is executed. Alternatively, the complex drawing instruction corresponding to the occluded area is skipped, and the complex instruction of the occluded area is executed.

[0076] The following is an introduction to the multi-window display processing method mentioned in the embodiment of the present application. Figure 7 A schematic flow chart of a multi-window display processing method is shown, and the multi-window display processing method can be executed by an electronic device, such as Figure 7 As shown, the multi-window display processing method may include:

[0077] 701: Obtain a set of windows to be drawn, and determine the occluded area and / or the occluded area of ​​each window to be drawn in the set of windows to be drawn.

[0078] It can be understood that the set of windows to be drawn may include multiple windows of multiple applications, or multiple windows of one application. Among them, the window may be a rounded corner window. After obtaining the set of windows to be drawn, the occlusion area identification and occlusion tuple information transmission can be performed on each window to be drawn in the set of windows to be drawn. Among them, the occlusion tuple information may include the coordinates of the point corresponding to the upper left corner of the rectangle corresponding to the occlusion area or the occluded area, the width of the rectangle, and the height of the rectangle.

[0079] In some optional instances, when an electronic device pre-displays multiple windows, the HWC in the electronic device can generate a Vsync signal and upload it to the upper-level software surface flinger. The surface flinger can obtain a set of windows to be drawn, and perform multi-rectangular overlap judgment on the windows to be drawn in the set of windows to be drawn, output the occlusion area and / or occluded area of ​​each window to be drawn, and report it to the main thread in the application corresponding to each window to be drawn in the form of occlusion tuple information along with the Vsync signal, that is, report it to the central processing unit.

[0080] In some optional instances, a window to be drawn may have multiple occlusion areas, that is, part or all of the area of ​​a window to be drawn may occlude part or all of the area of ​​each of multiple windows to be drawn; a window to be drawn may have multiple occluded areas, that is, part or all of the area of ​​a window to be drawn may be occluded by part or all of the area of ​​each of multiple windows to be drawn.

[0081] In some optional examples, the occlusion area and the occluded area of ​​the window to be drawn in the window to be drawn set can be represented in the form of occlusion tuple information. For example, for window 0 to be drawn corresponding to application 0, window 1 to be drawn corresponding to application 1, window 2 to be drawn corresponding to application 2..., the occlusion tuple information corresponding to the window to be drawn set can be represented as {<app0,rect0> ,<app1,rect1> ,<app2,rect2> ,...}, the occlusion tuple information of each window to be drawn can be expressed as<app i,rect i(x,y,w,h)> , where x, y can be expressed as the coordinates of the point corresponding to the upper left corner of the rectangle corresponding to the occluding area or the occluded area, w can be expressed as the width of the rectangle corresponding to the occluding area or the occluded area, and h can be expressed as the height of the rectangle corresponding to the occluding area or the occluded area.

[0082] In some optional instances, since a window to be drawn may have multiple occlusion areas and multiple occluded areas, the occlusion information rect i(x, y, w, h) in the occlusion tuple information of each window to be drawn may also include rect i1(x, y, w, h), rect i2(x, y, w, h), ..., wherein rect i1(x, y, w, h) may represent an occluded area, and rect i2(x, y, w, h) may represent another occluded area. rect i1(x, y, w, h) may also represent an occluded area, and rect i2(x, y, w, h) may represent an occlusion area. The embodiment of the present application does not specifically limit the specific form of the occlusion tuple information.

[0083] 702: Based on the blocking area and / or blocked area of ​​part or all of the windows to be drawn, generate drawing instructions corresponding to each window to be drawn.

[0084] It can be understood that after the CPU obtains the occlusion tuple information corresponding to the set of windows to be drawn, the occlusion culling process can be performed in the application, and before the rendering thread draws, that is, before submitting the drawing instruction, a solid color frame drawing instruction corresponding to the occluded area of ​​each window to be drawn is generated, and the solid color frame drawing instruction corresponding to the occluded area is used to instruct the drawing of a solid color frame. Among them, the solid color frame is not a hollow frame, but an integral area, and the execution of the solid color frame drawing instruction can be to draw the occluded area with the same RGB value.

[0085] In some specific implementations, the main thread in the application corresponding to each window to be drawn in the set of windows to be drawn can record the occlusion tuple information of the window to be drawn after receiving it, and generate a solid color frame drawing instruction corresponding to the occluded area of ​​the window to be drawn based on the occlusion tuple information, and first send the complex drawing instruction of the occluded area (such as the instruction to draw the texture of the occluded area and / or display special effects) to the graphics processing unit (GPU), and then send the solid color frame drawing instruction corresponding to the occluded area to the GPU. Among them, the display special effects can include special effects such as frosted glass special effects, transparent special effects or anti-aliasing special effects.

[0086] In some optional instances, a complex drawing instruction corresponding to the unobstructed window to be drawn may be generated, so that complex information such as texture and / or display effects of the unobstructed window to be drawn may be drawn on the display interface based on the complex drawing instruction.

[0087] In other optional instances, a solid color frame drawing instruction corresponding to the occluded area can be generated based on the occluded area of ​​the window to be drawn, so that a solid color frame can be first drawn on the display interface based on the solid color frame drawing instruction corresponding to the occluded area, and then the occluding window can be drawn to cover the solid color frame.

[0088] In some other optional examples, based on the occluded area of ​​the window to be drawn, a drawing instruction for drawing the maximum inscribed rectangle of the occluded area using the overlay mode and a drawing instruction for drawing the rounded corners of the occluded area using the transparent mode may be generated.

[0089] In some optional instances, for window 1 to be drawn corresponding to application 1, window 2 to be drawn corresponding to application 2, window 3 to be drawn corresponding to application 3, window 4 to be drawn corresponding to application 4, window 5 to be drawn corresponding to application 5, and window 0 to be drawn corresponding to application 0 that are stacked from bottom to top, based on each obscured area of ​​windows 1, 2, 3, 4, 5 to be drawn, a solid color frame drawing instruction corresponding to each obscured area can be generated, and based on each obscured area of ​​windows 1, 3, 5 to be drawn, a solid color frame drawing instruction corresponding to each obscured area can be generated.

[0090] 703: Based on the drawing instruction corresponding to the window to be drawn, perform simple drawing on the blocked area of ​​the window to be drawn, and perform complex drawing on the blocked area of ​​the window to be drawn.

[0091] It can be understood that after receiving the drawing instructions corresponding to each window to be drawn, the graphics processing unit (GPU) can execute the solid color frame drawing instructions corresponding to the obscured area when drawing a window to be drawn whose partial or complete area is obscured by partial or complete areas of other windows to be drawn, and does not execute the complex drawing instructions corresponding to the obscured area, that is, simply draw the obscured area and draw the obscured area as a solid color frame.

[0092] It can be understood that when synthesizing the layers corresponding to the windows to be drawn in the set of windows to be drawn, complex drawing can be performed on the windows to be drawn whose partial or complete areas occlude part or all of the areas of other windows to be drawn according to the stacking order of the windows to be drawn in the set of windows to be drawn, that is, drawing the texture and / or display effects of part or all of the occluded areas, partial areas (including occluded areas and unoccluded areas) or all areas (including occluded areas and unoccluded areas) of part or all of the windows to be drawn (that is, the occluded windows) with occluded areas.

[0093] The following introduces the multi-window display processing method mentioned above in combination with the architecture of the electronic device. Figure 8 A block diagram of an electronic device is shown. Figure 8 As shown, the electronic device may include a hardware hybrid renderer 810 and an interface drawing service 820, and the interface drawing service 820 may include a signal receiving module 821, an occlusion determination module 822, a solid color culling occlusion module 823, a rounded corner splitting culling module 824 and a synthesis display module 825.

[0094] Among them, the multi-window display processing method may include:

[0095] When the electronic device pre-displays multiple windows, the hardware hybrid renderer 810 in the electronic device can generate a Vsync signal and upload it to the signal receiving module 821 in the upper-layer software interface drawing service 820 .

[0096] The interface drawing service 820 can obtain a set of windows to be drawn when the signal receiving module 821 receives the Vsync signal, for example, obtains window 1 to be drawn corresponding to App1, window 2 to be drawn corresponding to App2, window 3 to be drawn corresponding to App3... The signal receiving module 821 in the interface drawing service 820 can also send a trigger signal to the occlusion judgment module 822 in the interface drawing service 820 after receiving the Vsync signal uploaded by the hardware hybrid renderer 810.

[0097] After receiving the trigger signal sent by the occlusion judgment module 822, the occlusion judgment module 822 in the interface drawing service 820 can perform multi-rectangular overlap judgment on the windows to be drawn in the set of windows to be drawn, output the occlusion area and / or the occluded area of ​​each window to be drawn, and output the occlusion tuple information {<app1,rect1> ,<app2,rect2> , ...} in the form of Vsync signal to the main thread of the application corresponding to each window to be drawn. Fig. 9 The occlusion tuple information of the occluded area of ​​the to-be-drawn window 5 corresponding to App5 being occluded by the to-be-drawn window 0 corresponding to App0<app5-pid,x,y,w,h> Report to the main thread in App5.

[0098] After receiving the occlusion tuple information corresponding to the window to be drawn, the main thread of the application can record the occlusion tuple information, and before the rendering thread draws the controls / graphics elements of the window to be drawn, that is, before submitting the drawing instruction of the window to be drawn, generate a solid color frame drawing instruction to draw the occluded area of ​​the window to be drawn as a solid color frame. For example, Fig.10 As shown, a pure color frame drawing instruction for drawing the blocked area of ​​the to-be-drawn window 5 corresponding to App5 as a pure color frame can be generated. Then, the complex drawing instruction of the to-be-drawn window can be submitted to the interface drawing service 820 first, and then the pure color frame drawing instruction of the blocked area (i.e., the instruction for drawing the blocked area as a pure color frame) can be submitted to the interface drawing service 820.

[0099] The occlusion identification module 822 in the interface drawing service 820 can identify the occlusion area of ​​each window to be drawn in the set of windows to be drawn, and then pass the occlusion area of ​​the window to be drawn to the solid color occlusion culling module 823 in the interface drawing service 820, and pass the occlusion culling drawing instruction corresponding to the occlusion area of ​​the window to be drawn to the solid color occlusion culling module 823 in the interface drawing service 820; or, pass the occlusion culling drawing instruction corresponding to the occlusion area of ​​the window to be drawn to the rounded corner splitting culling module 824. When drawing a window to be drawn whose part or all of the area is occluded by part or all of the area of ​​other windows to be drawn, the solid color occlusion culling module 823 can draw the occluded area as a solid color frame. When drawing a window to be drawn whose part or all of the area occludes part or all of the area of ​​other windows to be drawn, the solid color occlusion culling module 823 can draw a solid color frame in the area corresponding to the occlusion area. When drawing a window to be drawn whose partial or full area obscures the partial or full area of ​​other windows to be drawn, the rounded corner splitting and culling module 824 can divide the obstructed area into multiple areas based on the maximum inscribed rectangle of the obstructed area, draw the texture corresponding to the maximum inscribed rectangle based on the texture drawing mode, and draw the areas corresponding to the four rounded corners in the obstructed area based on the transparent mode. Then, the layers corresponding to each window to be drawn in the set of windows to be drawn can be transferred to the synthesis display module 825 in the interface drawing service 820.

[0100] The synthesis and display module 825 in the interface drawing service 820 can synthesize each window to be drawn according to the order in which the layers corresponding to each window to be drawn are stacked, and display the synthesized layers to the display interface.

[0101] like Fig.11 As shown, in some optional instances, a solid color frame can be drawn on a window to be drawn that partially or completely blocks part or all of the area of ​​other windows to be drawn, and then the layer corresponding to the blocking window is drawn to cover the solid color frame, that is, a solid color frame is first drawn on the area corresponding to the blocking area on the lower layer where the layer corresponding to the blocking window is to be set. Then the layer corresponding to the blocking window is set on the solid color frame corresponding to the blocking area on the lower layer. Specifically, the layer corresponding to the blocking window can be drawn on the layer corresponding to app5 first. Fig.11 The solid color box shown as A in the middle, and then cover the solid color box A with the layer corresponding to app0.

[0102] like Fig.12As shown, in other optional instances, an inscribed rectangle can be cropped out of a window to be drawn whose partial or complete area obscures part or complete area of ​​other windows to be drawn, for example, a maximum inscribed rectangle is cropped out, and the drawing mode of the inscribed rectangle is set to an overlay mode (BlendMode=kSrc), so that for the maximum inscribed rectangle, when drawing the layer corresponding to the obstructing window, the portion corresponding to the maximum inscribed rectangle is textured, and the portion corresponding to the rounded corner is transparently drawn.

[0103] For example, Fig.13 As shown, the layer corresponding to the occluding window can be divided into a first part and a second part, wherein the first part can be a part corresponding to the maximum inscribed rectangle of the occluding window, such as Fig.13 The second part may be the remaining part of the occlusion window except the part corresponding to the largest inscribed rectangle, such as Fig.13 The non-shaded areas shown in ①, ②, ③, and ④, among which, the four parts ① to ④ are drawn according to the original mode (because the four rounded corners need to be drawn as transparent parts and FPK is not enabled), and the drawing mode of part ⑤ can be set to overlay mode (BlendMode=kSrc), so that FPK can be enabled and texture drawing can be performed on the pixels of part ⑤, that is, the display information of the pixels of part ⑤ is set to the texture data corresponding to area ⑤.

[0104] In some optional instances, you can use dumpsys SurfaceFlinger to view the synthesis method of the layers corresponding to the windows to be drawn in the window set to be drawn. When there are many layers, most of the layers will use the Client synthesis method, that is, the GPU is used for synthesis, and the layers corresponding to the top window to be drawn are also synthesized by the GPU. Therefore, you can use the Client synthesis method for the layers corresponding to the top window to be drawn. Fig.13 The processing is performed in the manner shown, so that the repeated and complex drawing of pixels corresponding to the occluded area can be reduced, and the power consumption of the electronic device can be reduced.

[0105] In some optional examples, the above multi-window display method can be applied to a scenario where windows corresponding to multiple applications are displayed simultaneously.

[0106] For example, in a scenario where multiple windows corresponding to a chat application and a short video application are displayed at the same time, after the user opens the chat application, the window corresponding to the chat application can be displayed on the display interface of the electronic device. After the user opens the short video application, the window corresponding to the short video application can be displayed on the display interface of the electronic device, wherein the window corresponding to the short video application can partially cover the window corresponding to the chat application.

[0107] The following is an introduction to the hardware structure of electronic equipment. Fig.14 As shown, the electronic device 1400 may include a processor 1410, an external memory interface 1420, an internal memory 1421, a universal serial bus (USB) interface 1430, a charging management module 1440, a power management module 1441, a battery 1442, an antenna 1, an antenna 2, a mobile communication module 1450, a wireless communication module 1460, an audio module 1470, a speaker 1470A, a receiver 1470B, a microphone 1470C, an earphone interface 1470D, a sensor module 1480, a button 1490, a motor 1491, an indicator 1492, a camera 1493, a display screen 1494, and a subscriber identification module (SIM) card interface 1495, etc. The sensor module 1480 may include a pressure sensor 1480A, a gyroscope sensor 1480B, an air pressure sensor 1480C, a magnetic sensor 1480D, an acceleration sensor 1480E, a distance sensor 1480F, a proximity light sensor 1480G, a fingerprint sensor 1480H, a temperature sensor 1480J, a touch sensor 1480K, an ambient light sensor 1480L, a bone conduction sensor 1480M, and the like.

[0108] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device 1400 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0109] The processor 1410 may include one or more processing units, for example: the processor 1410 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural-network processing unit (neural-network processing unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors. For example, the processor 1410 can execute the multi-window display processing method mentioned in the embodiment of the present application.

[0110] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0111] A memory may also be provided in the processor 1410 for storing instructions and data. In some embodiments, the memory in the processor 1410 is a cache memory. The memory may store instructions or data that the processor 1410 has just used or circulated. If the processor 1410 needs to use the instruction or data again, it may be directly called from the memory. Repeated access is avoided, the waiting time of the processor 1410 is reduced, and the efficiency of the system is improved. For example, the memory may store the multi-window display processing method mentioned in the embodiment of the present application.

[0112] The electronic device 1400 implements the display function through a GPU, a display screen 1494, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 1494 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 1410 may include one or more GPUs that execute program instructions to generate or change display information.

[0113] The display screen 1494 is used to display images, videos, etc. The display screen 1494 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0114] The above describes the hardware structure that the electronic device may have. It is understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0115] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0116] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0117] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit, or a microprocessor.

[0118] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0119] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, floppy disks, optical disks, optical disks, read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Therefore, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a machine (e.g., computer) readable form.

[0120] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0121] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0122] Although the present application has been illustrated and described with reference to certain embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the present application.

Claims

1. A multi-window display processing method, used in an electronic device, characterized in that: include: Obtain a set of windows to be drawn corresponding to the first display interface; The set of windows to be drawn includes a first window to be drawn and a second window to be drawn which partially overlap; Acquire a first area of ​​a first window to be drawn in the set of windows to be drawn, wherein at least a portion of an area of ​​the second window to be drawn that overlaps with the first area is a second area; Acquire a first drawing instruction for the second area, where the first drawing instruction is used to draw a first solid color frame in the second area; When drawing the first display interface, skip the second drawing instruction and execute the first drawing instruction; the second drawing instruction includes a native drawing instruction of the second area of ​​the second window to be drawn.

2. The method according to claim 1, characterized in that The area of ​​the second region is smaller than or equal to the area of ​​a region in the second window to be drawn that overlaps with the first region.

3. The method according to claim 1, characterized in that Also includes: reporting the second area to a first drawing thread of a first application corresponding to the second window to be drawn, The first drawing thread of the first application generates the first drawing instruction based on the second area.

4. The method according to claim 1, characterized in that: The skipping the second drawing instruction includes deleting the second drawing instruction, discarding the second drawing instruction, or not executing the second drawing instruction.

5. The method according to claim 1, characterized in that The native drawing instruction is used to draw the texture and / or display special effects of the second area of ​​the second window to be drawn.

6. The method according to claim 1, characterized in that The first window to be drawn and the second window to be drawn belong to the same application, or The first window to be drawn and the second window to be drawn belong to different applications.

7. The method according to claim 1, characterized in that The drawing mode of the first drawing instruction is an overlay mode.

8. The method according to claim 1, characterized in that Also includes: Acquire a third area of ​​a third window to be drawn in the set of windows to be drawn, wherein at least a portion of the area of ​​the third window to be drawn that overlaps with the second area is the third area; Acquire a third drawing instruction for the third area, where the third drawing instruction is used to draw a second solid color frame in the third area; When drawing the first display interface, the fourth drawing instruction is skipped and the third drawing instruction is executed; the fourth drawing instruction includes a native drawing instruction of the third area of ​​the third window to be drawn.

9. A multi-window display processing method, used in electronic equipment, characterized in that: include: Obtain a set of windows to be drawn corresponding to the first display interface; The set of windows to be drawn includes a first window to be drawn with rounded corners and a second window to be drawn with rounded corners that partially overlap; Acquire a first part of a first rounded-corner window to be drawn in the set of windows to be drawn, wherein at least a partial area of ​​the first part of the first rounded-corner window to be drawn overlaps with a first area of ​​the second rounded-corner window to be drawn; Obtaining a first drawing instruction for a first part of the first window with rounded corners to be drawn, where the first drawing instruction includes a native drawing instruction for the first part; When drawing the first display interface, skip the second drawing instruction and execute the first drawing instruction; the second drawing instruction includes the native drawing instruction of the first area of ​​the second rounded-corner window to be drawn.

10. The method according to claim 9, characterized in that The first part of the first rounded-corner window to be drawn is a part corresponding to the inscribed rectangle in the first rounded-corner window to be drawn.

11. The method according to claim 9, characterized in that The skipping the second drawing instruction includes deleting the second drawing instruction, discarding the second drawing instruction, or not executing the second drawing instruction.

12. The method according to claim 9, characterized in that The native drawing instruction is used to draw the texture and / or display special effects of the first area of ​​the second rounded-corner window to be drawn.

13. The method according to claim 9, characterized in that The drawing mode of the first drawing instruction is an overlay mode.

14. The method according to claim 9, characterized in that The first window with rounded corners to be drawn and the second window with rounded corners to be drawn belong to the same application, or The first window with rounded corners to be drawn and the second window with rounded corners to be drawn belong to different applications.

15. The method according to claim 9, characterized in that Also includes: Acquire a second area of ​​a third window with rounded corners to be drawn in the window set to be drawn, wherein a partial area of ​​the third window with rounded corners to be drawn that overlaps with the first area is the second area; Acquire a third drawing instruction for the second area, where the third drawing instruction includes a native drawing instruction for the second area; When drawing the first display interface, the third drawing instruction is skipped and the first drawing instruction is executed.

16. An electronic device, characterized in that: include: A memory, used to store instructions executed by one or more processors of the electronic device, and a processor, which is one of the one or more processors of the electronic device, used to execute the multi-window display processing method described in any one of claims 1-15.

17. A readable storage medium, characterized in that: The readable medium stores instructions, and when the instructions are executed on an electronic device, the electronic device executes the multi-window display processing method according to any one of claims 1 to 15.

Images