An image processing method and apparatus
By combining hardware compositors and graphics processing units in electronic devices, the compositing process for multi-window displays is optimized, solving the problem of high compositing overhead in existing technologies and achieving higher compositing efficiency and resource utilization.
Patent Information
- Application Number
- CN202510625887.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-14
- Publication Date
- 2026-11-17
AI Technical Summary
In multi-window display scenarios, existing compositing methods result in significant compositing processing overhead during interface updates.
By employing a combination of hardware compositor (HWC) and graphics processing unit (GPU) in electronic devices, the first compositing process is performed on windows with updated content, and the second compositing result of windows without updated content is reused, thereby reducing compositing overhead through a caching optimization scheme.
It improves the compositing efficiency of multi-window display and reduces the resource consumption and time overhead of the compositing process.
Smart Images

Figure CN123114334A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of electronic device technology, and in particular to an image processing method and device. Background Technology
[0002] When an electronic device displays multiple windows, it can draw and combine the individual windows, and then display the combined result on the screen.
[0003] The compositing process can include different compositing methods. For example, different compositing methods can include device compositing performed by a hardware compositor (HWC) and CLIENT compositing performed by a graphics processing unit (GPU).
[0004] Electronic devices can determine the compositing method for each layer based on the Z-order of the layers in each window.
[0005] In this multi-window display scenario, the user can also interact with one of the windows. Correspondingly, the electronic device can update the interface based on the user's actions.
[0006] Based on the existing composition allocation mechanism, the composition processing overhead may be large during the interface update process. Summary of the Invention
[0007] This application provides an image processing method and apparatus that can effectively improve compositing efficiency and reduce the overhead of the compositing process in multi-window display scenarios.
[0008] To achieve the above technical objectives, this application adopts the following technical solution:
[0009] A first aspect provides an image processing method applied to an electronic device. The method includes: displaying a first interface, the first interface including a first window and a second window; receiving a first operation, the first operation being an operation on the first window; responding to the first operation, performing a first compositing process on at least one layer of the first window to obtain a first compositing result; obtaining a second compositing result corresponding to the second window; and displaying a second interface based on the first compositing result and the second compositing result.
[0010] Based on this scheme, in a multi-window display scenario, the electronic device can perform a first compositing process on the first window that has received an operation and has updated content, thereby obtaining the updated content. The electronic device can also reuse the existing second compositing result for the second window that has not been updated. This saves the compositing overhead of the second window before displaying the second interface.
[0011] Optionally, the first compositing process for at least one layer of the first window includes: compositing the at least one layer of the first window using a hardware compositor (HWC) of the electronic device.
[0012] In this example, it is explicitly stated that the first synthesis process can be an HWC-based synthesis process.
[0013] Optionally, the compositing process for at least one layer of the first window includes: performing a first compositing process on at least one layer of the first window via the video input gateway VIG channel of the HWC.
[0014] In this example, because the first window has window features, the electronic device can efficiently composite the layers in the first window through the HWC's VIG channel.
[0015] Optionally, before displaying receipt of the first operation, the method further includes: obtaining the second compositing result through a second compositing process. The second compositing process includes: transmitting the layer to be processed to the graphics processing unit (GPU) of the electronic device for compositing processing of the layer to be processed by the GPU.
[0016] This scheme clarifies the timing and method for acquiring the second synthesis result. Specifically, the second synthesis result can be obtained through CLIENT synthesis via the GPU before receiving the first operation. In this way, after receiving the first operation, the electronic device can reuse the existing second synthesis result, thereby saving corresponding synthesis overhead.
[0017] Optionally, after receiving the first operation, the method further includes: determining to perform the second compositing process on at least one layer in the second window. In this example, when compositing windows in the second interface, the electronic device can configure windows without content updates to be composited via the GPU. This allows for the reuse of existing second compositing results based on hardware-provided caching optimization schemes in the CLIENT compositing process.
[0018] Optionally, after determining that at least one layer in the second window has undergone a second compositing process, the method further includes: determining that the layer information of the second layer at a first moment is the same as the layer information of the second layer at a second moment. The first moment is earlier than the moment the first interface is displayed, and the second moment is later than the moment the first operation is received. The second layer is included in at least one layer in the second window. The layer information is used to indicate the display content of the corresponding layer. Obtaining the second compositing result includes: obtaining the second compositing result based on the layer information of the first layer at the first moment being the same as the layer information of the first layer at the second moment, so as to display the second interface based on the second compositing result.
[0019] This example provides a mechanism for triggering a cache optimization scheme in CLIENT compositing. For instance, an electronic device can determine to trigger the cache optimization scheme if the layer information of the same layer in the current frame image is identical to the layer information in a previous frame. That is, the second composition result already obtained in the composition of the previous frame is reused as the composition result of the current frame.
[0020] Optionally, after receiving the first operation and before obtaining the second composite result, the method further includes: obtaining first metadata and second metadata. The first metadata is used to indicate the layer information and index of the first layer in the first window. The second metadata is used to indicate the layer information and index of the second layer in the second window. The index is used to indicate the Z-order of the corresponding layer. Based on the first and second metadata, the composite method of the first layer and the second layer is determined. The composite method includes the first composite process and the second composite process.
[0021] In this example, the electronic device can determine, through the index in the metadata, to perform DEVICE compositing for the first layer corresponding to HWC and CLIENT compositing for the second layer corresponding to GPU.
[0022] Optionally, determining the compositing method for the first layer and the second layer based on the first metadata and the second metadata includes: determining that the first layer is a window layer based on the layer information of the first layer. The size of the window layer is smaller than the size of the electronic device's display screen. Determining that the second layer is the window layer based on the layer information of the second layer. Determining the compositing method for the first layer as the first compositing method based on the fact that the index of the first layer is smaller than the layer index of the second layer. The electronic device can preferentially allocate HWC compositing resources to each layer according to the ascending order of the indices. If the HWC compositing resources are exhausted, GPU compositing resources can be allocated to the layers.
[0023] Optionally, if the number of layers in the first window reaches a first quantity, the compositing method for the second layer is determined to be the second compositing method. The first quantity is the number of VIG channels supported by the HWC of the electronic device.
[0024] Optionally, before obtaining the first and second metadata, the method further includes: generating third and fourth metadata based on the first operation. The third metadata indicates that the index of the first layer is greater than the index of the second layer.
[0025] Optionally, if preset conditions are met, the first metadata can be obtained based on the third metadata. The second metadata can be obtained based on the fourth metadata.
[0026] In this example, the electronic device can configure the index of the first layer with updated content to be less than that of the second layer, thus allowing the first layer to preferentially acquire HWC compositing resources. Correspondingly, if the layers in the first window exhaust the HWC VIG channels, the electronic device can allocate GPU compositing resources to the layers in the second window (such as the second layer), thereby enabling the layers in the second window to trigger a caching optimization scheme and reuse existing compositing results.
[0027] Optionally, the preset condition includes at least one of the following: the first window is not obscured by the second window, and the number of layers in the first window does not exceed the number of VIG channels supported by the HWC of the electronic device.
[0028] Optionally, before displaying the first interface, the method further includes: receiving a second operation, the second operation instructing the electronic device to display the first window on the display screen; and receiving a third operation, the third operation instructing the electronic device to display the second window on the display screen.
[0029] Optionally, the third operation is received earlier than the second operation.
[0030] In this implementation, since the second operation is later than the third operation, the creation of layers in the second window indicated by the third operation occurs earlier than the creation of layers in the first window indicated by the second operation. In this scenario, because the layers in the second window are created earlier, the default index of the layers in the second window is lower than the index of the layers in the first window. In this case, the electronic device can, based on the above scheme, adjust the index of the layers in the first window to be lower than the index of the layers in the second window, provided that preset conditions are met. This allows the electronic device to prioritize allocating HWC VIG channels to the layers in the first window with the adjusted smaller index when allocating compositing resources based on the index. Correspondingly, if the layers in the first window occupy all VIG channels, the electronic device can allocate GPU compositing resources to the layers in the second window, thereby reusing existing compositing results.
[0031] This will improve synthesis efficiency and reduce synthesis costs.
[0032] Secondly, this application also provides an electronic device comprising: a memory and one or more processors. The memory and processors are coupled. The memory stores computer program code, which includes computer instructions. When the processor executes the computer instructions, it causes the electronic device to perform the technical solutions provided in the first aspect and any possible implementation thereof.
[0033] Thirdly, this application also provides a chip system applied to an electronic device; the chip system may include one or more interface circuits and one or more processors. The interface circuits and processors are interconnected via lines, and the interface circuits are used to receive signals from the electronic device's memory and send the signals to the processors, the signals including computer instructions stored in the memory. When the processor executes the aforementioned computer instructions, the electronic device executes the technical solutions provided in the first aspect and any possible implementation thereof.
[0034] Fourthly, this application also provides a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the technical solutions provided in the first aspect and any possible implementation thereof.
[0035] Fifthly, this application also provides a computer program product that, when run on a computer, causes the computer to execute the technical solutions provided in the first aspect and any possible implementation thereof.
[0036] It is understood that the solutions provided in the second to fifth aspects of this application can be respectively associated with the first aspect and any of its possible designs, and therefore the beneficial effects achieved are similar, which will not be elaborated here. Attached Figure Description
[0037] Figure 1 A schematic diagram of an interface provided for an embodiment of this application;
[0038] Figure 2 A schematic diagram of an interface composition provided in an embodiment of this application;
[0039] Figure 3 A schematic diagram of an interaction process provided for an embodiment of this application;
[0040] Figure 4 A schematic diagram illustrating yet another interface configuration provided in this application embodiment;
[0041] Figure 5 A logical schematic diagram of an image processing flow provided in an embodiment of this application;
[0042] Figure 6 A schematic diagram of yet another interactive process provided in an embodiment of this application;
[0043] Figure 7 A schematic diagram illustrating a CLIENT composition scene caching optimization provided in an embodiment of this application;
[0044] Figure 8 A schematic diagram illustrating the composition of an electronic device provided in an embodiment of this application;
[0045] Figure 9 This is a schematic diagram of the interactive flow of an image processing method provided in an embodiment of this application;
[0046] Figure 10 A schematic diagram of the interactive flow of another image processing method provided in an embodiment of this application;
[0047] Figure 11 A schematic diagram of the interactive flow of another image processing method provided in an embodiment of this application;
[0048] Figure 12 This is a schematic diagram illustrating a layer index adjustment process provided in an embodiment of this application;
[0049] Figure 13 A comparative diagram illustrating the effects of one solution provided in an embodiment of this application;
[0050] Figure 14 A schematic diagram illustrating the composition of yet another electronic device provided in this application embodiment;
[0051] Figure 15 This is a schematic diagram of the composition of a chip system provided in an embodiment of this application. Detailed Implementation
[0052] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.
[0053] Currently, electronic devices can provide users with a wide range of display functions.
[0054] For example, refer to Figure 1 This is a schematic diagram of an interface provided in an embodiment of this application.
[0055] Take a tablet computer as an example.
[0056] like Figure 1 As shown, the electronic device may include a display screen. After the electronic device is powered on, the display screen can show the corresponding interface 01.
[0057] refer to Figure 2 This is a schematic diagram of an interface composition provided in an embodiment of this application.
[0058] like Figure 2As shown, interface 01 can include icons, names, etc. of one or more installed applications. Electronic devices can display these installed applications in interface 01 through a desktop launcher.
[0059] This interface 01 may also include system status bars such as WIFI indicator, battery level indicator, and time indicator.
[0060] This interface 01 may also include a navigation bar.
[0061] Users can interact with the displayed interface on the screen, instructing the electronic device to display the interface of one or more applications.
[0062] In this application, the application interface may include a variety of display formats.
[0063] For example, the various display formats may include: full-screen display, window display, parallel interface, split-screen display, etc.
[0064] In this way, the content corresponding to the application can be displayed on the screen in at least one of the following forms: full-screen display, window display, parallel interface, or split-screen display.
[0065] The following description uses the example of displaying the application content in full-screen or windowed mode.
[0066] refer to Figure 3 This is a schematic diagram of an interactive process provided in an embodiment of this application.
[0067] In this example, application A, application B, and application C can be applications already installed on the electronic device. For example, application A, application B, or application C can be, for instance, applications that are already installed on the electronic device. Figure 1 or Figure 2 Any application displayed in interface 01 shown. For example, application A, application B, or application C may not be the same as... Figure 1 or Figure 2 Other applications of any of the applications shown in interface 01.
[0068] The user can input operation OP1 into the electronic device. Operation OP1 can be used to instruct the electronic device to display content A of application A.
[0069] In some embodiments, taking application A as an example where it is displayed in full-screen mode by default, operation OP1 may include: clicking the icon of application A (indicating that application A should be run).
[0070] In other embodiments, operation OP1 may include: clicking the icon of application A (instructing to run application A), and instructing to display the content A of application A in full screen.
[0071] Correspondingly, the electronic device's display screen can switch to display interface 31. In this interface 31, the electronic device can display the content A of application A to the user in full-screen mode.
[0072] After this, the user can input operation OP2 (third operation) on the electronic device. Operation OP2 can be used to instruct the electronic device to display content B of application B.
[0073] In some embodiments, taking the default window of application B as an example, operation OP2 may include: clicking the icon of application B (indicating the operation of running application B).
[0074] In other embodiments, operation OP2 may include clicking the icon of application B and instructing the content B of application B to be displayed in a window.
[0075] Correspondingly, the display screen of the electronic device can switch to display interface 32 (first interface). In this interface 32, the electronic device can display the content B of application B to the user through window B (second window) corresponding to application B.
[0076] Since application B is displayed in a window, its content B can be displayed on top. Therefore, in interface 32, some elements of content A that are not currently displayed will be obscured by content B. Thus, the content displayed in interface 32 can include content B, as well as the portions of content A that are not obscured by the window of application B.
[0077] After this, the user can input operation OP3 (second operation) into the electronic device. This operation OP3 can be used to instruct the electronic device to display the content C of application C.
[0078] In some embodiments, the default window of application C is used as an example. Operation OP3 may include: clicking the icon of application C (indicating that application C should be run).
[0079] In other embodiments, operation OP3 may include: clicking the icon of application C, and instructing the content C of application C to be displayed in a window.
[0080] Take, for example, the window C (the first window) of application C and the window B of application B, which do not overlap in any part.
[0081] Correspondingly, the electronic device's display screen can switch to display interface 33 (second interface). In this interface 33, the electronic device can display the content C of application C to the user through a window display.
[0082] Based on the description in interface 32, in interface 33, content C can also partially obscure the display elements of content A and remain on top. Therefore, in interface 33, content B, content C, and the portion of content A not obscured by content B and content C can be displayed.
[0083] In some cases, combined Figure 1 and Figure 2 This interface 33 can also include more display elements.
[0084] refer to Figure 4 This is a schematic diagram of another interface composition provided in the embodiments of this application.
[0085] This interface 33 may also include a system status bar, a navigation bar, etc. Each application's corresponding display area may also include a function bar. This function bar can be used to adjust the application window size, position, close the window, etc.
[0086] Each display element in this interface 33 (such as application content, system status bar, navigation bar, function bar, etc.) can be assigned to a different layer.
[0087] like Figure 4 As shown, layer A1 can be used to draw the content A of application A, and layer A2 can be used to draw the function bar A corresponding to application A; layer B1 can be used to draw the content B of application B, and layer B2 can be used to draw the function bar B corresponding to application B; layer C1 can be used to draw the content C of application C, and layer C2 can be used to draw the function bar C corresponding to application C. In addition, layer D1 can be used to draw the system status bar, and layer D2 can be used to draw the navigation bar.
[0088] It should be noted that, Figure 4 The examples of layers B2 and C2 are for illustrative purposes only and do not constitute a limitation on the scope of layers B2 and / or C2.
[0089] Take layer B2 as an example. In some embodiments, the size of layer B2 may be smaller than the size of window B of application B; for example, layer B2 may be completely contained within the area of window B. In other embodiments, the size of layer B2 may be the same as the size of window B of application B.
[0090] Layer C2 is similar. No further details will be provided.
[0091] In this way, through various interactive methods, electronic devices can display different applications on the screen in various display formats under the user's instructions.
[0092] To achieve the above Figures 1-4The interactive display mechanism allows electronic devices to control the display screen to show the corresponding interface after receiving user operations, through operations such as drawing processing, compositing processing, and display processing.
[0093] refer to Figure 5 This is a logical schematic diagram of an image processing flow provided in an embodiment of this application.
[0094] After receiving a user's operation, the electronic device can perform drawing processing through its drawing module.
[0095] For example, this drawing process can draw one or more layers to be displayed. For instance, combined with... Figure 4 After the drawing process is completed, the electronic device can acquire layers A1, A2, B1, B2, C1, C2, D1, and D2 respectively.
[0096] Subsequently, the electronic device can undergo compositing processing via the compositing module. This compositing process can be used to composite one or more layers onto a single canvas for display based on the composited result on that canvas.
[0097] like Figure 5 As shown, in this application, the synthesis module may include a first synthesis unit and a second synthesis unit. The synthesis processing mechanisms performed by the different synthesis units are different.
[0098] As an example, the first compositing unit may include a hardware composer (HWC). The first compositing process performed by the first compositing unit may include hardware compositing processing, or DEVICE compositing processing.
[0099] The first synthesis unit can provide at least two channels. The synthesis processing capabilities of different channels are different.
[0100] For example, the at least two channels may include a video input gateway (VIG) channel and a direct memory access (DMA) channel.
[0101] The VIG channel supports compositing of the first type of layers. These first-type layers have window features and / or rotation features. For example, layers B1, B2, C1, and C2 in interface 33 are all layers with window features.
[0102] The VIG channel enables compositing of the second type of layer. This second type of layer does not have window and / or rotation features.
[0103] Generally, the number of channels in the first compositing unit is limited. Each channel can only be used to composite one layer at a time.
[0104] Accordingly, the second compositing unit may include a graphics processor unit (GPU). The second compositing process performed by the second compositing unit may include software compositing processing, or CLIENT compositing processing.
[0105] The second compositing unit, under the control of an electronic device, can work in conjunction with the first compositing unit to perform compositing processing on the first type of layers and / or the second type of layers.
[0106] Generally, the synthesis efficiency of the first synthesis unit is higher than that of the second synthesis unit. This synthesis efficiency can be reflected in aspects such as synthesis time and resource consumption. Higher synthesis efficiency corresponds to shorter synthesis time and less resource consumption.
[0107] In this example, when the compositing module performs compositing on multiple layers, the electronic device can transmit the data of multiple layers to the first compositing unit. If all channels of the first compositing unit are occupied, the other layers can be transmitted to the second compositing unit for compositing.
[0108] Electronic devices can configure indexes for each layer to facilitate the rational transfer of multiple layers to the first or second compositing unit via the layer indexes. This index can also be called Z-order, or Z-order index.
[0109] For example, an electronic device can assign composition resources to each layer sequentially, from smallest to largest index. This process of assigning composition resources may include, for example, inputting the layers into a first composition unit or a second composition unit.
[0110] In some embodiments, the index of each layer may be related to the display format of the layer and / or the timing of when the layer is drawn.
[0111] For example, layers drawn earlier have smaller indices. Also, layers displayed in windowed mode have smaller indices than layers displayed in full-screen mode.
[0112] Combination Figure 3 The interactive examples in the document illustrate the index configuration for each layer.
[0113] exist Figure 3 In the example, after the user inputs operation OP1, layers A1, A2, D1, and D2 are drawn. Among them, layers A1 and A2 are displayed in full-screen mode.
[0114] Thus, the correspondence between each layer and its index can be shown in Table 1:
[0115] Table 1
[0116]
[0117]
[0118] As shown in Table 1, in this example, the index of layer A1 can be 0, the index of layer A2 can be 1, the index of layer D1 can be 2, and the index of layer D2 can be 3.
[0119] After the user inputs operation OP2, the drawing module of the electronic device can draw layers A1, A2, D1, and D2, as well as layers B1 and B2. Among them, layers A1 and A2 are displayed in full-screen mode; layers B1 and B2 are displayed in window mode.
[0120] Thus, the correspondence between each layer and its index can be shown in Table 2:
[0121] Table 2
[0122] Layers index Layer B1 0 Layer B2 1 Layer A1 2 Layer A2 3 Layer D1 4 Layer D2 5
[0123] Since layers B1 and B2 are displayed in windows, the electronic device can insert the indices of layers B1 and B2 before layers A1 and A2. Correspondingly, the indices of layers A1 and subsequent layers change sequentially.
[0124] As shown in Table 2, the index of layer B1 can be 0, the index of layer B2 can be 1, the index of layer A1 can be 2, the index of layer A2 can be 3, the index of layer D1 can be 4, and the index of layer D2 can be 5.
[0125] After the user inputs operation OP3, the drawing module of the electronic device can draw layers A1, A2, D1 and D2, B1, B2, C1 and C2. Among them, layers A1 and A2 are displayed in full-screen mode; layers B1, B2, C1, and C2 are displayed in window mode.
[0126] Thus, the correspondence between each layer and its index can be shown in Table 3:
[0127] Table 3
[0128] Layers index Layer B1 0 Layer B2 1 Layer C1 2 Layer C2 3 Layer A1 4 Layer A2 5 Layer D1 6 Layer D2 7
[0129] Since layers C1 and C2 are displayed in windows, the electronic device can insert the indices of layers C1 and C2 before layers A1 and A2. Correspondingly, the indices of layers A1 and subsequent layers change sequentially.
[0130] As shown in Table 3, the index of layer B1 can be 0, the index of layer B2 can be 1, the index of layer C1 can be 2, the index of layer C2 can be 3, the index of layer A1 can be 4, the index of layer A2 can be 5, the index of layer D1 can be 6, and the index of layer D2 can be 7.
[0131] In this way, the electronic device can control the compositing unit to perform compositing processing on each layer according to the index configuration results shown in Table 3.
[0132] For example, taking a first compositing unit comprising two VIG channels and two DMA channels as an example. In conjunction with the foregoing description, layers B1, B2, C1, and C2 can be classified as first-type layers, while layers A1, A2, D1, and D2 can be classified as second-type layers.
[0133] The electronic device can transmit layer B1 and layer B2, which have an index of 0, to two VIG channels for compositing.
[0134] When allocating composition resources for layer C1 with index 2, since layer C1 is a second-class layer and all VIG channels are occupied, the electronic device can input layer C1 into the second composition unit for compositing processing via the GPU. Layer C2 is similar.
[0135] For layer A1, since it is a second-type layer, the electronic device can input layer A1 into the DMA channel for compositing. Layer A2 is similar.
[0136] For layers D1 and D2, since all channels of the first composition unit are occupied, the electronic device can input layers D1 and D2 into the second composition unit for compositing.
[0137] Therefore, the resource allocation results for each layer are shown in Table 4.
[0138] Table 4
[0139] Layers Synthetic Resources Layer B1 DEVICE Layer B2 DEVICE Layer C1 CLIENT Layer C2 CLIENT Layer A1 DEVICE Layer A2 DEVICE Layer D1 CLIENT Layer D2 CLIENT
[0140] As shown in Table 4, layers B1, B2, A1, and A2 can be input into the first composition unit to perform the first composition process, i.e., the DEVICE composition process.
[0141] Layers C1, C2, D1, and D2 can be input into a second composition unit to perform a second composition process, i.e., CLIENT composition.
[0142] Therefore, electronic devices can complete tasks such as Figure 3 or Figure 4 The composite processing of interface 33 is performed to obtain the composite result corresponding to interface 33. The composite result corresponding to interface 33 can include all display elements in interface 33. This composite result can also be referred to as the display image.
[0143] After the synthesis process is completed, such as Figure 5 As shown, the electronic device can control the display module to display information. The display module may include at least one display screen.
[0144] For example, an electronic device can control a display module to display interface 33 based on the composite result of interface 33.
[0145] Understandably, the user's input OP3 is not usually an isolated operation. After instructing the electronic device to display content C of application C by operating OP3, the user can also input other operations for application C.
[0146] For example, refer to Figure 6 This is a schematic diagram of another interactive process provided in the embodiments of this application.
[0147] With the display screen showing interface 33, the user can input operation OP4 (first operation).
[0148] In this example, operation OP4 can be an input operation on the display area of application C. The display area of application C can be the area on the screen corresponding to the display window of application C in interface 33.
[0149] For example, operation OP4 could include a swipe gesture on the display area of application C. Thus, through operation OP4, the user can instruct the electronic device to swipe and update the displayed content of application C's window.
[0150] For example, operation OP4 could include clicking an interactive button in application C. Thus, through operation OP4, the user can instruct the electronic device to perform the interactive process corresponding to that interactive button via application C.
[0151] In this way, after receiving operation OP4, the electronic device can control the display screen to show interface 34.
[0152] Compared to interface 33, in interface 34, within the display area of application C, content C is updated to content C'. Display elements outside the display area of application C remain unchanged.
[0153] For example, the layer information of layer D1 in interface 33 is the same as that of layer D1 in interface 34, both of which include WIFI indicator, battery indicator, time indicator, etc.
[0154] For example, the layer information of layer C1 in interface 33 is different from that of layer C1 in interface 34. The layer information of layer C1 in interface 33 corresponds to content C, while the layer information of layer C1 in interface 34 corresponds to content C'.
[0155] In this application, based on Figure 5 The processing logic shown allows the electronic device to optimize the compositing process of the second compositing unit during the display of multiple frames of images, thereby improving the compositing efficiency of the second compositing unit. This optimization scheme can also be called CLIENT compositing scene cache optimization.
[0156] The following combination Figure 6 The scenario and the correspondence between layers and indexes provided in Table 3 are illustrated with examples.
[0157] It should be noted that, in this application, the layer information in the layer can be used to indicate the content to be drawn / displayed by the layer. When the layer has window features and / or rotation features, this layer information can indicate the characteristics of the layer.
[0158] In some implementations, layer information may also include information indicating the occlusion relationships between different layers. This layer information can be included in the layer's metadata. In some implementations, the layer's metadata may also include the layer's index (i.e., Z-order).
[0159] In other words, the metadata of a frame of an image can include layer information and Z-order.
[0160] The layer information may include at least one of the following:
[0161] Size and resolution, color space, bit depth, rotation angle, mirror information, timestamp, duration, content descriptor, rendering instructions (such as color correction, transparency, scaling mode, etc.).
[0162] by Figure 6 Take layers C1 and D1 as examples.
[0163] Layer C1 may have different layer information in interfaces 33 and 34; layer D1 may have the same layer information in interfaces 33 and 34.
[0164] refer to Figure 7 This is a schematic diagram illustrating a CLIENT compositing scene caching optimization provided in an embodiment of this application.
[0165] For layer D1, in the compositing process of interface 33, the electronic device can determine, after determining the index as shown in Table 4, that layer D1 can be input into the second compositing unit for compositing. The electronic device can put the metadata 1 of layer D1 into the CLIENT compositing queue. The second compositing unit can retrieve the metadata 1 of layer D1 from the CLIENT compositing queue and perform compositing of layer D1. Thus, the compositing result 1 of layer D1 can be obtained.
[0166] Similarly, for layer C1, in the compositing process of interface 33, the electronic device can determine, after determining the index as shown in Table 4, that layer C1 can be input into the second compositing unit for compositing. The electronic device can put the metadata 2 of layer C1 into the CLIENT compositing queue. The second compositing unit can retrieve the metadata 2 of layer C1 from the CLIENT compositing queue and perform compositing of layer C1. Thus, the compositing result 2 of layer C1 can be obtained.
[0167] After this, electronic devices can receive, such as Figure 6 The operation shown is OP4. Correspondingly, the electronic device can draw the contents of each layer in interface 34 and then perform the compositing process of interface 34.
[0168] In this example, the electronic device can determine whether the layer information in a given layer is consistent with the layer information in a previous frame before compositing any layer in the second compositing unit. If the layer information is the same in the current frame as in the previous frame, the electronic device can skip compositing that layer in the current frame and directly reuse the compositing result from the previous frame.
[0169] For example, during the processing of interface 34, for layer D1, the electronic device can determine, after determining the index shown in Table 4, that layer D1 can be input into the second compositing unit for compositing processing. The electronic device can, based on the fact that the layer information of layer D1 in interface 34 is the same as the layer information of layer D1 in a previous frame (such as interface 33), not put the metadata 1 of layer D1 into the CLIENT compositing queue. The electronic device can use the compositing result 1 of layer D1 in interface 33 as the compositing result of layer D1 in interface 34. For example, the electronic device can obtain the compositing result 1 by storing the buffer ID of the compositing result 1 during the compositing process of interface 33.
[0170] During the processing of interface 34, for layer C1, the electronic device can determine, after determining the index shown in Table 4, that layer C1 can be input into the second compositing unit for compositing processing. The electronic device can perform compositing processing on layer C1 in interface 34 according to a compositing processing logic similar to that of interface 33, based on the difference between the layer information of layer C1 in interface 34 and the layer information of layer C1 in the previous frame (such as interface 33).
[0171] For example, an electronic device can put the metadata 3 of layer C1 into the CLIENT composition queue. The second composition unit can retrieve the metadata 3 of layer C1 from the CLIENT composition queue and perform composition on layer C1. Thus, the composition result 3 of layer C1 can be obtained in interface 34.
[0172] Based on this, embodiments of this application provide an image processing method. This method enables an electronic device to flexibly adjust the compositing resources of each window layer in a multi-window display scenario, allowing the focused window to be efficiently composited through the first compositing unit, while windows with unchanged layer information can effectively improve compositing efficiency through CLIENT compositing scene caching optimization.
[0173] The solutions provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.
[0174] The electronic devices involved in the embodiments of this application may include at least one of the following: mobile phones, foldable electronic devices, tablet computers, desktop computers, laptop computers, handheld computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cellular phones, personal digital assistants (PDAs), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, in-vehicle devices, smart home devices, or smart city devices. The embodiments of this application do not impose any special limitations on the specific type of the electronic device.
[0175] In different implementations, electronic devices can be implemented using the solutions provided in the embodiments of this application to control the composition and display of windows.
[0176] In some embodiments, the software system of an electronic device may adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application uses the layered architecture of the Android system as an example to exemplify the software structure of the electronic device.
[0177] In this example, the layered architecture of the Android system divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system may include five layers, from top to bottom: the Application (APP) layer, the App Framework layer, the Android runtime (ART) and native C / C++ libraries, the Hardware Abstraction Layer (HAL), and the kernel layer.
[0178] The following examples use the application layer, application framework layer, and hardware abstraction layer of the Android system as examples to illustrate the composition of an electronic device.
[0179] refer to Figure 8 This is a schematic diagram of the composition of an electronic device provided in an embodiment of this application.
[0180] The application layer can include a series of application packages. The application layer can also be called the application layer or the APP layer.
[0181] like Figure 8 As shown in the example, the application layer may include application A, application B, application C, etc.
[0182] The application framework layer, also known as the framework layer or framework layer, provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The framework layer includes a set of predefined functions.
[0183] For example, the framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, an activity manager, an input manager, a graphics compositor, etc.
[0184] Among them, the window manager provides the Window Manager Service (WMS), which can be used for window management, window animation management, surface management, and as a relay station for the input system.
[0185] Content providers store and retrieve data, making that data accessible to applications. This data can include videos, images, audio, phone calls made and received, browsing history and bookmarks, phone books, etc.
[0186] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.
[0187] like Figure 8 As shown, the framework layer may also include a surface control module.
[0188] A Surface is a memory area used for drawing graphics, and views in the view system can be drawn onto the Surface. The Surface Control module provides low-level control and management functions for the Surface.
[0189] For example, the surface control module can be used to create, destroy, and manage the lifecycle of Surfaces. It can assign appropriate Surfaces to different windows or views according to the needs of the application, ensuring that each view has suitable drawing space.
[0190] For example, the surface control module can be used to set various properties of the Surface, such as size, format, and transparency. These properties affect the drawing effect and display characteristics of the view on the Surface.
[0191] For example, the surface control module can be used to coordinate the drawing order and synchronization between different Surfaces, ensuring that the drawing of multiple views can be correctly combined and displayed. At the same time, the surface control module can manage the drawing queue of Surfaces, processing the drawing requests of views in a certain order to ensure the smoothness and consistency of the interface.
[0192] The Input Manager Service (IMS) provides input management services, which can be used to manage system inputs such as touchscreen input, keypad input, and sensor input. IMS retrieves events from input device nodes and, through interaction with the WMS (Windows Management System), distributes these events to appropriate windows.
[0193] The Surface Flinger (SF) can be used for surface management. SF can also be used to control the compositing of layers.
[0194] For example, SF can control any one of the multiple layers in a frame image to be composited through the first compositing process described above, or through the second compositing process described above. Optionally, SF can be used to control the first compositing unit to continue compositing based on the compositing result obtained from the second compositing process and the compositing results obtained from other first compositing processes, thereby obtaining the compositing result corresponding to the frame image. The compositing result corresponding to the frame image may include all display elements in the frame image, that is, the display image corresponding to the frame image.
[0195] In this application, in a multi-window display scenario, SF can also, according to preset conditions, control the layer of the focused window to be composited through a first compositing unit, and control the layers of at least one other window to be processed through a second compositing unit. This allows for the reasonable application of... Figure 7 The CLIENT compositing scene caching optimization shown aims to improve compositing efficiency. The specific process will be detailed later.
[0196] Hardware Abstraction Layer (HAL) runs in user space, encapsulates kernel-level drivers, and provides calling interfaces to upper layers (such as framework layers and / or related unit modules in application layers). For example, the Hardware Abstraction Layer may include touch HAL, display HAL, and composite HAL (Composer HAL).
[0197] Among them, the touch HAL is used for interaction between the upper layer and the touch panel (TP) of the electronic device; the display HAL is used for interaction between the upper layer and the display module of the electronic device, and the display module may include the display driver integrated circuit (DDIC) and the display screen; the synthesis HAL is used for interaction between the upper layer and the synthesis unit of the electronic device, such as the first synthesis unit and the second synthesis unit mentioned above.
[0198] like Figure 8 As shown, the application layer, framework layer, HAL, and their respective unit modules can be configured in the software layer of the electronic device.
[0199] exist Figure 8 The document also provides examples of the hardware layer of electronic devices.
[0200] like Figure 8 As shown, the hardware layer can be configured with a display touch module, hardware synthesizer, GPU, etc.
[0201] The display touch module can include a touchpad, a display module, etc. The hardware synthesizer is called HWC. The GPU can be used for graphics rendering, secondary compositing, etc.
[0202] It should be noted that, Figure 8The electronic devices shown are merely examples and do not constitute a limitation on the electronic devices involved in this application. In other embodiments, the electronic devices may also have, for example... Figure 8 Show more or fewer components. Figure 8 One or more components can be integrated or separately installed in an electronic device.
[0203] In the following examples, electronic devices have, for example, Figure 8 The following is an example of the components shown.
[0204] refer to Figure 9 This is a schematic diagram of the interactive flow of an image processing method provided in an embodiment of this application.
[0205] Combination Figure 3 The example below uses user input operation OP2 as an example. Electronic devices can use this... Figure 9 The illustrated scheme controls the display screen 32 to show the interface based on the user's input operation OP2. In some implementations, the user can also input operation OP1 before inputting operation OP2, and the corresponding electronic device can control the display screen 31 to show the interface.
[0206] like Figure 9 As shown, the method may include:
[0207] S901, The display touch module sends touch event 91 to the input event management.
[0208] For example, the TP in the display touch module can generate a touch event 91 corresponding to the user's operation OP2 after receiving the operation OP2. As explained above, the operation OP2 can be used to instruct the electronic device to display the relevant content of application B through a window display format.
[0209] The display touch module can transmit the touch event 91 to the input event management for further processing.
[0210] S902, Input Event Management sends touch event 91 to application B.
[0211] For example, input event management can determine that the operation is related to application B based on touch event 91. Thus, the electronic device can run application B. Input event management can transmit the touch event 91 to application B so that application B can perform subsequent processing.
[0212] In this example, the information sent by the input event management to application B includes touch event 91. In other examples, the information sent by the input event management to application B may be generated based on touch event 91.
[0213] S903, Application B sends a window launch instruction 92 to the window management service. This window launch instruction 92 can be used to instruct the window management service to generate a window corresponding to Application B. This window of Application B can be used to display the content B of Application B.
[0214] S904, The window management service sends a window start instruction 92 to the surface control.
[0215] S905, Surface Control sends Creation Command 93 to SF.
[0216] For example, creation instruction 93 can be used to create a layer in SF corresponding to the window of application B. In some examples, the window management service can generate and send creation instruction 93 to SF based on the received window launch instruction 92.
[0217] S906, SF creates layers B1 and B2.
[0218] Thus, the layers of the window corresponding to application B (e.g., window B) can include layer B1 and layer B2.
[0219] Therefore, application B can generate corresponding drawing instructions based on touch event 91. Application B can also control the electronic device (such as calling the GPU or other components with graphics drawing capabilities) to draw according to these drawing instructions. This drawing process can be associated with the layers B1 and B2 created above. For example, the electronic device can draw content B on layer B1; or, for example, the electronic device can draw the function bar B on layer B2.
[0220] It should be noted that, in the description of this application, the example given is that each application includes a layer for drawing content and a layer for drawing the function bar. In other implementations, there can be multiple layers for drawing content, and there can also be multiple layers for drawing the function bar. In this case, the multiple layers for drawing content and the multiple layers for drawing the function bar can be associated with the application respectively.
[0221] Therefore, in Figure 9 In the example, after the electronic device completes the processing of S906, it can draw content B on layer B1 and function bar B on layer B2.
[0222] In some embodiments, combined with Figure 3 In the example above, if the user inputs operation OP1 before operation OP2, the electronic device can implement a solution similar to S901-S906, creating layers A1 and A2 for application A. Correspondingly, under the control of application A, the electronic device can also draw content A on layer A1 and function bar A on layer A2.
[0223] In other embodiments, combined with Figure 4 In the example provided, the electronic device can also be configured to display a system status bar and a navigation bar. This can be achieved using a scheme similar to S901-S906, creating corresponding layers for the system status bar and navigation bar respectively. For example, a layer D1 can be created for the system status bar, and a layer D2 can be created for the navigation bar. The electronic device can also draw the system status bar on layer D1 and the navigation bar on layer D2.
[0224] Thus, after the user inputs operation OP1 and operation OP2, the electronic device can create layers through the above S901-S906, including: layer A1 and layer A2 associated with application A, layer B1 and layer B2 associated with application B, layer D1 and layer D2.
[0225] After completing the drawing process on each layer, the SF of the electronic device can control the first compositing unit and / or the second compositing unit to perform compositing processing on these layers, thereby obtaining the display image of the interface 32 to be displayed.
[0226] In the following description, we take HWC as the first synthesis unit and GPU as the second synthesis unit as an example.
[0227] Based on the above explanation of the drawing process on layers, in this example, SF can also obtain the metadata of each layer before controlling the compositing unit to perform the compositing process. For example, the metadata A1 of layer A1, the metadata A2 of layer A2, the metadata B1 of layer B1, the metadata B2 of layer B2, the metadata D1 of layer D1, and the metadata D2 of layer D2, etc.
[0228] The metadata for each layer can include layer information, index (i.e., Z-order), occlusion relationships, etc., of the content drawn on that layer.
[0229] Based on the metadata of each layer, SF can interact with HWC to determine the compositing method for each layer. This compositing method can include DEVICE compositing and CLIENT compositing.
[0230] For example, the process may include S907:
[0231] S907 and SF send synthesis parameters 94 to HWC. HWC sends synthesis configuration 95 to SF.
[0232] For example, composition parameter 94 may include metadata for at least one layer.
[0233] For example, composition parameter 94 may include metadata A1 of layer A1, metadata A2 of layer A2, metadata B1 of layer B1, metadata B2 of layer B2, metadata D1 of layer D1, and metadata D2 of layer D2.
[0234] Correspondingly, HWC can determine the composition method of each layer based on hardware capabilities and the Z-order of each layer.
[0235] For example, this hardware capability may include: the number of VIG channels supported by HWC, the number of DMA channels supported, etc.
[0236] As an example, the correspondence between the layers and the index indicated by the composition parameter 94 is the same as the correspondence in Table 2 above.
[0237] With HWC supporting 2 VIG channels and 2 DMA channels, based on the correspondence in Table 2, HWC can determine the composition configuration 95 for each layer, which may include: layers B1 and B2 are composed through HWC's VIG channels, layers A1 and A2 are composed through HWC's DMA channels, and layers D1 and D2 are composed through the second composition unit (GPU).
[0238] In other words, composition configuration 95 can indicate the composition method for each layer. For example, the composition method (or composition resource) correspondence for each layer is as follows: layers B1, B2, A1, and A2 are assigned to DEVICE composition; layers D1 and D2 are assigned to CLIENT composition.
[0239] Therefore, SF can input each layer into the corresponding compositing unit for compositing processing according to the determined compositing method, based on the compositing configuration 95. For example, this process may include S908-S909.
[0240] S908, SF sends the first composite layer information to HWC 96. HWC sends the first composite result to SF 97.
[0241] For example, the first composite layer information 96 may include metadata for layer B1, layer B2, layer A1, and layer A2.
[0242] HWC can use the VIG channel to perform a first compositing process on layers B1 and B2, which have window characteristics, thereby obtaining the compositing result corresponding to application B.
[0243] HWC can use the DMA channel to perform the first compositing process on layers A1 and A2, thereby obtaining the compositing result corresponding to application A.
[0244] The synthesis results corresponding to application B and application A can be carried in the first synthesis result 97 and sent by HWC to SF.
[0245] S909, SF sends the second composite layer information to the GPU 98. The GPU sends the second composite result to SF 99.
[0246] For example, the second composite layer information 98 may include metadata for layer D1 and layer D2.
[0247] The GPU can perform a second compositing process on layers D1 and D2 to obtain the second compositing result 99 corresponding to layers D1 and D2.
[0248] The GPU can send the second synthesized result 99 to SF for further processing.
[0249] S910 and SF obtained the synthesis result 991.
[0250] For example, SF can determine synthesis result 991 based on the first synthesis result 97 and the second synthesis result 99.
[0251] As an example, SF can input the first composite result 97 and the second composite result 99 into HWC. This allows HWC to perform a composite process on the first composite result 97 and the second composite result 99, thereby compositing all display elements of interface 32 onto a single layer. This composite result, which includes all display elements of interface 32, is also known as composite result 991.
[0252] S911, SF sends the composite result 991 to the display touch module. This composite result 991 is also the display image corresponding to interface 32. For example, SF can put the composite result 991 into the display queue so that the display module in the display touch module can retrieve the composite result 991 from the display queue as needed for display.
[0253] S912, Display the touch module display interface 32. For example, the touch module can display interface 32 on the display screen according to the synthesis result 991.
[0254] Subsequently, upon user input of operation OP3, the electronic device can, according to, such as Figure 10 The process shown is displayed on the screen as interface 33.
[0255] refer to Figure 10 This is a schematic diagram of the interactive flow of another image processing method provided in the embodiments of this application.
[0256] like Figure 10 As shown, the method may include:
[0257] S1001, The display touch module sends touch event 101 to the input event management.
[0258] S1002, Input Event Management sends Touch Event 101 to Application C.
[0259] S1003, Application C sends a window start instruction 102 to the window management service.
[0260] S1004, The window management service sends a window start instruction 102 to the surface control.
[0261] S1005, Surface control sends creation command 103 to SF.
[0262] S1006, SF creates layers C1 and C2.
[0263] For example, the processing steps S1001-S1006 can be referred to Figure 9 S901-S906 in the series.
[0264] In this example, after receiving the user's operation OP3, the electronic device can interact with the application C corresponding to operation OP3, thereby realizing the above S1001-S1006.
[0265] Therefore, after receiving user input operations OP1-OP3, the layers created by the SF of the electronic device can include layers A1 and A2 of application A, layers B1 and B2 of application B, layers C1 and C2 of application C, as well as layers D1 and D2, etc.
[0266] Subsequently, under the control of application C, the electronic device can draw content C on layer C1 and draw the function bar C corresponding to application C1 on layer C2.
[0267] In addition, content A of application A1 can be drawn on layer A1, and function bar A corresponding to application A1 can be drawn on layer A2; content B of application B1 can be drawn on layer B1, and function bar B corresponding to application B1 can be drawn on layer B2; system status bar can be drawn on layer D1, and navigation bar can be drawn on layer D2.
[0268] similar Figure 9 The explanation in Figure 10 In the example, after completing the drawing process on each layer, the SF of the electronic device can control the first compositing unit and / or the second compositing unit to perform compositing processing on these layers, thereby obtaining the display image of the interface 33 to be displayed.
[0269] For example, before controlling the compositing unit to perform compositing processing, SF can also obtain the metadata of each layer. For example, the metadata A1 of layer A1, the metadata A2 of layer A2, the metadata B1 of layer B1, the metadata B2 of layer B2, the metadata C1 of layer C1, the metadata C2 of layer C2, the metadata D1 of layer D1, the metadata D2 of layer D2, etc.
[0270] The metadata for each layer can include layer information, index (i.e., Z-order), occlusion relationships, etc., of the content drawn on that layer.
[0271] The composition method for each layer can be determined based on the metadata SF and HWC of each layer.
[0272] S1007, SF sends synthesis parameters 104 to HWC. HWC sends synthesis configuration 105 to SF.
[0273] For example, composition parameter 104 may include metadata for each layer.
[0274] For example, composition parameter 104 may include metadata A1 of layer A1, metadata A2 of layer A2, metadata B1 of layer B1, metadata B2 of layer B2, metadata C1 of layer C1, metadata C2 of layer C2, metadata D1 of layer D1, and metadata D2 of layer D2.
[0275] Similar to the example in S907, in this S1007, HWC can determine the composition configuration 105 of each layer based on the composition parameters 104.
[0276] Among them, composition configuration 105 can be used to indicate the composition method of each layer.
[0277] As an example, referring to the explanation in Table 3, layers B1, B2, C1, and C2 all include window features. Therefore, the indices of layers B1, B2, C1, and C2 can precede those of layers A1, A2, D1, and D2. Layers C1 and C2 are inserted later at the beginning of the index, so their indices can be lower than those of layers B1 and B2.
[0278] Therefore, based on the indexes of each layer as shown in Table 3, HWC can combine hardware capabilities to determine the compositing method for each layer.
[0279] Let's continue with the example of HWC supporting a maximum of 2 VIG channels and 2 DMA channels.
[0280] In the composition configuration 105 determined by HWC, the composition methods for each layer can include:
[0281] Layers B1 and B2 use DEVICE compositing (e.g., first compositing via the VIG channel); layers C1 and C2 use CLIENT compositing; layers A1 and A2 use DEVICE compositing (e.g., first compositing via the DMA channel); layers D1 and D2 use CLIENT compositing. This compositing configuration 105 corresponds to the compositing resource configuration shown in Table 4.
[0282] Understandably, since all VIG channels are occupied, the DMA channels cannot be used for compositing window layers (i.e., layers containing window features). Therefore, layers C1 and C2, whose indices are less than those of layers A1 and A2, are configured to use the GPU for CLIENT compositing. However, when layers A1 and A2 are composited using the DMA channels, all HWC channels are occupied, so layers D1 and D2 are configured to use CLIENT compositing.
[0283] In this way, SF can control the compositing process of the corresponding layers according to the compositing method indicated by compositing configuration 105.
[0284] S1008, SF sends the first composite layer information to HWC 106. HWC sends the first composite result to SF 107.
[0285] For example, the first composite layer information 106 may include metadata for layers C1, C2, A1, and A2. This first composite layer information 106 can be used by HWC for first compositing processing.
[0286] According to the instructions in S1007, HWC can put layers C1 and C2 into the VIG channel for the first compositing process; HWC can also put layers A1 and A2 into the DMA channel for the first compositing process.
[0287] Therefore, HWC can obtain the first composite processing result 107 for layers C1, C2, A1, and A2.
[0288] S1009, SF sends the second composite layer information to the GPU 108. The GPU sends the second composite result to SF 109.
[0289] For example, the second compositing layer information 108 may include metadata for layers B1, B2, D1, and D2. This second compositing layer information 108 can be used by the GPU for second compositing processing.
[0290] Therefore, the GPU can perform a second compositing process based on the second compositing layer information 108 to obtain the second compositing results 109 corresponding to layers B1, B2, D1, and D2.
[0291] S1010 and SF obtained the synthesis result 1091.
[0292] Similar to the description in S910, in this example, SF can determine the composite result 1091 corresponding to interface 33 based on the first composite result 107 and the second composite result 109. The composite result 1091 can include all the display elements of interface 33.
[0293] S1011 and SF send the composite result 1091 to the display touch module.
[0294] S1012, Display touch module display interface 33.
[0295] Therefore, after completing the synthesis process through S1007-S1010, SF can send the synthesis result 1091 to the display touch module so that the display screen in the display touch module can display the interface 33 according to the synthesis result 1091.
[0296] It is understandable that, in the above Figure 9 In the example, for window B of application B, since window B is obtained through the first composition process performed by HWC, it does not involve Figure 7 The application of CLIENT composition scene caching optimization is shown.
[0297] In the above Figure 10 In the example, for window C of application C, since window C is obtained through the first composition process performed by HWC, it does not involve Figure 7 The example shown illustrates the application of CLIENT compositing scene caching optimization. For window B of application B, since window B is obtained through the first compositing process performed by HWC, it is not involved. Figure 7 The example shown illustrates the application of CLIENT compositing scene caching optimization. For window C of application C, since window C is a newly created window, the content within window C differs from previous frames (e.g., interface 32), therefore compositing processing for window C will not be triggered. Figure 7 The CLIENT compositing scene caching optimization is shown.
[0298] Combination Figure 6 In the interactive example, after the user inputs operation OP3, the electronic device can display interface 33. Subsequently, the user can input other operations (such as operation OP4) into window C of application C. Correspondingly, upon receiving the user's operation OP4, the electronic device can implement the drawing, composition, and display of interface 34 using the following scheme.
[0299] refer to Figure 11 This is a schematic diagram of the interactive flow of another image processing method provided in the embodiments of this application.
[0300] S1101, The display touch module sends touch event 111 to the input event management.
[0301] S1102, Input event management sends touch event 111 to application C.
[0302] For example, upon receiving user input operation OP4, the display touch module can generate touch event 111.
[0303] Combination Figure 6 As explained in the documentation, operation OP4 can be an operation on window C of application C. Operation OP4 can be used to instruct the electronic device to update the displayed content in window C.
[0304] If application C determines that its window C needs to be updated based on the touch event 111, application C can interact with the electronic device to update the display elements in the updated window C through the electronic device.
[0305] S1103, Application C sends a window update request 112 to the window management service.
[0306] For example, window update request 112 may include information about the update content corresponding to operation OP4.
[0307] The window update request 112 can be used by electronic devices to draw, composite, and display the display elements of the interface 34.
[0308] S1104, Window Management Service sends window update information 113 to Surface Control.
[0309] S1105, Surface control sends window update information 113 to SF.
[0310] The window update information 113 may include metadata of the layers C1 and C2 to be updated. For example, the window update information 113 may include metadata of layer C1 (such as layer information and index) and metadata of layer C2 (such as layer information and index).
[0311] As an example, since operation OP4 operates on an existing window, it is more efficient than... Figure 9 or Figure 10 In the example, the Figure 11 The process does not include the creation of new layers.
[0312] In this way, the indexes of each layer can be compared with... Figure 10 The layer indices are the same.
[0313] For example, the index of layer C1 can be 2, and the index of layer C2 can be 3.
[0314] In this example, the information sent by the window management service to the surface control (such as window update information 113) is the same as the information sent by the surface control to the SF. In other embodiments, the surface control may also send information indicating the elements displayed in window C to the SF via other messages based on the window update message 113. This allows the SF to update the elements displayed in window C based on the received information.
[0315] Therefore, SF can control the updating of window C of application C based on the received window update information 113.
[0316] The update of window C may include: drawing, compositing, and displaying layers C1 and C2 of application C.
[0317] The drawing of layer C1 can be performed based on the layer information in window update information 113.
[0318] Furthermore, while window C is being updated, applications A and B, as well as other display elements (such as the system status bar and navigation bar), remain unchanged. This allows SF to retrieve metadata for applications A and B, as well as the system status bar and navigation bar. For example, SF can retrieve layer information and indices for layers A1, A2, B1, B2, D1, and D2.
[0319] After that, SF can perform compositing based on the updated metadata of layers C1 and C2, as well as the metadata of layers A1, A2, B1, B2, D1, and D2.
[0320] Based on the above explanation, after completing S1105, the indices of each layer obtained by SF can be similar to those in Table 3.
[0321] For example, the index of layer B1 is 0, the index of layer B2 is 1, the index of layer C1 is 2, the index of layer C2 is 3, the index of layer A1 is 4, the index of layer A2 is 5, the index of layer D1 is 6, and the index of layer D2 is 7.
[0322] Thus, if the compositing method for each layer is determined by communicating with HWC based on the indexes of each layer obtained from the SF, the compositing method for each layer determined by HWC can be combined with... Figure 10 The logic is similar to that in [the text].
[0323] For example, as shown in Table 4, layers B1 and B2 are composited using DEVICE; layers C1 and C2 are composited using CLIENT; layers A1 and A2 are composited using DEVICE; and layers D1 and D2 are composited using CLIENT.
[0324] In this example, among the layers in interface 34, only layers C1 and C2 are updated; that is, the metadata of layers C1 and C2 changes compared to the previous frame. Therefore, when layers C1 and C2 are used for CLIENT compositing, it will not trigger the following: Figure 7 The CLIENT compositing scene caching optimization is shown.
[0325] For other layers, such as layer B1 and layer B2, since they are not updated in the frame corresponding to interface 34, using CLIENT compositing on layers B1 and B2 can trigger events such as... Figure 7 The CLIENT compositing scene caching optimization shown here not only avoids occupying the HWC VIG channel, but also effectively improves compositing efficiency.
[0326] Therefore, if we continue to use the metadata (such as indexes) of each layer obtained by SF to determine the compositing method for each layer, some inefficient compositing processing logic will occur. For example, layers such as B1 and B2 that have not been updated occupy the VIG channel in HWC, and cannot trigger CLIENT compositing scene cache optimization; or, updated layers such as C1 and C2 cannot be efficiently composed through HWC because the VIG channel is occupied.
[0327] Based on this, the embodiment provided in this application... Figure 11 In the scheme shown, SF can execute the following S1106 in a multi-window display scenario when at least one window content is updated, thereby effectively improving the compositing efficiency.
[0328] S1106 and SF adjust the layer index according to preset conditions.
[0329] For example, preset conditions may include at least one of the following:
[0330] The focused window is not obscured by any other window, and the number of layers associated with the focused window does not exceed the number of VIG channels.
[0331] Among them, the applications associated with any other window are different from the applications associated with the focused window.
[0332] Based on the example above, the focused window can be the window that receives user operation OP4, i.e., window C. The application associated with the focused window can be application C. The layers of application C can include layer C1 and layer C2. Any other window can include window B of application B.
[0333] In this example, SF can adjust the index of each layer if preset conditions are met.
[0334] As one possible implementation, the SF adjusts the indices of the various layers, which may include:
[0335] Insert the index of the layer associated with the first layer before the index of the layers in all windows.
[0336] Based on the description in S1105, before adjusting the index, the indexes of each layer can be as shown in Table 3.
[0337] In this way, under the preset conditions, SF can insert layers C1 and C2 of application C before the layers of other windows.
[0338] The layers of this other window may include layer B1 and layer B2 in window B of application B.
[0339] Therefore, the index of the adjusted layer can be as shown in Table 5.
[0340] Table 5
[0341] Layers index Layer C1 0 Layer C2 1 Layer B1 2 Layer B2 3 Layer A1 4 Layer A2 5 Layer D1 6 Layer D2 7
[0342] As shown in Table 5, in the adjusted layer indexes, the indices of layers C1 and C2 can be moved before the indices of all other layers in the window. For example, the index of layer C1 is 0, and the index of layer C2 is 1.
[0343] Correspondingly, due to the adjustment of the indices of layers C1 and C2, the indices of other layers are adjusted accordingly.
[0344] For example, since layers C1 and C2 are inserted before layer B1, the index of layer B1 is adjusted to 2. Correspondingly, the index of layer B2 is adjusted to 3.
[0345] In addition, the index of layer A1 is 4, the index of layer A2 is 5, the index of layer D1 is 6, and the index of layer D2 is 7.
[0346] Based on this adjusted index (as shown in Table 5), SF can interact with HWC to determine the composition method for each layer.
[0347] S1107, SF sends synthesis parameters 114 to HWC. HWC sends synthesis configuration 115 to SF.
[0348] For example, the composition parameter 114 may include metadata for each layer. The index in the metadata of each layer may be an adjusted index.
[0349] For example, composition parameter 114 may include: layer information and index of layer A1 (e.g., 4), layer information and index of layer A2 (e.g., 5), layer information and index of layer B1 (e.g., 2), layer information and index of layer B2 (e.g., 3), layer information and index of layer C1 (e.g., 0), layer information and index of layer C2 (e.g., 1), layer information and index of layer D1 (e.g., 6), and layer information and index of layer D2 (e.g., 7).
[0350] Correspondingly, HWC can determine the compositing method for each layer based on the index from smallest to largest.
[0351] For example, for layer C1 with index 0, HWC can determine whether to use the VIG channel to composite layer C1 based on the layer information of layer C1 indicating that layer C1 has window features.
[0352] Similarly, for layer C2 with index 1, HWC can determine whether to use the VIG channel to composite layer C2 based on the layer information of layer C2 indicating that layer C2 has window features.
[0353] As a result, the VIG channel of HWC is occupied by layers C1 and C2.
[0354] Furthermore, for layer B1 with index 2, HWC can determine whether to perform CLIENT compositing on layer B1 based on the layer information of layer B1, which indicates that layer B1 has window features and all VIG channels are occupied.
[0355] Similarly, for layer B2 with index 3, HWC can determine that layer B2 has window features and that all VIG channels are occupied, based on the layer information of layer B2.
[0356] For layers A1 and A2, HWC can determine to composite the two layers using HWC's DMA channels.
[0357] For layers D1 and D2, HWC can determine whether to perform CLIENT compositing on layers D1 and D2.
[0358] Therefore, HWC sends the composition configuration 115, which can be the composition method for each layer determined according to the adjusted layer index.
[0359] For example, composition configuration 115 can instruct: to composite layers C1 and C2 via the VIG channel of HWC, to composite layers A1 and A2 via the DMA channel of HWC, and to composite layers B1, B2, D1, and D2 via the GPU.
[0360] Therefore, SF can control the compositing process of each layer according to the compositing method indicated by compositing configuration 115.
[0361] S1108, SF sends the first composite layer information to HWC 116. HWC sends the first composite result to SF 117.
[0362] For example, the first composite layer information 116 may include: layer C1, layer C2, and metadata of layer A1 and layer A2.
[0363] Correspondingly, HWC can perform the first compositing process on layers C1 and C2 through the VIG channel. HWC can also perform the first compositing process on layers A1 and A2 through the DMA channel.
[0364] In this way, compared to compositing based on the index before adjustment, layers C1 and C2 in the window of application C with updated content can be efficiently composited using the VIG channel of HWC.
[0365] Therefore, HWC can send the first composite processing result 117, which includes layers C1, C2, A1, and A2, to SF.
[0366] S1109 and SF perform CLIENT synthesis according to synthesis configuration 115.
[0367] In conjunction with the description in S1107, the layers for CLIENT composition indicated by composition configuration 115 may include: layer B1, layer B2, and layers D1 and D2.
[0368] Combination Figure 7 The description of CLIENT scene caching optimization states that SF can directly reuse the compositing result of layer B1 in interface 33 as the compositing result of layer B1 in interface 34, based on the layer information of layer B1, which is the same as the layer information of layer B1 in the previous frame (such as interface 33).
[0369] Similarly, SF can reuse the composite results of layers B2, D1, and D2 in interface 33 as the composite results of the corresponding layers in interface 34.
[0370] Therefore, the SF can avoid compositing layers B1, B2, D1, and D2 again through the GPU, thereby improving CLIENT compositing efficiency by triggering CLIENT compositing scene cache optimization.
[0371] S1110 and SF obtained the synthesis result 118.
[0372] For example, after obtaining the synthesis results of HWC and GPU, SF can obtain the complete synthesis result 118 corresponding to the interface 34 based on each synthesis result. The synthesis result 118 may include all the display elements in the interface 34.
[0373] S1111 and SF send the composite result 118 to the display touch module.
[0374] S1112, Display touch module display interface 34.
[0375] Therefore, through this Figure 11 The scheme shown allows the electronic device to update the display of interface 34 upon receiving operation OP4.
[0376] like Figure 11 The explanations in S1106-S1110 state that in this example, when the content in the focus window is updated, SF adjusts the layer index, inserting the focus window's layer index before the indices of other windows. This allows the focus window's index to be smaller than that of other windows. Consequently, when determining the compositing method based on this index, HWC's VIG channel resources can be prioritized for the focus window with updated content, thus improving the compositing efficiency of the focus window. Furthermore, due to the corresponding adjustment of the indices of other windows, if the VIG channel is occupied by the focus window, other windows without updated content can be configured as CLIENT compositing based on their index. This effectively triggers... Figure 7 The CLIENT compositing scene caching optimization shown improves the compositing efficiency of windows that do not have content updates.
[0377] Therefore, this application improves the compositing efficiency of the focus window and other windows by reasonably adjusting the indexes of each layer, thereby improving the overall image compositing efficiency. This allows images to complete compositing processing faster, reducing time and resource overhead, and also supports faster image display.
[0378] refer to Figure 12 This is a schematic diagram illustrating a layer index adjustment process provided in an embodiment of this application.
[0379] SF can be through, for example Figure 12 The solution shown is implemented as follows: Figure 11S1106 in the example accurately adjusts the index of the layers in window C. In this example, window C is the focus window, and layer C1 is the layer in the focus window where content C is drawn. The corresponding layers applied to C include layer C1 and layer C2. The size of layer C2 is the same as the size of layer C1, taking the focus window size as an example.
[0380] like Figure 12 As shown, the solution may include:
[0381] S1201, Determine the application C corresponding to the focus window.
[0382] For example, SF can obtain the PID of the focused application and filter the name (i.e., package name) and focus index of the focused application (i.e., application C) in / proc / pid / cmdline.
[0383] In other implementations, SF can determine the user's current position using WMS and / or an input manager. SF can also determine the application C corresponding to the focused window by combining the positions of various application windows on the screen.
[0384] In this example, SF can also determine the layer corresponding to C, that is, the layer in the focus window.
[0385] Generally, there is a relationship between the application and the name of the corresponding layer.
[0386] Thus, in some implementations, SF can traverse all current layers and determine the layers in the focus window, including layer C1 and layer C2, based on the names of each layer, including the package name of application C, or the layer names being related to the package name of application C.
[0387] Furthermore, the name of the layer used to draw the function bar can include a corresponding identifier. Therefore, SF can use this identifier to determine that layer C2 is the layer in the focus window where the function bar is drawn.
[0388] In this application, the preset conditions include that the focus window is not obscured by any other window and that the number of layers of the application related to the focus window does not exceed the number of VIG channels.
[0389] In this way, SF can execute S1204, that is, adjust the index of each layer of application C, if both of the above conditions are met at the same time.
[0390] In some embodiments, SF can execute the two conditions mentioned above sequentially.
[0391] For example, SF can determine whether the number of layers associated with the focused window does not exceed the number of VIG channels if the focused window is not obscured by any other window. However, if the focused window is obscured by at least one window, SF may not perform the determination of whether the number of layers associated with the focused window exceeds the number of VIG channels.
[0392] For example, SF can determine whether the focused window is obscured by any other window if the number of layers associated with the focused window does not exceed the number of VIG channels. However, if the number of layers associated with the focused window exceeds the number of VIG channels, SF may choose not to perform the determination of whether the focused window is obscured by any other window.
[0393] In other embodiments, such as Figure 12 As shown, SF can execute the two conditional judgments mentioned above simultaneously, and execute S1204 if both conditional judgments are true.
[0394] S1202. Ensure that the focus window C does not overlap with other windows.
[0395] For example, take a layer C2 in the focused window C that is used to draw the function bar, which is the same size as the window C.
[0396] Therefore, SF can traverse all layers of other window applications (such as application B), and determine that the focused window C does not overlap with other windows if layer C2 does not overlap with any of the layers of application B.
[0397] As an example, SF can determine the parameters of the four sides of layer C2, such as Left1, Right1, Top1, and Bottom1. For any layer (Left2, Right2, Top2, Bottom2) of application B, if Left1 is greater than Right2, or Right1 is greater than Left2, or Top1 is greater than Bottom2, or Bottom1 is greater than Top2, then layer C2 will not overlap with any other layer of application B. That is, the focused window C will not overlap with other windows.
[0398] S1203. Determine that the number of layers in application C is less than the number of VIG channels.
[0399] For example, referring to the description in S1201, SF can determine the number of layers corresponding to application C based on the package name of application C. For instance, SF can determine that the layers corresponding to application C may include layer C1 and layer C2, with a total quantity of 2.
[0400] SF can also determine the number of VIG channels based on the hardware capabilities provided by HWC. For example, the number of VIGs is 2.
[0401] It is understandable that if the number of layers in C is less than the number of channels in VIG, then according to Figure 11 The method shown, by inserting the indices of layers C1 and C2 forward, ensures that all layers in the focus window can be quickly composited using the VIG channel. This avoids the problem of slow compositing of some layers in the focus window due to insufficient VIG channels, which requires some layers to be composited using CLIENT.
[0402] Therefore, if both S1202 and S1203 are true, SF can execute S1204.
[0403] S1204. Adjust the indexes of each layer in application C.
[0404] Based on the description in S1106, SF can adjust the indexes of layers C1 and C2 of application C to the top of all windows' layers.
[0405] For example, SF can iterate through all layers and determine the layer with the smallest index and window characteristics based on the layer information of each layer.
[0406] Based on the relationship between layers and indices shown in Table 3, SF can determine whether a layer is a window layer by sequentially determining the layer in ascending order of the index.
[0407] For example, SF can start traversing from layer B1 with index 0. SF can determine that layer B1 is the layer with the smallest index that has window features, based on the layer information of layer B1 indicating that layer B1 contains window features.
[0408] Therefore, SF can insert layers C1 and C2 of application C into the header of layer B1 and reconfigure the indexes of each layer.
[0409] For example, referring to Table 5, after SF inserts layers C1 and C2 (which apply C) at the top of layer B1, the index of layer C1 is replaced with the original index of layer B1, which is 0. The index of layer C2 is changed to 1. Since two layers are inserted at the top, the index of layer B1 is increased by 2, becoming 2. The indices of other layers are similarly calculated.
[0410] Therefore, SF can complete the index adjustment of each layer of application C.
[0411] Thus, through such Figure 11 and Figure 12The proposed solution, in a multi-window display scenario, allows SF to adjust the indices (i.e., Z-order) of each layer in a multi-window display, enabling efficient compositing of the layers of windows with updated content via HWC. Furthermore, due to the layer index adjustment, layers of windows without updated content can be composited via the GPU for CLIENT compositing, thereby improving compositing efficiency by triggering CLIENT compositing scene caching optimization.
[0412] In the specific implementation, we take application B as a music application and application C as a browser application with a 90Hz refresh rate as an example.
[0413] refer to Figure 13 This is a comparative schematic diagram illustrating the effects of a solution provided in an embodiment of this application.
[0414] In such Figure 11 Before the proposed solution took effect, the GPU rendering pipeline had a high composition processing frequency, resulting in high power consumption. In the case of... Figure 11 After the solution shown takes effect, the compositing frequency of the GPU rendering pipeline is significantly reduced, resulting in a corresponding reduction in power consumption.
[0415] The following data can be obtained through actual measurement: (The following is a continuation of the data...) Figure 11 The solution shown, after adjusting the layer index, can improve compositing efficiency and reduce power consumption by 60mA by triggering CLIENT compositing scene cache optimization.
[0416] It is understood that the electronic device provided in this application embodiment includes hardware structures and / or software modules corresponding to perform each function in order to achieve the above-mentioned functions. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.
[0417] This application embodiment can divide the above-described electronic device into functional modules based on the method example described above. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware or as software functional modules. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.
[0418] The foregoing mainly describes the solutions provided by the embodiments of this application from the perspective of various functional modules. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0419] The integrated modules described above can be implemented in hardware or as software functional modules. It should be noted that the module division in this embodiment is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used.
[0420] For example, Figure 14 A schematic diagram of the composition of an electronic device 1400 is shown. (As shown...) Figure 14 As shown, the electronic device 1400 may include a processor 1401 and a memory 1402. The memory 1402 is used to store computer execution instructions. Exemplarily, in some embodiments, when the processor 1401 executes the instructions stored in the memory 1402, the electronic device 1400 may perform any of the methods shown in the above embodiments.
[0421] It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.
[0422] Figure 15 A schematic diagram of a chip system 1500 is shown. The chip system 1500 may include a processor 1501 and a communication interface 1502, used to support related devices in implementing the functions involved in the above embodiments. In one possible design, the chip system also includes a memory for storing necessary program instructions and data of the electronic device. This chip system may be composed of chips or may include chips and other discrete devices. It should be noted that in some implementations of this application, the communication interface 1502 may also be referred to as an interface circuit.
[0423] It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.
[0424] This application also provides a computer-readable storage medium storing a computer program thereon. When executed by a computer, the computer program implements the method flow related to the electronic device in any of the above method embodiments. Specifically, the computer can be the aforementioned electronic device.
[0425] This application also provides a computer program or a computer program product including a computer program, which, when executed on a computer, causes the computer to implement the method flow related to the electronic device in any of the above method embodiments. Specifically, the computer can be the aforementioned electronic device.
[0426] The functions, actions, operations, or steps in the above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. When implemented using software programs, they can be implemented, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or include one or more data storage devices such as servers and data centers that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).
[0427] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of the application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.
Claims
1. An image processing method, characterized in that, The method is applied to an electronic device, and the method includes: Display a first interface, which includes a first window and a second window; Receive a first operation, which is an operation on the first window; In response to the first operation, a first compositing process is performed on at least one layer of the first window to obtain a first compositing result; a second compositing result is obtained, the second compositing result corresponding to the second window; Based on the first synthesis result and the second synthesis result, a second interface is displayed.
2. The method according to claim 1, characterized in that, The first compositing process on at least one layer of the first window includes: At least one layer of the first window is composited using the hardware compositor HWC of the electronic device.
3. The method according to claim 2, characterized in that, The compositing process for at least one layer of the first window includes: The first compositing process is performed on at least one layer of the first window through the video input gateway VIG channel of the HWC.
4. The method according to any one of claims 1-3, characterized in that, Before displaying receipt of the first operation, the method further includes: The second synthesis result is obtained through a second synthesis process; The second synthesis process includes: The layer to be processed is transmitted to the graphics processing unit (GPU) of the electronic device so that the GPU can perform composite processing on the layer to be processed.
5. The method according to claim 4, characterized in that, After receiving the first operation, the method further includes: Determine that at least one layer in the second window will be subjected to the second compositing process.
6. The method according to claim 5, characterized in that, After determining that at least one layer in the second window will undergo a second compositing process, the method further includes: The layer information of the second layer at the first moment is determined to be the same as the layer information of the second layer at the second moment; wherein, the first moment is earlier than the moment when the first interface is displayed, and the second moment is later than the moment when the first operation is received; the second layer is included in at least one layer in the second window; the layer information is used to indicate the display content of the corresponding layer. The process of obtaining the second synthesis result includes: Based on the layer information of the first layer at the first time point, which is the same as the layer information of the first layer at the second time point, the second composite result is obtained so as to display the second interface based on the second composite result.
7. The method according to claim 6, characterized in that, After receiving the first operation and before obtaining the second synthesis result, the method further includes: Obtain first metadata and second metadata; wherein, the first metadata is used to indicate the layer information and index of the first layer in the first window; the second metadata is used to indicate the layer information and index of the second layer in the second window; the index is used to indicate the Z-order of the corresponding layer; Based on the first metadata and the second metadata, the compositing method of the first layer and the second layer is determined; the compositing method includes the first compositing process and the second compositing process.
8. The method according to claim 7, characterized in that, Determining the compositing method of the first layer and the second layer based on the first metadata and the second metadata includes: Based on the layer information of the first layer, the first layer is determined to be a window layer; the size of the window layer is smaller than the size of the electronic device's display screen. Based on the layer information of the second layer, the second layer is determined to be the window layer; Based on the fact that the index of the first layer is less than the layer index of the second layer, the compositing method of the first layer is determined to be the first compositing method corresponding to the first compositing process.
9. The method according to claim 8, characterized in that, If the number of layers in the first window reaches a first quantity, the compositing method of the second layer is determined to be the second compositing method corresponding to the second compositing process; the first quantity is the number of VIG channels supported by the HWC of the electronic device.
10. The method according to any one of claims 7-9, characterized in that, Before obtaining the first metadata and the second metadata, the method further includes: Based on the first operation, third metadata and fourth metadata are generated; the third metadata indicates that the index of the first layer is greater than the index of the second layer.
11. The method according to claim 10, characterized in that, Under the condition that the preset conditions are met, the first metadata is obtained according to the third metadata; and the second metadata is obtained according to the fourth metadata.
12. The method according to claim 11, characterized in that, The preset conditions include at least one of the following: The first window is not obscured by the second window, and the number of layers in the first window does not exceed the number of VIG channels supported by the HWC of the electronic device.
13. The method according to any one of claims 1-12, characterized in that, Before displaying the first interface, the method further includes: Receive a second operation, the second operation instructing the electronic device to display the first window on the display screen; A third operation is received, which instructs the electronic device to display the second window on the display screen.
14. The method according to claim 13, characterized in that, The third operation is received earlier than the second operation.
15. An electronic device, characterized in that, The electronic device includes: a memory and one or more processors; the memory and the processors are coupled. The memory is used to store computer program code, which includes computer instructions. When the processor executes the computer instructions, the electronic device performs the method as described in any one of claims 1-14.
16. A chip system, characterized in that, The chip system is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuits and the processors are interconnected via lines; the interface circuits are used to receive signals from the memory of the electronic device and send the signals to the processors, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device performs the method as described in any one of claims 1-14.
17. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-14.
