Method and device for optimizing layer synthesis, electronic equipment and storage medium

Abstract

The invention provides a method and a device for optimizing layer synthesis, electronic equipment and a storage medium, the method is applied to the electronic equipment, and the electronic equipment is provided with hardware for realizing layer synthesis. The method comprises the following steps: acquiring a to-be-synthesized layer set and metadata information respectively corresponding to each to-be-synthesized layer in the to-be-synthesized layer set; matching the metadata information based on a system side synthesis strategy configured on the electronic equipment, and determining a to-be-synthesized layer corresponding to the matched metadata information as a target layer; and when the to-be-synthesized layer set is processed based on a hardware side synthesis strategy provided by the hardware, preferentially selecting the target layer for the hardware to perform layer synthesis. According to the method provided by the invention, by adopting the system side synthesis strategy, the layer selection can be realized from the operating system level, and the limitation of the hardware side synthesis strategy on the layer selection is eliminated.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a method, device, electronic device and storage medium for optimizing layer synthesis. Background Art

[0002] Layer compositing is a user interface rendering technology that combines all visual content created by the operating system and applications into a final image that can be displayed on the screen. These visual contents can be any combination of layers such as icons, taskbars, desktop backgrounds, high-definition video layers, pictures, or 3D games rendered by the Graphics Processing Unit (GPU). The faster and smoother the layer compositing process, the faster the electronic device responds to user operations, and the more vivid and smooth the performance.

[0003] In the related art, by equipping electronic devices with hardware specifically used for layer synthesis, in the process of synthesizing all the layers to be synthesized in the interface of the electronic device, the hardware produced by different hardware manufacturers will select some of the layers to be synthesized to perform layer synthesis on the hardware according to their respective hardware-side synthesis strategies, so as to reduce the burden of the central processing unit (CPU) and the graphics processor in processing layer synthesis.

[0004] However, the hardware-side synthesis strategies provided by hardware manufacturers cannot adapt to the flexible and changeable layer synthesis optimization scenarios. For example, when faced with scenarios that require specific optimization effects such as improving the visual effects of image display in the user interface or improving the smoothness of image display, the hardware-side synthesis strategies provided by hardware manufacturers cannot correctly select the layer objects that need to be synthesized on the hardware according to the optimization requirements of the specific scenario to achieve the corresponding optimization effect. Summary of the invention

[0005] In view of this, the present disclosure provides a method, device, electronic device and storage medium for optimizing layer synthesis, so as to get rid of the limitation of hardware-side synthesis strategy on layer selection.

[0006] According to a first aspect of an embodiment of the present disclosure, a method for optimizing layer synthesis is provided, which is applied to an electronic device, wherein the electronic device is equipped with hardware for implementing layer synthesis, and the method includes:

[0007] Obtaining a set of layers to be synthesized and metadata information corresponding to each layer to be synthesized in the set of layers to be synthesized;

[0008] Matching the metadata information based on the system-side synthesis strategy configured on the electronic device, and determining the to-be-synthesized layer corresponding to the matched metadata information as the target layer;

[0009] When the set of layers to be synthesized is processed based on the hardware-side synthesis strategy provided by the hardware, the target layer is preferentially selected for the hardware to perform layer synthesis.

[0010] Optionally, the method further includes: setting the priority of each layer to be synthesized in the set of layers to be synthesized based on the system-side synthesis strategy, so that the priority of the target layer is higher than the priority of other layers in the set of layers to be synthesized;

[0011] The priority level is positively correlated with the probability of being selected for layer synthesis by the hardware.

[0012] Optionally, the setting of the priority of each to-be-synthesized layer in the to-be-synthesized layer set includes:

[0013] Adding high priority marking information to the target layer, wherein the high priority marking information is used to indicate that the priority of the target layer is higher than the priority of other layers in the set of layers to be synthesized; and / or,

[0014] Add low priority marking information to other layers in the set of layers to be synthesized, wherein the low priority marking information is used to indicate that the priority of other layers in the set of layers to be synthesized is lower than the priority of the target layer.

[0015] Optionally, the method further includes: sending the remaining layers in the set of layers to be synthesized that are not selected by the hardware-side synthesis strategy for layer synthesis by the hardware to a graphics processor installed on the electronic device for layer synthesis.

[0016] Optionally, the method further includes: when it is determined based on the hardware-side synthesis strategy that the processing resource amount of the hardware exceeds the resource consumption of all target layers, selecting a corresponding number of other layers from the set of layers to be synthesized according to the remaining processing resources for the hardware to perform layer synthesis;

[0017] When it is determined based on the hardware-side synthesis strategy that the processing resources of the hardware are less than the resource consumption of all target layers, the target layers not selected by the hardware-side synthesis strategy are sent to a graphics processor installed on the electronic device for layer synthesis.

[0018] Optionally, the method further includes: editing the system-side synthesis strategy according to a received editing instruction for the system-side synthesis strategy.

[0019] Optionally, it further includes: generating the system-side synthesis strategy locally in the electronic device; and / or,

[0020] The system-side synthesis strategy sent from the cloud is received and stored in the electronic device.

[0021] Optionally, the method for generating the system-side synthesis strategy includes: determining, according to a layer synthesis method with the best display effect determined in a test environment, a layer for layer synthesis by the hardware in the layer synthesis method, and generating the system-side synthesis strategy based on the determined layer; and / or,

[0022] Acquire actual operation data of the electronic device or other devices of the same model as the electronic device, determine a freeze moment in the layer synthesis process according to the actual operation data, and generate the system-side synthesis strategy based on the layer in the synthesis process at the freeze moment.

[0023] According to a second aspect of an embodiment of the present disclosure, there is provided a device for optimizing layer synthesis, which is applied to an electronic device, wherein the electronic device is equipped with hardware for implementing layer synthesis, and the device includes:

[0024] An acquisition module, used for acquiring a set of layers to be synthesized and metadata information corresponding to each layer to be synthesized in the set of layers to be synthesized;

[0025] A target layer determination module is used to match the metadata information based on the system-side synthesis strategy configured on the electronic device, and determine the to-be-synthesized layer corresponding to the matched metadata information as the target layer;

[0026] The resource allocation module is used to preferentially select the target layer for the hardware to perform layer synthesis when processing the set of layers to be synthesized based on the hardware-side synthesis strategy provided by the hardware.

[0027] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0028] processor;

[0029] a memory for storing a computer program executable by the processor;

[0030] The processor is configured to execute the computer program in the memory to implement the steps of the method described in the first aspect.

[0031] According to a fourth aspect of an embodiment of the present disclosure, there is provided a non-temporary computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0032] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0033] After obtaining the set of layers to be synthesized and the metadata information corresponding to each layer to be synthesized in the set of layers to be synthesized, the target layer to be synthesized by the hardware is determined according to the metadata information of the layer to be synthesized through the system-side synthesis strategy configured on the electronic device, and the hardware-side synthesis strategy provided by the hardware is instructed to give priority to the target layer for the hardware to perform layer synthesis. When it is necessary to achieve a specific optimization effect of layer synthesis, by adopting the system-side synthesis strategy, the selection of the target layer can be realized from the operating system level, getting rid of the restrictions of the hardware-side synthesis strategy on layer selection, so that the selected target layer can achieve the corresponding specific optimization effect after layer synthesis by the hardware.

[0034] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0036] Figure 1 It is a schematic diagram of a hardware architecture for implementing layer synthesis according to an exemplary embodiment of the present disclosure.

[0037] Figure 2 It is a schematic diagram of a layer synthesis method shown in the present disclosure according to an exemplary embodiment.

[0038] Figure 3 The present invention is a flowchart of a method for optimizing layer synthesis according to an exemplary embodiment.

[0039] Figure 4 It is a schematic diagram of a method for optimizing layer synthesis according to an exemplary embodiment of the present disclosure.

[0040] Figure 5 It is a schematic diagram of a video playback interface shown in accordance with an exemplary embodiment of the present disclosure.

[0041] Figure 6 It is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure.

[0042] Figure 7 The present invention is a block diagram of a device for optimizing layer synthesis according to an exemplary embodiment. DETAILED DESCRIPTION

[0043] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0044] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0045] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0046] Figure 1 It is a schematic diagram of a hardware architecture for implementing layer synthesis according to an exemplary embodiment of the present disclosure.

[0047] like Figure 1 As shown, the hardware architecture for implementing layer synthesis may include GPU 101 and CPU 102. Specifically, when the electronic device is equipped with GPU 101 and CPU 102, in the task of implementing layer synthesis, the layer synthesis may be performed by the GPU 101 or the CPU 102, or by the GPU 101 and the CPU 102 together.

[0048] In order to ease the processing burden of the CPU and GPU, related technologies have proposed that layer synthesis can be achieved through dedicated hardware. During the layer synthesis process, the hardware can accelerate the synthesis of layers to provide a fast layer synthesis effect. Figure 1As shown, the hardware can be a data processing unit (Data Processing Unit, DPU103) for implementing layer synthesis. In the process of synthesizing all the layers to be synthesized on the electronic device, at least a part of the layers to be synthesized can be synthesized on DPU103 to reduce the burden of the central processing unit (Central Processing Unit, CPU102) and the graphics processing unit (Graphics Processing Unit, GPU101) in processing layer synthesis.

[0049] Among them, DPU103 can be used as an independent hardware accelerator or as a module integrated in other processors, for example, it can be integrated in a module in any processor such as a system-level chip, a field-programmable gate array (FPGA) and an application-specific integrated circuit (ASIC). DPU103 can be used as hardware specifically used for layer synthesis in the present disclosure. Of course, in addition to DPU103, other hardware capable of realizing layer synthesis can also replace DPU103 to be applicable to the technical solution of the present disclosure, and the present disclosure does not impose any restrictions.

[0050] Below Figure 1 As an example, the hardware architecture shown in Figure 2 This article introduces the process of implementing layer synthesis in related technologies. Figure 2 As shown, the buffer data 201 is the rendered image pixel data stored in the frame buffer. Before synthesizing the layer, the buffer data 201 at a specific position can be read from the frame buffer, and data conversion can be performed on the read buffer data 201. For example, the color value can be converted from the original format to the color space required by the layer to be synthesized to ensure that the layer to be synthesized is consistent with other layers in color, and the metadata information of the layer to be synthesized, such as position, size and other metadata information, can be set. Then, after the above processing, these obtained buffer data 201 can form a layer set 202 to be synthesized. According to the hardware-side synthesis strategy provided by the manufacturer of DPU103, the metadata information of the layer to be synthesized is matched, and the layer set 203 that needs to be synthesized by DPU103 is obtained from the layer set 202 to be synthesized, and it is sent to DPU103 for layer synthesis, and the remaining layer set 204 is sent to other hardware 206 for layer synthesis, where other hardware 206 can refer to CPU102 or GPU103.

[0051] Since the hardware produced by different hardware manufacturers has different characteristics and design principles, the hardware manufacturers will design corresponding hardware-side synthesis strategies for the hardware architecture. For example, layers 1-10 (labeled 1-10 is used to distinguish different layers) are a set of layers to be synthesized. According to the hardware-side synthesis strategy provided by DPU103, layers 1-5 are selected to be synthesized by DPU103, and layers 6-10 are synthesized by other hardware 206. However, in order to achieve a specific optimization effect of layer synthesis, the technicians in this field hope to give priority to DPU103 for layer synthesis of layers 1-3 and 7-8, and the remaining layers 4-6 and 9-10 are synthesized by other hardware 206. However, since the hardware-side synthesis strategies provided by some hardware manufacturers are in a closed-source state, the developers' in-depth understanding and customization capabilities of the algorithms and implementation details are limited, resulting in the inability of developers to improve the hardware-side synthesis strategies, and thus the inability to achieve a specific optimization effect of layer synthesis by changing the algorithm logic of the hardware-side synthesis strategies.

[0052] In order to solve the above problems, the present invention proposes a method for optimizing layer synthesis. Without changing the algorithm logic of layer synthesis of the hardware-side synthesis strategy, a system-side synthesis strategy is customized to replace the hardware-side synthesis strategy to select the target layer for layer synthesis by hardware. When it is necessary to achieve a specific optimization effect of layer synthesis, the system-side synthesis strategy is used to instruct the hardware-side synthesis strategy to select the target layer determined by the system-side synthesis strategy, so that the selected target layer can achieve the corresponding specific optimization effect after layer synthesis is performed by the hardware.

[0053] Next, the embodiments of the present disclosure are described in detail.

[0054] like Figure 3 As shown, Figure 3 1 is a flowchart of a method for optimizing layer synthesis according to an exemplary embodiment of the present disclosure, comprising the following steps:

[0055] Step 301: Obtain a set of layers to be synthesized and metadata information corresponding to each layer to be synthesized in the set of layers to be synthesized.

[0056] The to-be-synthesized layer disclosed in the present invention may refer to a layer that has been processed in the rendering stage and is to be synthesized with other layers in the next stage to obtain a final synthesized image output by display hardware.

[0057] Before compositing the layer to be composited with other layers, the metadata information of the layer will be added to the layer to be composited. The metadata information may include but is not limited to the layer name, layer position, layer size, transparency, blending mode and other properties used to describe and manage the layer. In addition to the above common metadata information, those skilled in the art can also define and use other custom properties according to actual needs, which can be used to describe and manage the layer in more detail.

[0058] Step 302: Match the metadata information based on the system-side synthesis strategy configured on the electronic device, and determine the to-be-synthesized layer corresponding to the matched metadata information as the target layer.

[0059] In an illustrated embodiment, a system-side synthesis strategy is configured on the electronic device, and the electronic device is equipped with hardware dedicated to layer synthesis, such as the aforementioned DPU. Figure 4 As shown, Figure 2 Compared with the layer synthesis process shown in the figure, the difference is that before selecting a layer set 404 for layer synthesis by DPU 103 from the layer set 202 to be synthesized according to the hardware-side synthesis strategy, this embodiment adds a system-side synthesis strategy on the electronic device to process the layer set 202 to be synthesized, and configures the target layer that needs to be synthesized by hardware according to the metadata information of each layer in the layer set 202 to be synthesized. According to the hardware-side synthesis strategy, a layer set 404 for layer synthesis by DPU 103 is selected from the layer set 403 to be synthesized after being processed by the system-side synthesis strategy, and the remaining layer set 405 is sent to other hardware 206 for layer synthesis.

[0060] For example, in an actual layer synthesis scenario, it has been found through experiments that the video in HDR video format can significantly improve the video display effect when processed by DPU103. Accordingly, the system-side synthesis strategy can match the layer containing the HDR video format in the metadata information and configure it as the target layer that needs to be synthesized by DPU103.

[0061] In this embodiment, Figure 2 Compared with the technical solution shown, this embodiment customizes the system-side synthesis strategy at the operating system level, which can instruct the hardware-side synthesis strategy to correctly select the HDR format layer as the layer to be synthesized by DPU103, thereby solving the uncontrollable problems caused by using the hardware-side synthesis strategy alone and ensuring that all HDR format layers are synthesized by DPU103.

[0062] Step 303: When processing the set of layers to be synthesized based on the hardware-side synthesis strategy provided by the hardware, preferentially select the target layer for the hardware to perform layer synthesis.

[0063] like Figure 4 As shown, the hardware-side synthesis strategy obtains the set of layers to be synthesized 403 after being processed by the system-side synthesis strategy. The processed set of layers to be synthesized 403 includes the target layer and other layers to be synthesized determined by the system-side synthesis strategy. When the processed set of layers to be synthesized 403 is processed according to the hardware-side synthesis strategy provided by the hardware side, the target layer is preferentially selected for the DPU to perform layer synthesis.

[0064] This embodiment utilizes the system-side synthesis strategy to select the target layer that needs to be synthesized by the hardware, and instructs the hardware-side synthesis strategy to give priority to the target layer for the hardware to perform layer synthesis. Therefore, when faced with a specific scenario of optimizing layer synthesis, the system-side synthesis strategy helps the hardware-side synthesis strategy to correctly select the target layer object that needs to be synthesized on the hardware, thereby achieving a specific optimization effect.

[0065] In an illustrated embodiment, after matching the target layer according to the system-side synthesis strategy, the priority of each layer to be synthesized in the set of layers to be synthesized is set so that the priority of the target layer is higher than the priority of other layers in the set of layers to be synthesized; wherein the priority is positively correlated with the probability of being selected for layer synthesis by the hardware.

[0066] For example, the priority of the layers to be synthesized can be reflected by adjusting the order of the layers in the set of layers to be synthesized, placing the target layer in front of other layers in the set of layers to be synthesized, and instructing the hardware-side synthesis strategy to give priority to the target layer in the front of the set of layers to be synthesized. For example, by adjusting the order of layers 1-10 in the set of layers to be synthesized, placing the target layers 6-10 in the set of layers to be synthesized in front of other layers in the set of layers to be synthesized, and instructing the hardware-side synthesis strategy to give priority to the target layers 6-10 in the front of the set of layers to be synthesized.

[0067] For another example, in an illustrated embodiment, the priority of different layers to be synthesized can be reflected by adding tag information to each layer. Optionally, high priority tag information is added to the target layer, wherein the high priority tag information is used to indicate that the priority of the target layer is higher than the priority of other layers in the set of layers to be synthesized. The marked priority information can be added to the metadata information of the target layer, and the metadata information of each layer is matched according to the hardware-side synthesis strategy, indicating that the hardware processes the target layer with priority tag information. For example, for layers 1-10 of the set of layers to be synthesized, the metadata information of layers 1-5 is added with high priority tag information, then when the metadata information of each layer is matched according to the hardware-side synthesis strategy, the hardware is instructed to process layers 1-5 with priority tag information. Optionally, low priority tag information is added to other layers of the set of layers to be synthesized, wherein the low priority tag information is used to indicate that the priority of other layers in the set of layers to be synthesized is lower than the priority of the target layer. Similar to the way of adding high priority tag information to the target layer, by adding low priority tag information to other layers of the set of layers to be synthesized, the metadata information of each layer can also be matched according to the hardware-side synthesis strategy, indicating that the hardware is preferentially processed. The target layer that does not contain low priority tag information. For example, for layers 1-10 of the layer set to be synthesized, low priority information is added to the metadata information of layers 6-10. When matching the metadata information of each layer according to the hardware-side synthesis strategy, the target layer for layer synthesis by hardware is preferentially selected from layers 1-5 that do not contain low priority information. Of course, there can also be other ways to reflect the processing priority of different layers. For example, layers that do not require hardware processing can be marked as marking information that forces other hardware to process them. The other hardware can be a graphics processor.

[0068] The above embodiment shows a method for processing a target layer according to a hardware-side synthesis strategy. The present disclosure also provides a method for processing other layers in a set of layers to be synthesized according to a hardware-side synthesis strategy.

[0069] In an illustrated embodiment, the remaining layers in the set of layers to be synthesized that are not selected by the hardware-side synthesis strategy as layers to be synthesized by hardware are sent to other hardware on the electronic device for layer synthesis, and the layers synthesized by other hardware are synthesized with the target layers selected by the hardware-side synthesis strategy, where the other hardware may be a graphics processor. For example, for layers 1-10 of the set of layers to be synthesized, layers 1-5 are selected as layers to be synthesized by hardware according to the hardware-side synthesis strategy, and the remaining layers 6-10 are sent to other hardware on the electronic device for layer synthesis, and the layers synthesized by other hardware are synthesized with the target layers selected by the hardware-side synthesis strategy.

[0070] In an illustrated embodiment, if the processing resources of the hardware determined according to the hardware-side synthesis strategy exceed the resource consumption of all target layers, the hardware-side synthesis strategy can select a corresponding number of other layers from the set of layers to be synthesized based on the remaining processing resources for the hardware to perform layer synthesis. Through this embodiment, the processing resources of the hardware can be utilized to the maximum extent to improve the efficiency of layer synthesis. For example, for layers 1-10 of the set of layers to be synthesized, if layers 1-5 are selected as target layers according to the system-side synthesis strategy, but the processing resources of the hardware determined according to the hardware-side synthesis strategy exceed the resource consumption of all target layers, the hardware-side synthesis strategy can select a corresponding number of other layers from layers 6-10 based on the remaining processing resources for the hardware to perform layer synthesis.

[0071] In another illustrated embodiment, it is also possible to face a situation where the processing resources of the hardware are less than the resource consumption of all target layers. In this case, it cannot be guaranteed that all target layers are synthesized by the hardware. Therefore, when it is determined according to the hardware-side synthesis strategy that the processing resources of the hardware are less than the resource consumption of all target layers, the target layers not selected by the hardware-side synthesis strategy can be sent to the graphics processor installed on the electronic device for layer synthesis. For example, for layers 1-10 of the set of layers to be synthesized, if layers 1-5 are selected as target layers according to the system-side synthesis strategy, but it is determined according to the hardware-side synthesis strategy that the processing resources of the hardware are less than the resource consumption of all target layers, the hardware-side synthesis strategy can select layers for layer synthesis by hardware from layers 1-5, and send the remaining target layers to the graphics processor installed on the electronic device for layer synthesis. It should be noted that as to how to determine the target layers that cannot be synthesized by the hardware according to the hardware-side synthesis strategy, the processing methods of the hardware-side synthesis strategy provided by each hardware manufacturer are different, and this disclosure will not be repeated here.

[0072] like Figure 5 As shown, Figure 5 501, a video playback interface, a video area 502, a search bar 503, a browsing area 504, an icon 505, and other different layers form a final image that can be displayed on the screen. Each layer has metadata information for describing the attribute information of the layer. The system-side synthesis strategy can match the target layer that needs to be synthesized by hardware according to the metadata information of each layer. Figure 5 This paper introduces how the optimized layer synthesis method proposed in the present invention can adapt to flexible and changeable application scenarios to achieve optimization effects in specific scenarios.

[0073] For example, the human eye generally browses the middle area of ​​an electronic device, so the layer located in the middle can be used as the target layer. In specific implementation, the system-side synthesis strategy can mark the layer located in the middle in the metadata information as the target layer. Figure 5 In the video playback interface, the layer where the video area 502 located in the middle area of ​​the interface is located can be marked as the target layer. When the to-be-synthesized layer set is processed according to the hardware-side synthesis strategy provided by the hardware side, the target layer is preferentially selected for the hardware to perform layer synthesis. In this way, the layer synthesis efficiency of the main browsing area of ​​the user can be guaranteed to be the highest, thereby achieving the best display effect and improving the user experience.

[0074] In addition, the system-side synthesis strategy marks different sequential layers in the same position area as target layers, so that the hardware-side synthesis strategy can give priority to different layers in the same position area, and can also solve the abnormal display effect problem caused by the hardware-side synthesis strategy provided by different hardware installed on the electronic device in the related art processing different parts of the layers in the same position area. For example, in the related art, when different layers in the same area use different hardware-side synthesis strategies to synthesize the pixels of the layer with the pixels of the layer below, the difference in synthesis mode may cause the screen flickering problem.

[0075] Still taking the video in the HDR video format as an example, by specifying a layer of a specific format as the target layer through the system-side synthesis strategy, it can be ensured that the layer that is more suitable for hardware processing is preferentially selected by the hardware-side synthesis strategy as the layer for layer synthesis by the hardware.

[0076] In an illustrated embodiment, the system-side synthesis strategy can also determine the target layer according to the size of the layer, and prioritize the larger layer as the target layer, which can ensure that the more complex layer is preferentially selected by the hardware-side synthesis strategy as the layer for layer synthesis by the hardware, thereby improving the efficiency of layer synthesis and making the layer synthesis process faster and smoother. Figure 5 In the video playback interface, the metadata information of each layer records the size information of the layer. Through the system-side synthesis strategy, the layer where the video area 502 with a larger size is located can be preferentially selected as the target layer.

[0077] It is not difficult to see from the embodiments shown above that when faced with scenarios that require specific optimization effects such as improving the image display visual effect of the user interface or improving the image display smoothness, the present invention determines the target layer based on the system-side synthesis strategy, and helps the hardware-side synthesis strategy to correctly select the layer objects that need to be synthesized on the hardware, thereby achieving specific optimization effects.

[0078] In an illustrated embodiment, the user may edit the system-side synthesis strategy according to the scene requirements of different optimization effects and according to the received editing instructions for the system-side synthesis strategy, thereby continuously optimizing the system-side synthesis strategy.

[0079] In an illustrated embodiment, a system-side synthesis strategy can be generated locally on the electronic device according to the scenario requirements of different electronic devices. For example, for smart electronic devices such as mobile phones and tablets, better display visual effects are usually required. Therefore, a system-side synthesis strategy for improving the display visual effect can be configured. For electronic devices with lower configurations, the image display smoothness of the electronic device can be improved by configuring a system-side synthesis strategy for improving the image display smoothness. Of course, the system-side synthesis strategy can also have different optimization effects such as improving the image display visual effect of the user interface and improving the image display smoothness, and the present disclosure is not limited to this.

[0080] In another illustrated embodiment, the system-side synthesis strategy stored in the cloud server can be sent to each electronic device and stored in the electronic device. Of course, the electronic device can also edit the system-side synthesis strategy according to the received editing instruction for the system-side synthesis strategy.

[0081] In an illustrated embodiment, the electronic device can not only generate a corresponding system-side synthesis strategy locally according to actual application requirements, but also receive the system-side synthesis strategy sent by the cloud server to the electronic device.

[0082] The present disclosure also provides a method for generating a system-side synthesis strategy. In an illustrated embodiment, the layer synthesis method for the best display effect determined in the test environment can be used to determine the layer for which the hardware performs layer synthesis in the layer synthesis method, and the system-side synthesis strategy is generated based on the determined layer. It should be noted that the test environment can be either in the test environment of the backend server or in the test environment of the electronic device.

[0083] In another illustrated embodiment, actual operation data of the electronic device or other devices of the same model as the electronic device is obtained, a freeze moment in the layer synthesis process is determined according to the actual operation data, and a system-side synthesis strategy is generated based on the layer in the synthesis process at the freeze moment. Figure 5As shown, at this time, the video area 502 in the interface of the electronic device is playing a video. When the electronic device is detected to be stuck according to the actual operation data of the device, the layer information in the layer synthesis process at the time of the stuck is recorded, and the key layer is selected from the layer information. Among them, the key layer can be the largest layer in the interface of the electronic device, or it can be a layer in a certain area, such as the middle area, or a layer in a certain format. The relevant metadata information of the key layer can be recorded to the system-side synthesis strategy. The next time a similar stuck scene is encountered, the system-side synthesis strategy will mark the key layer as the target layer, and instruct the hardware-side synthesis strategy to give priority to the target layer for the hardware to perform layer synthesis, thereby improving the efficiency of layer synthesis and avoiding stuck problems in the operation of the electronic device.

[0084] For the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present disclosure is not limited by the described order of actions, because according to the present disclosure, certain steps can be performed in other orders or simultaneously.

[0085] Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0086] Corresponding to the aforementioned application function implementation method embodiment, the present disclosure also provides an application function implementation device and a corresponding terminal embodiment.

[0087] like Figure 6 As shown, Figure 6 The present disclosure is a block diagram of a device for optimizing layer synthesis according to an exemplary embodiment. The device can be applied to an electronic device, and the electronic device can be equipped with hardware for implementing layer synthesis. The device may include:

[0088] An acquisition module 602 is used to acquire a set of layers to be synthesized and metadata information corresponding to each layer to be synthesized in the set of layers to be synthesized;

[0089] The target layer determination module 604 is used to match the metadata information based on the system-side synthesis strategy configured on the electronic device, and determine the to-be-synthesized layer corresponding to the matched metadata information as the target layer;

[0090] The resource allocation module 606 is used to preferentially select the target layer for the hardware to perform layer synthesis when processing the set of layers to be synthesized based on the hardware-side synthesis strategy provided by the hardware.

[0091] Optionally, the device also includes a priority determination module, which is used to set the priority of each to-be-synthesized layer in the set of to-be-synthesized layers based on the system-side synthesis strategy, so that the priority of the target layer is higher than the priorities of other layers in the set of to-be-synthesized layers; wherein the priority is positively correlated with the probability of being selected for layer synthesis by the hardware.

[0092] Optionally, the priority determination module is specifically used to add high priority marking information to the target layer, wherein the high priority marking information is used to indicate that the priority of the target layer is higher than the priority of other layers in the set of layers to be synthesized; and / or, to add low priority marking information to other layers in the set of layers to be synthesized, wherein the low priority marking information is used to indicate that the priority of other layers in the set of layers to be synthesized is lower than the priority of the target layer.

[0093] Optionally, the device also includes other layer processing modules, which are used to send the remaining layers in the set of layers to be synthesized that are not selected by the hardware-side synthesis strategy for layer synthesis by the hardware to a graphics processor installed on the electronic device for layer synthesis.

[0094] Optionally, the device also includes a layer selection module, which is used to select a corresponding number of other layers from the set of layers to be synthesized according to the remaining processing resources for the hardware to perform layer synthesis when it is determined based on the hardware-side synthesis strategy that the processing resources of the hardware exceed the resource consumption of all target layers; when it is determined based on the hardware-side synthesis strategy that the processing resources of the hardware are less than the resource consumption of all target layers, the target layers not selected by the hardware-side synthesis strategy are sent to the graphics processor installed on the electronic device for layer synthesis.

[0095] Optionally, the device further comprises an instruction receiving module, configured to edit the system-side synthesis strategy according to a received editing instruction for the system-side synthesis strategy.

[0096] Optionally, the device also includes a system-side synthesis strategy receiving module, which is used to generate the system-side synthesis strategy locally on the electronic device; and / or receive the system-side synthesis strategy sent from the cloud and store it in the electronic device.

[0097] Optionally, the device also includes a module for generating a system-side synthesis strategy, which is used to determine the layers to be synthesized by the hardware in the layer synthesis method according to the layer synthesis method with the best display effect determined in the test environment, and generate the system-side synthesis strategy based on the determined layers; and / or, obtain actual operation data of the electronic device or other devices of the same model as the electronic device, determine the freeze moment in the layer synthesis process according to the actual operation data, and generate the system-side synthesis strategy based on the layers that are in the synthesis process at the freeze moment.

[0098] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0099] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0100] Figure 7 It is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. The electronic device 700 may be a user device, and may specifically be a mobile phone, a computer, a digital broadcast terminal, a message transceiver device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, a wearable device such as a smart watch, a smart glasses, a smart bracelet, a smart running shoe, etc.

[0101] like Figure 7 As shown, electronic device 700 may include one or more of the following components: a processing component 702 , a memory 704 , a power component 706 , a multimedia component 708 , an audio component 710 , an input / output (I / O) interface 712 , a sensor component 714 , and a communication component 716 .

[0102] The processing component 702 generally controls the overall operation of the electronic device 700, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 702 may include one or more modules to facilitate the interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate the interaction between the multimedia component 708 and the processing component 702. The processor 720 may include a DPU and a GPU (not shown in the figure), and the DPU may be used as the hardware for implementing layer synthesis in the present disclosure, and the GPU may be used as other hardware for synthesizing the remaining layers not processed by the DPU in the present disclosure.

[0103] The memory 704 is configured to store various types of data to support operations on the electronic device 700. Examples of such data include instructions for any application or method operating on the electronic device 700, contact data, phone book data, messages, pictures, videos, etc. The memory 704 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0104] The power supply component 706 provides power to the various components of the electronic device 700. The power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 700.

[0105] The multimedia component 708 includes a screen that provides an output interface between the electronic device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 708 includes a front camera and / or a rear camera. When the electronic device 700 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0106] The audio component 710 is configured to output and / or input audio signals. For example, the audio component 710 includes a microphone (MIC), and when the electronic device 700 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 704 or sent via the communication component 716. In some embodiments, the audio component 710 also includes a speaker for outputting audio signals.

[0107] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0108] The sensor assembly 714 includes one or more sensors for providing various aspects of status assessment for the electronic device 700. For example, the sensor assembly 714 can detect the open / closed state of the electronic device 700, the relative positioning of the components, such as the display and keypad of the electronic device 700, and the sensor assembly 714 can also detect the position change of the electronic device 700 or a component of the electronic device 700, the presence or absence of contact between the user and the electronic device 700, the orientation or acceleration / deceleration of the electronic device 700, and the temperature change of the electronic device 700. The sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 714 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 714 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0109] The communication component 716 is configured to facilitate wired or wireless communication between the electronic device 700 and other devices. The electronic device 700 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G LTE, 5G NR or a combination thereof. In an exemplary embodiment, the communication component 716 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 716 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0110] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0111] The present disclosure also provides an executable non-temporary computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of any of the above-mentioned methods for optimizing layer synthesis provided by the present disclosure are implemented. The non-temporary computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0112] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0113] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for optimizing layer synthesis, characterized in that: Applied to an electronic device, the electronic device is equipped with hardware for implementing layer synthesis, and the method includes: Obtaining a set of layers to be synthesized and metadata information corresponding to each layer to be synthesized in the set of layers to be synthesized; Matching the metadata information based on the system-side synthesis strategy configured on the electronic device, and determining the to-be-synthesized layer corresponding to the matched metadata information as the target layer; When the set of layers to be synthesized is processed based on the hardware-side synthesis strategy provided by the hardware, the target layer is preferentially selected for the hardware to perform layer synthesis.

2. The method according to claim 1, characterized in that Also includes: Based on the system-side synthesis strategy, setting the priority of each layer to be synthesized in the set of layers to be synthesized so that the priority of the target layer is higher than the priority of other layers in the set of layers to be synthesized; The priority level is positively correlated with the probability of being selected for layer synthesis by the hardware.

3. The method according to claim 2, characterized in that The step of setting the priority of each layer to be synthesized in the set of layers to be synthesized includes: Adding high priority marking information to the target layer, wherein the high priority marking information is used to indicate that the priority of the target layer is higher than the priority of other layers in the set of layers to be synthesized; and / or, Add low priority marking information to other layers in the set of layers to be synthesized, wherein the low priority marking information is used to indicate that the priority of other layers in the set of layers to be synthesized is lower than the priority of the target layer.

4. The method according to claim 1, characterized in that: Also includes: The remaining layers in the set of layers to be synthesized that are not selected by the hardware-side synthesis strategy for layer synthesis by the hardware are sent to a graphics processor mounted on the electronic device for layer synthesis.

5. The method according to claim 1, characterized in that Also includes: When it is determined based on the hardware-side synthesis strategy that the processing resource amount of the hardware exceeds the resource consumption of all target layers, a corresponding number of other layers are selected from the set of layers to be synthesized according to the remaining processing resources for the hardware to perform layer synthesis; When it is determined based on the hardware-side synthesis strategy that the processing resources of the hardware are less than the resource consumption of all target layers, the target layers not selected by the hardware-side synthesis strategy are sent to a graphics processor installed on the electronic device for layer synthesis.

6. The method according to claim 1, characterized in that Also includes: The system-side synthesis policy is edited according to the received editing instruction for the system-side synthesis policy.

7. The method according to claim 1, characterized in that Also includes: Generating the system-side synthesis strategy locally on the electronic device; and / or, The system-side synthesis strategy sent from the cloud is received and stored in the electronic device.

8. The method according to claim 1, characterized in that: The generation method of the system side synthesis strategy includes: According to the layer synthesis mode with the best display effect determined in the test environment, determine the layer to be synthesized by the hardware in the layer synthesis mode, and generate the system-side synthesis strategy based on the determined layer; and / or, Acquire actual operation data of the electronic device or other devices of the same model as the electronic device, determine a freeze moment in the layer synthesis process according to the actual operation data, and generate the system-side synthesis strategy based on the layer that is in the synthesis process at the freeze moment.

9. A device for optimizing layer synthesis, characterized in that: Applied to an electronic device, the electronic device is equipped with hardware for realizing layer synthesis, and the device includes: An acquisition module, used for acquiring a set of layers to be synthesized and metadata information corresponding to each layer to be synthesized in the set of layers to be synthesized; A target layer determination module is used to match the metadata information based on the system-side synthesis strategy configured on the electronic device, and determine the to-be-synthesized layer corresponding to the matched metadata information as the target layer; The resource allocation module is used to preferentially select the target layer for the hardware to perform layer synthesis when processing the set of layers to be synthesized based on the hardware-side synthesis strategy provided by the hardware.

10. An electronic device, characterized in that: include: processor; a memory for storing a computer program executable by the processor; The processor is configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images