Image composition method and electronic device

Abstract

The present application relates to the field of terminals, and discloses an image composition method and an electronic device. In the process of composing an image comprising a mini window, a GPU capability is invoked to acquire a shadowed screenshot of the mini window, and the shadowed screenshot is stored in a cache layer. In this way, HWC composition, rather than GPU composition, can be used to compose, on the basis of a mini window layer and the cache layer, an image comprising a mini window having a shadow, thereby avoiding the problems of high load and low performance caused by GPU composition, and reducing power consumption and load of electronic devices in mini window scenarios.

Description

Image synthesis method and electronic device

[0001] The present application claims priority from the Chinese patent application No. 202410870706.7 filed on June 28, 2024, and entitled "Image synthesis method and electronic device", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of terminals, and in particular to an image synthesis method and an electronic device. BACKGROUND

[0003] More and more electronic devices can provide the function of playing an application window small window. The small window can also be referred to as a floating window, a picture-in-picture, etc. When the application window small window is played, the small window is suspended above other application windows. For example, as shown in FIG. 1, an electronic device 10 displays a desktop interface 101 and a small window 102, and the small window 102 is suspended above the application window of the desktop application. The small window and the full-screen displayed application window can be different application windows of the same application, or can be application windows of different applications.

[0004] Generally, the small window edge has a shadow effect, and the shadow area changes with the movement of the small window, so as to provide a better display effect to the user. For example, as shown in FIG. 2, there is a shadow outside the bottom frame of the small window 102.

[0005] Each frame of image displayed by the electronic device is synthesized by multiple layers. For example, as shown in FIG. 3, the image displayed by the electronic device 10 in FIG. 1 is synthesized by a wallpaper layer, a navigation bar layer, a status bar layer, an application icon layer, a small window layer, etc. Optionally, in some systems, such as the Android system, the synthesis manner of the layers includes a hardware synthesis (HWC) manner and a graphics processing unit (GPU) synthesis manner. Generally, based on the characteristics of low power consumption and high efficiency of the HWC synthesis manner, the layers without special requirements are synthesized by the HWC synthesis manner, and some layers that need special processing, such as a layer that needs to perform a rounded corner processing on elements in the layer, a layer that needs to perform a blur processing, and a layer that has a shadow effect on elements in the layer, are generally synthesized by the GPU synthesis manner.

[0006] The GPU synthesis manner has low efficiency and high power consumption, and if too many layers are synthesized by the GPU synthesis manner when synthesizing an image, the electronic device will have a problem of high power consumption. SUMMARY

[0007] ​The embodiment of the present application provides an image synthesis method and an electronic device, which can reduce the layers synthesized by the GPU synthesis mode and reduce the power consumption of the electronic device.

[0008] To achieve the above object, the embodiment of the present application adopts the following technical scheme:

[0009] In a first aspect, an image synthesis method is provided, which comprises: after obtaining data of each layer of a first image, generating a shadow screenshot of a small window in the first image by GPU capability according to data of the small window in the first image, and saving the shadow screenshot of the small window in a cache layer; and generating the first image by HWC according to data of the small window in the first image and the cache layer containing the shadow screenshot; wherein the first image comprises the small window with a shadow.

[0010] In the method, the GPU capability is called to obtain the shadow screenshot, and the shadow screenshot is saved in the cache layer. In this way, the HWC synthesis mode can be used to generate a small window layer according to the shadow screenshot, and an image comprising a small window with a shadow can be synthesized according to the small window layer and the cache layer, thereby avoiding the high load and low performance problems caused by GPU synthesis, and reducing the power consumption and load of the electronic device in the small window scenario.

[0011] In combination with the first aspect, in a possible implementation, after the first image is generated by HWC, data of each layer of a second image is obtained; the second image is generated by HWC according to data of a small window in the second image and the cache layer containing the shadow screenshot; wherein the second image comprises the small window with a shadow.

[0012] In a possible implementation, the first image is a copy frame, and the second image is a skip frame. In the copy frame, the GPU capability is called to obtain the shadow screenshot, and the shadow screenshot is saved in the cache layer. In this way, the shadow screenshot can be reused in the skip frame, and the image can be synthesized by HWC, thereby avoiding the high load and low performance problems caused by GPU synthesis. Moreover, the shadow screenshot is generated only once in the copy frame, and the shadow screenshot does not need to be repeatedly generated in the skip frame, thereby reducing the number of times of calling the GPU capability and further improving the performance of image synthesis.

[0013] In combination with the first aspect, in a possible implementation, after the data of each layer of the first image is obtained, if it is determined that the position and size of the small window in the image displayed by the electronic device do not change within a preset time length, the shadow screenshot of the small window is generated by the GPU capability.

[0014] In the method, after the position and size of the small window are stable, the shadow screenshot generated in the copy frame is reused in the skip frame, and the position and size of the shadow of the small window do not change any more, so that the shadow matched with the small window can be realized.

[0015] With reference to the first aspect, in a possible implementation, the data of each layer of the third image is acquired before the data of each layer of the first image is acquired; if it is determined that the small window is not included in the previous frame of the third image and the small window is included in the third image, an empty cache layer is created.

[0016] In the method, the determination that the small window is not included in the previous frame of the third image and the small window is included in the third image indicates that the third image is the frame in which the small window first appears, i.e., the frame state of the electronic device is switched from the closed state to the normal frame state for the first time, and an empty cache layer is created to store the shadow screenshot, so that the shadow screenshot can be multiplexed in the frame skipping.

[0017] With reference to the first aspect, in a possible implementation, the method further includes acquiring data of each layer of a fourth image; and if it is determined that the small window is not included in the fourth image and the small window is included in the previous frame of the fourth image, the cache layer is deleted. That is, when the small window scene is exited, the cache layer is deleted, system resources are recycled, and waste of system resources is avoided.

[0018] With reference to the first aspect, in a possible implementation, the operating system of the electronic device includes a layer management unit and a surface composition unit, the surface composition unit acquires the data of each layer of the first image from a system cache of the operating system; the layer management unit acquires the data of each layer of the first image from the surface composition unit; the layer management unit generates a shadow screenshot of the small window by using GPU capability according to the data of the small window in the first image; and the surface composition unit generates the first image by using HWC according to the data of the small window in the first image and the cache layer containing the shadow screenshot.

[0019] In a possible implementation, the layer management unit further notifies the surface composition unit to close the GPU composition, so that the surface composition unit synthesizes all layers of the image by using HWC, and high load caused by the GPU composition is avoided.

[0020] In the method, the layer management unit of the operating system framework layer acquires the data of each layer from the surface composition unit, so that the layer management unit can determine the synthesis manner of each layer according to the data of each layer, and generate the cache layer containing the shadow screenshot and close the GPU composition. Thus, when the surface composition unit allocates the synthesis manner of each layer, the surface composition unit can synthesize the small window with the shadow by using HWC according to the existing strategy according to the cache layer containing the shadow screenshot and the small window layer, without modifying the original processing logic of the surface composition unit in the Android system, avoiding modification of the original processing mechanism of the Android system, and not affecting the stability of the operating system. In the method, the layer management unit of the operating system framework layer acquires the data of each layer from the surface composition unit, so that the layer management unit can determine the synthesis manner of each layer according to the data of each layer, and generate the cache layer containing the shadow screenshot and close the GPU composition. Thus, when the surface composition unit allocates the synthesis manner of each layer, the surface composition unit can synthesize the small window with the shadow by using HWC according to the existing strategy according to the cache layer containing the shadow screenshot and the small window layer, without modifying the original processing logic of the surface composition unit in the Android system, avoiding modification of the original processing mechanism of the Android system, and not affecting the stability of the operating system.

[0021] ​With reference to the first aspect, in a possible implementation, after N frames of images are generated by the HWC according to the cache layer containing the shadow screenshot, the shadow screenshot of the small window in the (N+1)th frame of image is generated by the GPU capability; where N is a preset value, and N is greater than 1.

[0022] The duration of the frame skipping state of the electronic device is less than a preset threshold. For example, if it is determined that the number of frames of the synthesized image in the frame skipping state is greater than or equal to a preset value (for example, the preset value is N, and N is greater than 1), the frame state of the electronic device is switched to the copy frame state. In this way, the shadow screenshot can be updated periodically, and the shadow screenshot can also be prevented from being misaligned, and the system can also be prevented from being in a dead loop.

[0023] With reference to the first aspect, in a possible implementation, the small window in the first image is a small window of a video application. The image in the video small window changes greatly and quickly, and too many layers of images synthesized by the GPU in this scenario can affect the overall power consumption and performance of the system and affect the user experience. The image synthesis method provided in the embodiment of the present application reduces the number of times of synthesis by the GPU and improves the system performance.

[0024] The second aspect provides an electronic device having a function of implementing the method of the first aspect. The function can be implemented by hardware, or the function can be implemented by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0025] The third aspect provides an electronic device, including a processor, a memory, and a display screen. The memory is used to store computer execution instructions. When the electronic device is running, the processor executes the computer execution instructions stored in the memory, so that the electronic device executes the method according to any one of the possible implementation manners of the first aspect.

[0026] The fourth aspect provides a computer readable storage medium, which stores instructions. When the instructions are executed on a computer, the computer can execute the method according to any one of the possible implementation manners of the first aspect.

[0027] The fifth aspect provides a computer program product containing instructions. When the instructions are executed on a computer, the computer can execute the method according to any one of the possible implementation manners of the first aspect.

[0028] The technical effects brought by any one of the possible implementation manners of the second aspect to the fifth aspect can be referred to the technical effects brought by different implementation manners of the first aspect, and will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0029] FIG. 1 is a schematic diagram of a scenario to which a method for synthesizing images provided in an embodiment of the present application is applied;

[0030] FIG. 2 is a schematic diagram of a scenario to which a method for synthesizing images provided in an embodiment of the present application is applied;

[0031] FIG. 3 is a schematic diagram of a scenario example of a method for synthesizing images provided in an embodiment of the present application;

[0032] FIG. 4 is a schematic diagram of a software architecture of an electronic device provided in an embodiment of the present application;

[0033] FIG. 5 is a schematic diagram of a scenario example of a method for synthesizing images provided in an embodiment of the present application;

[0034] FIG. 6 is a schematic diagram of a scenario example of a method for synthesizing images provided in an embodiment of the present application;

[0035] FIG. 7 is a schematic diagram of a method for synthesizing images provided in an embodiment of the present application;

[0036] FIG. 8 is a schematic diagram of switching between four frame states of an electronic device provided in an embodiment of the present application;

[0037] FIG. 9 is a schematic diagram of effect analysis of a method for synthesizing images provided in an embodiment of the present application;

[0038] FIG. 10 is a schematic diagram of a hardware structure of an electronic device provided in an embodiment of the present application;

[0039] FIG. 11 is a schematic diagram of an electronic device structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing the specific embodiments and are not intended to be limiting of the present application. As used in the specification and the appended claims of the application, the singular forms “a,” “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and / or “comprising,” when used in the following embodiments of the present application, specify the presence of stated features, integers, steps, operations, objects, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, objects, and / or components thereof. The term “and / or” used in the context of the following embodiments of the present application refers to a conjunctive relationship with the associated objects, indicating that there can be three kinds of relationships; for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character “ / ” generally represents an “or” relationship between the associated objects.

[0041] Reference within the specification to "one embodiment" or "an embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The appearances of the phrase "in one embodiment" or "in some embodiments" in various places within specified

[0042] In the present application, the word "exemplary" or "for example" is used to mean serving as an example, instance, or illustration. Any implementation described as "exemplary" or "for example" in the present application is not necessarily to be construed as preferred or advantageous over other implementations. The

[0043] The image synthesis method provided by the embodiments of the present application can be applied to an electronic device including a display screen. The electronic device can include a mobile phone, a tablet computer, a notebook computer, a personal computer (PC), an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a smart home device (such as a smart television, a smart screen, a large screen, a smart speaker, a smart air conditioner, etc.), a personal digital assistant (PDA), a wearable device (such as a smart watch, a smart bracelet, etc.), a vehicle-mounted device, a virtual reality device, etc., and the embodiments of the present application do not make any limitation thereto.

[0044] In the embodiments of the present application, the electronic device is an electronic device that can run an operating system and install an application. Optionally, the operating system running on the electronic device can be an Android system, a Windows system, an iOS system, etc.

[0045] FIG. 4 is a schematic diagram of a software architecture of an electronic device provided by the embodiments of the present application.

[0046] It can be appreciated that the layered architecture divides software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, Android The system can include an application (App) layer, a framework (FWK) layer, a hardware abstraction layer (HAL), and a kernel layer.

[0047] The application layer can include a series of application packages. For example, the application packages can include gallery, calendar, call, short message, camera, desktop, video, and the like.

[0048] The framework layer provides application programming interfaces (APIs) and programming frameworks for the applications of the application layer. The framework layer includes some pre-defined functions. The framework layer provides programming services to the application layer through the API interface, enabling the application to interact with hardware and the operating system. For example, the framework layer can include a window manager service (WMS), a content provider service, a view system, a resource management service, a surface composition unit, a rendering engine, a layer management unit, and the like.

[0049] The window manager service is used to manage window programs. For example, adding a window, deleting a window, modifying a window, and the like. The content provider service is used to store and obtain data, and enables the data to be accessed by the application. The data can include video, image, audio, dialed and received calls, browsing history and bookmarks, phonebook, and the like. The view system includes visual controls, such as a control for displaying text, a control for displaying pictures, and the like. The view system can be used to build an application. The resource management service provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and the like.

[0050] The surface composition unit (SurfaceFlinger) is a graphics compositor in the Android system, responsible for composing multiple layers and displaying them on the display screen; SurfaceFlinger coordinates and manages the rendering and display process of graphics. The rendering engine (RenderEngine) is a graphics rendering engine in the Android system, responsible for presenting layers to the display screen; it implements graphics drawing and composition operations. In the embodiments of the present application, the layer management unit is used to manage the layer composition process, such as managing the composition of each layer through the HWC composition method or the GPU composition method.

[0051] The hardware abstraction layer is used to abstract hardware and encapsulate the drivers of the kernel layer, and provides an interface upwardly. For example, as shown in FIG. 4, the hardware abstraction layer includes HWC, etc. The HWC is a hardware compositor in the Android system, which is used to compose layers into a displayable frame, and cooperates with the SurfaceFlinger to realize hardware accelerated rendering and composition of graphics.

[0052] The kernel layer provides underlying drivers for various hardware of the electronic device. For example, the kernel layer can include display drivers, etc.

[0053] Each frame of image displayed by the electronic device is composed of multiple layers, which can come from the same application or from multiple applications. For example, the electronic device 10 shown in FIG. 1 displays a desktop interface 101 and a small window 102, the desktop interface 101 is a user interface of a desktop application, and the small window 102 can be an application window of a video application, a chat application, a music playing application, etc. The desktop interface 101 and the small window 102 are composed of one or more layers respectively. The operating system of the electronic device obtains information of each layer in a frame of image (for example, elements included in the layer, layout mode of the elements, etc.), composes each layer, generates an image and sends the image for display.

[0054] For example, the following takes an Android system as an example to introduce the process of composing a frame of image. For example, the following takes an Android system as an example to introduce the process of composing a frame of image. SurfaceFlinger in the system acquires the data of each layer of the current frame from the system Buffer when the system's synchronization signal arrives. The data of each layer of the current frame is generated by each application in the application layer. For example, the data of the wallpaper layer in FIG. 3 is generated by a system application and stored in the system Buffer, the data of the application icon layer is generated by a desktop application and stored in the system Buffer, and the data of the small window layer is generated by a video application and stored in the system Buffer. ① SurfaceFlinger sends the list information of all layers included in a frame of image to the HWC. ② The HWC determines the synthesis method of these layers according to the hardware capability. The synthesis method includes the HWC synthesis method (usually hardware synthesis) or the GPU synthesis method (usually software synthesis). The HWC will mark the corresponding synthesis method for each layer, that is, whether the layer is synthesized by the GPU synthesis method or the HWC synthesis method. ③ SurfaceFlinger is responsible for calling the GPU to synthesize (for example, the GPU uses the OpenGL ES instruction to synthesize the layer) all layers marked with the GPU synthesis method into an output Buffer. ④ SurfaceFlinger gives the output Buffer and other layers marked with the HWC synthesis method to the HWC. ⑤ The HWC calls the display processing unit (DPU) to complete the synthesis of the remaining layers (layers marked with the HWC synthesis method). Finally, the HWC is responsible for displaying the final synthesis result on the display screen.

[0055] Generally, the HWC determines the synthesis method of the layer according to the hardware capability (for example, the GPU capability). For some layers that need special processing, such as layers that need to be rounded, blurred, and have a shadow effect, the GPU synthesis method is generally used for synthesis.

[0056] In the following embodiments, the synthesis by the GPU synthesis method is referred to as synthesis by the GPU, and the synthesis by the HWC synthesis method is referred to as synthesis by the HWC.

[0057] When the electronic device displays the small window, generally, the small window has a shadow effect. The effects of the shadow effect of the small window include: 1, emphasizing the small window. The shadow can help highlight the small window, make the small window more obvious in the interface, distinguish the small window from the background by creating a visual boundary around the small window, and improve the user's attention to the small window. 2, the shadow can improve the visibility of the small window. When the small window is located on a complex or diverse background, it can cause insufficient contrast between the small window content and the background, making it difficult to identify the small window content. By adding a shadow, a relatively simple background can be provided for the small window, making the small window content easier to read and view. 3, indicating the sense of hierarchy. The shadow can add a sense of hierarchy to different elements in the application interface. The small window usually needs to be suspended above other content of the application to highlight its importance. By adding a shadow effect to the small window, a virtual three-dimensional effect can be created visually for the small window, making it easier for users to perceive that the small window is above the background. 4, providing aesthetics. The shadow can increase the aesthetics and visual appeal of the application interface. By skillfully designing the color, transparency, and shape of the shadow, a certain visual artistic effect can be added to the small window, making it more attractive.

[0058] Because the small window has a shadow effect, and optionally, some small windows also have a rounded corner effect, generally, the small window layer is synthesized by the GPU, and the power consumption of the GPU synthesis is large. Especially when the small window is a video small window, the image in the video small window changes greatly and quickly, which will cause the number of layers synthesized by the GPU to be too large in this scenario, thereby affecting the overall power consumption and performance of the system and affecting the user experience.

[0059] In an example, taking the scenario that the electronic device 10 shown in FIG. 1 displays the desktop interface 101 and the small window 102 as an example. As shown in FIG. 5(a), the user interface is generated by the wallpaper layer, the navigation bar layer, the application icon layer, the small window layer, and the status bar layer. In an implementation, the small window element in the small window layer needs to be shadowed, and the information of the wallpaper layer, the navigation bar layer, and the application icon layer needs to be combined to generate the shadow effect of the small window. Therefore, the wallpaper layer, the navigation bar layer, the application icon layer, and the small window layer are synthesized by the GPU, for example, the layers are generated by the GPU and the shadow processing is performed; and the status bar layer is synthesized by the HWC; and the wallpaper layer, the navigation bar layer, the application icon layer, the small window layer, and the status bar layer are synthesized by the HWC synthesis manner to generate an image containing the desktop interface 101 and the small window 102. In this process, a large number of layers are synthesized by the GPU, and the small window layer is also included in the layers, which has a large and fast image change.

[0060] A large number of layers are synthesized by the GPU, which is inefficient and consumes a large amount of power. For example, as shown in (b) of FIG. 5, in the process of generating the desktop interface 101 and the small window 102, because a plurality of layers are synthesized by the GPU, the rendering engine thread is called, and the time length during which the surface synthesis thread is called includes the time length during which the rendering engine thread is executed. After the GPU synthesis and the HWC synthesis are completed, the surface synthesis thread notifies the HWC to display.

[0061] In another example, as shown in FIG. 6, in one scenario, the user interface does not include shadows, and all layers are synthesized by the HWC, and it takes about 4 ms to generate a frame of image. In another scenario, the user interface includes shadows, and part of the layers are synthesized by the GPU, and it takes about 6-8 ms to generate a frame of image.

[0062] It can be seen that the implementation of the small window layer by the GPU synthesis causes the problem of low image synthesis efficiency and high power consumption of the electronic device.

[0063] Embodiments of the present application provide an image synthesis method, which synthesizes an image including a small window with a shadow by using an HWC synthesis manner, to reduce the power consumption and load of the electronic device in the small window scenario.

[0064] When the user interface includes a small window, a frame of image is taken as a copy frame, and a shadow screenshot of the small window is obtained in the copy frame. In this way, in the synthesis of a subsequent frame of image (referred to as a skip frame), the small window without a shadow can be synthesized by the HWC, and the small window layer and the shadow screenshot are further merged to generate the small window with a shadow.

[0065] In some embodiments, as shown in FIG. 7, a layer is generated in advance, for example, referred to as a cache layer.

[0066] In the copy frame, data of each layer of the frame of image is obtained, for example, wallpaper layer data, navigation bar layer data, application icon layer data, small window layer data, status bar layer data, and the like. GPU capability is called, and each layer is processed according to the data to generate a shadow screenshot of the small window. The shadow screenshot of the small window is saved in the cache layer.

[0067] In one implementation, in the copy frame, the GPU synthesis is closed, and all layers are set to be synthesized by the HWC. A partial screenshot interface is called to obtain the shadow screenshot of the small window. The partial screenshot interface calls the GPU capability to process each layer according to the data to generate the shadow screenshot of the small window. Then, the shadow screenshot of the small window is saved in the cache layer (StageLayer). In this implementation, the GPU capability is called to generate the shadow screenshot, instead of generating the small window layer with a shadow by the GPU synthesis, so that the frequency of the GPU synthesis is reduced, the power consumption of the GPU synthesis is saved, and the performance is improved.

[0068] Then, the data of each layer of the image frame and the buffer layer containing the shadow screenshot are synthesized by the HWC to generate an image frame.

[0069] In the skip frame, the GPU synthesis is closed and all layers are set to be synthesized by the HWC. The data of each layer of the image frame is obtained, for example, the wallpaper layer data, the navigation bar layer data, the application icon layer data, the small window layer data, the status bar layer data, and the like. Then, the data of each layer of the image frame and the buffer layer containing the shadow screenshot are synthesized by the HWC to generate an image frame.

[0070] In this way, in the copy frame and the skip frame, the small window with the shadow does not need to be generated by the GPU synthesis, and the problem of low performance and high power consumption caused by the generation of the small window layer by the GPU synthesis is avoided.

[0071] Optionally, in an implementation, in the case where the position and size of the small window do not change, the GPU synthesis is closed and all layers are set to be synthesized by the HWC. The first image frame that meets the case where the position and size of the small window do not change is determined as the copy frame, and the image frames that meet the case where the position and size of the small window do not change except the copy frame are determined as the skip frames.

[0072] In the case where the position and size of the small window do not change, the position and shape of the shadow of the small window also do not change, and the shadow screenshot is obtained in the copy frame, so that the shadow screenshot can be reused in the skip frame, and the GPU capability does not need to be called again to obtain the shadow screenshot in the skip frame, thereby reducing the frequency of calling the GPU capability and further improving the performance.

[0073] In an implementation, according to the shadow of the small window of the image frame, the electronic device includes four frame states, which are a closed state, a normal frame state, a copy frame state, and a skip frame state.

[0074] For example, FIG. 8 shows a schematic diagram of switching between the four frame states of the electronic device.

[0075] The frame state of the electronic device is the closed state by default. Optionally, if the image frame does not include the small window, the frame state of the electronic device is the closed state. For example, before starting the small window to play or after exiting the small window to play, the image frame does not include the small window. Optionally, in the case where the image frame includes the small window but the small window does not have the shadow, the frame state of the electronic device is the closed state. For example, according to a preset display strategy on the electronic device, in the process where the size of the small window changes or in the process where the position of the small window changes, the small window does not have the shadow.

[0076] In an example, the electronic device compares data of a current image frame with data of a previous image frame. If it is determined that the current image frame is the first image frame in which the small window with the shadow appears, the frame state of the electronic device is switched from the closed state to the normal frame state. If it is determined that the position and size of the small window do not change within a preset time length (or a preset number of frames), the frame state of the electronic device is switched from the normal frame state to the copy frame state. After the electronic device synthesizes an image frame in the copy frame state, the frame state of the electronic device is switched to the skip frame state. If it is determined that the position or size of the small window changes in the copy frame state or the skip frame state, the frame state of the electronic device is switched to the normal frame state. If the small window playback is exited or the shadow of the small window disappears in the copy frame state or the skip frame state or the normal frame state, the frame state of the electronic device is switched to the closed state. In an implementation, before the frame state of the electronic device is switched to the closed state, the cache layer is deleted. Optionally, the electronic device is in the skip frame state for a time length less than a preset threshold. For example, if it is determined that the number of image frames synthesized in the skip frame state is greater than or equal to a preset value (for example, the preset value is N, and N is greater than 1), the frame state of the electronic device is switched to the copy frame state. In this way, the shadow screenshot can be periodically updated, the shadow screenshot can be prevented from being misaligned, and the system can be prevented from being in a dead loop.

[0077] In an implementation, when the electronic device is in the normal frame state, the electronic device can generate an image by using a method in a conventional technology, for example, the small window layer with the shadow can be synthesized by using a GPU. Optionally, when the frame state of the electronic device is switched from the closed state to the normal frame state for the first time, an empty cache layer is created.

[0078] When the electronic device is in the copy frame state, the GPU synthesis is closed, and all layers are set to be synthesized by using the HWC. The shadow screenshot of the small window is generated by using the GPU capability, and the shadow screenshot is saved in the cache layer. Then, data of each layer of a current image frame and the cache layer containing the shadow screenshot are used to generate an image frame by using the HWC synthesis.

[0079] When the electronic device is in the skip frame state, the GPU synthesis is closed, and all layers are set to be synthesized by using the HWC. Data of each layer of a current image frame and the cache layer containing the shadow screenshot are used to generate an image frame by using the HWC synthesis.

[0080] The image synthesis method provided in the embodiments of the present application can be used to obtain the shadow screenshot by using the GPU capability when the position and size of the small window with the shadow do not change, and the shadow screenshot is saved in the cache layer. In this way, the image can be synthesized by using the HWC in the case where the position and size of the small window do not change, and the high load and low performance problems caused by the GPU synthesis can be avoided.

[0081] For example, FIG. 9 shows an effect analysis schematic diagram of the image synthesis method provided in the embodiments of the present application.

[0082] In the case that the image includes a scene with a small window with a shadow, as shown in (a) of FIG. 9, if the image is generated by the method in the conventional technique, for example, the small window layer with the shadow can be synthesized by the GPU, and the rendering engine thread is invoked, i.e. the GPU capability is used, to execute the time-consuming surface synthesis thread when synthesizing each frame of image. That is, the performance of synthesizing the image is low.

[0083] As shown in (b) of FIG. 9, if the image synthesis method provided by the embodiment of the present application is used, the shadow screenshot is obtained in the copy frame, and the image is synthesized by the HWC in the copy frame and the skip frame, and the rendering engine thread is invoked (the GPU capability is invoked to obtain the shadow screenshot) only when the image is synthesized in the copy frame, and the rendering engine thread is not invoked when the image is synthesized in the skip frame. When synthesizing each frame of image, the time-consuming of executing the surface synthesis thread is short. Compared with (a) of FIG. 9, the performance of synthesizing the image is high.

[0084] For example, FIG. 10 shows a schematic diagram of a hardware structure of the electronic device. As shown in FIG. 10, the electronic device 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charge management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset jack 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 can include a temperature sensor, an ambient light sensor, etc.

[0085] It can be understood that the structure shown in the embodiment does not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 can include more or fewer components than shown, or combine certain components, or split certain components, or different arrangement of components. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0086] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a display processing unit (DPU), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices, or can be integrated in one or more processors.

[0087] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to instruction operation codes and timing signals, complete the control of fetching instructions and executing instructions.

[0088] The memory can also be provided in the processor 110, for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can save instructions or data that have just been used or are used repeatedly by the processor 110. If the processor 110 needs to use the instructions or data again, it can directly call from the memory. This avoids repeated access and reduces the waiting time of the processor 110, thus improving the efficiency of the system.

[0089] In some embodiments, the processor 110 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0090] It can be understood that the interface connection relationship between the modules shown in the embodiments is only illustrative and does not limit the structure of the electronic device. In other embodiments, the electronic device can also use different interface connection manners or combinations of multiple interface connection manners.

[0091] The charging management module 140 is configured to receive charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input through a wireless charging coil of the electronic device. The charging management module 140 can charge the battery 142 and also supply power to the electronic device through the power management module 141.

[0092] The power management module 141 is configured to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to supply power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160, etc. The power management module 141 can also be configured to monitor parameters such as battery capacity, battery cycle count, battery health status (leakage, impedance), etc. In other embodiments, the power management module 141 can also be arranged in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be arranged in the same device.

[0093] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, etc.

[0094] The electronic device 100 implements the display function through the GPU, the display screen 194, and the application processor, etc. The GPU is a microprocessor for image processing, connecting the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering, and plays a crucial role in improving graphics performance and efficiency. The processor 110 can include one or more GPUs that execute program instructions to generate or change display information.

[0095] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini-LED, a Micro-OLED, a quantum dot light emitting diode (QLED), etc.

[0096] The electronic device 100 can implement the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, and the application processor, etc.

[0097] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. The ISP can also optimize the algorithm of the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature, and other parameters of the shooting scene. In some embodiments, the ISP can be arranged in the camera 193.

[0098] The camera 193 is used to capture still images or videos. Objects project optical images through the lens to the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then transmitted to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into a standard RGB, YUV, or the like format image signal. In some embodiments, the electronic device can include one or N cameras 193, where N is a positive integer greater than one.

[0099] The digital signal processor is used to process digital signals, in addition to being able to process digital image signals, it can also process other digital signals. For example, when the electronic device selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0100] The video codec is used to compress or decompress digital video. The electronic device 100 can support one or more video codecs. In this way, the electronic device can play or record videos in multiple encoding formats, such as: moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0101] The NPU is a neural-network (NN) computing processor that learns from the structure of biological neural networks, such as the transmission mode between human brain neurons, to quickly process input information and continuously self-learn. Through the NPU, the electronic device can achieve intelligent cognitive applications, such as: image recognition, face recognition, speech recognition, text understanding, etc.

[0102] The DPU is a display processing unit that is responsible for processing and optimizing display image and video content to provide a better visual experience.

[0103] The electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor, etc. For example, music playback, recording, etc.

[0104] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with the processor 110 through the external memory interface 120 to implement data storage functions. For example, audio, video, and the like files are saved in the external memory card.

[0105] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 performs various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 121. For example, in the embodiments of the present application, the processor 110 can execute the instructions stored in the internal memory 121, and the internal memory 121 can include a program storage area and a data storage area. The program storage area can store an operating system and at least one application program required for a function (such as a sound playing function, an image playing function, etc.). The data storage area can store data (such as a video file) created during use of the electronic device. In addition, the internal memory 121 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0106] The keys 190 include a power-on key, a volume key, etc. The keys 190 can be mechanical keys. They can also be touch keys. The motor 191 can generate a vibration prompt. The motor 191 can be used for incoming call vibration prompt, and can also be used for touch vibration feedback. The indicator 192 can be an indicator light, which can be used to indicate a charging state, a power change, and can also be used to indicate a message, a missed call, a notification, etc. The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or pulled out of the SIM card interface 195 to realize contact and separation with the electronic device. The electronic device can support one or N SIM card interfaces, and N is a positive integer greater than 1. The SIM card interface 195 can support a Nano SIM card, a Micro SIM card, a SIM card, etc.

[0107] It can be understood that the electronic device provided by the embodiments of the present application contains the corresponding hardware structure and / or software modules for executing various functions in order to realize the above functions. Those skilled in the art should easily realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed in the present application can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of the present application.

[0108] The embodiments of the present application can divide the functional modules of the electronic device according to the above method examples. For example, each functional module can be divided according to each function, or two or more functions can be integrated in one processing module. The integrated module can be realized in the form of hardware or in the form of a software functional module. It should be noted that the division of the modules in the embodiments of the present application is illustrative, and is only a logical functional division. In actual implementation, another division manner can be used.

[0109] In an example, in combination with FIG. 4, the framework layer of the operating system of the electronic device includes a layer management unit, which is used to manage the layer composition process. For example, after the surface composition unit obtains the data of each layer of the current image frame, the layer management unit obtains the data of each layer of the current image frame from the surface composition unit, for example, obtains the handle of each layer, manages the frame state of the electronic device according to the data of each layer of the current image frame, judges and executes the switching between the frame states, and determines that each layer is synthesized by the HWC synthesis manner or the GPU synthesis manner. In this example, the layer management unit of the framework layer obtains the data of each layer from the surface composition unit, so as to determine the synthesis manner of each layer according to the data of each layer, and generate a cache layer containing a shadow screenshot, and notify the surface composition unit to close the GPU synthesis. In this way, when the surface composition unit allocates the synthesis manner of each layer, the surface composition unit can synthesize the small window with a shadow through the HWC synthesis according to the existing strategy according to the cache layer containing the shadow screenshot and the small window layer, without modifying the original processing logic of the surface composition unit in the Android system, avoiding the modification of the original processing mechanism of the Android system, and not affecting the stability of the operating system. In an example, in combination with FIG. 4, the framework layer of the operating system of the electronic device includes a layer management unit, which is used to manage the layer composition process. For example, after the surface composition unit obtains the data of each layer of the current image frame, the layer management unit obtains the data of each layer of the current image frame from the surface composition unit, for example, obtains the handle of each layer, manages the frame state of the electronic device according to the data of each layer of the current image frame, judges and executes the switching between the frame states, and determines that each layer is synthesized by the HWC synthesis manner or the GPU synthesis manner. In this example, the layer management unit of the framework layer obtains the data of each layer from the surface composition unit, so as to determine the synthesis manner of each layer according to the data of each layer, and generate a cache layer containing a shadow screenshot, and notify the surface composition unit to close the GPU synthesis. In this way, when the surface composition unit allocates the synthesis manner of each layer, the surface composition unit can synthesize the small window with a shadow through the HWC synthesis according to the existing strategy according to the cache layer containing the shadow screenshot and the small window layer, without modifying the original processing logic of the surface composition unit in the Android system, avoiding the modification of the original processing mechanism of the Android system, and not affecting the stability of the operating system.

[0110] In an implementation manner, the layer management unit includes a parameter management module, a frame state decision module, a copy frame control module, and a skip frame control module.

[0111] The parameter management module is configured to initialize parameters, monitor, collect, and save the data of each layer of each image frame, and the like.

[0112] The frame state decision module is configured to determine the frame state according to the data of each image frame and execute the frame state switching, determine the synthesis manner of each layer according to the frame state, adjust the order of each layer, and the like. Optionally, the frame state decision module is further configured to generate and delete a cache layer. Optionally, the frame state decision module is further configured to close the GPU synthesis, and the like.

[0113] The copy frame control module is configured to generate and save a shadow screenshot, and control each layer of the copy frame to be synthesized through the HWC synthesis.

[0114] ​The frame skipping control module is configured to control a cache layer containing a shadow screenshot generated by frame skipping multiplexing a copy frame, synthesize the cache layer containing the shadow screenshot with each layer of the frame skipping through HWC, and close GPU synthesis.

[0115] In an example, refer to FIG. 11, which shows a possible structural schematic diagram of the electronic device involved in the above embodiments. The electronic device 200 includes a processing unit 210, a storage unit 220, and a display unit 230.

[0116] The processing unit 210 is configured to control and manage the actions of the electronic device 200. The storage unit 220 is configured to store the program codes and data of the electronic device 200. The display unit 230 is configured to display the user interface of the electronic device 200.

[0117] Of course, the unit modules in the electronic device 200 include but are not limited to the above-mentioned processing unit 210, storage unit 220, and display unit 230.

[0118] Optionally, the electronic device 200 can further include an image acquisition unit configured to acquire images.

[0119] Optionally, the electronic device 200 can further include an audio unit configured to acquire audio, play audio, and the like.

[0120] Optionally, the electronic device 200 can further include a communication unit configured to support the communication between the electronic device 200 and other devices.

[0121] The processing unit 210 can be a processor or a controller, for example, a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. The storage unit 220 can be a memory. The display unit 230 can be a display screen and the like. The image acquisition unit can be a camera and the like. The audio unit can include a microphone, a speaker, and the like. The communication unit can include a mobile communication unit and / or a wireless communication unit.

[0122] For example, the processing unit 210 is a processor (e.g., the processor 110 shown in FIG. 10), the storage unit 220 can be a memory (e.g., the internal memory 121 shown in FIG. 10), and the display unit 230 can be a display screen (e.g., the display screen 194 shown in FIG. 10). The image acquisition unit can be a camera (e.g., the camera 193 shown in FIG. 10). The audio unit can be an audio module (e.g., the audio module 170 shown in FIG. 10). The communication unit can include a mobile communication unit (e.g., the mobile communication module 150 shown in FIG. 10) and a wireless communication unit (e.g., the wireless communication module 160 shown in FIG. 10). The electronic device 200 provided in the embodiments of the present application can be the electronic device 100 shown in FIG. 10. The above processor, memory, display screen, camera, audio module, mobile communication unit, wireless communication unit, and the like can be connected together, for example, through a bus.

[0123] The embodiments of the present application also provide a chip system, which includes at least one processor and at least one interface circuit. The processor and the interface circuit can be interconnected through a line. For example, the interface circuit can be used to receive signals from other devices (e.g., a memory of an electronic device). For another example, the interface circuit can be used to send signals to other devices (e.g., a processor). Illustratively, the interface circuit can read instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the electronic device can perform various steps in the above embodiments. Of course, the chip system can also include other discrete devices, which are not limited in the embodiments of the present application.

[0124] The embodiments of the present application also provide a computer readable storage medium, which includes computer instructions. When the computer instructions are run on the above electronic device, the electronic device performs various functions or steps performed by the mobile phone in the above method embodiments.

[0125] The embodiments of the present application also provide a computer program product, which makes the computer perform various functions or steps performed by the mobile phone in the above method embodiments when the computer program product is run on the computer.

[0126] From the above description of the embodiments, those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example for illustration. In actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above described functions.

[0127] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the modules or units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another apparatus, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0128] The units described as separate components can or can not be physically separate, and the components shown as units can be one physical unit or a plurality of physical units, that is, can be located in one place, or can be distributed to a plurality of different places. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0129] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0130] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application essentially or the parts that make contributions to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing an apparatus (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the method described in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0131] The above is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any change or replacement within the technical scope disclosed in the present application should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An image synthesis method applied to an electronic device, comprising: The method comprises: obtaining data of each layer of a first image; generating a shadow screenshot of the small window in the first image by a display processing unit (GPU) according to data of the small window in the first image; the shadow screenshot of the small window is saved in a cache layer; generating the first image by a hardware composition (HWC) according to data of the small window in the first image and the cache layer containing the shadow screenshot; the first image comprises the small window with the shadow.

2. The method of claim 1, wherein, After the first image is generated by the HWC, the method further comprises: obtaining data of each layer of a second image; generating the second image by the HWC according to data of the small window in the second image and the cache layer containing the shadow screenshot; the second image comprises the small window with the shadow.

3. The method according to claim 1 or 2, characterized in that, Before the shadow screenshot of the small window is generated by the display processing unit (GPU), the method further comprises: determining that the position and size of the small window in the image displayed by the electronic device do not change within a preset time length.

4. The method according to any one of claims 1 to 3, characterized in that, Before the data of each layer of the first image is obtained, the method further comprises: obtaining data of each layer of a third image; if it is determined that the small window is not included in a previous frame of the third image and the small window is included in the third image, creating an empty cache layer.

5. The method of claim 4, wherein, The method further comprises: obtaining data of each layer of a fourth image; if it is determined that the small window is not included in the fourth image and the small window is included in a previous frame of the fourth image, deleting the cache layer.

6. The method according to any one of claims 1 to 5, characterized in that, An operating system of the electronic device comprises a layer management unit and a surface composition unit, the data of each layer of the first image is obtained; the shadow screenshot of the small window is generated by a display processing unit (GPU) according to data of the small window in the first image; the first image is generated by a hardware composition (HWC) according to data of the small window in the first image and the cache layer containing the shadow screenshot, comprising: the surface composition unit obtains the data of each layer of the first image from a system cache of the operating system; the layer management unit obtains the data of each layer of the first image from the surface composition unit; the layer management unit generates the shadow screenshot of the small window by the GPU according to data of the small window in the first image; the surface composition unit generates the first image by the HWC according to data of the small window in the first image and the cache layer containing the shadow screenshot.

7. The method of claim 6, wherein, The method further comprises: the layer management unit notifies the surface composition unit to close the GPU composition.

8. The method according to any one of claims 1 to 7, characterized in that, The method further comprises: after N frames of images are generated by the HWC according to the cache layer containing the shadow screenshot, the shadow screenshot of the small window in the (N+1)th frame of image is generated by the GPU; wherein N is a preset value, and N is greater than 1.

9. The method according to any one of claims 1 to 8, characterized in that, The small window in the first image is a small window of a video application.

10. An electronic device, comprising: The electronic device comprises a memory, a processor and a display screen; the display screen is used to display an image; the memory is coupled with the processor; the memory is used to store computer program codes; the computer program codes comprise computer instructions; the processor invokes the computer instructions to enable the electronic device to execute the method according to any one of claims 1-9.

11. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, The computer program / instructions, which are executed by the processor, implement the method according to any one of claims 1-9.

12. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions, which are executed by the processor, implement the method according to any one of claims 1-9.

Images