Multi-screen synthesis method and device, electronic equipment and storage medium

Abstract

The invention provides a multi-screen synthesis method and apparatus, an electronic device and a storage medium. The method comprises the steps of obtaining layer information corresponding to a to-be-synthesized layer in each target screen; under the condition that it is determined that the to-be-synthesized layers need to be updated according to the layer information, obtaining updating information corresponding to each to-be-synthesized layer, and updating the to-be-synthesized layers in each target screen through the updating information to obtain updated layers; and obtaining a preset synthesis mode corresponding to each target screen, and performing layer synthesis processing on the updated layer in each target screen according to the preset synthesis mode to obtain a synthesized target layer corresponding to each target screen. Compared with the prior art, the image layer synthesis processing is performed on the updated image layer in each target screen, and the synthesis of each screen is independently processed, so that the overall time consumption of SurfaceFinger and DPU synthesis is not influenced, the screen lagging is reduced, and the system display fluency is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a method and device for multi-screen synthesis, an electronic device, and a storage medium. Background Art

[0002] At present, most mainstream mobile devices, in-vehicle devices, etc., already support simultaneous display of multiple screens in order to provide users with a diverse and rich interactive experience. Compared with single-screen devices, when displaying on multiple screens simultaneously, the Android system as a whole needs to draw and synthesize more windows, and the central processing unit (CPU), graphics display controller (GPU), and display controller (DPU) are more heavily loaded.

[0003] When displaying on multiple screens at the same time, multi-screen synthesis is required. Multi-screen synthesis is mainly achieved through SurfaceFlinger (a system service in the Android operating system). The logic of SurfaceFlinger synthesis is to traverse all screens in the order of reporting screen pclk (Peripheral Clock) signals, and synthesize visible windows in sequence before displaying them on the screen.

[0004] However, the multi-screen synthesis method in the existing technology requires serial synthesis in the main thread. When the number of visible windows on the same system-on-chip (SOC) is large, the overall synthesis time of SurfaceFlinger and DPU will increase, and it is easy to exceed a vertical synchronization (Vertical Sync, Vsync) cycle, causing frame loss and freeze in the screen display. Summary of the Invention

[0005] The present disclosure provides a method and apparatus for multi-screen synthesis, an electronic device, and a storage medium. Its main purpose is to address the existing multi-screen synthesis method, which requires serial synthesis in the main thread. When the number of visible windows on the same system-on-chip (SOC) is large, the overall synthesis time consumed by SurfaceFlinger and DPU increases, easily exceeding one Vsync cycle, resulting in frame drops and freezes on the screen.

[0006] According to a first aspect of the present disclosure, a method for multi-screen synthesis is provided, comprising:

[0007] Get the layer information corresponding to the layer to be synthesized in each target screen;

[0008] When it is determined according to the layer information that the layer to be synthesized needs to be updated, obtaining update information corresponding to each layer to be synthesized, and updating the layer to be synthesized in each target screen according to the update information to obtain an updated layer;

[0009] A preset synthesis mode corresponding to each target screen is obtained, and layer synthesis processing is performed on the updated layer in each target screen according to the preset synthesis mode to obtain a synthesized target layer corresponding to each target screen.

[0010] Optionally, obtaining update information corresponding to each layer to be synthesized includes:

[0011] Performing hierarchical construction processing on the layers to be synthesized using a preset construction algorithm to obtain a hierarchical structure of the layers to be synthesized;

[0012] The layer sequence information corresponding to each layer to be synthesized is obtained in the hierarchical structure respectively, and based on a preset visual algorithm, a visible area is calculated according to the layer sequence information and layer attribute information to obtain the update information; wherein the layer information includes the layer attribute information.

[0013] Optionally, performing layer synthesis processing on the updated layer in each target screen according to the preset synthesis method to obtain a synthesized target layer corresponding to each target screen includes:

[0014] Based on a preset synthesis algorithm, creating a synthesis processing process for each corresponding target screen according to the preset synthesis method;

[0015] In each of the synthesis processing processes, the updated layer is synchronously subjected to layer synthesis processing to obtain the synthesized target layer.

[0016] Optionally, after performing layer synthesis processing on the updated layer in each target screen according to the preset synthesis method to obtain a synthesized target layer corresponding to each target screen, the method further includes:

[0017] Generate an image buffer area, and store the target layer in the image buffer area;

[0018] Transferring the image buffer to a preset hardware frame buffer; wherein the preset hardware frame buffer is an area of ​​the target screen that stores the target layer;

[0019] The target layer in the preset hardware frame buffer is displayed on the target screen.

[0020] Optionally, before obtaining the layer information corresponding to the layer to be synthesized in each target screen, the following steps are further included:

[0021] In response to the multi-screen merging instruction, acquiring layer data in each of the target screens;

[0022] The layer data is optimized using a preset processing algorithm to obtain the layer to be synthesized corresponding to each target screen.

[0023] Optionally, after displaying the target layer in the preset hardware frame buffer on the target screen, the method further includes:

[0024] Acquire operation information corresponding to the target layer; wherein the operation information is used to implement the update of the target layer in the target screen;

[0025] The target layer displayed on the target screen is updated according to the operation information.

[0026] According to a second aspect of the present disclosure, a multi-screen synthesis device is provided, comprising:

[0027] An acquisition unit, used to acquire layer information corresponding to the layer to be synthesized in each target screen;

[0028] The acquisition unit is further configured to, when it is determined according to the layer information that the layers to be synthesized need to be updated, acquire update information corresponding to each of the layers to be synthesized;

[0029] an updating unit, configured to update the layer to be synthesized in each target screen using the update information to obtain an updated layer;

[0030] The acquisition unit is further configured to acquire a preset synthesis mode corresponding to each target screen;

[0031] The synthesis unit is used to perform layer synthesis processing on the updated layer in each target screen according to the preset synthesis method to obtain a synthesized target layer corresponding to each target screen.

[0032] Optionally, the acquiring unit includes:

[0033] A construction module, configured to perform a hierarchical construction process on the layers to be synthesized using a preset construction algorithm to obtain a hierarchical structure of the layers to be synthesized;

[0034] An acquisition module, configured to respectively acquire layer sequence information corresponding to each layer to be synthesized in the hierarchical structure;

[0035] The calculation module is used to calculate the visible area based on a preset visual algorithm according to the layer sequence information and the layer attribute information to obtain the update information; wherein the layer information includes the layer attribute information.

[0036] Optionally, the synthesis unit includes:

[0037] A creation module, configured to create a synthesis processing process for each corresponding target screen according to the preset synthesis method based on a preset synthesis algorithm;

[0038] The synthesis module is used to synchronously perform layer synthesis processing on the updated layer in each synthesis processing process to obtain the synthesized target layer.

[0039] Optionally, the device further includes:

[0040] A generating unit, configured to generate an image buffer and store the target layer in the image buffer;

[0041] A transmission unit, configured to transmit the image buffer area to a preset hardware frame buffer area; wherein the preset hardware frame buffer area is an area of ​​the target screen that stores the target layer;

[0042] A display unit is configured to display the target layer in the preset hardware frame buffer on the target screen.

[0043] Optionally, the device further includes:

[0044] The acquisition unit is further configured to, in response to a multi-screen merging instruction, acquire the layer data in each of the target screens;

[0045] The optimization unit is used to perform data optimization processing on the layer data through a preset processing algorithm to obtain the layer to be synthesized corresponding to each target screen.

[0046] Optionally, the acquisition unit is further configured to acquire operation information corresponding to the target layer; wherein the operation information is used to implement an update of the target layer in the target screen;

[0047] The updating unit is further configured to update the target layer displayed on the target screen according to the operation information.

[0048] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0049] at least one processor; and

[0050] a memory communicatively connected to the at least one processor; wherein,

[0051] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0052] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0053] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method as described in the first aspect above.

[0054] The method and device, electronic device and storage medium for multi-screen synthesis provided by the present disclosure obtain layer information corresponding to the layer to be synthesized in each target screen; when it is determined that the layer to be synthesized needs to be updated based on the layer information, the update information corresponding to each layer to be synthesized is obtained, and the layer to be synthesized is updated in each target screen according to the update information to obtain the updated layer; the preset synthesis method corresponding to each target screen is obtained, and the updated layer is layer synthesized in each target screen according to the preset synthesis method to obtain the synthesized target layer corresponding to each target screen. Compared with the related art, the embodiment of the present disclosure decomposes the multi-screen synthesis process by performing layer synthesis on the updated layer in each target screen respectively, and independently processes the synthesis of each screen, thereby improving the synthesis load-bearing capacity in the case of concurrent multi-screen scenes, without affecting the overall time consumption of SurfaceFinger and DPU synthesis, reducing screen freezes, and improving system display fluency.

[0055] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0057] Figure 1 A flowchart of a multi-screen synthesis method provided by an embodiment of the present disclosure;

[0058] Figure 2 A schematic diagram of the principle of multi-screen synthesis provided by an embodiment of the present disclosure;

[0059] Figure 3A schematic diagram of a target layer display process provided by an embodiment of the present disclosure;

[0060] Figure 4 A schematic structural diagram of a multi-screen synthesis device provided by an embodiment of the present disclosure;

[0061] Figure 5 A schematic structural diagram of another multi-screen synthesis device provided by an embodiment of the present disclosure;

[0062] Figure 6 A schematic block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0063] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0064] The following describes a method and apparatus for multi-screen synthesis, an electronic device, and a storage medium according to embodiments of the present disclosure with reference to the accompanying drawings.

[0065] Figure 1 A flowchart of a multi-screen synthesis method provided by an embodiment of the present disclosure is provided.

[0066] like Figure 1 As shown, the method comprises the following steps:

[0067] Step 101: Obtain layer information corresponding to the layer to be synthesized in each target screen.

[0068] In the embodiment of the present disclosure, the layer information includes but is not limited to: attribute information of the layer to be synthesized, corresponding information between the target screen and the layer to be synthesized, Vsync cycle information, etc. Specifically, the embodiment of the present disclosure does not impose any restrictions on the layer information.

[0069] The Vsync cycle (vertical synchronization cycle) refers to the period at which the display refreshes its image, which determines how many frames per second the display can display. Vsync is a technology used to synchronize the refresh rate of a video display device with the frame rate generated by a video source (such as a graphics card) to avoid image tearing and motion blur.

[0070] Step 102, when it is determined based on the layer information that the layer to be synthesized needs to be updated, the update information corresponding to each layer to be synthesized is obtained, and the layer to be synthesized is updated in each target screen using the update information to obtain an updated layer.

[0071] In the embodiment of the present disclosure, the update information includes but is not limited to: a visualization area of ​​the layer to be synthesized, visual elements of the layer to be synthesized, etc. Specifically, the embodiment of the present disclosure does not limit the update information.

[0072] Regarding the update of the layers to be synthesized, it can be implemented in the following ways, but not limited to: rebuild layerstack, traverse the Z axis to calculate the visible area of ​​each layer, and update the layer-related information to the corresponding display. Among them, Layer: refers to a part that can be edited and manipulated independently in an image or design. It can contain elements such as images, text, and shapes. Each layer can be moved, scaled, rotated, and have transparency adjusted independently without affecting other layers. Designers can perform different edits on different layers, making the entire design more flexible and diverse. Layerstack: It usually refers to a stacking structure of all layers in the graphics rendering process. Each layer may contain different visual elements, such as text, pictures, etc. Display: Here refers to the display device, which may be a single screen or a display area composed of multiple screens. The concept of the Z axis: It represents the order in which the layers are stacked. The smaller the value on the Z axis, the closer the layer is to the front.

[0073] Rebuild Layerstack: Determine the hierarchical structure of the layerstack that needs to be created based on the needs of the application. Normally, the new layerstack will be modified or expanded based on the previous stack. Traverse the Z axis to calculate the visible area of ​​each layer: Traverse each layer in the layerstack and determine its visible area based on its position on the Z axis and other properties (such as transparency, occlusion relationship, etc.). The visible area refers to the part of the layer that is actually displayed on the display, which may be blocked by other layers. Update Layer information to the corresponding Display: Update the display on the display based on the calculated visible area of ​​each layer. This can be done through OpenGL, DirectX or other graphics APIs.

[0074] It should be noted that multi-screen compositing and layer processing can consume significant computing resources, so it's important to consider optimizing performance, such as using hardware acceleration and reducing unnecessary rendering operations. Furthermore, multiple tests and adjustments may be necessary in practice to ensure synchronization and correct display between the layerstack and the display.

[0075] Step 103: Obtain a preset synthesis method corresponding to each target screen, and perform layer synthesis processing on the updated layer in each target screen according to the preset synthesis method to obtain a synthesized target layer corresponding to each target screen.

[0076] In the embodiment of the present disclosure, the preset synthesis method is a synthesis method determined by considering the properties of the layers, such as format, size, rotation, etc., and how they are merged with other layers. Specifically, the preset synthesis method can be determined according to actual conditions, and the embodiment of the present disclosure does not limit it.

[0077] In order to facilitate understanding of the implementation process of the embodiment of the present disclosure, the present disclosure provides a schematic diagram of the principle of multi-screen synthesis, as shown in FIG. Figure 2 As shown, it means that when performing layer synthesis processing, each screen is synthesized independently, without serial processing, and will not cause its own display delay and jamming due to the long synthesis time of the previous screen. For example: Figure 2 As shown in the , the synthesis method described in this scheme is multi-threaded synthesis, that is, the first target screen...the third target screen are subjected to layer synthesis processing at the same time, and the overall synthesis time is: the synthesis time of a target screen with the longest synthesis time among all target screens (the time of layer synthesis processing in the first target screen). Compared with the overall synthesis time in the prior art being: the sum of the synthesis times of all target screens, the overall screen synthesis takes a shorter time, and at the same time, the display will not be delayed or stuck due to the long synthesis time of the previous screen.

[0078] Regarding the layer synthesis processing of the updated layer, it can be achieved by but not limited to the following methods: first, update the Layer data to the hardware compositor (HWC), traverse the Z-axis layer, save the visible OutputLayer information in CompositionState, and write it to HWC, traverse all displays, and at the same time, call beginFrame, prepareFrame, doDebugFlashRegions, doComposition, SurfaceFlinger.prepareFrame->RenderSurface.prepareFrame->HWComposer.prepar e in sequence, enter the HWComposer.prepare() method, determine which synthesis method the current display supports, and determine the synthesis method for each layer.

[0079] In the Android system, multi-screen composition is handled by SurfaceFlinger, a service responsible for compositing multiple Surfaces and rendering them on the display. The SurfaceFlinger.prepareFrame: method, part of SurfaceFlinger, prepares the frame to be composited. During this stage, SurfaceFlinger processes the SurfaceLayer components from different applications and determines how they should be composited.

[0080] RenderSurface.prepareFrame:RenderSurface is a class used internally by SurfaceFlinger that encapsulates rendering operations on the underlying hardware. In this method, the frame data is further processed to prepare for synthesis.

[0081] HWComposer.prepare:HWComposer is the hardware compositor in SurfaceFl inger, which is responsible for hardware-related composition operations. In this method, it is determined which composition methods the current display supports and which composition method each layer should use.

[0082] Compositing methods include software compositing and hardware compositing. Software compositing uses the GPU, while hardware compositing utilizes the DPU or a dedicated hardware unit. Hardware compositing generally offers better performance than software compositing. The hardware compositor (HWComposer) determines which compositing method to use based on the current hardware capabilities and system settings. This decision is based on various factors, such as the availability of hardware devices that support hardware compositing, the system's power management settings, and the current system load.

[0083] The compositing method for each layer is also determined at this stage, which includes considering the layer's properties, such as format, size, rotation, etc., and how they are merged with other layers. Correctly configuring the compositing method for each layer is crucial to ensuring overall performance and correct rendering.

[0084] When deciding how to use composition, the system also considers the balance between performance and power. For example, if the device is set to power saving mode, the system may prefer to use software composition, even though this may reduce performance. Conversely, if the device is set to performance priority mode, the system may choose hardware composition to achieve better performance.

[0085] Therefore, this call order and composition decision process is key to ensuring that Android devices can efficiently and correctly render multiple layers of content to the screen.

[0086] The method for multi-screen synthesis provided by the present disclosure obtains layer information corresponding to the layer to be synthesized in each target screen; when it is determined that the layer to be synthesized needs to be updated based on the layer information, obtains update information corresponding to each layer to be synthesized, and updates the layer to be synthesized in each target screen through the update information to obtain an updated layer; obtains a preset synthesis method corresponding to each target screen, and performs layer synthesis processing on the updated layer in each target screen according to the preset synthesis method to obtain the synthesized target layer corresponding to each target screen. Compared with the related art, the embodiment of the present disclosure decomposes the multi-screen synthesis process by performing layer synthesis processing on the updated layer in each target screen respectively, and independently processes the synthesis of each screen, thereby improving the synthesis load-bearing capacity in the case of concurrent multi-screen scenes, without affecting the overall time consumption of SurfaceFinger and DPU synthesis, reducing screen freezes, and improving system display fluency.

[0087] In one implementable manner of the embodiment of the present disclosure, in order to ensure efficient and correct display output in a multi-screen environment, it is necessary to obtain update information corresponding to the layer to be synthesized, and perform update processing on the layer to be synthesized in the target screen. Regarding the update process, it can be implemented in but not limited to the following manner: hierarchically constructing the layer to be synthesized through a preset construction algorithm to obtain a hierarchical structure of the layer to be synthesized; obtaining layer sequence information corresponding to each layer to be synthesized in the hierarchical structure, and based on a preset visual algorithm, performing visible area calculation according to the layer sequence information and layer attribute information to obtain the update information; wherein, the layer information includes the layer attribute information.

[0088] In the disclosed embodiment, the preset construction algorithm is an algorithm with custom settings, such as an algorithm modified or extended according to a previous stack, Adobe After Effects, Nuke, Blender, etc., wherein the hierarchical construction process is to construct layerstack (layer stack), Layerstack: refers to a stacked structure of all layers in the graphics rendering process, each layer contains different visual elements, such as text, pictures, etc. When performing the hierarchical construction process, the different elements and materials of the layers to be synthesized are organized into a hierarchical structure for synthesis and rendering, and the layers to be synthesized are input to the corresponding positions of the layer stack. According to the properties of each layer to be synthesized, such as transparency, blending mode, opacity, position, scale, etc., the appearance of the layer to be synthesized in the final output is determined. Finally, the layers to be synthesized are subjected to color correction, filters, animation keyframes and other operations. The hierarchical structure of the layers to be synthesized is obtained.

[0089] The preset visual algorithm is also an algorithm with custom settings, such as: OpenGL, DirectX and other graphics API operation algorithms, wherein the process of calculating the visible area includes but is not limited to: traversing the Z axis to calculate the visible area of ​​each layer, updating the layer-related information to the corresponding display, the Z axis represents the order of layer stacking, the smaller the value on the Z axis, the closer the layer is to the front, traversing each layer in the layerstack, and determining its visible area based on its position on the Z axis and other properties (such as transparency, occlusion relationship, etc.).

[0090] Since the visible area refers to the portion of the layer that is actually displayed on the display, which may be blocked by other layers, the calculation of the visible area can ensure synchronization and correct display between the layerstack and the display. Specifically, the embodiments of this disclosure do not limit the preset construction algorithm.

[0091] In one implementable method of the embodiment of the present disclosure, when performing layer synthesis processing, in order not to affect the overall synthesis time of SurfaceFinger, reduce the probability of application freeze, and improve the system display smoothness, it is necessary to create a synthesis processing thread for each screen, and the synthesis of each screen is processed independently. Therefore, the layer synthesis processing can be implemented in but not limited to the following ways: based on a preset synthesis algorithm, a synthesis processing process is created for the corresponding target screen according to the preset synthesis method; in each of the synthesis processing processes, the updated layer is synchronously subjected to layer synthesis processing to obtain the synthesized target layer.

[0092] In the embodiment of the present disclosure, the preset synthesis algorithm is a custom-selected algorithm, for example: the calling order of SurfaceFlinger.prepareFrame->RenderSurface.prepareFrame->HWComposer.prepar e, entering the HWComposer.prepare() method, etc. Specifically, the embodiment of the present disclosure does not limit the preset synthesis algorithm.

[0093] In one possible implementation of the embodiment of the present disclosure, after performing multi-screen synthesis processing, it is necessary to apply for GraphicBuffer, fill it with frame data, and finally submit it to the hardware frame buffer so that it can be displayed on the display device. Therefore, in order to correctly and clearly display the target layer, the present disclosure provides a flow chart of target layer display, such as Figure 3 Shown, including:

[0094] Step 301: Generate an image cache area and store the target layer in the image cache area.

[0095] In the disclosed embodiment, the image buffer is a GraphicBuffer, which is an abstraction for an image buffer in the Android system. It provides a memory area for storing pixel data. After compositing the layers, a GraphicBuffer needs to be applied to store the frame data to be displayed.

[0096] It's important to note that before generating the image buffer, the rendering engine composites all layers. In a multi-screen display system, there may be multiple layers, such as background and foreground layers. The rendering engine is responsible for compositing these layers according to a specific order and rules to produce the final video stream or image. This process may include operations such as blending, scaling, and rotating layers.

[0097] Storing the target layer in the image buffer means filling it with frame data (target layer). Once the GraphicBuffer is ready, the rendering engine will fill the synthesized image data into this buffer. This process involves memory management and data copying to ensure that the image data is correctly copied to the GraphicBuffer.

[0098] Step 302: Transfer the image buffer to a preset hardware frame buffer; wherein the preset hardware frame buffer is an area of ​​the target screen that stores the target layer.

[0099] In the embodiment of the present disclosure, transferring the image buffer area to the preset hardware frame buffer means: finally submitting it to the hardware frame buffer. Finally, the GraphicBuffer (image buffer area) filled with frame data is submitted to the hardware frame buffer. The hardware frame buffer is a memory area used by the display device to quickly display images. After submitting the data to this area, the display device can read the data directly from the hardware frame buffer and display it on the screen, so that the user can see the synthesized image.

[0100] Step 303: Display the target layer in the preset hardware frame buffer on the target screen.

[0101] In the embodiment of the present disclosure, the target layer in the preset hardware frame buffer is directly called by the target screen, and can be displayed in the target screen.

[0102] In one implementable method of the embodiment of the present disclosure, before the synthesis processing is performed, the layer data will also be optimized to optimize the final display effect. Therefore, in order to achieve a better display effect, the following method can also be adopted but is not limited to: in response to a multi-screen merging instruction, the layer data in each of the target screens is obtained; the layer data is optimized through a preset processing algorithm to obtain the layer to be synthesized corresponding to each of the target screens.

[0103] In the embodiment of the present disclosure, the preset processing algorithm is a custom-selected algorithm, which can perform a series of processing on the content of each screen before the actual rendering operation to optimize the final display effect. Specifically, the embodiment of the present disclosure does not limit the preset processing algorithm.

[0104] The data optimization processing of the layer data can also be achieved through, but is not limited to, the following methods: Geometric transformation: Performing geometric transformations such as scaling, rotation, and translation on screen content to adapt to the sizes and proportions of different screens. Color correction: Adjusting the color balance of different screens to ensure color consistency after compositing. Occlusion processing: Determining the occlusion relationship between screens and correctly occluding overlapping areas to ensure accurate display of screen content. Effect preprocessing: Applying pre-rendering effects such as shadows, lighting, and filters to graphic elements on the screen to reduce the pressure of real-time rendering. Material preparation: Pre-processing the materials to be composited, such as pre-rendering animation frames and pre-calculating dynamic effects. Performance optimization: Optimizing the content to be composited to reduce the amount of computation required during rendering and improve performance, such as using texture mapping to reduce repeated rendering. Composition pre-calculation: Pre-calculating the blending mode and transparency required for interactive effects between multiple screens. Dynamic adjustment: Dynamically adjusting pre-processed content based on user operations or application logic to adapt to different display requirements. Resource management: Rationally managing graphics card resources, caching and reusing pre-composited resources to improve efficiency.

[0105] The above preprocessing steps can significantly improve the efficiency of multi-screen synthesis and the quality of the final presentation. In practical applications, this step may be completed automatically through software programming, or in some hardware-accelerated environments, it may be accelerated by hardware units such as GPUs.

[0106] In one implementable manner of the embodiment of the present disclosure, after the layers are merged, the target layer also needs to be post-merged to ensure that the target layer can perform status updates and interactive processing at the correct time. Therefore, after the target layer is displayed, the following method can also be used but is not limited to: obtaining operation information corresponding to the target layer; wherein the operation information is used to implement the update of the target layer in the target screen; and updating the target layer displayed in the target screen according to the operation information.

[0107] In the embodiment of the present disclosure, the operation information includes but is not limited to: layer blending, color space conversion, filter application, etc. Specifically, the embodiment of the present disclosure does not limit the operation information.

[0108] The implementation process of the disclosed embodiments can be achieved through, but not limited to, the following methods: Calling back each layer's onPostComposition callback. In a multi-screen or multi-window environment, each layer may have its own rendering and update logic. The onPostComposition callback is a notification mechanism that tells the layer to perform additional operations after completing a composition operation.

[0109] Composition generally refers to the process of drawing a layer's contents to the screen, which may include layer blending, color space conversion, filter application, etc. When these operations are complete, the layer's onPostComposition callback is called, allowing the layer to perform any necessary post-processing, such as updating the display, handling user interactions, or transitioning between states.

[0110] In this callback, the layer can access the results of the composite operation that just completed and process them accordingly. For example, an animated layer can update its animation state in the onPostComposition callback or adjust the displayed content based on the current animation progress.

[0111] In the Android system, SurfaceView and View layers can use the onDraw method to handle drawing logic, but when it comes to composite layers, such as using hardware-accelerated Surface or Overlay layers, you need to use the onPostComposition callback to handle subsequent operations of these layers after the composite is completed.

[0112] In general, the onPostComposition callback is an important mechanism used in multi-screen merging to ensure that the layer can perform status updates and interaction processing at the correct time.

[0113] In summary, the embodiments of the present disclosure can achieve the following effects:

[0114] The disclosed embodiment decomposes the multi-screen synthesis process by performing layer synthesis processing on the updated layer in each of the target screens respectively, and independently processes the synthesis of each screen, thereby improving the synthesis load-bearing capacity in concurrent multi-screen scenes, without affecting the overall synthesis time of SurfaceFinger and DPU, reducing screen freezes, and improving system display smoothness.

[0115] Corresponding to the above-mentioned multi-screen synthesis method, the present invention also provides a multi-screen synthesis device. Since the device embodiment of the present invention corresponds to the above-mentioned method embodiment, details not disclosed in the device embodiment can be referred to the above-mentioned method embodiment and will not be repeated in this invention.

[0116] Figure 4 A schematic diagram of the structure of a multi-screen synthesis device provided by an embodiment of the present disclosure is shown as follows: Figure 4 As shown, including:

[0117] An acquisition unit 41 is used to acquire layer information corresponding to the layer to be synthesized in each target screen;

[0118] The acquisition unit 41 is further configured to acquire update information corresponding to each of the layers to be synthesized when it is determined according to the layer information that the layers to be synthesized need to be updated;

[0119] An updating unit 42, configured to update the layer to be synthesized in each target screen using the update information to obtain an updated layer;

[0120] The acquisition unit 41 is further configured to acquire a preset synthesis mode corresponding to each target screen;

[0121] The synthesis unit 43 is configured to perform layer synthesis processing on the updated layer in each target screen according to the preset synthesis method to obtain a synthesized target layer corresponding to each target screen.

[0122] The multi-screen synthesis device provided by the present disclosure obtains layer information corresponding to the layer to be synthesized in each target screen; when it is determined that the layer to be synthesized needs to be updated based on the layer information, the update information corresponding to each layer to be synthesized is obtained, and the layer to be synthesized is updated in each target screen according to the update information to obtain an updated layer; the preset synthesis method corresponding to each target screen is obtained, and the updated layer is subjected to layer synthesis processing in each target screen according to the preset synthesis method to obtain the synthesized target layer corresponding to each target screen. Compared with the related art, the embodiment of the present disclosure decomposes the multi-screen synthesis process by performing layer synthesis processing on the updated layer in each target screen respectively, and independently processes the synthesis of each screen, thereby improving the synthesis load-bearing capacity in the case of concurrent multi-screen scenes, without affecting the overall time consumption of SurfaceFinger and DPU synthesis, reducing screen freezes, and improving system display fluency.

[0123] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 5As shown, the acquisition unit 41 includes:

[0124] A construction module 411 is configured to perform a hierarchical construction process on the layers to be synthesized using a preset construction algorithm to obtain a hierarchical structure of the layers to be synthesized;

[0125] An acquisition module 412 is configured to acquire layer sequence information corresponding to each layer to be synthesized in the hierarchical structure;

[0126] The calculation module 413 is configured to calculate the visible area based on a preset visual algorithm and the layer sequence information and layer attribute information to obtain the update information; wherein the layer information includes the layer attribute information.

[0127] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 5 As shown, the synthesis unit 43 includes:

[0128] A creation module 431 is configured to create a synthesis process for each target screen based on a preset synthesis algorithm and according to the preset synthesis method;

[0129] The synthesis module 432 is used to synchronously perform layer synthesis processing on the updated layer in each synthesis processing process to obtain the synthesized target layer.

[0130] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 5 As shown, the device also includes:

[0131] A generating unit 44 is configured to generate an image buffer and store the target layer in the image buffer;

[0132] The transmission unit 45 is configured to transmit the image buffer to a preset hardware frame buffer; wherein the preset hardware frame buffer is an area of ​​the target screen that stores the target layer;

[0133] The display unit 46 is configured to display the target layer in the preset hardware frame buffer on the target screen.

[0134] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 5 As shown, the device also includes:

[0135] The acquisition unit 41 is further configured to, in response to a multi-screen merging instruction, acquire the layer data in each target screen;

[0136] The optimization unit 47 is configured to perform data optimization processing on the layer data using a preset processing algorithm to obtain the layer to be synthesized corresponding to each target screen.

[0137] Furthermore, in a possible implementation of the embodiment of the present disclosure, the acquisition unit 41 is further configured to acquire operation information corresponding to the target layer; wherein the operation information is used to implement an update of the target layer in the target screen;

[0138] The updating unit 42 is further configured to update the target layer displayed on the target screen according to the operation information.

[0139] It should be noted that the above explanation of the method embodiment is also applicable to the device of the embodiment of the present disclosure, and the principles are the same, which is no longer limited in the embodiment of the present disclosure.

[0140] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0141] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0142] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 602 or a computer program loaded from a storage unit 608 into a RAM (Random Access Memory) 603. Various programs and data required for the operation of the device 600 can also be stored in the RAM 603. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An I / O (Input / Output) interface 605 is also connected to the bus 604.

[0143] Various components in device 600 are connected to I / O interface 605, including an input unit 606, such as a keyboard, mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, optical disk, etc.; and a communication unit 609, such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0144] The computing unit 601 can be various general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), various specialized AI (Artificial Intelligence) computing chips, various computing units that run machine learning model algorithms, a DSP (Digital Signal Processor), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the multi-screen synthesis method. For example, in some embodiments, the multi-screen synthesis method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the method described above can be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the aforementioned multi-screen synthesis method in any other appropriate manner (for example, by means of firmware).

[0145] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System on Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0146] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0147] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0148] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0149] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0150] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship is established by computer programs running on the respective computers, establishing a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and VPS services ("Virtual Private Servers" or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0151] It's important to note that artificial intelligence (AI) is the study of how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). This encompasses both hardware and software technologies. AI hardware technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily encompass computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graphs.

[0152] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0153] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for multi-screen synthesis, characterized in that: include: Get the layer information corresponding to the layer to be synthesized in each target screen; When it is determined according to the layer information that the layer to be synthesized needs to be updated, obtaining update information corresponding to each layer to be synthesized, and updating the layer to be synthesized in each target screen according to the update information to obtain an updated layer; A preset synthesis mode corresponding to each target screen is obtained, and layer synthesis processing is performed on the updated layer in each target screen according to the preset synthesis mode to obtain a synthesized target layer corresponding to each target screen.

2. The method according to claim 1, characterized in that The obtaining of update information corresponding to each layer to be synthesized includes: Performing hierarchical construction processing on the layers to be synthesized using a preset construction algorithm to obtain a hierarchical structure of the layers to be synthesized; The layer sequence information corresponding to each layer to be synthesized is obtained in the hierarchical structure respectively, and based on a preset visual algorithm, a visible area is calculated according to the layer sequence information and layer attribute information to obtain the update information; wherein the layer information includes the layer attribute information.

3. The method according to claim 1, characterized in that The performing layer synthesis processing on the updated layer in each target screen according to the preset synthesis method to obtain the synthesized target layer corresponding to each target screen includes: Based on a preset synthesis algorithm, creating a synthesis processing process for each corresponding target screen according to the preset synthesis method; In each of the synthesis processing processes, layer synthesis processing is synchronously performed on the updated layer to obtain the synthesized target layer corresponding to each of the target screens.

4. The method according to claim 1, wherein After performing layer synthesis processing on the updated layer in each target screen according to the preset synthesis method to obtain a synthesized target layer corresponding to each target screen, the method further includes: Generate an image buffer area, and store the target layer in the image buffer area; Transferring the image buffer to a preset hardware frame buffer; wherein the preset hardware frame buffer is an area of ​​the target screen that stores the target layer; The target layer in the preset hardware frame buffer is displayed on the target screen.

5. The method according to claim 1, wherein Before obtaining the layer information corresponding to the layer to be synthesized in each target screen, the following steps are also included: In response to the multi-screen merging instruction, acquiring layer data in each of the target screens; The layer data is optimized using a preset processing algorithm to obtain the layer to be synthesized corresponding to each target screen.

6. The method according to claim 4, characterized in that After displaying the target layer in the preset hardware frame buffer on the target screen, the method further includes: Acquire operation information corresponding to the target layer; wherein the operation information is used to implement the update of the target layer in the target screen; The target layer displayed on the target screen is updated according to the operation information.

7. A multi-screen synthesis device, characterized in that: include: An acquisition unit, used to acquire layer information corresponding to the layer to be synthesized in each target screen; The acquisition unit is further configured to, when it is determined according to the layer information that the layers to be synthesized need to be updated, acquire update information corresponding to each of the layers to be synthesized; an updating unit, configured to update the layer to be synthesized in each target screen using the update information to obtain an updated layer; The acquisition unit is further configured to acquire a preset synthesis mode corresponding to each target screen; The synthesis unit is used to perform layer synthesis processing on the updated layer in each target screen according to the preset synthesis method to obtain a synthesized target layer corresponding to each target screen.

8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images