Layer composition method, terminal device, and storage medium

Abstract

The present application relates to the technical field of terminals. Provided are a layer composition method, a terminal device, and a storage medium. In the solution, during the process where a user uses a large screen, a terminal device may identify each layer to be subjected to composition, so as to determine whether the layer requires the use of GPU preprocessing, e.g., the rotation of a wallpaper layer; and if the use of GPU preprocessing is required, a GPU is used to render a preprocessed layer, and the preprocessed layer is stored in a cache region. When there is subsequently the same layer processing task, the preprocessed layer in the cache region may be directly sent to an HWC for hardware composition, without requiring the use of GPU composition, such that the overall power consumption and load of the terminal device are reduced, thereby improving the overall performance.

Description

Layer synthesis method, terminal device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on September 13, 2023, with application number 202311187697.3 and application name “Layer Synthesis Method, Terminal Device and Storage Medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to a layer synthesis method, terminal device and storage medium. Background Art

[0003] Large-screen devices are increasingly being used. Users can adjust the screen orientation at any time while using these devices. Large-screen devices can rotate wallpaper resources so that the wallpaper's orientation remains consistent from the user's perspective.

[0004] Currently, most large-screen devices use large-sized high-definition wallpapers. When the height of the wallpaper resource exceeds the maximum rotation height supported by the hardware composer (HWC), a graphics processing unit (GPU) can be used for synthesis. The GPU first rotates the wallpaper resource 90°, then sends the rotated wallpaper resource to the HWC for hardware synthesis, and then sends it to the large screen for display, so that the user sees that the wallpaper direction remains consistent before and after adjusting the screen direction. However, GPU synthesis has the disadvantage of high power consumption. When multiple frames of images are synthesized based on the same wallpaper resource, the GPU may need to perform multiple rotation operations on the same wallpaper resource, resulting in high power consumption and poor performance of the large-screen device.

[0005] Summary of the Invention

[0006] The present application provides a layer synthesis method, a terminal device, and a storage medium, which solve the technical problem that the GPU frequently rotates wallpaper layers, resulting in high power consumption and poor performance of the terminal device.

[0007] To achieve the above objectives, this application adopts the following technical solutions:

[0008] In a first aspect, an embodiment of the present application provides a layer synthesis method. The method can be applied to a terminal device. Taking a folding screen device as an example, the folding screen device may include a GPU, a HWC, a large screen, and a small screen. The method may include:

[0009] When the folding screen device is at a first folding angle (i.e., a folded state), if the first wallpaper layer is obtained, the first wallpaper layer, the first desktop layer, and the first status bar layer are synthesized into a first image, and the first image is displayed on the small screen. If the user receives an operation to adjust the folding screen device from the first folding angle to the second folding angle (such as switching from the folded state to the unfolded state), then in response to the operation, the first wallpaper layer is obtained, and the first wallpaper layer is rotated by the GPU to obtain a first pre-processed layer, the first pre-processed layer, the second desktop layer, and the second status bar layer are synthesized into a second image, the first pre-processed layer is stored in the first cache area, and the second image is displayed on the large screen. While maintaining the unfolded state, the first pre-processed layer is obtained from the first cache area, the first pre-processed layer, the third desktop layer, and the third status bar layer are synthesized into a third image, and the third image is displayed on the large screen.

[0010] The second folding angle is greater than the first folding angle. The first folding angle is less than or equal to a first preset angle, which may be 30° or 45°, and the second folding angle is greater than or equal to a second preset angle, which may be 90° or 120°.

[0011] In the above scheme, when the folding screen device is in the folded state, the installation direction of the small screen is consistent with the direction of the wallpaper, so there is no need to rotate the wallpaper layer, and the composite image of the wallpaper layer and other layers is directly displayed on the small screen. When the folding screen device switches from the folded state to the unfolded state, the installation direction of the large screen is inconsistent with the wallpaper direction, and the height of the first wallpaper layer is greater than the maximum rotation height supported by the HWC. Therefore, the GPU is required to rotate the first wallpaper layer in advance to obtain a pre-processed layer, and store the pre-processed layer in the cache. In this way, when there are the same layer processing tasks, the pre-processed layer in the cache can be directly sent to the HWC for hardware synthesis, without having to use the GPU to rotate the wallpaper layer, thereby reducing the overall power consumption and load of the folding screen device and improving the overall performance of the folding screen device.

[0012] It should be noted that the first desktop layer, the second desktop layer, and the third desktop layer can be the same layer or different layers; the first status bar layer, the second status bar layer, and the third status bar layer can be the same layer or different layers. In addition, this application does not specifically limit the execution order of "combining the first pre-processed layer, the second desktop layer, and the second status bar layer into the second image" and "storing the first pre-processed layer in the first cache area". The two can be executed at the same time, or "combining the first pre-processed layer, the second desktop layer, and the second status bar layer into the second image" can be executed first and then "storing the first pre-processed layer in the first cache area". It can also be executed first and then "storing the first pre-processed layer in the first cache area" can be executed.

[0013] In one possible implementation, the folding screen device is an inward folding screen device. The inward folding screen device is provided with an inner screen and an outer screen. The size of the inner screen is larger than that of the outer screen, so the inner screen is also called the large screen, and the outer screen is also called the small screen. The large screen can be bent into two display areas. When the inward folding screen device is in a folded state, the folding angle between the two display areas of the large screen is less than or equal to a first preset angle, the small screen faces the user, and the large screen is in a hidden state; when the inward folding screen device is in an unfolded state, the folding angle between the two display areas of the large screen is greater than or equal to a second preset angle, the large screen faces the user, and the small screen is away from the user. When in the folded state, the small screen displays the wallpaper; after switching from the folded state to the unfolded state, the large screen displays the wallpaper. Among them, the large screen and the small screen can use the same wallpaper resources.

[0014] In one possible implementation, the folding screen device is an external folding screen device. The external folding screen device is provided with a screen that can be bent into at least two display areas. The larger of the at least two display areas is called the main screen, and the smaller display area is called the back screen or sub-screen. Usually, the main screen is also called the small screen, and the main screen and the back screen are collectively referred to as the large screen. When the external folding screen device is in a folded state, the folding angle between the main screen and the back screen is less than or equal to a first preset angle, the main screen faces the user, and the back screen faces away from the user; when the external folding screen device is in an unfolded state, the folding angle between the main screen and the back screen is greater than or equal to a second preset angle, and the main screen and the back screen face the user. When in the folded state, the small screen displays the wallpaper; after switching from the folded state to the unfolded state, the large screen displays the wallpaper. The large screen and the small screen can use the same wallpaper resource.

[0015] In one possible implementation, when the foldable screen device switches from a folded state to an unfolded state, obtaining the first wallpaper layer includes: obtaining a first Vsync-SF signal; and if a second buffer area has been obtained before obtaining the first Vsync-SF signal, obtaining the first wallpaper layer from the second buffer area. The second buffer area is used to store the first wallpaper layer.

[0016] In the above solution, when the wallpaper of the wallpaper application changes, the changed wallpaper layer can be stored in the second buffer area. In this way, when the first Vsync-SF signal is obtained, the wallpaper layer can be directly read from the second buffer area.

[0017] In one possible implementation, after obtaining the first Vsync-SF signal, a second Vsync-SF signal may also be obtained. SurfaceFlinger then determines whether, after obtaining the first Vsync-SF signal and before obtaining the second Vsync-SF signal, other cache areas other than the second cache area are obtained, where other cache areas refer to cache areas storing other wallpaper layers. In one case, if no other cache areas are obtained before obtaining the second Vsync-SF signal, the first pre-processed layer is obtained from the first cache area. In another case, if the third cache area is obtained before obtaining the second Vsync-SF signal, the second wallpaper layer is obtained from the third cache area, the second wallpaper layer is rotated by the GPU to obtain the second pre-processed layer (the height of the second wallpaper layer is greater than the maximum rotation height supported by the HWC), and the second pre-processed layer is stored in the fourth cache area; the second pre-processed layer, the fourth desktop layer, and the fourth status bar layer are synthesized into a fourth image; and the fourth image is displayed on the large screen. The second Vsync-SF signal is a signal generated after the first Vsync-SF signal. The second Vsync-SF signal differs from the first Vsync-SF signal by N Vsync-SF signal cycles, where N is a positive integer.

[0018] In the above solution, when the wallpaper resource is a video resource, the wallpaper is updated frequently from the first frame to the last frame of the video resource. The GPU can temporarily store each frame of wallpaper in the cache after rotating it. If no new wallpaper layer is obtained after obtaining the first Vsync-SF signal and before obtaining the second Vsync-SF signal (i.e., within m Vsync cycles after playing a certain frame), it means that the last frame of the video resource has been played, and the pre-processed layer in the cache can be used.

[0019] In one possible implementation, the row scanning direction of the small screen is parallel to the first frame, and the column scanning direction of the small screen is parallel to the second frame. The row scanning direction of the large screen is parallel to the second frame, and the column scanning direction of the large screen is parallel to the first frame. The first frame and the second frame are two adjacent frames on the body of the foldable screen device, and the length of the first frame is shorter than the length of the second frame.

[0020] In the above solution, the installation directions of the small screen and the large screen are inconsistent, and the display direction of the small screen and the wallpaper resource are consistent, so the small screen does not need to rotate the wallpaper, but the large screen needs to rotate the wallpaper.

[0021] In a second aspect, an embodiment of the present application provides a layer synthesis method. The method can be applied to a terminal device. The terminal device can be a large-screen device, such as a foldable screen device or a PAD. The large-screen device may include a GPU, a HWC, and a large screen. The method may include:

[0022] When the large-screen device is in landscape mode, if the first wallpaper layer is obtained, the first wallpaper layer, the first desktop layer, and the first status bar layer are synthesized into a first image, and the first image is displayed on the large screen. If the user's operation of adjusting the large-screen device from landscape mode to portrait mode is obtained, then in response to the operation, the first wallpaper layer is obtained, and the first wallpaper layer is rotated by the GPU to obtain a first pre-processed layer, the first pre-processed layer, the second desktop layer, and the second status bar layer are synthesized into a second image, the first pre-processed layer is stored in the first buffer area, and the second image is displayed on the large screen. Then, while maintaining the portrait mode, the first pre-processed layer is obtained from the first buffer area, the first pre-processed layer, the third desktop layer, and the third status bar layer are synthesized into a third image, and the third image is displayed on the large screen.

[0023] In the above solution, when the large-screen device is in landscape mode, the display direction of the large screen is consistent with the wallpaper direction, and there is no need to rotate the wallpaper layer. The composite image of the wallpaper layer and other layers is directly displayed on the large screen. When the large-screen device is adjusted from landscape mode to portrait mode, the display direction of the large screen is inconsistent with the wallpaper direction, and the wallpaper layer needs to be rotated. The GPU can pre-rotate the first wallpaper layer to obtain a pre-processed layer, and store the pre-processed layer in the cache. Therefore, when there are the same layer processing tasks, the pre-processed layer in the cache can be directly sent to the HWC for hardware synthesis without using GPU synthesis, thereby reducing the overall power consumption and load of the large-screen device and improving the overall performance of the large-screen device.

[0024] In one possible implementation, when the screen is adjusted from a landscape orientation to a portrait orientation, obtaining the first wallpaper layer includes: obtaining a first Vsync-SF signal; and if a second buffer is obtained before the first Vsync-SF signal is obtained, obtaining the first wallpaper layer from the second buffer. The second buffer is used to store the first wallpaper layer.

[0025] In one possible implementation, after obtaining the first Vsync-SF signal, a second Vsync-SF signal may also be obtained. SurfaceFlinger then determines whether, after obtaining the first Vsync-SF signal and before obtaining the second Vsync-SF signal, other cache areas other than the second cache area are obtained, where other cache areas refer to cache areas storing other wallpaper layers. In one case, if no other cache areas are obtained before obtaining the second Vsync-SF signal, the first pre-processed layer is obtained from the first cache area. In another case, if the third cache area is obtained before obtaining the second Vsync-SF signal, the second wallpaper layer is obtained from the third cache area, the second wallpaper layer is rotated by the GPU to obtain the second pre-processed layer (the height of the second wallpaper layer is greater than the maximum rotation height supported by the HWC), and the second pre-processed layer is stored in the fourth cache area; the second pre-processed layer, the fourth desktop layer, and the fourth status bar layer are synthesized into a fourth image; and the fourth image is displayed on the large screen. The second Vsync-SF signal is a signal generated after the first Vsync-SF signal. The second Vsync-SF signal differs from the first Vsync-SF signal by N Vsync-SF signal cycles, where N is a positive integer.

[0026] In the above solution, when the wallpaper resource is a video resource, the wallpaper is updated frequently from the first frame to the last frame of the video resource. The GPU can temporarily store each frame of wallpaper in the cache after rotating it. If no new wallpaper layer is obtained after obtaining the first Vsync-SF signal and before obtaining the second Vsync-SF signal (i.e., within m Vsync cycles after playing a certain frame), it means that the last frame of the video resource has been played, and the pre-processed layer in the cache can be used.

[0027] In one possible implementation, the row scanning direction of the large screen is parallel to the second frame, and the column scanning direction of the large screen is parallel to the first frame. The first frame and the second frame are two adjacent frames on the main body of the large screen device, and the length of the first frame is shorter than the length of the second frame.

[0028] In a third aspect, an embodiment of the present application provides a layer synthesis method. This method can be applied to a terminal device. Taking a folding screen device as an example, the folding screen device may include a GPU, a HWC, a large screen, and a small screen. The method may include:

[0029] When the foldable screen device is at a first folding angle (unfolded state), if a first wallpaper layer is obtained, the first wallpaper layer is rotated by the GPU to obtain a first pre-processed layer, the first pre-processed layer, the first desktop layer, and the first status bar layer are synthesized into a first image, the first pre-processed layer is stored in a first buffer area, and the first image is displayed on the large screen. While maintaining the first folding angle, the first pre-processed layer can be obtained from the first buffer area, the first pre-processed layer, the second desktop layer, and the second status bar layer are synthesized into a second image, and the second image is displayed on the large screen.

[0030] If a user operation is received to adjust the folding screen device from a first folding angle to a second folding angle (the second folding angle is not equal to the first folding angle and is still in an unfolded state), then in response to the operation, a second wallpaper layer is obtained, and the second wallpaper layer is rotated by the GPU to obtain a second pre-processed layer, the second pre-processed layer, the third desktop layer, and the third status bar layer are synthesized into a third image, the second pre-processed layer is stored in the second cache area, and the third image is displayed on the large screen. While maintaining the second folding angle, the second pre-processed layer can be obtained from the second cache area, the second pre-processed layer, the fourth desktop layer, and the fourth status bar layer are synthesized into a fourth image, and the fourth image is displayed on the large screen.

[0031] In the above scheme, when the folding screen device is in the unfolded state, if the folding angle is increased or decreased, the wallpaper layer is updated, and the GPU rotates the new wallpaper layer in advance to obtain a pre-processed layer, and stores the pre-processed layer in the cache area. Therefore, the pre-processed layer in the cache area can be directly sent to the HWC for hardware synthesis without using GPU synthesis, thereby reducing the overall power consumption and load of the folding screen device and improving the overall performance of the folding screen device.

[0032] In one possible implementation, when the foldable screen device is at a first folded angle, obtaining a first wallpaper layer includes: obtaining a first Vsync-SF signal; and if buffer area a is obtained before obtaining the first Vsync-SF signal, obtaining the first wallpaper layer from buffer area a. Buffer area a is used to store the first wallpaper layer.

[0033] Correspondingly, after obtaining the first Vsync-SF signal, a second Vsync-SF signal can also be obtained. Then, SurfaceFlinger determines whether other buffer areas other than buffer area a are obtained after obtaining the first Vsync-SF signal and before obtaining the second Vsync-SF signal. Other buffer areas refer to buffer areas storing other wallpaper layers. If no other buffer areas are obtained before obtaining the second Vsync-SF signal, the first pre-processed layer is obtained from the first buffer area.

[0034] In one possible implementation, when the foldable screen device is at the second folding angle, obtaining the second wallpaper layer includes: obtaining a third Vsync-SF signal; and if buffer area b is obtained before the third Vsync-SF signal is obtained, obtaining the second wallpaper layer from buffer area b. Buffer area b is used to store the second wallpaper layer.

[0035] Correspondingly, after obtaining the third Vsync-SF signal, a fourth Vsync-SF signal can also be obtained. Then, SurfaceFlinger determines whether other buffer areas other than buffer area b are obtained after obtaining the third Vsync-SF signal and before obtaining the fourth Vsync-SF signal. Other buffer areas refer to buffer areas storing other wallpaper layers. If no other buffer areas are obtained before obtaining the fourth Vsync-SF signal, the second pre-processed layer is obtained from the second buffer area.

[0036] In one possible implementation, the row scanning direction of the small screen is parallel to the first frame, and the column scanning direction of the small screen is parallel to the second frame. The row scanning direction of the large screen is parallel to the second frame, and the column scanning direction of the large screen is parallel to the first frame. The first frame and the second frame are two adjacent frames on the body of the foldable screen device, and the length of the first frame is shorter than the length of the second frame.

[0037] In a fourth aspect, an embodiment of the present application provides a layer synthesis method. The method can be applied to a terminal device. The terminal device is provided with a GPU, a HWC, and a display screen. For example, the terminal device can be a folding screen device or a large screen device. The method may include: obtaining a first wallpaper layer, the height of the first wallpaper layer is greater than the maximum rotation height supported by the HWC; rotating the first wallpaper layer by the GPU to obtain a first pre-processed layer; synthesizing the first pre-processed layer, the first desktop layer, and the first status bar layer into a first image; storing the first pre-processed layer in a first buffer area; displaying the first image on the display screen; obtaining the first pre-processed layer from the first buffer area; synthesizing the first pre-processed layer, the second desktop layer, and the second status bar layer into a second image; and displaying the second image on the display screen.

[0038] In the above solution, when the terminal device first obtains the first wallpaper layer and the height of the first wallpaper layer is greater than the maximum rotation height supported by the HWC, the GPU can be used to rotate the first wallpaper layer to obtain a pre-processed layer, and the pre-processed layer can be stored in a cache. Since the GPU pre-rotates the first wallpaper layer to obtain the pre-processed layer and stores the pre-processed layer in the cache, when the terminal device obtains the synthesis task of the first wallpaper layer again, the pre-processed layer in the cache can be directly sent to the HWC for hardware synthesis without using GPU synthesis, thereby reducing the overall power consumption and load of the terminal device and improving the overall performance of the terminal device.

[0039] In one possible implementation, obtaining the first wallpaper layer includes: obtaining a first Vsync-SF signal; if the second buffer is already obtained before the first Vsync-SF signal is obtained, obtaining the first wallpaper layer from the second buffer. Accordingly, obtaining the first pre-processed layer from the first buffer includes: obtaining a second Vsync-SF signal; if no other buffer is obtained before the second Vsync-SF signal is obtained, obtaining the first pre-processed layer from the first buffer.

[0040] In one possible implementation, after obtaining the second Vsync-SF signal, the method further includes: if the third cache area is obtained before the second Vsync-SF signal is obtained, obtaining the second wallpaper layer from the third cache area; rotating the second wallpaper layer through the GPU to obtain a second pre-processed layer, and the height of the second wallpaper layer is greater than the maximum rotation height supported by the HWC; storing the second pre-processed layer in the fourth cache area; synthesizing the second pre-processed layer, the third desktop layer and the third status bar layer into a third image; and displaying the third image on the display screen.

[0041] In one possible implementation, the screen of the terminal device can be mounted horizontally. For example, the row scanning direction of the display screen is parallel to the second frame, and the column scanning direction of the display screen is parallel to the first frame. The first frame and the second frame are two adjacent frames on the body of the terminal device, and the length of the first frame is shorter than the length of the second frame.

[0042] In the above solution, since the direction of the wallpaper resource is consistent with the horizontal screen state, when in the vertical screen state, the wallpaper resource needs to be rotated first, and then the rotated wallpaper is displayed on the screen, so that from the user's visual perspective, the direction of the wallpaper displayed on the small screen and the large screen is consistent, thereby improving the user experience.

[0043] In one possible implementation, obtaining the first wallpaper layer from the second cache includes obtaining the first wallpaper layer and layer information of the first wallpaper layer from the second cache; wherein the layer information of the first wallpaper layer includes a first status bit and a rotation parameter, the first status bit indicating that the first wallpaper layer has not been rotated, and the rotation parameter indicating a rotation direction and angle. Accordingly, rotating the first wallpaper layer using a GPU to obtain a first pre-processed layer includes rotating the first wallpaper layer using the GPU based on the first status bit and the rotation parameter to obtain the first pre-processed layer.

[0044] In one possible implementation, before obtaining the first wallpaper layer, the method further includes: determining a rotation parameter if a preset condition is met; sending the rotation parameter to a wallpaper application on the terminal device; generating a first status bit via the wallpaper application, and storing the first status bit, the rotation parameter, and the first wallpaper layer in a second buffer. The first status bit indicates that the resources of the first wallpaper layer have not yet been rotated. For example, the status bar layer and the desktop layer are layers drawn by the application and rotated, and the "rotated" status bit is used. For another example, the wallpaper layer and the video layer are layers read by the video codec and not yet rotated, and the "unrotated" status bit is used. The rotation parameter indicates the direction and angle of rotation, such as 90° counterclockwise rotation or 90° clockwise rotation. The preset condition includes any of the following: the display screen switches from a landscape mode to a portrait mode; the terminal device is a foldable screen device, and the foldable screen device switches from a folded state to an unfolded state.

[0045] In the above solution, the wallpaper application writes the first state bit and rotation parameters in the cache area, so that SurfaceFlinger determines whether the wallpaper layer needs to be rotated and how to rotate the wallpaper layer according to the first state bit and rotation parameters.

[0046] In one possible implementation, the method may further include: in response to a user operation, turning on the landscape and portrait screen functions; wherein the landscape and portrait screen functions allow display in a first direction when the terminal device is in a landscape state, and display in a second direction when the terminal device is in a portrait state; the first direction and the second direction differ by 90° or 270° clockwise.

[0047] In one possible implementation, when the terminal device is in landscape mode, the angle between a first frame of the terminal device's body and the ground plane is greater than the angle between a second frame of the terminal device's body and the ground plane; and when the terminal device is in portrait mode, the angle between the first frame and the ground plane is less than or equal to the angle between the second frame and the ground plane. The length of the first frame is less than the length of the second frame.

[0048] In one possible implementation, the layer information of the wallpaper layer may further include the layer overlay order of the first wallpaper layer. Combining the first preprocessing layer, the first desktop layer, and the first status bar layer into a first image includes: according to the layer overlay order of the first wallpaper layer, the layer overlay order of the first desktop layer, and the layer overlay order of the first status bar layer, the first preprocessing layer, the first desktop layer, and the first status bar layer are overlaid to obtain the first image. Combining the first preprocessing layer, the second desktop layer, and the second status bar layer into a second image includes: according to the layer overlay order of the first wallpaper layer, the layer overlay order of the second desktop layer, and the layer overlay order of the second status bar layer, the first preprocessing layer, the second desktop layer, and the second status bar layer are overlaid to obtain the second image.

[0049] In the above solution, each application writes the layer overlay order in the cache area corresponding to each application, so that SurfaceFlinger can determine the synthesis method according to the overlay order of each layer.

[0050] In one possible implementation, before rotating the first wallpaper layer through the GPU, the method may further include: determining that the first wallpaper layer is the first frame or the change frame of the wallpaper resource. The first frame refers to other wallpaper layers for which wallpaper resources have not been obtained before the first wallpaper layer is obtained; the change frame refers to other wallpaper layers for which wallpaper resources have been obtained before the first wallpaper layer is obtained, and the other wallpaper layers are different from the first wallpaper layer. Taking the first wallpaper layer as layer 1 as an example, the first frame means that before SurfaceFlinger receives the synthesis task of layer 1, other wallpaper layers have not been stored in the cache area corresponding to the wallpaper application, that is, layer 1 is the first synthesis task of the wallpaper application received by SurfaceFlinger; the change frame means that before SurfaceFlinger receives the synthesis task of layer 1, it has received the synthesis task of other layers of the wallpaper application, and the other layers and layer 1 are stored in different cache areas.

[0051] In the above solution, determining whether a wallpaper layer is the first frame or a changing frame of a wallpaper resource can determine whether it needs to be preprocessed. It should be understood that when a wallpaper layer is the first frame or a changing frame of a wallpaper resource, subsequent wallpaper layers may be the same layer. Therefore, preprocessing this wallpaper layer is appropriate, allowing direct use of the preprocessed layer when compositing tasks involving the same subsequent wallpaper layers occur.

[0052] In one possible implementation, before obtaining the first wallpaper layer at the second time, the method may further include: determining a rotation parameter when a preset condition is met; sending the rotation parameter to a wallpaper application, a desktop application, and a system user interface application; generating a first status bit through the wallpaper application, and storing the first status bit, the rotation parameter, and the resources of the first wallpaper layer in a second buffer; generating a second status bit through the desktop application, and storing the second status bit, the rotation parameter, and the resources of the second desktop layer in a third buffer, the second status bit being used to indicate that a rotation operation has been performed on the resources of the second desktop layer; generating a third status bit through the system user interface application, and storing the third status bit, the rotation parameter, and the resources of the second status bar layer in a fourth buffer, the third status bit being used to indicate that a rotation operation has been performed on the resources of the second status bar layer. The preset condition includes any one of the following: the screen of the terminal device switches from a landscape state to a portrait state; the terminal device is a foldable screen device, and the foldable screen device switches from a folded state to an unfolded state.

[0053] In the above scheme, when the screen of the terminal device switches from landscape to portrait, or from folded to unfolded, the sensor of the terminal device can collect data such as folding angle and acceleration in real time, and send these data to the hardware configuration module. The screen management service can obtain information such as screen direction, window direction and folding state from the hardware configuration module. Then, WMS can calculate the rotation parameters according to the installation direction of the screen and the current landscape and portrait state of the screen, and send these rotation parameters to each application. Each application can generate a status bit indicating whether the layer has been rotated based on the generation method of its own layer. In this way, SurfaceFlinger can use the status bit to determine whether each layer needs to be rotated.

[0054] In a fifth aspect, the present application provides an apparatus comprising units for executing the method described in the first aspect. This apparatus may be configured to execute the layer composition method described in the first aspect. For a detailed description of the units in this apparatus, please refer to the description of the first aspect above and will not be repeated here for the sake of brevity.

[0055] The method described in the first aspect above can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a processing module or unit, a display module or unit, etc.

[0056] In a sixth aspect, the present application provides a terminal device. For example, the terminal device may be a foldable screen device or a large screen device. The terminal device includes a memory and one or more processors. The memory is used to store computer program code, and the computer program code includes computer instructions. When the computer instructions are called by the processor, the terminal device executes the layer synthesis method provided in any one of the first aspects.

[0057] In a seventh aspect, the present application provides a computer-readable storage medium. The computer-readable storage medium includes computer instructions. When the computer instructions are executed on a foldable screen device, the foldable screen device executes the layer composition method provided in the first aspect and any possible implementation thereof.

[0058] In an eighth aspect, the present application provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the layer composition method provided in the first aspect and any possible implementation thereof.

[0059] In a ninth aspect, the present application provides a chip system. The chip system includes one or more interface circuits and one or more processors. The interface circuit and the processor are interconnected by lines. The chip system can be applied to a folding screen device including a communication module and a memory. The interface circuit is used to receive a signal from the memory of the folding screen device and send the received signal to the processor, where the signal includes a computer instruction stored in the memory. When the processor calls the computer instruction, the folding screen device can execute the layer synthesis method provided in any one of the first aspects.

[0060] It can be understood that the beneficial effects that can be achieved by the fifth aspect apparatus, the sixth aspect terminal device, the seventh aspect computer-readable storage medium, the eighth aspect computer program product, and the ninth aspect chip system provided above can refer to the beneficial effects in the first aspect and any possible implementation thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] FIG1 is a schematic diagram of synthesizing multiple layers into a single image frame according to an embodiment of the present application;

[0062] Figures 2A-2C are schematic diagrams of an inward-folding screen mobile phone provided in an embodiment of the present application;

[0063] Figures 3A-3C are schematic diagrams of an outward-folding screen mobile phone provided in an embodiment of the present application;

[0064] Figures 4A-4G are schematic diagrams of a small screen, a large screen, wallpaper, GPU synthesis, and HWC synthesis provided in an embodiment of the present application;

[0065] 5A-5B are schematic diagrams of synthesizing multiple layers into a single image frame and displaying the image on a large screen according to an embodiment of the present application;

[0066] FIG6 is a schematic diagram of transmitting multiple frames of images to a large screen for display according to an embodiment of the present application;

[0067] FIG7 is a schematic diagram of the architecture of a folding screen device provided in an embodiment of the present application;

[0068] FIG8 is a flow chart of a layer synthesis method based on intelligent scene recognition provided in an embodiment of the present application;

[0069] FIG9 is a schematic diagram of using a GPU to render a pre-processed layer according to an embodiment of the present application;

[0070] FIG10 is a schematic diagram of synthesizing multiple layers into one frame of image based on pre-processed layers according to an embodiment of the present application;

[0071] FIG11A is a schematic diagram of synthesizing a video layer and a wallpaper layer into a multi-frame image based on a pre-processing layer according to an embodiment of the present application;

[0072] FIG11B is a schematic diagram of synthesizing a status bar layer and a wallpaper layer into a multi-frame image based on a pre-processed layer according to an embodiment of the present application;

[0073] FIG12 is a schematic diagram of synthesizing the Nth frame image and the N+1th frame image provided by an embodiment of the present application;

[0074] FIG13 is a flow chart of another layer synthesis method based on intelligent scene recognition provided in an embodiment of the present application;

[0075] FIG14A is a schematic diagram of a scenario in which a pre-processed layer is applied, according to an embodiment of the present application;

[0076] FIG14B is a schematic diagram of another scenario of applying a pre-processed layer according to an embodiment of the present application;

[0077] FIG14C is a schematic diagram of another scenario of applying a pre-processed layer according to an embodiment of the present application;

[0078] FIG14D is a schematic diagram of another scenario of applying a pre-processed layer according to an embodiment of the present application;

[0079] FIG15 is a flow chart of another layer synthesis method based on intelligent scene recognition provided in an embodiment of the present application;

[0080] FIG16 is a flow chart of another layer synthesis method based on intelligent scene recognition provided in an embodiment of the present application;

[0081] FIG17 is a schematic diagram of another scenario of applying a pre-processed layer according to an embodiment of the present application;

[0082] Figure 18 is a schematic diagram of the hardware structure of the folding screen device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0084] In the description of this application, unless otherwise specified, " / " means or. For example, A / B can mean A or B. In the description of this application, "and / or" is simply a way to describe the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0085] In the specification and claims of this application, the terms "first" and "second" are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects. For example, "first frame" and "first frame" are used to distinguish different video frames, rather than to describe a specific order of video frames. In the embodiments of this application, "plurality" refers to two or more.

[0086] The following is an explanation of some terms used in the embodiments of this application.

[0087] The picture displayed on the screen of the terminal device is usually composed of multiple layers. For example, Figure 1 shows a schematic diagram of synthesizing the wallpaper layer, desktop layer, video layer and status bar layer into a frame of image. Usually, the terminal device uses a system service called surface manager (SurfaceFlinger) to manage the rendering and synthesis process of the layer. When each application needs to display a window, each application sends the relevant information of the window to SurfaceFlinger. After receiving relevant information from multiple applications, SurfaceFlinger uses GPU and / or HWC to complete rendering, synthesis and other operations, and finally obtains the image to be displayed, and sends this frame of image to the screen.

[0088] The above-mentioned GPU synthesis and HWC synthesis are two different image synthesis technologies.

[0089] The following examples illustrate the advantages and disadvantages of these two image synthesis technologies.

[0090] 1) HWC synthesis

[0091] HWC compositing is a hardware-accelerated compositing technology that leverages hardware-level image compositing capabilities to improve performance and efficiency. In HWC compositing, image compositing is performed by hardware modules, potentially involving a small amount of central processing unit (CPU) compositing operations, without requiring software-level compositing by the GPU.

[0092] HWC synthesis has the following advantages: 1. Due to the use of hardware acceleration, HWC synthesis can provide higher frame rates and smoother animation effects, that is, high performance; 2. By directly utilizing hardware resources for synthesis, HWC synthesis can reduce the power consumption of the CPU and GPU, that is, low power consumption; 3. HWC synthesis can display the graphical content of the application on the screen faster, providing faster user interface response time, that is, real-time response.

[0093] However, the HWC hardware module is a hardware module in the system-on-a-chip (SOC). HWC hardware modules provided by different manufacturers have different HWC synthesis specifications. Currently, the HWC hardware module has conditional restrictions on images. For example, some HWC hardware modules have restrictions on the height of the image when rotating the image, some HWC hardware modules do not support output rounded corners, some HWC hardware modules have restrictions on the number of layers, conversion format, layer size and zoom ratio during synthesis, and some HWC hardware modules do not support window blur, etc. When the synthesis requirements of the layers exceed the capabilities of the HWC hardware module, HWC synthesis can no longer be used.

[0094] 2) GPU compositing

[0095] GPU compositing is a software-based image compositing technology, also known as software compositing. GPU compositing uses the GPU to perform graphics compositing operations. In GPU compositing, the CPU sends application graphics operations to the GPU, which then performs compositing operations in software. The GPU then passes the composited image data to the HWC for hardware compositing, which then displays it on the screen.

[0096] GPU compositing offers flexibility. Because it's software-based, GPU compositing can support more complex graphics compositing operations, such as rotations, special effects, filters, transformations, and rounded corners. When the compositing requirements for a layer fall within the capabilities of the HWC hardware module, HWC compositing can be used; when the compositing requirements for a layer exceed the capabilities of the HWC hardware module, GPU compositing can be used. However, GPU compositing has the disadvantage of high power consumption, and its performance is lower than that of HWC compositing.

[0097] Currently, most terminal devices are equipped with high-definition dynamic wallpapers. High-definition dynamic wallpapers have a larger resolution. If the screen is installed in a horizontal manner, the high-definition dynamic wallpaper needs to be rotated when the screen is in portrait mode. When the height of the high-definition dynamic wallpaper before rotation exceeds the maximum rotation height supported by the HWC hardware module (for example, 2176 pixels), GPU synthesis can be selected. For horizontal installation, please refer to the relevant description of Figures 4A to 4G in the following embodiments.

[0098] The following uses a folding screen terminal device as an example to introduce the principle of rotating wallpaper using these two synthesis technologies.

[0099] The above-mentioned folding screen device can be an inward folding screen device or an outward folding screen device.

[0100] Among them, both the inner folding screen device and the outer folding screen device can be folded into three folding states: folded state, hovering state, and unfolded state. Specifically, when the folding screen device is in the folded state, the folding angle between the main screen and the back screen is less than or equal to α, and the small screen faces the user; when the folding screen device is in the unfolded state, the folding angle between the main screen and the back screen is greater than or equal to β, and the large screen faces the user; when the folding screen device is in the hovering state, the folding angle between the main screen and the back screen is greater than α and less than β. For example, α is equal to or close to the folding angle of 0°, and β is equal to or close to the folding angle of 180°.

[0101] For example, Figures 2A to 2C show schematic diagrams of an inward-folding screen mobile phone.

[0102] The inward folding screen mobile phone is provided with an inner screen (also called a large screen) and an outer screen (also called a small screen), and the inner screen can be bent into at least two display areas. As shown in Figure 2A, when the inward folding screen mobile phone is in a folded state, the inner surfaces of the two folding bodies are opposite, and the inner screen is in a hidden state. As shown in Figure 2B, when the inward folding screen mobile phone is in a hovering state, the folding angle between the two display areas of the inner screen is greater than 0° and less than 180°. As shown in Figure 2C, when the inward folding screen mobile phone is in an unfolded state, the folding angle between the two display areas of the inner screen is equal to or close to 180°, so that the two display areas of the inner screen are located on the same surface. Usually, when the inward folding screen mobile phone is in a folded state, the outer screen displays wallpaper; after the inward folding screen mobile phone switches from a folded state to an unfolded state, the inner screen displays wallpaper.

[0103] For example, Figures 3A to 3C show schematic diagrams of an outward-folding screen mobile phone.

[0104] An external folding screen mobile phone is provided with a screen that can be bent into at least two display areas. Among them, the larger display area of ​​the at least two display areas is called the main screen, and the smaller display area is called the back screen. Generally, the main screen is also called the small screen, and the main screen and the back screen are collectively called the large screen. As shown in Figure 3A, the back screen is folded to be flush with the surface where the rear camera is located, with the main screen facing the user and the back screen facing away from the user. As shown in Figure 3B, when the external folding screen mobile phone is in a hovering state, the folding angle between the main screen and the back screen is greater than 0° and less than 180°. As shown in Figure 3C, when the external folding screen mobile phone is in an unfolded state, the folding angle between the two display areas of the inner screen is equal to or close to 180°. Generally, when the external folding screen mobile phone is in a folded state, the main screen displays the wallpaper; after the external folding screen mobile phone switches from the folded state to the unfolded state, the main screen and the back screen are used to display the wallpaper at the same time.

[0105] According to the description of the above embodiment, although the physical forms of the inward-folding screen mobile phone and the outward-folding screen mobile phone are different, they both have the following common characteristics: when in the folded state, the small screen is used to display a smaller-sized wallpaper; when in the unfolded state, the large screen is used to display a larger-sized wallpaper.

[0106] Typically, the resolution and orientation of small and large screens may differ. For example, if the resolution of the small screen is 2300*1176 pixels and the resolution of the large screen is 2300*2100 pixels, and the small screen is mounted in portrait orientation and the large screen in landscape orientation, the small screen's orientation may differ by 90° or 270°. When the small and large screens use the same dynamic wallpaper video resource, to ensure that the wallpaper displayed in portrait mode on the small screen and the large screen is aligned in the user's visual orientation, the wallpaper application notifies SurfaceFlinger based on the folding configuration to determine whether to rotate the wallpaper. For example, if the currently active screen is a small screen in portrait mode, wallpaper rotation is not required, and low-power HWC compositing is selected. However, if the currently active screen is a large screen in portrait mode, wallpaper rotation is required, and the pre-rotation wallpaper height (e.g., 2300 pixels) exceeds the maximum rotation height supported by the HWC hardware module (e.g., 2176 pixels), GPU compositing, which supports more complex compositing operations, is selected. After rotating, scaling, or cropping the wallpaper layer based on GPU synthesis and / or HWC synthesis, HWC uses hardware synthesis to synthesize the wallpaper layer with other layers into a frame of image and sends it to the small or large screen for display.

[0107] It should be noted that if the small screen is on, it can be divided into two states: the small screen is in portrait state, and the angle between the long border of the small screen and the ground plane is greater than or equal to the angle between the short border of the small screen and the ground plane; or, the small screen is in landscape state, and the angle between the long border of the small screen and the ground plane is smaller than the angle between the short border of the small screen and the ground plane.

[0108] If the large screen is on, it can be divided into two states: the large screen is in portrait mode, and the angle between the long frame of the large screen (parallel to the fold direction) and the ground plane is greater than or equal to the angle between the short frame of the large screen (perpendicular to the fold direction) and the ground plane. At this time, the large screen is displayed in the first direction; or the large screen is in landscape mode, and the angle between the long frame of the large screen (parallel to the fold direction) and the ground plane is less than the angle between the short frame of the large screen (perpendicular to the fold direction) and the ground plane. At this time, the large screen is displayed in the second direction. The first direction and the second direction differ by 90°. In this way, whether the large screen is in portrait mode or landscape mode, from the user's perspective, the screen always remains in the normal display and does not rotate.

[0109] Below, taking the inward folding screen mobile phone shown in Figures 2A to 2C as an example, combined with Figures 4A to 4G, the relationship between the small screen, large screen, wallpaper, GPU synthesis and HWC synthesis is illustrated.

[0110] For example, taking the small and large screens of an inward-folding screen mobile phone as an example, the type of organic light-emitting diode (OLED) is used. Regardless of the small or large screen, the most basic unit element that makes up the image is the pixel. The resolution of the screen is usually expressed as "row pixel value * column pixel value". For example, the resolution of a screen is m (height) * n (width), which means that the screen has m rows and n columns. It can be understood that there are m pixels in each column and n pixels in each row.

[0111] As shown in Figure 4A , the resolution of the small screen is 2300 (height) by 1176 (width) pixels, indicating that the small screen has 2300 rows and 1176 columns. A two-dimensional coordinate system XY is established with row 1 by column 1 as the origin. When the small screen is mounted on the phone body in the vertical installation configuration shown in Figure 4A , the positive X-axis of the two-dimensional coordinate system points to the right along the first side of the phone body, and the positive Y-axis of the two-dimensional coordinate system points downward along the second side of the phone body, where the first side is perpendicular to the second side. The small screen transmits pixel data for a frame of image to each pixel in the following scanning manner: starting from row 1*column 1, the pixels of the first row are scanned in sequence along the positive direction of the X-axis until row 1*column 1176 is reached; after the transmission of the first row of pixel data is completed, starting from row 2*column 1, the pixels of the second row are scanned in sequence along the positive direction of the X-axis until row 2*column 1176 is reached; and so on, starting from row 2300*column 1, the pixels of the 2300th row are scanned in sequence along the positive direction of the X-axis until row 2300*column 1176 is reached. It can be understood that when the installation direction of the small screen on the mobile phone body is changed (for example, the small screen is rotated 180° relative to the mobile phone body), the starting point and scanning direction of the small screen scanning will change with the mobile phone body as the reference.

[0112] As shown in Figure 4B , the resolution of the large screen is 2100 (height) * 2300 (width) pixels, indicating that the large screen has 2100 rows and 2300 columns. A two-dimensional coordinate system XY is established with row 1 * column 1 as the origin. When the large screen is mounted on the mobile phone body in the horizontal installation method shown in Figure 4B , the positive X-axis of the two-dimensional coordinate system is downward along the third side (i.e., the first side) of the mobile phone body, and the positive Y-axis of the two-dimensional coordinate system is leftward along the fourth side (the fourth side is parallel to the second side) of the mobile phone body, where the third side is perpendicular to the fourth side. The large screen transmits pixel data for a frame of image to each pixel using the following scanning method: starting from row 1*column 1, the pixels in the first row are scanned in the positive direction along the X-axis until row 1*column 2300 is reached; after the first row of pixel data is transmitted, starting from row 2*column 1, the pixels in the second row are scanned in the positive direction along the X-axis until row 2*column 2300 is reached; and so on, starting from row 2100*column 1, the pixels in row 2100 are scanned in the positive direction along the X-axis until row 2100*column 2300 is reached. It can be understood that when the large screen is installed in the mobile phone body and the mounting orientation is changed, the starting point and scanning direction of the large screen will change with the mobile phone body as the reference.

[0113] According to the above two paragraphs, if the mobile phone body is used as a reference, the coordinate system of the large screen shown in Figure 4B is rotated 90° clockwise relative to the coordinate system of the small screen shown in Figure 4A, that is, the installation directions of the small screen and the large screen differ by 90°.

[0114] The direction of the wallpaper resources stored inside the mobile phone is fixed. Usually, the direction of the wallpaper resources is consistent with the installation direction of the small screen on the mobile phone body, and is 90° or 270° clockwise from the installation direction of the large screen on the mobile phone body. As an example, if the small screen is installed on the mobile phone body in a vertical installation manner as shown in Figure 4A, and the large screen is installed on the mobile phone body in a horizontal installation manner as shown in Figure 4B, then as shown in Figure 4C, the direction of the wallpaper resources is consistent with the installation direction of the small screen on the mobile phone body, and the direction of the wallpaper resources is 90° clockwise from the installation direction of the large screen on the mobile phone body.

[0115] When the user holds the mobile phone with both hands, the folding form of the mobile phone can be changed by changing the folding angle, such as switching from a folded state to an unfolded state, or from an unfolded state to a folded state. If switching from a folded state to an unfolded state, the wallpaper is first displayed on the small screen, and then on the large screen. If switching from an unfolded state to a folded state, the wallpaper is first displayed on the large screen, and then on the small screen. It should be understood that during the switching process of these two folding forms, the user may prefer to see the wallpaper directions of the two screens remain consistent. As shown in Figure 4C, when the small screen and the large screen are in portrait mode, the phoenix displayed on the small screen and the phoenix displayed on the large screen both fly toward the upper right. In conjunction with the description of the above embodiment, since the direction of the wallpaper resource is consistent with the installation direction of the small screen on the mobile phone body and inconsistent with the installation direction of the large screen on the mobile phone body, it is necessary to rotate the wallpaper resource first, and then send the rotated wallpaper to the large screen for display, so that from the user's visual perspective, the wallpaper directions displayed on the small screen and the large screen are consistent, thereby improving the user experience.

[0116] As an example, based on Figure 4C, Figure 4D shows a schematic diagram of sending the display to the small screen when the mobile phone is in a folded state and the small screen is in a vertical state. The direction of the wallpaper resource is consistent with the installation direction of the small screen on the mobile phone body. There is no need to rotate the wallpaper, and HWC can be used for synthesis. The resolution of the wallpaper resource is 2300 (height) * 2100 (width) pixels, and the resolution of the small screen is 2300 (height) * 1176 (width) pixels. HWC can crop a 2300 (height) * 1176 (width) partial image from the wallpaper resource according to the dotted box shown in Figure 4D, and send the partial image to the small screen for display.

[0117] As another example, based on Figure 4C, Figure 4E shows a schematic diagram of sending the display to the small screen when the mobile phone is in a folded state and the small screen is in a horizontal state. The direction of the wallpaper resource is consistent with the installation direction of the small screen on the mobile phone body. There is no need to rotate the wallpaper, and HWC can be used for synthesis. The resolution of the wallpaper resource is 2300 (height) * 2100 (width) pixels, and the resolution of the small screen is 2300 (height) * 1176 (width) pixels. The HWC can cut out a 2300 (height) * 1176 (width) partial image from the wallpaper resource according to the dotted box shown in Figure 4E, and send the partial image to the small screen for display. It should be understood that regardless of whether the small screen is in a vertical or horizontal state, the wallpaper is not rotated.

[0118] It should be noted that Figures 4D and 4E illustrate the cropping operation of wallpaper resources and do not limit the present application. In actual implementation, the wallpaper resource can also be scaled according to a certain scaling factor based on the resolution of the wallpaper and the small screen, or the cropping and scaling operations can be performed simultaneously to obtain a wallpaper that is adapted to the resolution of the small screen.

[0119] As another example, based on Figure 4C, Figure 4F shows a schematic diagram of sending the display to the large screen when the mobile phone is in the unfolded state and the large screen is in the vertical screen state. The direction of the wallpaper resource is 90° clockwise from the installation direction of the large screen on the mobile phone body. The wallpaper needs to be rotated, and the GPU can be used for synthesis. The resolution of the wallpaper resource is 2300 (height) * 2100 (width) pixels, and the resolution of the large screen is 2100 (height) * 2300 (width) pixels. Since the height of the wallpaper resource is 2300 pixels, it exceeds the maximum rotation height supported by the HWC hardware module (for example, 2176 pixels). Therefore, the GPU can rotate the wallpaper resource 90° counterclockwise and then send the rotated wallpaper resource to the large screen for display.

[0120] It should be noted that the "large screen resolution of 2100 (height) * 2300 (width) pixels" in the embodiments of this application is described with reference to the scanning direction of the large screen and does not limit this application. As shown in Figure 4B, after the large screen is mounted on the mobile phone body, the relative positional relationship between the large screen and the mobile phone body is fixed. If the mobile phone body is used as a reference, 2300 pixels can also be used as the height of the large screen and 2100 pixels as the width of the small screen.

[0121] In the above embodiment, since the resolution of the rotated wallpaper is 2100 (height) * 2300 (width) pixels and the resolution of the large screen is 2100 (height) * 2300 (width) pixels, that is, the resolution of the rotated wallpaper is the same as the resolution of the large screen, there is no need to crop and / or scale the wallpaper. It should be understood that when the resolution of the rotated wallpaper is different from the resolution of the large screen, cropping and / or scaling operations can be performed according to usage requirements.

[0122] In addition, Figures 4B, 4C, and 4F are illustrated by taking the direction of the wallpaper resource and the direction of the large screen being installed on the mobile phone body as an example, which does not limit the present application. As another example, as shown in Figure 4G, the direction of the wallpaper resource and the direction of the large screen being installed on the mobile phone body are 270° clockwise (i.e., 90° counterclockwise), and the wallpaper height before rotation (e.g., 2300 pixels) exceeds the maximum rotation height supported by the HWC hardware module (e.g., 2176 pixels). The GPU can rotate the wallpaper resource 90° clockwise and then send the rotated wallpaper resource to the large screen for display. It should be noted that the following embodiments are all illustrated by taking the large screen installation direction shown in Figures 4B, 4C, and 4F as an example. In actual implementation, the large screen installation direction shown in Figure 4G can also be adopted.

[0123] The above embodiment, in conjunction with Figures 4A to 4G, illustrates the relationship between small screen, large screen, wallpaper, GPU synthesis and HWC synthesis. In actual implementation, the mobile phone screen will not only display the wallpaper layer, but may also display other layers. For example, the mobile phone screen will superimpose the desktop layer, video layer and status bar layer on the wallpaper layer in sequence, and synthesize multiple layers into one frame of image. It should be understood that in addition to these four types of layers, there may be other types of layers. The number and type of layers can be adjusted according to usage requirements, and the embodiments of this application are not specifically limited.

[0124] The following example illustrates how to combine multiple layers into one image when in the expanded state.

[0125] 5A and 5B are schematic diagrams showing how multiple layers are synthesized into one image frame and displayed on a large screen.

[0126] As shown in Figure 5A, when the phone is in the unfolded state, SurfaceFlinger receives the synthesis tasks of the wallpaper layer, desktop layer, video layer, and status bar layer in sequence. Among them, the synthesis task of the wallpaper layer specifies overlay order 1, which means it is at the bottom layer; the synthesis task of the desktop layer specifies overlay order 2, which means it is located above the bottom layer; the synthesis task of the video layer specifies overlay order 3, which means it is located two layers above the bottom layer; and the synthesis task of the status bar layer specifies overlay order 4, which means it is located at the top layer. In other words, the overlay order of the layers from bottom to top is: wallpaper layer, desktop layer, video layer, and status bar layer.

[0127] The desktop layer is the layer drawn by desktop applications, and the status bar layer is the layer drawn by the system user interface (UI). During the drawing process, the desktop application and the system UI have already rotated the desktop layer and the status bar layer 90° counterclockwise, and no further rotation is required. In principle, SurfaceFlinger can give priority to assigning the synthesis task of the desktop layer and the status bar layer to the HWC. However, since the wallpaper layer and the video layer are obtained by decoding the wallpaper resources and video resources through the video codec, and the video codec does not have the rotation capability, the wallpaper layer and the video layer are layers that have not yet been rotated. In addition, the video layer needs to be cropped with rounded corners, and HWC synthesis does not support cropping with rounded corners. Therefore, SurfaceFlinger can assign the synthesis task of the wallpaper layer and the video layer to the GPU. In addition, when at least one layer is located between layer a and layer b, and layer a and layer b can only be synthesized by the GPU, regardless of whether the at least one layer is suitable for HWC synthesis, SurfaceFlinger will assign the synthesis task of the at least one layer, layer a and layer b to the GPU. Because the desktop layer is located between the wallpaper layer and the video layer, SurfaceFlinger ultimately assigns the desktop layer synthesis task to the GPU instead of the HWC. Among them, the wallpaper resource can be a video resource composed of multiple pictures (i.e. dynamic wallpaper) or a single picture (i.e. static wallpaper).

[0128] The GPU rotates the wallpaper layer 90° counterclockwise, rotates the video layer 90° counterclockwise and rounds the corners of the video layer. The GPU superimposes the desktop layer and the rotated video layer on the rotated wallpaper layer in sequence to obtain a composite layer. Then, the GPU sends the composite layer to the HWC. The HWC can superimpose the status bar layer on the composite layer, generate a frame of image to be displayed, and send the image to be displayed to the large screen. When the large screen is in portrait mode, the phoenix in the wallpaper displayed on the large screen flies toward the upper right (as shown in Figure 5B), which is consistent with the flying direction of the phoenix in the wallpaper displayed on the small screen (as shown in Figure 4D). In this way, although the installation directions of the large screen and the small screen are inconsistent, the wallpaper directions of the large screen and the small screen that the user finally sees are consistent, which is more in line with user habits.

[0129] The above embodiment, combined with Figures 5A and 5B, illustrates how to synthesize multiple layers into a single image frame when the foldable screen device is in the unfolded state. In actual implementation, the large screen will use a periodic tearing effect (TE) signal to continuously refresh the interface. The TE signal can represent the refresh rate of the large screen. After the large screen is powered on and initialized, the large screen begins to periodically send TE signals to SurfaceFlinger. The frequency of the TE signal determines the refresh rate of the large screen. For example, when the refresh frame rate of the large screen is 120Hz, the transmission period of the TE signal is 8.33 milliseconds. For another example, when the refresh frame rate of the large screen is 30Hz, the transmission period of the TE signal is 33.33 milliseconds. After each TE signal is sent by the large screen, SurfaceFlinger performs a refresh action in response to the TE signal. Based on the TE signal, SurfaceFlinger also maintains the Vsync-SF (vertical synchronization (Vsync) - SurfaceFlinger) signal. Each time a Vsync-SF signal is obtained, the layer synthesis process is performed. Take obtaining a Vsync-SF signal as an example. When a Vsync-SF signal is obtained, SurfaceFlinger performs layer synthesis and rendering through the GPU or HWC to generate a new image to be displayed, and then sends it to the large screen for display.

[0130] It should be noted that this application involves two types of Vsync signals: Vsync-SF signal and Vsync-APP signal. Among them, the Vsync-SF signal is used to control the synthesis speed of the layer, and the Vsync-APP signal is used to control the speed of the APP drawing layer. The APP can request the Vsync-APP signal from SurfaceFlinger as needed.

[0131] The following example illustrates how the large screen continuously refreshes its interface, taking the example of SurfaceFlinger receiving at least the synthesis task of the wallpaper layer and the video layer when the large screen is in the portrait state as shown in Figure 4F, in conjunction with Figure 6. It should be understood that in actual implementation, in addition to the wallpaper layer and the video layer, it may also include a desktop layer, a status bar layer and / or a web page layer, etc., which is not limited in the embodiment of this application.

[0132] Exemplarily, FIG6 shows a schematic diagram of synthesizing a wallpaper layer and a video layer into a multi-frame image and displaying the multi-frame image on a large screen.

[0133] The frequency at which each application updates its window may vary, resulting in different image feed cycles for each application. For example, a wallpaper application may not refresh its wallpaper for tens of seconds, minutes, or even hours, a video application window may refresh 40 to 60 frames per second, a news window may refresh 120 frames per second, and a game window may refresh 160 frames per second. When the image feed cycles of each application are different, SurfaceFlinger calculates the refresh rate based on the minimum image feed cycle, that is, the period of the Vsync-SF signal, to avoid frame drops caused by using a larger image feed cycle.

[0134] For example, a video application can send a task to SurfaceFlinger every 33.33 milliseconds. Take task A1 as an example. The video application can notify the video codec to decode the video resource. The video resource stores the decoded resource a1 in the buffer and sends the buffer to SurfaceFlinger. SurfaceFlinger obtains resource a1 from the buffer. Although the height of resource a1 does not exceed the maximum rotation height supported by the HWC hardware module (for example, 2176 pixels), because the HWC hardware module does not support rounded corners, SurfaceFlinger hands task A1 to the GPU. The GPU rotates and rounds the corners of resource a1 to obtain layer A1, and then sends layer A1 to the HWC.

[0135] The wallpaper application can issue a new synthesis task to SurfaceFlinger each time the wallpaper is updated. For example, a wallpaper update is triggered when the user changes the folding angle of the foldable screen, or when the user reselects a wallpaper in the Settings app. Therefore, wallpaper updates are closely related to user actions. If the wallpaper is not updated for several seconds, minutes, or even hours, the wallpaper application will not issue new tasks during this period. Take tasks B1 and B2 as an example. The wallpaper application can notify the video codec to decode the wallpaper resource. The video resource stores the decoded resource b1 in a buffer and sends this buffer to SurfaceFlinger. SurfaceFlinger retrieves resource b1 from this buffer. Because the height of resource b1 exceeds the maximum rotation height supported by the HWC hardware module (for example, 2176 pixels), SurfaceFlinger assigns task B1 to the GPU, which rotates resource b1 to obtain layer B1. Before the arrival of task B2, the next task after task B1, to maintain the same refresh rate between the wallpaper layer and the video layer, the GPU continuously rotates resource b1 to obtain layer B1 and sends the resulting layer B1 to the HWC.

[0136] HWC superimposes layer A1 on the first layer B1 to form frame 1, superimposes layer A2 on the second layer B1 to form frame 2, and superimposes layer A3 on the third layer B1 to form frame 3... In response to the received TE signal, HWC sends these images to the large screen in sequence.

[0137] In conjunction with the description of the above embodiment, when a large screen is in portrait mode, if multiple frames are synthesized based on the same wallpaper resource, the GPU may need to rotate the same wallpaper resource multiple times. However, GPU synthesis has the disadvantage of high power consumption. Frequent rotation of wallpaper layers leads to high power consumption, high load, poor performance, and severe heat generation.

[0138] In view of the above problems, an embodiment of the present application provides a layer synthesis method based on intelligent scene recognition. This method can be applied to a terminal device that is provided with at least one large screen, and the large screen is installed in a horizontal manner. In this method, when the large screen is in vertical screen mode, each layer to be synthesized is identified to determine whether the layer requires GPU preprocessing, such as wallpaper layer rotation. If GPU preprocessing is required, the preprocessed layer is rendered using the GPU, and the preprocessed layer is placed in a cache area. In this way, when the same layer processing task is subsequently performed, the preprocessed layer can be directly obtained from the cache area, and the preprocessed layer can be sent to the HWC for hardware synthesis, thereby reducing the overall power consumption and load of the terminal device and improving the overall performance of the terminal device.

[0139] The above-mentioned terminal devices are not limited to foldable screen devices, but can also be large-screen devices such as tablet computers (PADs).

[0140] In order to better understand the layer synthesis method based on intelligent scene recognition provided by the embodiment of the present application, the architecture of the folding screen device is introduced below using the folding screen device as an example.

[0141] For example, FIG7 shows a schematic diagram of the architecture of the folding screen device provided in an embodiment of the present application.

[0142] As shown in Figure 7, the folding screen device can adopt a layered architecture to divide the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the software layers of the software structure are divided from top to bottom into: application (APP) layer, application framework (FW) layer, system library (FWK LIB), hardware abstraction layer (HAL layer) and kernel layer. The above software architecture runs on the hardware layer, which may include screen, HWC hardware module, GPU and sensor, etc.

[0143] The application layer, also known as the application layer, can include a series of application packages, such as wallpaper applications, desktop applications, system UI, video applications, and browsers. The system UI is a native application of the Android system. When these application packages are running, they can access the various service modules provided by the application framework layer through the application programming interface (API) and perform corresponding intelligent services.

[0144] The application framework layer provides APIs and a programming framework for applications. The application framework layer includes some predefined functions. As shown in Figure 7, the application framework layer can include the activity manager service (AMS), the window manager service (WMS), SurfaceFlinger, the screen management service, and video codecs (such as MediaCodec). The AMS manages the lifecycle of each application. The WMS manages all windows in the system. The screen management service is responsible for calculating the folding angle of the foldable screen device, the landscape or portrait orientation of the foldable screen device, and other information based on data collected by sensors using preset algorithms. SurfaceFlinger is responsible for the synthesis and display of layer data. For example, SurfaceFlinger can provide layer information to the hardware composer (HWC) HAL. The HWC HAL can decide whether to use the HWC underlying hardware or the GPU for synthesis based on hardware performance. MediaCodec is an audio and video codec tool provided in the Android system. It mainly encapsulates the upper-level interface and provides it to developers. The encoding and decoding functions are actually performed in the local layer service.

[0145] The system library can include multiple functional modules, such as a surface manager, media libraries, a two-dimensional (2D) graphics engine (e.g., SGL), and a three-dimensional (3D) graphics processing library (e.g., OpenGL ES). The surface manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG. The 2D graphics engine is a drawing engine for 2D graphics. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0146] Within the system libraries, the Android Runtime includes the core libraries and a virtual machine. The Android Runtime is responsible for scheduling and management of the Android system. The core libraries consist of two parts: one for the Java language's callable functions and the other for the Android core libraries. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files in the application layer and application framework layer as binary files. The virtual machine is responsible for performing functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0147] The Hardware Abstraction Layer (HAL) provides standard interfaces implemented by hardware vendors. For example, the HAL may include the HWC HAL and the Audio HAL. The HWC HAL is responsible for determining whether the compositing requirements of the layer to be composited exceed the capabilities of the HWC hardware module. It should be understood that the capabilities of HWC hardware modules manufactured by different manufacturers may vary.

[0148] The kernel layer is the layer between hardware and software and belongs to the bottom layer of the Android system. The kernel layer can contain various driver interfaces, such as display drivers, audio drivers, and sensor drivers.

[0149] It should be noted that although the embodiments of the present application are described using the Android system as an example, its basic principles are also applicable to foldable screen devices based on operating systems such as iOS or Windows.

[0150] Based on the system architecture of the folding screen device shown in Figure 7 above, the specific interaction process between the various functional structures of the folding screen device in the process of implementing the layer synthesis method of the present application is specifically explained below in conjunction with the interaction flow diagram.

[0151] Figure 8 is a flow chart of a layer synthesis method based on intelligent scene recognition provided by an embodiment of the present application. This method can be applied to a terminal device equipped with at least one large screen, where the large screen is mounted horizontally. When the large screen is in portrait mode, as shown in Figure 8, the method includes the following steps S1 to S5.

[0152] S1, SurfaceFlinger obtains layer 1 to be synthesized.

[0153] Among them, layer 1 is a layer corresponding to the window of any application, such as a wallpaper layer, a desktop layer, a status bar layer or a video layer.

[0154] In some embodiments, the metadata / resources of layer 1 can be stored in a cache. The metadata / resources of different layers are stored in different caches. For example, a desktop application can draw a desktop layer through a render thread and store the desktop layer in one cache; a system UI can draw a status bar layer through a render thread and store the status bar layer in another cache. In an embodiment of the present application, the cache used to store the metadata / resources of a layer can be an anonymous shared memory.

[0155] In one case, layer 1 is the desktop layer or the status bar layer, and the synthesis task of these layers is sent to SurfaceFlinger by the Vsync-APP signal.

[0156] Take the desktop layer as an example. A desktop application can use WMS to notify SurfaceFlinger to create a desktop window, which is used to display the desktop. When the desktop changes, such as when an icon's position changes or the user clicks an icon, the desktop application can execute the setView() method to add a view. When the desktop application executes setView(), it first executes the requestLayout() method for layout, and then executes the addWindow() method for adding a window. The requestLayout() method is used to trigger the redrawing of the view, and the addWindow() method is used to add a window to the view. When executing the requestLayout() method, the desktop application interacts with SurfaceFlinger and requests registration. After successful registration, SurfaceFlinger returns a Vsync-APP signal to the desktop application. The desktop application then draws the desktop layer through the render thread, stores the desktop layer in a buffer, and calls a function to notify SurfaceFlinger of this buffer. This buffer contains the layer information for layer 1.

[0157] In another case, layer 1 is a wallpaper layer or a video layer, and these layer synthesis tasks are not sent to SurfaceFlinger based on the Vsync-APP signal.

[0158] Take the video layer as an example. The video application can notify the video codec to decode the video resource. The video resource stores the decoded resource in a buffer and calls a function to send the buffer to SurfaceFlinger. This buffer contains the layer information of layer 1.

[0159] For example, the above-mentioned layer information may include layer name, status bit, layer overlay order, and / or rotation parameters, etc.

[0160] The layer name can be used to indicate the name of layer 1, such as status bar layer, desktop layer, video layer, browser layer, and wallpaper layer.

[0161] The status bit indicates whether layer 1 has been rotated. For example, the status bar layer and desktop layer are rotated layers drawn by applications and have a "rotated" status bit. Another example is the wallpaper layer and video layer, which are layers read from the video codec and have not yet been rotated. They have a "not rotated" status bit.

[0162] The layer stacking order is used to indicate the stacking order of layer 1 in multiple layers. For example, the wallpaper layer is at the bottom of multiple layers, and the status bar layer is at the top of multiple layers.

[0163] The rotation parameter is used to indicate the rotation direction and angle of layer 1, such as 90° counterclockwise and 90° clockwise.

[0164] S2, SurfaceFlinger determines whether to preprocess layer 1.

[0165] As an example, SurfaceFlinger can include a layer identification module. The layer identification module can determine whether layer 1 needs to be preprocessed based on the status bit and cache of layer 1. The status bit is used to indicate whether application 1 has rotated layer 1. The cache is used to store metadata / resources of layer 1.

[0166] Since different frames are stored in different buffers, whenever the application sends a new buffer to SurfaceFlinger, the layer identification module can determine that a new frame (i.e., the first frame or the changed frame) has been received, and determine whether to perform preprocessing based on the status bit. In other cases, when the application does not send a new buffer to SurfaceFlinger, in order to synchronize the refresh of multiple layers, the previously received frames can be sent to the layer identification module. The layer identification module can determine that the first frame and the changed frame have not been received in this round, and determine whether to perform preprocessing based on the status bit.

[0167] Among them, the first frame means that before SurfaceFlinger receives the synthesis task of layer 1, the other layers of application 1 have not been stored in the cache area corresponding to application 1, that is, layer 1 is the first synthesis task of application 1 received by SurfaceFlinger. The change frame means that before SurfaceFlinger receives the synthesis task of layer 1, it has received the synthesis task of other layers of application 1, and the other layers and layer 1 are stored in different cache areas.

[0168] Table 1 shows the correspondence between layer information, whether to perform preprocessing, and whether to use the preprocessing layer.

[0169] Table 1

[0170] S3, if the status bit of the layer information is rotated, SurfaceFlinger notifies HWC to use layer 1 to synthesize the image.

[0171] When the foldable screen device is in the unfolded state and the large screen is in the portrait state, if the status bit of layer 1 (such as the status bar layer and desktop layer shown in Table 1) is rotated, no matter whether it is the first frame or the change frame, there is no need to rotate the layer. There is no problem that the height of the layer to be rotated exceeds the maximum rotation height supported by the HWC hardware module (such as 2176 pixels), and HWC synthesis can be used. In this case, SurfaceFlinger can send a synthesis request to HWC. HWC synthesizes layer 1 with other layers to generate a frame of image and sends it to the large screen for display.

[0172] S4: If the status bit is not rotated and layer 1 belongs to the first frame or the change frame, SurfaceFlinger notifies the GPU to preprocess layer 1, generate a preprocessed layer, and store the preprocessed layer in the cache.

[0173] When the foldable screen device is in the unfolded state and the large screen is in the portrait state, if the status bit of layer 1 (such as the wallpaper layer shown in Table 1) is not rotated, and layer 1 belongs to the first frame or the change frame, it means that the pre-processed layer of layer 1 has not been stored previously, and layer 1 needs to be rotated. The height of layer 1 (such as 2300 pixels) exceeds the maximum rotation height supported by the HWC hardware module (such as 2176 pixels), and GPU synthesis can be used. In this case, SurfaceFlinger sends a synthesis request to the GPU. The GPU rotates layer 1, generates a pre-processed layer, and stores the pre-processed layer in the buffer area (such as buffer area 1) for layer synthesis of the next frame. In addition, in this frame synthesis, the GPU can send the rotated pre-processed layer to the HWC, and the HWC synthesizes the rotated pre-processed layer with other layers to generate a frame of image, and sends it to the large screen for display.

[0174] Exemplarily, FIG9 shows a process of rendering a pre-processed layer using a GPU.

[0175] As shown in Figure 9, Application 1 first creates a canvas / drawing surface. In the Android system, each activity has a separate canvas (called a layer on the SurfaceFlinger side). No matter how complex the view structure of an activity, it is ultimately rendered on the canvas of its corresponding activity. For example, the process of creating an application's surface is as follows: WMS creates a connection to SurfaceFlinger for the application; the application executes the relayoutWindow() method, triggering WMS to create a SurfaceControl object; and WMS obtains the surface through the SurfaceControl object.

[0176] After the surface creation is completed, the pre-processed layer can be rendered on the surface. The specific process of rendering the pre-processed layer on the surface is as follows: after the video codec decodes the wallpaper resource, the decoded wallpaper resource is stored in the layer buffer; WMS sets the rotation parameters for the wallpaper layer, such as the rotation direction and rotation angle; SurfaceFlinger can read the metadata / resources of the wallpaper layer from the buffer (referred to as wallpaper data for short), and then execute the DrawCall() instruction to call the image programming interface, such as the OpenGL interface or the Vulkan interface, to command the GPU to perform rendering operations. Among them, the DrawCall() instruction carries the rotation parameters. Then, the GPU can render the rotated wallpaper layer according to the rotation parameters.

[0177] S5, if the status bit is not rotated, and layer 1 does not belong to the first frame or the change frame, then SurfaceFlinger notifies HWC to use the previously stored pre-processed layer to synthesize the image.

[0178] When the foldable screen device is in the unfolded state and the large screen is in the portrait state, if the status bit of layer 1 (such as the wallpaper layer shown in Table 1) is not rotated, and layer 1 does not belong to the first frame or the change frame, it means that the pre-processed layer of layer 1 has been stored previously and the pre-processed layer can be used directly. In this case, SurfaceFlinger reads the pre-processed layer from the cache (such as cache 2) and then sends the pre-processed layer to the HWC. The HWC synthesizes the pre-processed layer with other layers to generate a frame of image and sends it to the large screen for display.

[0179] The cache area 1 in S4 and the cache area 2 in S5 may be the same cache.

[0180] In the above layer synthesis method, when the large screen is in portrait mode, the layer to be synthesized is identified to determine whether the layer requires GPU pre-processing, such as wallpaper layer rotation. If GPU pre-processing is required, the pre-processed layer is rendered using the GPU. In this way, when the same layer processing task is subsequently performed, the pre-processed layer can be directly sent to the HWC for hardware synthesis, thereby reducing the power consumption of the entire device and improving the overall performance.

[0181] It should be noted that the preprocessing operation of the above-mentioned layer synthesis method is illustrated by taking the rotating layer as an example, which does not limit the present application. In actual implementation, when the HWC hardware module is limited in its ability to crop corners, number of layers, conversion format, layer size, zoom ratio and window blur, and the layer synthesis requirements exceed the capabilities of the HWC hardware module, you can refer to the above-mentioned layer synthesis method to determine whether the layer needs to be preprocessed using the GPU. If GPU preprocessing is required, then use the GPU to render the preprocessed layer, so that when the same layer processing task is subsequently performed, the preprocessed layer can be sent to the HWC for hardware synthesis. You can refer to the description of the above embodiment and will not repeat it here.

[0182] The above embodiment takes a layer as an example to introduce the specific process of whether to pre-process a layer. For ease of understanding, the layer synthesis method based on intelligent scene recognition is introduced below in conjunction with Figures 10, 11A, and 11B.

[0183] FIG10 shows a schematic diagram of synthesizing multiple layers into one frame of image based on pre-processing layers.

[0184] When the large screen is in portrait mode, SurfaceFlinger first performs the synthesis task of the Nth frame, and then performs the synthesis task of the N+ith frame. Among them, the Nth frame and the N+ith frame are both composed of a wallpaper layer, a desktop layer, a video layer, and a status bar layer. The order of superposition of these layers is: wallpaper layer, desktop layer, video layer, and status bar layer. It should be understood that in actual implementation, the Nth frame and the N+ith frame may also include more or fewer layers, such as replacing the video layer with a browser layer, etc., which is not limited in the embodiments of the present application.

[0185] For the Nth frame synthesis task: the height of the wallpaper resource (for example, 2300 pixels) exceeds the maximum rotation height supported by the HWC hardware module (for example, 2176 pixels), so SurfaceFlinger assigns the synthesis task of the wallpaper layer to the GPU. The GPU rotates the wallpaper layer 90° counterclockwise to obtain a pre-processed layer and places the pre-processed layer in the cache.

[0186] For the synthesis task of the N+i frame: If the wallpaper layer of the N+i frame and the wallpaper layer of the N frame are the same wallpaper resource, SurfaceFlinger reads the pre-processed layer from the cache area, and then sends the pre-processed layer to the HWC. In addition, SurfaceFlinger sends the desktop layer and the status bar layer to the HWC. In combination with the description of the above embodiment, the video layer is read out by the video codec, and the video codec does not have the ability to rotate. The video layer is a layer that has not yet been rotated. In addition, the HWC does not support rounded corners, so SurfaceFlinger can assign the synthesis task of the video layer to the GPU. The GPU rotates the video layer 90° counterclockwise and rounds the corners, and then sends the processed video layer to the HWC. The HWC superimposes the wallpaper layer, desktop layer, video layer and status bar layer in sequence to obtain a frame of video image, and sends it to the large screen for display. When the large screen is in portrait mode, the phoenix displayed on the large screen flies toward the upper right (as shown in Figure 5B), which is consistent with the flying direction of the phoenix displayed on the small screen (as shown in Figure 4D). In this way, although the installation directions of the large screen and the small screen are inconsistent, the wallpapers displayed on the large screen and the small screen that the user finally sees are in the same direction, which is more in line with user habits.

[0187] As an example, FIG11A shows a schematic diagram of synthesizing a video layer and a wallpaper layer into a multi-frame image based on a pre-processing layer.

[0188] As shown in Figure 11A, the frequency of the video application window update is much greater than the frequency of the wallpaper application window update. In order to keep the wallpaper layer and the video layer refresh rate the same, SurfaceFlinger can synthesize the wallpaper layer based on the same wallpaper resource before the wallpaper application updates the window.

[0189] The synthesis process of frame 1 is as follows: the video application sends task A1 to SurfaceFlinger; because the video layer needs to be rotated and rounded, SurfaceFlinger hands task A1 to the GPU; the GPU rotates the video resource a1 obtained from the cache 90° counterclockwise and rounds the corners to obtain layer A1, and then sends layer A1 to HWC. At the same time, the wallpaper application sends task B1 to SurfaceFlinger; because the wallpaper layer needs to be rotated, and the height of the wallpaper layer exceeds the maximum rotation height supported by the HWC hardware module (for example, 2176 pixels), SurfaceFlinger hands task B1 to the GPU; the GPU rotates the video resource b1 obtained from the cache 90° counterclockwise to obtain layer B1, and then sends layer B1 to HWC. HWC superimposes layer A1 on layer B1 to form frame 1.

[0190] The synthesis process for frame 2 is as follows: the video application sends task A2 to SurfaceFlinger; because the video layer needs to be rotated and rounded, SurfaceFlinger assigns task A2 to the GPU; the GPU obtains video resource a2 from the cache, rotates it 90° counterclockwise and rounds the corners to obtain layer A2, and then sends layer A2 to the HWC. At the same time, because the wallpaper application does not send new tasks to SurfaceFlinger, SurfaceFlinger can read the pre-processed layer B1 from the cache and send it to the HWC. The HWC overlays layer A2 on layer B1 to form frame 2.

[0191] As another example, FIG11B shows a schematic diagram of synthesizing a status bar layer and a wallpaper layer into a multi-frame image based on a pre-processing layer.

[0192] During device operation, the system UI and wallpaper application may update windows at different frequencies. For example, when the battery level changes, the cellular network changes, or the wireless network changes, the system UI will notify SurfaceFlinger to update the status bar layer. For another example, when the wallpaper application's wallpaper changes, the wallpaper application will notify SurfaceFlinger to update the wallpaper layer.

[0193] Assume that at a certain moment, the system UI calls the render thread to draw the rotated layer C1 and stores layer C1 in the cache. SurfaceFlinger reads the rotated layer C1 from the cache and sends the rotated layer C1 to the HWC. At the same time, the wallpaper application sends task B1 to SurfaceFlinger. Since the wallpaper layer needs to be rotated and the height of the wallpaper layer exceeds the maximum rotation height supported by the HWC hardware module, SurfaceFlinger hands task B1 to the GPU. The GPU rotates the video resource b1 obtained from the cache 90° counterclockwise to obtain the pre-processed layer B1, and sends the pre-processed layer B1 to the HWC. The HWC superimposes layer C1 on the pre-processed layer B1 to form frame 1.

[0194] M seconds after this moment, the status bar is updated, and N seconds after this moment, the wallpaper is updated, where N is greater than M. Before the Mth second, neither the wallpaper layer nor the status bar layer is updated. The HWC continues to overlay layer C1 on the pre-processed layer B1, forming frame 2. After the Mth second, the status bar is updated, and the system UI calls the render thread to draw the rotated layer C2 and store it in the buffer. SurfaceFlinger reads the rotated layer C2 from the buffer and sends it to the HWC. The HWC overlays layer C2 on the pre-processed layer B1, forming frame x. After the Nth second, the wallpaper layer is updated, and the wallpaper application sends task B2 to SurfaceFlinger. Because the wallpaper layer needs to be rotated, and the height of the wallpaper layer exceeds the maximum rotation height supported by the HWC hardware module, SurfaceFlinger assigns task B1 to the GPU. The GPU rotates the video resource b2 obtained from the buffer counterclockwise by 90° to obtain the pre-processed layer B2, which it sends to the HWC.

[0195] In conjunction with the description of Figures 5A, 5B, and 6 in the above embodiment, if the layer synthesis scheme shown in Figures 5A, 5B, and 6 is adopted, multiple wallpaper layers are based on the same wallpaper resource. The GPU may need to perform multiple rotation operations on the same wallpaper resource. However, GPU synthesis has the disadvantage of high power consumption. Frequent rotation of wallpaper layers leads to high power consumption, poor performance, and severe heat generation of the entire machine. In the synthesis scheme provided by Figures 11A and 11B, the GPU can use the pre-processed layer stored when synthesizing frame 1. For multiple frames located after frame 1, if the wallpaper layers of multiple frames and the wallpaper layer of frame 1 are based on the same wallpaper resource, then there is no need to frequently rotate the wallpaper layers. Using the pre-processed layer reduces the power consumption and load of the entire machine and improves the performance of the entire machine.

[0196] Based on Figures 10, 11A, and 11B, Figure 12 shows a schematic diagram of the synthesis of the Nth frame image and the N+1th frame image. As shown in Figure 12, during the synthesis of the Nth frame image, the GPU rotates the wallpaper layer and sends the rotated wallpaper layer (pre-processed layer) to the cache. During the synthesis of the N+1th frame image, the wallpaper resources of the N+1th frame image are the same as those of the Nth frame image, and the pre-processed layer can be directly used for synthesis.

[0197] The following uses the functional modules in the folding screen device shown in Figure 7 as the execution body, and combines the following examples 1 and 2 to introduce the specific method processes for synthesizing the Nth frame image and the N+1th frame image respectively.

[0198] Example 1

[0199] Figure 13 shows a flow chart of a method for synthesizing the Nth frame of image based on preprocessed layers. This method can be applied to a foldable screen device in an unfolded state. As shown in Figure 13, the method may include the following steps A1 to A16.

[0200] A1: The screen management service determines that the foldable screen device switches from the folded state to the unfolded state.

[0201] In some embodiments, sensors installed on the foldable screen device can collect data such as the folding angle and acceleration of the foldable screen device in real time and send this data to the hardware configuration module of the foldable screen device. The screen management service can obtain information such as screen orientation, window orientation, and folding state from the hardware configuration module and determine the current screen state based on this information, such as when the foldable screen device changes from a folded state to an unfolded state. The screen management service can then notify the WMS of the current screen state.

[0202] For example, as shown in Figure 14A, at the initial moment, the foldable screen device is in a folded state, and the small screen displays wallpaper, desktop, video window, status bar, etc. When the user wants to use the large screen, he can hold the foldable screen device with both hands and bend it outward, so that the foldable screen device switches from the folded state to the unfolded state. The sensors set on the foldable screen device can collect data such as folding angle and acceleration in real time, and send this data to the hardware configuration module of the foldable screen device. In this way, the screen management service can obtain information such as screen direction, window direction and folding state from the hardware configuration module. After determining that the foldable screen device has switched from the folded state to the unfolded state based on this information, the screen management service can light up the large screen and notify the WMS. It should be noted that in the scenario example shown in Figure 14A, after determining to switch to the unfolded state, regardless of whether the large screen is in portrait or landscape mode at this moment, the foldable screen device will default to being in portrait mode, and the wallpaper resource needs to be rotated.

[0203] It should be noted that Figure 14A is illustrated using an inward-folding screen mobile phone as an example, and does not limit this application. The foldable screen device can also be an outward-folding screen mobile phone. Specifically, when the outward-folding screen mobile phone is in the folded state, the small screen displays the wallpaper, and there is no need to rotate the wallpaper resource at this time; after switching from the folded state to the unfolded state, the large screen displays the wallpaper, and the wallpaper resource needs to be rotated at this time. The large and small screens can use the same wallpaper resource.

[0204] In addition, the above A1 is described using the switching from the folded state to the unfolded state as an example scenario, which does not limit the present application. In other example scenarios, the image synthesis method provided in the embodiment of the present application can also be executed.

[0205] As another scenario example, as shown in FIG14B , at the initial moment, the folding screen device is in the unfolded state, and the large screen displays wallpaper, desktop, video window, status bar, etc. in landscape mode. If the user rotates the large screen clockwise or counterclockwise, the sensor provided on the folding screen device can collect data such as the folding angle and acceleration in real time, and send these data to the hardware configuration module of the folding screen device. In this way, the screen management service can obtain the screen direction from the hardware configuration module, such as switching from the large screen landscape state to the large screen portrait state, and notify the WMS, so that the WMS determines the rotation parameters of the layer (such as 90° counterclockwise rotation) according to the screen direction, folding state and installation direction of the large screen, and executes the following A2-A16. It should be noted that before the user rotates the large screen clockwise or counterclockwise, the user can also trigger the folding screen device to turn on the landscape and portrait functions by operating the settings application or control center. Among them, the horizontal and vertical screen functions allow display in a first direction when the large screen is in horizontal state, and display in a second direction when the large screen is in vertical state, for example, the first direction and the second direction differ by 90° or 270° clockwise.

[0206] As another scenario example, as shown in Figure 14C, the PAD is provided with a large screen, and the large screen is installed in a horizontal manner. When the large screen is in a horizontal state, the direction of the wallpaper is consistent with the direction of the large screen. In this case, there is no need to rotate the wallpaper. When the user rotates the large screen 90° clockwise, the large screen switches from a horizontal state to a vertical state. In order to make the wallpaper visually seen by the user consistent with the direction of the large screen, the screen management service of the PAD can calculate the rotation parameter as a counterclockwise rotation of 90° and execute the following A2-A16. The horizontal installation method of the large screen of the PAD is similar to the horizontal installation method of the large screen of the folding screen device. You can refer to the description of Figures 4A to 4G in the above embodiment, and will not repeat them here.

[0207] As another example scenario, as shown in Figure 14D, at the initial moment, the foldable screen device is in the unfolded state, and the large screen displays wallpaper, desktop, video window, status bar, etc. in portrait mode. The user holds the foldable screen device with both hands and bends it inward, for example, reducing the folding angle from 180° to 150°. The sensor set on the foldable screen device can collect data such as folding angle and acceleration in real time, and send this data to the hardware configuration module of the foldable screen device. In this way, the screen management service can obtain the folding angle change and screen orientation from the hardware configuration module and notify the WMS. The embodiment of the present application also provides a method for changing dynamic wallpaper based on folding angle: each folding angle corresponds to a frame of wallpaper resource. When a change from a first folding angle D(A) to a second folding angle D(B) is detected, a frame of wallpaper resource corresponding to the second folding angle D(B) is determined. In this way, when the folding angle is changed in the large screen portrait mode, the WMS will still calculate the rotation parameters of the layer (for example, rotate 90° counterclockwise) and execute the following A2-A16.

[0208] The following uses the example of a foldable screen device switching from folded to unfolded state to describe the method for synthesizing the Nth frame image, with references A2-A16. The Nth frame image can be a static wallpaper or the last frame of a wallpaper video resource.

[0209] A2, WMS determines the layer's rotation parameters based on the screen orientation, folding status, and installation orientation of the large screen.

[0210] If the large screen is mounted on the phone as shown in Figure 4B, the wallpaper resource's orientation is 90° clockwise from the orientation of the large screen on the phone. Based on the screen's orientation and folded state, the WMS determines that the large screen is in portrait mode. The WMS calculates the layer's rotation parameter as a 90° counterclockwise rotation. The WMS then notifies the wallpaper app, desktop app, video app, and system UI of the rotation parameter.

[0211] When the window of any application in the wallpaper application, desktop application, video application and system UI changes, the changed application can create a layer and hand over the synthesis task of the layer to SurfaceFlinger. The following describes the rendering and synthesis process of the wallpaper layer, desktop layer, video layer and status bar layer through A3-A5, A6-A8, A9-A10, and A12-A15 respectively. It should be noted that the embodiment of this application does not specifically limit the execution order of these steps.

[0212] A3, the wallpaper application notifies SurfaceFlinger to create a blank wallpaper layer / canvas.

[0213] A4: The wallpaper application notifies the video codec to decode the video resource. The video resource stores the decoded resource in a buffer (for example, buffer b1) and calls a function to send buffer b1 to SurfaceFlinger.

[0214] The buffer b1 may include the layer name of the wallpaper layer, the status bit of the wallpaper layer, the layer stacking order of the wallpaper layer, and / or the rotation parameter layer information of the wallpaper layer.

[0215] A5, SurfaceFlinger determines whether to pre-process the wallpaper layer.

[0216] Whenever the application sends a new buffer area to SurfaceFlinger, the layer identification module can determine that a new frame has been received. Taking the received buffer b1 as an example, which stores the first frame of the wallpaper resource and the status bit of the wallpaper layer is not rotated, SurfaceFlinger can determine that the wallpaper layer needs to be preprocessed. SurfaceFlinger sends a synthesis request for the wallpaper layer to the GPU. The GPU rotates the wallpaper layer according to the rotation parameters, such as rotating it 90° counterclockwise, and renders the rotated wallpaper layer on the canvas using a line-by-line scanning method to obtain a preprocessed layer. On the one hand, the GPU can store the preprocessed layer in a buffer area (such as buffer area a) for use in the synthesis of the wallpaper layer in the N+1th frame. On the other hand, the GPU can execute the following A11 to use the preprocessed layer for the synthesis of the wallpaper layer in the Nth frame.

[0217] It should be noted that Example 1 involves two types of buffers: the first type of buffer is used to store the layer data to be synthesized, for example, buffer b1 stores metadata for the wallpaper layer, buffer b2 stores metadata for the desktop layer, buffer b3 stores metadata for the video layer, and buffer b4 stores metadata for the status bar layer; the second type of buffer is used to store pre-processed layers, for example, buffer a stores a wallpaper layer that has been rotated 90° counterclockwise. It should be understood that when pre-processing is required for the layers of multiple applications, the pre-processed layers of each application can be stored in different buffers.

[0218] A6: When a desktop application's desktop elements (such as icons) change, the desktop application sends a registration request to SurfaceFlinger. After successful registration, SurfaceFlinger begins returning Vsync-APP signals to the desktop application. The wallpaper application notifies SurfaceFlinger to create a blank desktop layer.

[0219] A7: The desktop application calls the render thread to draw the desktop layer and stores the drawn desktop layer in a buffer (e.g., buffer b2). That is, the desktop layer is drawn according to the rotation parameters and rotated (e.g., 90° counterclockwise).

[0220] Then, the desktop application calls a function to send buffer b2 to SurfaceFlinger. Buffer b2 may include layer information such as the layer name of the desktop layer, the status bit of the desktop layer (rotated), and the stacking order of the desktop layer.

[0221] A8, SurfaceFlinger determines whether to pre-process the desktop layer based on the layer information of the desktop layer.

[0222] Whenever the application sends a new buffer to SurfaceFlinger, the layer recognition module can determine that a new frame has been received. Since the desktop layer in buffer b2 is a rotated layer, SurfaceFlinger can determine that no preprocessing of the wallpaper layer is required.

[0223] Referring to the description of the above embodiment, the desktop application has rotated the desktop layer 90° counterclockwise during the drawing process, and no further rotation is required. In principle, SurfaceFlinger can assign the synthesis task of the desktop layer to the HWC. However, since the wallpaper layer located below the desktop layer and the video layer located above the desktop layer are both read out through the video codec, and the video codec does not have the ability to rotate, the wallpaper layer and the video layer are layers that have not been rotated. SurfaceFlinger can assign the synthesis task of the wallpaper layer, desktop layer, and video layer to the GPU instead of to the HWC.

[0224] A9, the video application notifies SurfaceFlinger to create a blank video layer / canvas.

[0225] A10: The video application notifies the video codec to decode the video resource. The video resource stores the decoded resource in a buffer (for example, buffer b3) and calls a function to send buffer b3 to SurfaceFlinger.

[0226] The buffer b3 may include the layer name of the video layer, the status bit of the video layer, the layer overlay order of the video layer, and / or the rotation parameter layer information of the video layer.

[0227] It should be noted that, since each frame of the video application will be updated, SurfaceFlinger can determine whether the video layer needs to be preprocessed based on the layer name of the video layer, but directly synthesize the video frame.

[0228] For example, SurfaceFlinger requests video composition from the GPU. The GPU rotates the video layer according to the rotation parameters, such as rotating it 90 degrees counterclockwise, and rounds the corners of the video layer to obtain the processed video layer. The GPU can then execute the following A11 to use the processed video layer for layer composition in the Nth frame.

[0229] A11, GPU synthesizes the wallpaper layer, desktop layer and video layer, and then sends the synthesized layer to HWC.

[0230] A12: When a status bar element in the system UI (such as the signal strength indicator or the remaining battery indicator) changes, the system UI sends a registration request to SurfaceFlinger. After successful registration, SurfaceFlinger begins returning Vsync-APP signals to the system UI.

[0231] A13, the system UI notifies SurfaceFlinger to create a blank status bar layer.

[0232] A14: The system UI calls the render thread to draw the status bar layer and stores the drawn status bar layer in a buffer (e.g., buffer b4). That is, the status bar layer is drawn according to the rotation parameters and rotated (e.g., 90° counterclockwise).

[0233] Then, the system UI calls a function to send buffer b4 to SurfaceFlinger, which may include layer information such as the layer name of the status bar layer, the status bit of the status bar layer (rotated), and the overlay order of the status bar layer.

[0234] A15, SurfaceFlinger determines whether to pre-process the status bar layer based on the layer information of the status bar layer.

[0235] Whenever the application sends a new buffer to SurfaceFlinger, the layer recognition module can determine that a new frame has been received. Since the status bar layer in buffer b4 is a rotated layer, SurfaceFlinger can determine that no preprocessing is required for the status bar layer and sends the status bar layer to HWC.

[0236] Different from the desktop layer, the status bar layer of A15 is located above the wallpaper layer, status bar layer and video layer, so SurfaceFlinger can directly assign the synthesis task of the status bar layer to HWC.

[0237] A16, HWC synthesizes the composite layer and the status bar layer to obtain the Nth frame image, and sends the Nth frame image to the large screen for display.

[0238] It should be understood that when the wallpaper layer to be synthesized is the first frame or a changing frame, SurfaceFlinger can notify the GPU to render the rotated wallpaper layer and put the rotated wallpaper layer into the cache. In this way, when the same layer processing task is subsequently performed, the rotated wallpaper layer in the cache can be directly used, and the GPU does not need to rotate each wallpaper layer, thereby reducing the power consumption and load of the entire machine.

[0239] Example 2

[0240] When the folding state of a foldable device changes, or the landscape or portrait orientation of a large screen changes, the WMS can notify wallpaper apps, desktop apps, and video apps of the updated rotation parameters. Based on Example 1, if an application's window changes but the application doesn't receive the updated rotation parameters, the application can continue to process the layer according to the rotation parameters obtained in Example 1.

[0241] The following takes the example of a case where after synthesizing the Nth frame image and before synthesizing the N+1th frame image, the windows of the wallpaper application and the desktop application do not change, while the windows of the video application and the status bar application change. In combination with Figure 15, the process of the method for synthesizing the N+1th frame image based on the preprocessing layer is introduced.

[0242] As shown in FIG. 15 , after A1 - A16 , the method may further include the following A17 to A22 .

[0243] A17, SurfaceFlinger determines whether to preprocess the wallpaper layer.

[0244] Whenever the application sends a new buffer area to SurfaceFlinger, the layer recognition module can determine that a new frame has been received. If the synthesis task of the next wallpaper layer is not received within the preset time after SurfaceFlinger receives the synthesis task of the wallpaper layer in the i-th frame image, then SurfaceFlinger can send the wallpaper layer last obtained from the buffer area b1 to the scene recognition module. Since the cache area of ​​the wallpaper layer in A17 and the cache area of ​​the wallpaper layer in A5 are both buffer area b1, the wallpaper resource is not the first frame and the change frame. In addition, the status bit of the wallpaper layer is not rotated, so the scene recognition module recognizes that the N+1 frame can use the pre-processed layer and does not need GPU synthesis, thereby obtaining the pre-processed layer (that is, the rotated wallpaper layer) from buffer area a, and then sending the pre-processed layer to the HWC.

[0245] A18, the desktop application determines whether to pre-process the desktop layer.

[0246] Whenever the application sends a new buffer area to SurfaceFlinger, the layer recognition module can determine that a new frame has been received. If SurfaceFlinger does not receive the next desktop layer synthesis task within the preset time after receiving the synthesis task of the desktop layer in the i-th frame image, SurfaceFlinger can send the desktop layer last obtained from the buffer area b2 to the scene recognition module. Since the status bit of the desktop layer is rotated, SurfaceFlinger can determine that the desktop layer does not need to be preprocessed and sends the desktop layer to HWC.

[0247] It should be noted that unlike the above-mentioned A8 where SurfaceFlinger sends the desktop layer to the GPU, the A17 uses a pre-processing layer for wallpaper, so the GPU does not need to process the wallpaper layer. In this way, in the A18, the desktop layer between the wallpaper layer and the video layer does not need to be synthesized by the GPU. SurfaceFlinger can send the desktop layer directly to the HWC.

[0248] A19: The video application notifies the video codec to decode the video resource. The video resource stores the decoded resource in a buffer (for example, buffer b3') and calls a function to send buffer b3' to SurfaceFlinger. SurfaceFlinger requests video synthesis from the GPU. The GPU rotates the video layer according to the rotation parameters (for example, 90° counterclockwise) and rounds the corners of the video layer to obtain the processed video layer. The GPU sends the video layer to the HWC.

[0249] It should be noted that the buffer b3' and the buffer b3 of A10 are different buffer areas. For example, the buffer b3 is used to store the first frame of the video, and the buffer b3' is used to store the second frame of the video.

[0250] For the specific implementation of A19, please refer to the description of A10 above, which will not be repeated here.

[0251] A20, when the status bar element of the system UI changes again, the system UI calls the render thread again to draw the status bar layer and stores the drawn status bar layer in the buffer (such as buffer b4'). In other words, the status bar layer is drawn according to the rotation parameters and rotated (for example, rotated 90 degrees counterclockwise).

[0252] Then, the system UI calls the function to send buffer area b4' to SurfaceFlinger.

[0253] It should be noted that the buffer area b4 ′ and the buffer area b4 of A14 are different cache areas.

[0254] A21, SurfaceFlinger determines whether to pre-process the status bar layer based on the layer information of the status bar layer.

[0255] Whenever the application sends a new buffer to SurfaceFlinger, the layer recognition module can determine that a new frame has been received. Since the status bar layer in buffer b4' is a rotated layer, SurfaceFlinger can determine that no preprocessing is required for the status bar layer and sends the status bar layer to the HWC.

[0256] A22, HWC synthesizes the pre-processing layer (i.e., wallpaper layer), desktop layer, video layer, and status bar layer to obtain the N+1th frame image, and sends the N+1th frame image to the large screen for display.

[0257] It should be understood that since the GPU sends the rotated wallpaper layer to cache a in advance, when there is a subsequent same layer processing task, the rotated wallpaper layer in cache a can be directly used to finally generate the N+1th frame image, thereby reducing the power consumption of the entire machine and improving the performance of the entire machine.

[0258] In the above-mentioned Examples 1 and 2, the wallpaper image obtained from the cache area b1 can be the last frame of a video resource. Usually, after the large screen is turned on, the large screen will start playing from the first frame of the video resource until the last frame of the video resource, and then the large screen will continue to display the last frame of the video resource. It should be understood that during the period from the first frame of the video resource to the last frame of the video resource, the wallpaper is updated more frequently, and the next frame of each frame of wallpaper may not require the use of a pre-processing layer; after playing to the last frame of the video resource, the wallpaper remains basically unchanged, and a pre-processing layer may need to be used subsequently. In this regard, in combination with Figure 16, an embodiment of the present application provides another layer synthesis method.

[0259] B1. The wallpaper application notifies SurfaceFlinger to create a wallpaper layer.

[0260] B2, the wallpaper application notifies the video codec to decode the video resource.

[0261] Assume that the video resource of the wallpaper consists of M frames. When the large screen is turned on, the wallpaper application can notify the video codec to decode the video resource. The video codec first stores the decoded first frame in Buffer1, the decoded second frame in Buffer2, the decoded third frame in Buffer3, ..., and so on, and directly stores the last frame in BufferM. Then, the video codec sends each Buffer to SurfaceFlinger in sequence. Among them, each Buffer can contain the layer name of the wallpaper layer, the status bit of the wallpaper layer, the layer overlay order of the wallpaper layer, and / or the rotation parameter layer information of the wallpaper layer. Every time SurfaceFlinger obtains a Buffer (a Buffer is equivalent to a synthesis task), it can perform the following judgment steps:

[0262] SurfaceFlinger obtains the i-th composition task and begins counting. Because the wallpaper resource for the i-th composition task is marked as unrotated, SurfaceFlinger determines that the wallpaper layer needs to be preprocessed. SurfaceFlinger sends a composition request to the GPU for the wallpaper layer. The GPU rotates the wallpaper layer according to the rotation parameters, for example, 90° counterclockwise, to obtain the preprocessed layer for the i-th composition task. The GPU stores the preprocessed layer in buffer a_i and sends the preprocessed layer for the i-th composition task to the HWC.

[0263] If SurfaceFlinger obtains the i+1th synthesis task before the timing reaches the maximum timing (for example, N Vsync-SF signal cycles, where N is a positive integer), it can be determined that the wallpaper is updated frequently and the pre-processed layer in the cache a_i does not need to be used. On the one hand, the pre-processed layer in the cache a_i can be cleared. On the other hand, SurfaceFlinger starts retiming. Since the status of the wallpaper resource of the i+1th synthesis task is marked as unrotated, SurfaceFlinger determines that the wallpaper layer needs to be pre-processed. SurfaceFlinger sends a synthesis request for the wallpaper layer to the GPU. The GPU rotates the wallpaper layer according to the rotation parameters, such as rotating it 90° counterclockwise, to obtain the pre-processed layer of the i+1th synthesis task. Then, the GPU stores the pre-processed layer of the i+1th synthesis task in the cache a_i+1, and sends the pre-processed layer of the i+1th synthesis task to the HWC.

[0264] If SurfaceFlinger does not get the i+1th synthesis task before the timing reaches the maximum timing (for example, N Vsync-SF signal cycles), it can be determined that the wallpaper layer has not been updated and the pre-processed layer in the buffer a_i needs to be used. SurfaceFlinger reads the pre-processed layer of the i-th synthesis task from the buffer a_i and sends the pre-processed layer to the HWC.

[0265] It should be understood that when the wallpaper resource is a video resource, the wallpaper updates frequently from the first frame to the last frame of the video resource. The GPU can temporarily store each frame of wallpaper in the cache after rotating it. If no new tasks are received within N Vsync-SF signal cycles after playing a certain frame, it means that the video resource has been played to the last frame, and the pre-processed layer in the cache can be used.

[0266] In each of the above embodiments, the preprocessing operation performed by the GPU is to rotate the wallpaper. The wallpaper application may not refresh the wallpaper for tens of seconds, minutes or even hours. Therefore, after preprocessing the wallpaper once, the preprocessed wallpaper can be reused for a long time, and the power consumption benefit is relatively large. Based on the preprocessing principle, the present application also provides a generalized layer synthesis method: when multiple applications send images to SurfaceFlinger, and the refresh rates of multiple applications are different, SurfaceFlinger determines the maximum refresh rate as the system frame rate. In the process of sending images by the first application, if the synthesis requirements of the first application layer exceed the capabilities of the HWC hardware module (such as cropping radius, number of layers, conversion format, layer size, zoom ratio and window blur, etc.), the GPU is used to preprocess the first application layer. Among them, the refresh rate of the first application is less than the system frame rate.

[0267] For example, Figure 17 is a schematic diagram of a scenario in which a pre-processed layer is applied, as provided in an embodiment of the present application. The refresh rate of the video application is 30fps, and the refresh rate of the headline application is 120fps. In order to keep the refresh rates of various applications consistent, SurfaceFlinger determines 120fps as the system frame rate, so that both the video application and the headline application send images to SurfaceFlinger at a refresh rate of 120fps. In this case, every time the headline application refreshes three frames, such as frame M1, frame M2, and frame M3, the video application also refreshes three frames N1. In combination with the description of the above embodiment, since the video layer needs to be rotated and rounded by the GPU, if all three frames N1 are rotated and rounded by the GPU, the power consumption of the entire device will be higher. Therefore, the layer synthesis method provided in an embodiment of the present application can be used to pre-process the first frame N1 to obtain a pre-processed layer, and then the second frame N1 and the third frame N1 pre-multiplex the pre-processed layer, thereby reducing the power consumption of the entire folding screen device and improving the performance of the entire device.

[0268] For example, FIG18 is a schematic diagram of the hardware structure of a folding screen device 100 provided in an embodiment of the present application.

[0269] As shown in Figure 18, the folding screen device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a SIM card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, and a bone conduction sensor 180M, etc.

[0270] The processor 110 may include one or more processing units. For example, the processor 110 may include a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), an image signal processor (ISP), a neural-network processing unit (NPU), a modem processor (modem), a data processing unit (DPU) and / or a baseband processor, etc.

[0271] The CPU is the final execution unit for information processing and program execution. Its main tasks include processing instructions, executing operations, controlling time, and processing data. The CPU may include a controller, an arithmetic unit, a cache memory, and a bus for connecting these components. The controller is the nerve center and command center of the folding screen device 100. The controller can generate operation control signals based on the instruction opcode and timing signal to complete the control of instruction fetching and execution. The arithmetic unit refers to a component that performs various arithmetic and logical operations. The cache memory can store instructions or data that have just been used or recycled by the processor 110.

[0272] The GPU, also known as a display core, visual processor, or display chip, is a microprocessor for graphics and image processing. The GPU connects the display 194, the AP, and the CPU. The GPU can perform complex mathematical and geometric calculations, floating-point operations, parallel computing, and graphics rendering. The GPU reduces graphics card reliance on the CPU, especially when processing 3D graphics. The GPU utilizes core technologies such as cubic environment material mapping and vertex blending, allowing the GPU to perform some of the work previously performed by the CPU.

[0273] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the folding screen device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store the operating system and the application required for at least one function. The data storage area can store configuration files of various applications, as well as data created during the use of the folding screen device 100.

[0274] The foldable screen device 100 can implement audio functions through the audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and AP.

[0275] The display screen 194 includes a large screen and a small screen as shown in FIG. 2A to FIG. 2C or FIG. 3A to FIG. 3C .

[0276] It is understood that the hardware structure illustrated in the embodiments of this application does not constitute a specific limitation on the foldable screen device 100. In other embodiments, the foldable screen device 100 may include more or fewer components than shown, or combine or split certain components, or arrange the components differently. The components shown in Figure 18 can be implemented in hardware, software, or a combination of software and hardware.

[0277] An embodiment of the present application also provides a terminal device, such as a folding screen device, including a processor, which is coupled to a memory, and the processor is used to execute computer programs or instructions stored in the memory to implement the methods in the above embodiments.

[0278] The embodiment of the present application also provides a computer-readable storage medium having computer instructions stored therein; when the computer-readable storage medium is run on a computer, the computer is caused to execute the method shown above. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more servers that can be integrated with the medium. The available medium can be a magnetic medium (e.g., a floppy disk, hard disk, or tape), an optical medium, or a semiconductor medium (e.g., a solid state disk (SSD)).

[0279] An embodiment of the present application further provides a computer program product, which includes computer program code. When the computer program code runs on a computer, the computer executes the methods in the above embodiments.

[0280] The present application also provides a chip that is coupled to a memory and is used to read and execute computer programs or instructions stored in the memory to perform the methods in the above embodiments. The chip can be a general-purpose processor or a dedicated processor. It should be noted that the chip can be implemented using the following circuits or devices: one or more field programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gate logic, discrete hardware components, any other suitable circuits, or any combination of circuits capable of performing the various functions described throughout this application.

[0281] The folding screen device, computer-readable storage medium, computer program product and chip provided in the above-mentioned embodiments of the present application are all used to execute the method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects corresponding to the method provided above, and will not be repeated here.

[0282] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0283] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules or units is merely a logical functional division. In actual implementation, other division methods may be used, such as combining or integrating multiple units or components into another device, or ignoring or not implementing certain features.

[0284] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0285] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0286] The above content is only a specific embodiment of this application, but the scope of protection of this application is not limited to this. Any changes or replacements within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A layer synthesis method, characterized in that: The method is applied to a terminal device, the terminal device includes a graphics processor GPU, a hardware synthesizer HWC, a large screen and a small screen, and the method includes: When the terminal device is at a first folding angle, obtaining a first wallpaper layer; Combining the first wallpaper layer, the first desktop layer and the first status bar layer into a first image; Displaying the first image on the small screen; In response to a user adjusting the terminal device from the first folding angle to a second folding angle, acquiring the first wallpaper layer, the second folding angle being greater than the first folding angle; Rotating the first wallpaper layer by the GPU to obtain a first preprocessing layer, wherein the height of the first wallpaper layer is greater than the maximum rotation height supported by the HWC; Storing the first preprocessed layer in a first buffer area; Combining the first preprocessing layer, the second desktop layer, and the second status bar layer into a second image; Displaying the second image on the large screen; Acquire the first preprocessing layer from the first buffer area; Combining the first preprocessing layer, the third desktop layer and the third status bar layer into a third image; The third image is displayed on the large screen.

2. The method according to claim 1, characterized in that The obtaining the first wallpaper layer includes: Acquire a first Vsync-SF signal; If the second buffer area has been acquired before the first Vsync-SF signal is acquired, acquiring the first wallpaper layer from the second buffer area; The acquiring the first preprocessing layer from the first buffer area includes: Obtaining a second Vsync-SF signal; If no other buffer area is obtained before the second Vsync-SF signal is obtained, the first pre-processed layer is obtained from the first buffer area.

3. The method according to claim 2, characterized in that After acquiring the second Vsync-SF signal, the method further includes: If the third buffer area is acquired before the second Vsync-SF signal is acquired, acquiring the second wallpaper layer from the third buffer area; Rotating the second wallpaper layer by the GPU to obtain a second preprocessing layer, wherein the height of the second wallpaper layer is greater than the maximum rotation height supported by the HWC; storing the second preprocessed layer in a fourth buffer area; Combining the second preprocessing layer, the fourth desktop layer and the fourth status bar layer into a fourth image; The fourth image is displayed on the large screen.

4. The method according to any one of claims 1 to 3, characterized in that The row scanning direction of the small screen is parallel to the first frame, and the column scanning direction of the small screen is parallel to the second frame; The row scanning direction of the large screen is parallel to the second frame, and the column scanning direction of the large screen is parallel to the first frame; The first frame and the second frame are two adjacent frames on the body of the terminal device, and the length of the first frame is smaller than the length of the second frame.

5. A layer synthesis method, characterized in that: The method is applied to a terminal device, the terminal device includes a GPU, a HWC and a display screen, and the method includes: Acquire a first wallpaper layer, where the height of the first wallpaper layer is greater than the maximum rotation height supported by the HWC; Rotating the first wallpaper layer by the GPU to obtain a first preprocessing layer; Combining the first preprocessing layer, the first desktop layer and the first status bar layer into a first image; Storing the first preprocessed layer in a first buffer area; displaying the first image on the display screen; Acquire the first preprocessing layer from the first buffer area; Combining the first preprocessing layer, the second desktop layer, and the second status bar layer into a second image; The second image is displayed on the display screen.

6. The method according to claim 5, characterized in that The step of obtaining the first wallpaper layer includes: Acquire a first Vsync-SF signal; If the second buffer area has been acquired before the first Vsync-SF signal is acquired, acquiring the first wallpaper layer from the second buffer area; The acquiring the first preprocessing layer from the first buffer area includes: Obtaining a second Vsync-SF signal; If no other buffer area is obtained before the second Vsync-SF signal is obtained, the first pre-processed layer is obtained from the first buffer area.

7. The method according to claim 6, characterized in that After acquiring the second Vsync-SF signal, the method further includes: If the third buffer area is acquired before the second Vsync-SF signal is acquired, acquiring the second wallpaper layer from the third buffer area; Rotating the second wallpaper layer by the GPU to obtain a second preprocessing layer, wherein the height of the second wallpaper layer is greater than the maximum rotation height supported by the HWC; storing the second preprocessed layer in a fourth buffer area; Combining the second preprocessing layer, the third desktop layer and the third status bar layer into a third image; The third image is displayed on the display screen.

8. The method according to any one of claims 5 to 7, characterized in that The row scanning direction of the display screen is parallel to the second frame, and the column scanning direction of the display screen is parallel to the first frame; The first frame and the second frame are two adjacent frames on the body of the terminal device, and the length of the first frame is smaller than the length of the second frame.

9. The method according to claim 6, characterized in that The acquiring the first wallpaper layer from the second buffer area includes: Acquire the first wallpaper layer and layer information of the first wallpaper layer from the second buffer area; wherein the layer information of the first wallpaper layer includes a state bit and a rotation parameter, the state bit is used to indicate that the first wallpaper layer has not been rotated, and the rotation parameter is used to indicate a rotation direction and a rotation angle of the first wallpaper layer; The step of rotating the first wallpaper layer by the GPU to obtain a first preprocessing layer includes: The GPU rotates the first wallpaper layer based on the state bit and the rotation parameter to obtain the first preprocessing layer.

10. The method according to claim 9, characterized in that Before obtaining the first wallpaper layer, the method further includes: When the preset conditions are met, determining the rotation parameters; Sending the rotation parameter to a wallpaper application of the terminal device; Generate a status bit through the wallpaper application, and store the status bit, the rotation parameter and the first wallpaper layer in the second buffer area; The preset condition includes any one of the following: The display screen switches from a horizontal screen state to a vertical screen state; The terminal device switches from a folded state to an unfolded state.

11. The method according to claim 10, characterized in that The method further comprises: In response to user operation, the horizontal and vertical screen functions are turned on; wherein, the horizontal and vertical screen functions allow display in a first direction when the terminal device is in the horizontal screen state, and display in a second direction when the terminal device is in the vertical screen state; the first direction differs from the second direction by 90° or 270° clockwise.

12. The method according to claim 11, characterized in that When the terminal device is in the horizontal screen state, the angle between the first frame of the terminal device body and the ground plane is greater than the angle between the second frame of the terminal device body and the ground plane; when the terminal device is in the vertical screen state, the angle between the first frame and the ground plane is less than or equal to the angle between the second frame and the ground plane; Wherein, the length of the first frame is smaller than the length of the second frame.

13. The method according to claim 9, characterized in that The layer information of the first wallpaper layer also includes a layer stacking order of the first wallpaper layer; The synthesizing the first preprocessing layer, the first desktop layer and the first status bar layer into the first image comprises: superimposing the first preprocessing layer, the first desktop layer and the first status bar layer according to the layer superposition order of the first wallpaper layer, the layer superposition order of the first desktop layer and the layer superposition order of the first status bar layer to obtain the first image; The synthesizing the first preprocessing layer, the second desktop layer and the second status bar layer into the second image includes: superimposing the first preprocessing layer, the second desktop layer and the second status bar layer according to the layer superposition order of the first wallpaper layer, the layer superposition order of the second desktop layer and the layer superposition order of the second status bar layer to obtain the second image.

14. A terminal device, characterized in that: comprising a processor, and a memory coupled to the processor; The memory stores instructions, and the processor calls the instructions so that the terminal device executes the layer synthesis method as described in any one of claims 1 to 13.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on a terminal device, the terminal device executes the layer synthesis method according to any one of claims 1 to 13.

Images