Layer synthesis method, terminal equipment and storage medium

Abstract

The invention provides a layer synthesis method, terminal equipment and a storage medium, and relates to the technical field of terminals. In the scheme, in the process that a user uses a large screen, terminal equipment can identify each to-be-synthesized layer to determine whether the layer needs to be preprocessed by a GPU (Graphics Processing Unit) or not, such as wallpaper layer rotation; if the GPU needs to be used for preprocessing, the GPU is used for rendering a preprocessed layer, and the preprocessed layer is stored in the cache region. And when the same layer processing task exists subsequently, the preprocessed layers in the cache region can be directly sent to the HWC for hardware synthesis, and GPU synthesis is not needed, so that the overall power consumption and load of the terminal equipment are reduced, and the overall performance is improved.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a layer synthesis method, a terminal device and a storage medium. Background Art

[0002] Large-screen devices are increasingly being used. When using large-screen devices, users can adjust the screen orientation at any time. Large-screen devices can rotate wallpaper resources so that the wallpaper orientation remains consistent from the user's visual perspective before and after the screen orientation is adjusted.

[0003] At present, most large-screen devices use large-sized high-definition wallpapers. When the height of the wallpaper resource exceeds the maximum rotation height supported by the hardware composer (HWC), the graphics processing unit (GPU) can be used for synthesis. The GPU first rotates the wallpaper resource 90°, then sends the rotated wallpaper resource to the HWC for hardware synthesis, and then sends it to the large screen for display, so that the user can see that the wallpaper direction remains consistent before and after adjusting the screen direction. However, GPU synthesis has the disadvantage of high power consumption. When multiple frames of images are synthesized based on the same wallpaper resource, the GPU may need to perform multiple rotation operations on the same wallpaper resource, resulting in high power consumption and poor performance of the large-screen device. Summary of the invention

[0004] The present application provides a layer synthesis method, a terminal device and a storage medium, which solve the technical problem that the GPU frequently rotates the wallpaper layer, resulting in high power consumption and poor performance of the terminal device.

[0005] In order to achieve the above objectives, this application adopts the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a layer synthesis method. The method can be applied to a terminal device. Taking the terminal device as a folding screen device as an example, the folding screen device may include a GPU, a HWC, a large screen, and a small screen. The method may include:

[0007] When the folding screen device is at a first folding angle (i.e., a folded state), if the first wallpaper layer is obtained, the first wallpaper layer, the first desktop layer, and the first status bar layer are synthesized into a first image, and the first image is displayed on the small screen. If an operation is received in which the user adjusts the folding screen device from the first folding angle to the second folding angle (such as switching from a folded state to an unfolded state), then in response to the operation, the first wallpaper layer is obtained, and the first wallpaper layer is rotated by the GPU to obtain a first preprocessing layer, the first preprocessing layer, the second desktop layer, and the second status bar layer are synthesized into a second image, the first preprocessing layer is stored in the first cache area, and the second image is displayed on the large screen. While maintaining the unfolded state, the first preprocessing layer is obtained from the first cache area, the first preprocessing layer, the third desktop layer, and the third status bar layer are synthesized into a third image, and the third image is displayed on the large screen.

[0008] The second folding angle is greater than the first folding angle. The first folding angle is less than or equal to a first preset angle, which may be 30° or 45°, etc. The second folding angle is greater than or equal to a second preset angle, which may be 90° or 120°, etc.

[0009] In the above scheme, when the folding screen device is in the folded state, the installation direction of the small screen is consistent with the direction of the wallpaper, so there is no need to rotate the wallpaper layer, and the composite image of the wallpaper layer and other layers is directly displayed on the small screen. When the folding screen device switches from the folded state to the unfolded state, the installation direction of the large screen is inconsistent with the wallpaper direction, and the height of the first wallpaper layer is greater than the maximum rotation height supported by the HWC. Therefore, the GPU is required to rotate the first wallpaper layer in advance to obtain a preprocessed layer, and store the preprocessed layer in the cache area. In this way, when there is the same layer processing task, the preprocessed layer in the cache area can be directly sent to the HWC for hardware synthesis, without using the GPU to rotate the wallpaper layer, thereby reducing the overall power consumption and load of the folding screen device and improving the overall performance of the folding screen device.

[0010] It should be noted that the first desktop layer, the second desktop layer and the third desktop layer can be the same layer or different layers; the first status bar layer, the second status bar layer and the third status bar layer can be the same layer or different layers. In addition, the present application does not specifically limit the execution order of "combining the first preprocessing layer, the second desktop layer and the second status bar layer into the second image" and "storing the first preprocessing layer in the first cache area". The two can be executed at the same time, or "combining the first preprocessing layer, the second desktop layer and the second status bar layer into the second image" can be executed first and then "storing the first preprocessing layer in the first cache area". It can also be executed first "storing the first preprocessing layer in the first cache area" and then "combining the first preprocessing layer, the second desktop layer and the second status bar layer into the second image".

[0011] In one possible implementation, the folding screen device is an inner folding screen device. The inner folding screen device is provided with an inner screen and an outer screen. The size of the inner screen is larger than that of the outer screen, so the inner screen is also called the large screen, and the outer screen is also called the small screen. The large screen can be bent into two display areas. When the inner folding screen device is in a folded state, the folding angle between the two display areas of the large screen is less than or equal to a first preset angle, the small screen faces the user, and the large screen is in a hidden state; when the inner folding screen device is in an unfolded state, the folding angle between the two display areas of the large screen is greater than or equal to a second preset angle, the large screen faces the user, and the small screen is away from the user. When in the folded state, the small screen displays the wallpaper; after switching from the folded state to the unfolded state, the large screen displays the wallpaper. Among them, the large screen and the small screen can use the same wallpaper resource.

[0012] In one possible implementation, the folding screen device is an external folding screen device. The external folding screen device is provided with a screen, which can be folded into at least two display areas. The larger display area of ​​the at least two display areas is called the main screen, and the smaller display area is called the back screen or the secondary screen. Usually, the main screen is also called the small screen, and the main screen and the back screen are collectively referred to as the large screen. When the external folding screen device is in a folded state, the folding angle between the main screen and the back screen is less than or equal to the first preset angle, the main screen faces the user, and the back screen faces away from the user; when the external folding screen device is in an unfolded state, the folding angle between the main screen and the back screen is greater than or equal to the second preset angle, and the main screen and the back screen face the user. When in the folded state, the small screen displays the wallpaper; after switching from the folded state to the unfolded state, the large screen displays the wallpaper. Among them, the large screen and the small screen can use the same wallpaper resource.

[0013] In a possible implementation, when the folding screen device switches from a folded state to an unfolded state, obtaining the first wallpaper layer includes: obtaining a first Vsync-SF signal; if the second buffer area has been obtained before the first Vsync-SF signal is obtained, obtaining the first wallpaper layer from the second buffer area. The second buffer area is used to store the first wallpaper layer.

[0014] In the above solution, when the wallpaper of the wallpaper application changes, the changed wallpaper layer can be stored in the second buffer area. In this way, when the first Vsync-SF signal is obtained, the wallpaper layer can be directly read from the second buffer area.

[0015] In a possible implementation, after obtaining the first Vsync-SF signal, a second Vsync-SF signal may also be obtained. Then, SurfaceFlinger determines whether other cache areas other than the second cache area are obtained after obtaining the first Vsync-SF signal and before obtaining the second Vsync-SF signal, and other cache areas refer to cache areas storing other wallpaper layers. In one case, if other cache areas are not obtained before obtaining the second Vsync-SF signal, the first pre-processed layer is obtained from the first cache area. In another case, if the third cache area is obtained before obtaining the second Vsync-SF signal, the second wallpaper layer is obtained from the third cache area, the second wallpaper layer is rotated by the GPU to obtain the second pre-processed layer (the height of the second wallpaper layer is greater than the maximum rotation height supported by the HWC), and the second pre-processed layer is stored in the fourth cache area; the second pre-processed layer, the fourth desktop layer and the fourth status bar layer are synthesized into a fourth image; and the fourth image is displayed on the large screen. Among them, the second Vsync-SF signal is a signal generated after the first Vsync-SF signal. The second Vsync-SF signal differs from the first Vsync-SF signal by N Vsync-SF signal cycles, where N is a positive integer.

[0016] In the above scheme, when the wallpaper resource is a video resource, the wallpaper is updated frequently during the period from the first frame of the video resource to the last frame of the video resource. The GPU can temporarily store each frame of the wallpaper in the cache after rotating it. After obtaining the first Vsync-SF signal and before obtaining the second Vsync-SF signal (i.e., within m Vsync cycles after a certain frame is played), if no new wallpaper layer is obtained, it means that the last frame of the video resource has been played, and the pre-processing layer in the cache can be used.

[0017] In a possible implementation, the row scanning direction of the small screen is parallel to the first frame, and the column scanning direction of the small screen is parallel to the second frame. The row scanning direction of the large screen is parallel to the second frame, and the column scanning direction of the large screen is parallel to the first frame. The first frame and the second frame are two adjacent frames on the body of the folding screen device, and the length of the first frame is less than the length of the second frame.

[0018] In the above solution, the installation directions of the small screen and the large screen are inconsistent, and the display direction of the small screen and the wallpaper resource is consistent, so the small screen does not need to rotate the wallpaper, but the large screen does.

[0019] In a second aspect, an embodiment of the present application provides a layer synthesis method. The method can be applied to a terminal device. The terminal device can be a large-screen device, such as a folding screen device or a PAD. The large-screen device can include a GPU, a HWC, and a large screen. The method can include:

[0020] When the large-screen device is in a horizontal state, if the first wallpaper layer is obtained, the first wallpaper layer, the first desktop layer, and the first status bar layer are synthesized into a first image, and the first image is displayed on the large screen. If the user's operation of adjusting the large-screen device from a horizontal state to a vertical state is obtained, then in response to the operation, the first wallpaper layer is obtained, and the first wallpaper layer is rotated by the GPU to obtain the first preprocessing layer, the first preprocessing layer, the second desktop layer, and the second status bar layer are synthesized into a second image, the first preprocessing layer is stored in the first buffer area, and the second image is displayed on the large screen. Then, in the process of maintaining the vertical state, the first preprocessing layer is obtained from the first buffer area, the first preprocessing layer, the third desktop layer, and the third status bar layer are synthesized into a third image, and the third image is displayed on the large screen.

[0021] In the above scheme, when the large-screen device is in landscape mode, the display direction of the large screen is consistent with the wallpaper direction, and there is no need to rotate the wallpaper layer. The composite image of the wallpaper layer and other layers is directly displayed on the large screen. When the large-screen device is adjusted from landscape mode to portrait mode, the display direction of the large screen is inconsistent with the wallpaper direction, and the wallpaper layer needs to be rotated. The GPU can pre-rotate the first wallpaper layer to obtain a pre-processed layer, and store the pre-processed layer in the cache area. Therefore, when there are the same layer processing tasks, the pre-processed layer in the cache area can be directly sent to the HWC for hardware synthesis without using GPU synthesis, thereby reducing the overall power consumption and load of the large-screen device and improving the overall performance of the large-screen device.

[0022] In a possible implementation, when the horizontal screen state is adjusted to the vertical screen state, obtaining the first wallpaper layer includes: obtaining a first Vsync-SF signal; if the second buffer area has been obtained before the first Vsync-SF signal is obtained, obtaining the first wallpaper layer from the second buffer area. The second buffer area is used to store the first wallpaper layer.

[0023] In a possible implementation, after obtaining the first Vsync-SF signal, a second Vsync-SF signal may also be obtained. Then, SurfaceFlinger determines whether other cache areas other than the second cache area are obtained after obtaining the first Vsync-SF signal and before obtaining the second Vsync-SF signal, and other cache areas refer to cache areas storing other wallpaper layers. In one case, if other cache areas are not obtained before obtaining the second Vsync-SF signal, the first pre-processed layer is obtained from the first cache area. In another case, if the third cache area is obtained before obtaining the second Vsync-SF signal, the second wallpaper layer is obtained from the third cache area, the second wallpaper layer is rotated by the GPU to obtain the second pre-processed layer (the height of the second wallpaper layer is greater than the maximum rotation height supported by the HWC), and the second pre-processed layer is stored in the fourth cache area; the second pre-processed layer, the fourth desktop layer and the fourth status bar layer are synthesized into a fourth image; and the fourth image is displayed on the large screen. Among them, the second Vsync-SF signal is a signal generated after the first Vsync-SF signal. The second Vsync-SF signal differs from the first Vsync-SF signal by N Vsync-SF signal cycles, where N is a positive integer.

[0024] In the above scheme, when the wallpaper resource is a video resource, the wallpaper is updated frequently during the period from the first frame of the video resource to the last frame of the video resource. The GPU can temporarily store each frame of the wallpaper in the cache after rotating it. After obtaining the first Vsync-SF signal and before obtaining the second Vsync-SF signal (i.e., within m Vsync cycles after a certain frame is played), if no new wallpaper layer is obtained, it means that the last frame of the video resource has been played, and the pre-processing layer in the cache can be used.

[0025] In a possible implementation, the row scanning direction of the large screen is parallel to the second frame, and the column scanning direction of the large screen is parallel to the first frame. The first frame and the second frame are two adjacent frames on the body of the large screen device, and the length of the first frame is less than the length of the second frame.

[0026] In a third aspect, an embodiment of the present application provides a layer synthesis method. The method can be applied to a terminal device. Taking the terminal device as a folding screen device as an example, the folding screen device may include a GPU, a HWC, a large screen, and a small screen. The method may include:

[0027] When the folding screen device is at a first folding angle (unfolded state), if a first wallpaper layer is obtained, the first wallpaper layer is rotated by the GPU to obtain a first preprocessing layer, the first preprocessing layer, the first desktop layer, and the first status bar layer are synthesized into a first image, the first preprocessing layer is stored in a first buffer area, and the first image is displayed on a large screen. In the process of maintaining the first folding angle, the first preprocessing layer can be obtained from the first buffer area, the first preprocessing layer, the second desktop layer, and the second status bar layer are synthesized into a second image, and the second image is displayed on a large screen.

[0028] If an operation is received in which the user adjusts the folding screen device from the first folding angle to the second folding angle (the second folding angle is not equal to the first folding angle and is still in the unfolded state), then in response to the operation, the second wallpaper layer is obtained, and the second wallpaper layer is rotated by the GPU to obtain the second pre-processed layer, the second pre-processed layer, the third desktop layer and the third status bar layer are synthesized into a third image, the second pre-processed layer is stored in the second cache area, and the third image is displayed on the large screen. In the process of maintaining the second folding angle, the second pre-processed layer can be obtained from the second cache area, the second pre-processed layer, the fourth desktop layer and the fourth status bar layer are synthesized into a fourth image, and the fourth image is displayed on the large screen.

[0029] In the above scheme, when the folding screen device is in the unfolded state, if the folding angle is increased or decreased, the wallpaper layer is updated, and the GPU rotates the new wallpaper layer in advance to obtain a preprocessed layer, and stores the preprocessed layer in the cache area. Therefore, the preprocessed layer in the cache area can be directly sent to the HWC for hardware synthesis without using GPU synthesis, thereby reducing the overall power consumption and load of the folding screen device and improving the overall performance of the folding screen device.

[0030] In a possible implementation, when the folding screen device is at a first folding angle, obtaining a first wallpaper layer includes: obtaining a first Vsync-SF signal; if buffer area a has been obtained before the first Vsync-SF signal is obtained, obtaining the first wallpaper layer from buffer area a. Buffer area a is used to store the first wallpaper layer.

[0031] Correspondingly, after obtaining the first Vsync-SF signal, the second Vsync-SF signal can also be obtained. Then, SurfaceFlinger determines whether other cache areas other than cache area a are obtained after obtaining the first Vsync-SF signal and before obtaining the second Vsync-SF signal. Other cache areas refer to cache areas storing other wallpaper layers. If other cache areas are not obtained before obtaining the second Vsync-SF signal, the first pre-processed layer is obtained from the first cache area.

[0032] In a possible implementation, when the folding screen device is at the second folding angle, obtaining the second wallpaper layer includes: obtaining a third Vsync-SF signal; if the buffer area b is obtained before the third Vsync-SF signal is obtained, obtaining the second wallpaper layer from the buffer area b. The buffer area b is used to store the second wallpaper layer.

[0033] Correspondingly, after obtaining the third Vsync-SF signal, the fourth Vsync-SF signal can also be obtained. Then, SurfaceFlinger determines whether other cache areas other than cache area b are obtained after obtaining the third Vsync-SF signal and before obtaining the fourth Vsync-SF signal. Other cache areas refer to cache areas storing other wallpaper layers. If other cache areas are not obtained before obtaining the fourth Vsync-SF signal, the second pre-processed layer is obtained from the second cache area.

[0034] In a possible implementation, the row scanning direction of the small screen is parallel to the first frame, and the column scanning direction of the small screen is parallel to the second frame. The row scanning direction of the large screen is parallel to the second frame, and the column scanning direction of the large screen is parallel to the first frame. The first frame and the second frame are two adjacent frames on the body of the folding screen device, and the length of the first frame is less than the length of the second frame.

[0035] In a fourth aspect, an embodiment of the present application provides a layer synthesis method. The method can be applied to a terminal device. The terminal device is provided with a GPU, a HWC, and a display screen. For example, the terminal device may be a folding screen device or a large screen device, etc. The method may include: obtaining a first wallpaper layer, the height of the first wallpaper layer is greater than the maximum rotation height supported by the HWC; rotating the first wallpaper layer by the GPU to obtain a first preprocessing layer; synthesizing the first preprocessing layer, the first desktop layer, and the first status bar layer into a first image; storing the first preprocessing layer in a first buffer area; displaying the first image on the display screen; obtaining the first preprocessing layer from the first buffer area; synthesizing the first preprocessing layer, the second desktop layer, and the second status bar layer into a second image; displaying the second image on the display screen.

[0036] In the above scheme, when the terminal device first obtains the first wallpaper layer, and the height of the first wallpaper layer is greater than the maximum rotation height supported by the HWC, the GPU can be used to rotate the first wallpaper layer to obtain a preprocessed layer, and the preprocessed layer is stored in the cache. Since the GPU pre-rotates the first wallpaper layer to obtain the preprocessed layer and stores the preprocessed layer in the cache, when the terminal device obtains the synthesis task of the first wallpaper layer again, the preprocessed layer in the cache can be directly sent to the HWC for hardware synthesis without using GPU synthesis, thereby reducing the overall power consumption and load of the terminal device and improving the overall performance of the terminal device.

[0037] In a possible implementation, obtaining the first wallpaper layer includes: obtaining a first Vsync-SF signal; if the second buffer area is obtained before the first Vsync-SF signal is obtained, obtaining the first wallpaper layer from the second buffer area. Correspondingly, obtaining the first pre-processed layer from the first buffer area includes: obtaining a second Vsync-SF signal; if no other buffer area is obtained before the second Vsync-SF signal is obtained, obtaining the first pre-processed layer from the first buffer area.

[0038] In one possible implementation, after obtaining the second Vsync-SF signal, the method also includes: if the third cache area is obtained before the second Vsync-SF signal is obtained, obtaining the second wallpaper layer from the third cache area; rotating the second wallpaper layer through the GPU to obtain a second preprocessing layer, and the height of the second wallpaper layer is greater than the maximum rotation height supported by the HWC; storing the second preprocessing layer in the fourth cache area; synthesizing the second preprocessing layer, the third desktop layer and the third status bar layer into a third image; and displaying the third image on the display screen.

[0039] In a possible implementation, the screen of the terminal device may be installed in a horizontal manner. For example, the row scanning direction of the display screen is parallel to the second frame, and the column scanning direction of the display screen is parallel to the first frame; wherein the first frame and the second frame are two adjacent frames on the body of the terminal device, and the length of the first frame is less than the length of the second frame.

[0040] In the above solution, since the direction of the wallpaper resource is consistent with the horizontal screen state, when in the vertical screen state, the wallpaper resource needs to be rotated first, and then the rotated wallpaper is displayed on the screen, so that from the user's visual perspective, the direction of the wallpaper displayed on the small screen and the large screen is consistent, thereby improving the user experience.

[0041] In a possible implementation, obtaining the first wallpaper layer from the second buffer includes: obtaining the first wallpaper layer and layer information of the first wallpaper layer from the second buffer; wherein the layer information of the first wallpaper layer includes a first state bit and a rotation parameter, the first state bit is used to indicate that the first wallpaper layer has not been rotated, and the rotation parameter is used to indicate the rotation direction and rotation angle. Accordingly, rotating the first wallpaper layer by the GPU to obtain the first preprocessing layer includes: rotating the first wallpaper layer by the GPU based on the first state bit and the rotation parameter to obtain the first preprocessing layer.

[0042] In a possible implementation, before obtaining the first wallpaper layer, the method further includes: determining a rotation parameter when a preset condition is met; sending the rotation parameter to a wallpaper application of a terminal device; generating a first status bit through the wallpaper application, and storing the first status bit, the rotation parameter, and the first wallpaper layer in a second buffer. The first status bit is used to indicate that the resource of the first wallpaper layer has not been rotated. For example, the status bar layer and the desktop layer are layers drawn by the application and rotated, and the "rotated" identification status bit is used. For another example, the wallpaper layer and the video layer are layers read out by the video codec and have not been rotated, and the "unrotated" identification status bit is used. The rotation parameter is used to indicate the rotation direction and rotation angle, such as 90° counterclockwise rotation, 90° clockwise rotation, etc. The preset condition includes any one of the following: the display screen switches from a horizontal screen state to a vertical screen state; the terminal device is a folding screen device, and the folding screen device switches from a folded state to an unfolded state.

[0043] In the above solution, the wallpaper application writes the first state bit and the rotation parameter in the buffer area, so that SurfaceFlinger determines whether the wallpaper layer needs to be rotated and how to rotate the wallpaper layer according to the first state bit and the rotation parameter.

[0044] In a possible implementation, the method may further include: in response to a user operation, turning on the horizontal and vertical screen function; wherein the horizontal and vertical screen function allows display in a first direction when the terminal device is in a horizontal state, and display in a second direction when the terminal device is in a vertical state; the first direction differs from the second direction by 90° or 270° clockwise.

[0045] In a possible implementation, when the terminal device is in a horizontal screen state, the angle between the first frame of the terminal device body and the ground plane is greater than the angle between the second frame of the terminal device body and the ground plane; when the terminal device is in a vertical screen state, the angle between the first frame and the ground plane is less than or equal to the angle between the second frame and the ground plane. The length of the first frame is less than the length of the second frame.

[0046] In a possible implementation, the layer information of the wallpaper layer may also include the layer stacking order of the first wallpaper layer. The first preprocessing layer, the first desktop layer, and the first status bar layer are synthesized into a first image, including: according to the layer stacking order of the first wallpaper layer, the layer stacking order of the first desktop layer, and the layer stacking order of the first status bar layer, the first preprocessing layer, the first desktop layer, and the first status bar layer are stacked to obtain the first image. The first preprocessing layer, the second desktop layer, and the second status bar layer are synthesized into a second image, including: according to the layer stacking order of the first wallpaper layer, the layer stacking order of the second desktop layer, and the layer stacking order of the second status bar layer, the first preprocessing layer, the second desktop layer, and the second status bar layer are stacked to obtain the second image.

[0047] In the above scheme, each application writes the layer overlay order in the cache area corresponding to each application, so that SurfaceFlinger can determine the synthesis method according to the overlay order of each layer.

[0048] In a possible implementation, before rotating the first wallpaper layer through the GPU, the method may further include: determining that the first wallpaper layer is the first frame or a change frame of the wallpaper resource. The first frame refers to other wallpaper layers for which wallpaper resources have not been acquired before the first wallpaper layer is acquired; the change frame refers to other wallpaper layers for which wallpaper resources have been acquired before the first wallpaper layer is acquired, and the other wallpaper layers are different from the first wallpaper layer. Taking the first wallpaper layer as layer 1 as an example, the first frame means that before SurfaceFlinger receives the synthesis task of layer 1, other wallpaper layers have not been stored in the cache area corresponding to the wallpaper application, that is, layer 1 is the first synthesis task of the wallpaper application received by SurfaceFlinger; the change frame means that before SurfaceFlinger receives the synthesis task of layer 1, it has received the synthesis task of other layers of the wallpaper application, and the other layers and layer 1 are stored in different cache areas.

[0049] In the above scheme, by judging whether a wallpaper layer is the first frame or the change frame of the wallpaper resource, it can be determined whether the wallpaper layer needs to be preprocessed. It should be understood that when a wallpaper layer is the first frame or the change frame of the wallpaper resource, the subsequent wallpaper layer and the wallpaper layer may be the same layer, so the wallpaper layer can be preprocessed, so that the preprocessed layer can be directly used when there is a synthesis task of the same subsequent wallpaper layer.

[0050] In a possible implementation, before acquiring the first wallpaper layer at the second time, the method may further include: determining the rotation parameter when the preset condition is met; sending the rotation parameter to the wallpaper application, the desktop application, and the system user interface application; generating a first status bit through the wallpaper application, and storing the first status bit, the rotation parameter, and the resources of the first wallpaper layer in the second buffer area; generating a second status bit through the desktop application, storing the second status bit, the rotation parameter, and the resources of the second desktop layer in the third buffer area, the second status bit is used to indicate that the resources of the second desktop layer have been rotated; generating a third status bit through the system user interface application, storing the third status bit, the rotation parameter, and the resources of the second status bar layer in the fourth buffer area, the third status bit is used to indicate that the resources of the second status bar layer have been rotated. Wherein, the preset condition includes any one of the following: the screen of the terminal device switches from the horizontal screen state to the vertical screen state; the terminal device is a folding screen device, and the folding screen device switches from the folded state to the unfolded state.

[0051] In the above scheme, when the screen of the terminal device switches from the horizontal screen state to the vertical screen state, or from the folded state to the unfolded state, the sensor of the terminal device can collect data such as the folding angle and acceleration in real time, and send these data to the hardware configuration module. The screen management service can obtain information such as the screen direction, window direction and folding state from the hardware configuration module, and then the WMS can calculate the rotation parameters according to the installation direction of the screen and the horizontal and vertical screen state of the current screen, and send these rotation parameters to each application. Each application can generate a status bit indicating whether the layer has been rotated according to the generation method of each layer. In this way, SurfaceFlinger can determine whether each layer needs to be rotated based on the status bit.

[0052] In a fifth aspect, the present application provides a device, which includes a unit for executing the method in the first aspect. The device may correspond to executing the layer synthesis method described in the first aspect. For the relevant description of the units in the device, please refer to the description of the first aspect, and for the sake of brevity, it will not be repeated here.

[0053] The method described in the first aspect above can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a processing module or unit, a display module or unit, etc.

[0054] In a sixth aspect, the present application provides a terminal device. For example, the terminal device may be a folding screen device or a large screen device. The terminal device includes a memory and one or more processors. The memory is used to store computer program code, and the computer program code includes computer instructions. When the computer instructions are called by the processor, the terminal device executes the layer synthesis method provided in any one of the first aspects.

[0055] In a seventh aspect, the present application provides a computer-readable storage medium. The computer-readable storage medium includes computer instructions. When the computer instructions are executed on a folding screen device, the folding screen device executes the layer synthesis method provided in the first aspect and any possible implementation thereof.

[0056] In an eighth aspect, the present application provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the layer synthesis method provided in the first aspect and any possible implementation thereof.

[0057] In the ninth aspect, the present application provides a chip system. The chip system includes one or more interface circuits and one or more processors. The interface circuit and the processor are interconnected by lines. The chip system can be applied to a folding screen device including a communication module and a memory. The interface circuit is used to receive a signal from the memory of the folding screen device and send the received signal to the processor, where the signal includes a computer instruction stored in the memory. When the processor calls the computer instruction, the folding screen device can execute the layer synthesis method provided in any one of the first aspects.

[0058] It can be understood that the beneficial effects that can be achieved by the fifth aspect apparatus, the sixth aspect terminal device, the seventh aspect computer-readable storage medium, the eighth aspect computer program product, and the ninth aspect chip system provided above can be referred to as the beneficial effects in the first aspect and any possible implementation method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A schematic diagram of synthesizing multiple layers into one frame of image provided in an embodiment of the present application;

[0060] Figure 2A-2C A schematic diagram of an inward folding screen mobile phone provided in an embodiment of the present application;

[0061] Figure 3A-3C A schematic diagram of an outward folding screen mobile phone provided in an embodiment of the present application;

[0062] Figure 4A-4G Schematic diagram of small screen, large screen, wallpaper, GPU synthesis and HWC synthesis provided in the embodiment of the present application;

[0063] Figure 5A-Figure 5B A schematic diagram of synthesizing multiple layers into one frame of image and displaying it on a large screen provided in an embodiment of the present application;

[0064] Figure 6 A schematic diagram of sending multiple frames of images to a large screen for display provided in an embodiment of the present application;

[0065] Figure 7 A schematic diagram of the architecture of a folding screen device provided in an embodiment of the present application;

[0066] Figure 8 A schematic diagram of a process flow of a layer synthesis method based on intelligent scene recognition provided in an embodiment of the present application;

[0067] Fig. 9 A schematic diagram of using a GPU to render a pre-processed layer provided in an embodiment of the present application;

[0068] Fig.10 A schematic diagram of synthesizing multiple layers into one frame of image based on preprocessing layers provided in an embodiment of the present application;

[0069] Fig.11A A schematic diagram of synthesizing a video layer and a wallpaper layer into a multi-frame image based on a preprocessing layer provided in an embodiment of the present application;

[0070] Fig. 11B A schematic diagram of synthesizing a status bar layer and a wallpaper layer into a multi-frame image based on a preprocessing layer provided in an embodiment of the present application;

[0071] Fig.12 A schematic diagram of synthesizing the Nth frame image and the N+1th frame image provided in an embodiment of the present application;

[0072] Fig.13 A schematic diagram of a process flow of another layer synthesis method based on intelligent scene recognition provided in an embodiment of the present application;

[0073] Fig.14A A schematic diagram of a scenario for applying a preprocessing layer provided in an embodiment of the present application;

[0074] Fig. 14B A schematic diagram of another scenario of applying a preprocessing layer provided in an embodiment of the present application;

[0075] Fig. 14C A schematic diagram of another scenario of applying a preprocessing layer provided in an embodiment of the present application;

[0076] Fig.14D A schematic diagram of another scenario of applying a preprocessing layer provided in an embodiment of the present application;

[0077] Fig.15A schematic diagram of a process flow of another layer synthesis method based on intelligent scene recognition provided in an embodiment of the present application;

[0078] Fig.16 A schematic diagram of a process flow of another layer synthesis method based on intelligent scene recognition provided in an embodiment of the present application;

[0079] Fig.17 A schematic diagram of another scenario of applying a preprocessing layer provided in an embodiment of the present application;

[0080] Fig.18 A schematic diagram of the hardware structure of the folding screen device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0082] In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B. In the description of this application, "and / or" is only a way to describe the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0083] The terms "first" and "second" in the specification and claims of this application are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects. For example, the first frame and the first frame are used to distinguish different video frames, rather than to describe a specific order of video frames. In the embodiments of this application, "multiple" refers to two or more.

[0084] Some terms used in the embodiments of the present application are explained below.

[0085] The images displayed on the screen of the terminal device are usually composed of multiple layers. For example, Figure 1A schematic diagram of synthesizing a wallpaper layer, a desktop layer, a video layer, and a status bar layer into a frame of image is shown. Typically, a terminal device uses a system service called SurfaceFlinger to manage the rendering and synthesis process of the layers. When each application needs to display a window, each application sends the relevant information of the window to SurfaceFlinger. After receiving relevant information from multiple applications, SurfaceFlinger uses GPU and / or HWC to complete rendering, synthesis and other operations, and finally obtains the image to be displayed, and sends this frame of image to the screen for display.

[0086] The above-mentioned GPU synthesis and HWC synthesis are two different image synthesis technologies.

[0087] The advantages and disadvantages of these two image synthesis technologies are illustrated below.

[0088] 1) HWC synthesis

[0089] HWC compositing is a hardware-accelerated compositing technology that uses hardware-level image compositing capabilities to improve performance and efficiency. In HWC compositing, the image compositing task is performed by the hardware module, which may involve a small amount of central processing unit (CPU) compositing operations, without requiring the GPU to perform software-level compositing operations.

[0090] HWC synthesis has the following advantages: 1. Due to the use of hardware acceleration, HWC synthesis can provide higher frame rates and smoother animation effects, that is, high performance; 2. By directly utilizing hardware resources for synthesis, HWC synthesis can reduce the power consumption of the CPU and GPU, that is, low power consumption; 3. HWC synthesis can display the graphics content of the application on the screen faster, providing faster user interface response time, that is, real-time response.

[0091] However, the HWC hardware module is a hardware module in the system-on-a-chip (SOC). The HWC hardware modules provided by different manufacturers have different HWC synthesis specifications. At present, the HWC hardware module has conditional restrictions on images. For example, some HWC hardware modules have restrictions on the height of the image when rotating the image, some HWC hardware modules do not support output rounded corners, some HWC hardware modules have restrictions on the number of layers, conversion formats, layer sizes and zoom ratios during synthesis, and some HWC hardware modules do not support window blur, etc. When the synthesis requirements of the layers exceed the capabilities of the HWC hardware module, HWC synthesis can no longer be used.

[0092] 2) GPU compositing

[0093] GPU compositing is a software-based image compositing technology, so GPU compositing is also called software compositing. GPU compositing uses the GPU to perform graphics compositing operations. In GPU compositing, the graphics operations of the application are sent by the CPU to the GPU, and then the GPU performs the compositing operations at the software level, and then passes the composite image data to the HWC for hardware compositing, and the HWC sends it to the screen for display.

[0094] GPU synthesis is flexible. Because it is based on software implementation, GPU synthesis can support more complex graphics synthesis operations, such as rotation, special effects, filters, transformations, and rounded corners. When the synthesis requirements of the layer do not exceed the capabilities of the HWC hardware module, HWC synthesis can be used; when the synthesis requirements of the layer exceed the capabilities of the HWC hardware module, GPU synthesis can be used. However, GPU synthesis has the disadvantage of high power consumption, and the performance of GPU synthesis is lower than that of HWC synthesis.

[0095] At present, most terminal devices are equipped with high-definition dynamic wallpapers. The resolution of high-definition dynamic wallpapers is relatively large. If the screen is installed in a horizontal manner, the high-definition dynamic wallpaper needs to be rotated when the screen is in a vertical state. When the height of the high-definition dynamic wallpaper before rotation exceeds the maximum rotation height supported by the HWC hardware module (for example, 2176 pixels), GPU synthesis can be selected. For horizontal installation, you can refer to the following embodiment for Figure 4A-4G Related description.

[0096] The following takes a folding screen device as an example to introduce the principle of rotating wallpaper using these two synthesis technologies.

[0097] The above-mentioned folding screen device can be an inner folding screen device or an outer folding screen device.

[0098] Among them, both the inner folding screen device and the outer folding screen device can be folded into three folding states: folded state, hovering state, and unfolded state. Specifically, when the folding screen device is in the folded state, the folding angle between the main screen and the back screen is less than or equal to α, and the small screen faces the user; when the folding screen device is in the unfolded state, the folding angle between the main screen and the back screen is greater than or equal to β, and the large screen faces the user; when the folding screen device is in the hovering state, the folding angle between the main screen and the back screen is greater than α and less than β. For example, α is equal to or close to the folding angle of 0°, and β is equal to or close to the folding angle of 180°.

[0099] For example, FIG. 2A to FIG. 2C A schematic diagram of an inward-folding screen mobile phone is shown.

[0100] The inner folding screen mobile phone is provided with an inner screen (also called a large screen) and an outer screen (also called a small screen), and the inner screen can be bent into at least two display areas. Figure 2AAs shown, when the inner folding screen mobile phone is in the folded state, the inner surfaces of the two folding bodies are opposite to each other, and the inner screen is in a hidden state. Figure 2B As shown, when the inner folding screen mobile phone is in a hovering state, the folding angle between the two display areas of the inner screen is greater than 0° and less than 180°. Figure 2C As shown in the figure, when the inner folding screen mobile phone is in the unfolded state, the folding angle between the two display areas of the inner screen is equal to or close to 180°, so that the two display areas of the inner screen are located on the same surface. Usually, when the inner folding screen mobile phone is in the folded state, the outer screen displays the wallpaper; after the inner folding screen mobile phone switches from the folded state to the unfolded state, the inner screen displays the wallpaper.

[0101] For example, FIG. 3A to FIG. 3C A schematic diagram of an outward folding screen mobile phone is shown.

[0102] The foldable phone has a screen that can be folded into at least two display areas. The larger of the two display areas is called the main screen, and the smaller display area is called the back screen. Usually, the main screen is also called the small screen, and the main screen and the back screen are collectively called the large screen. Figure 3A As shown in the figure, the back screen is folded to be flush with the surface where the rear camera is located, with the main screen facing the user and the back screen facing away from the user. Figure 3B As shown, when the external folding screen mobile phone is in a hovering state, the folding angle between the main screen and the back screen is greater than 0° and less than 180°. Figure 3C As shown in the figure, when the external folding screen mobile phone is in the unfolded state, the folding angle between the two display areas of the inner screen is equal to or close to 180°. Usually, when the external folding screen mobile phone is in the folded state, the main screen displays the wallpaper; after the external folding screen mobile phone switches from the folded state to the unfolded state, the main screen and the back screen are used to display the wallpaper at the same time.

[0103] According to the description of the above embodiments, although the physical forms of the inward-folding screen mobile phone and the outward-folding screen mobile phone are different, they both have the following common characteristics: when in a folded state, the small screen is used to display a smaller wallpaper; when in an unfolded state, the large screen is used to display a larger wallpaper.

[0104] Usually, the resolution and installation direction of the small screen and the large screen may be different. For example, the resolution of the small screen is 2300*1176 pixels, the resolution of the large screen is 2300*2100 pixels, the small screen is installed vertically, and the large screen is installed horizontally, that is, the installation direction of the small screen differs from that of the large screen by 90° or 270°. When the small screen and the large screen use the same dynamic wallpaper video resource, in order to make the wallpaper displayed on the small screen in portrait mode and the large screen in portrait mode consistent in the user's visual direction, the wallpaper application will notify SurfaceFlinger to determine whether to rotate the wallpaper according to the folding form. For example, if the currently lit screen is a small screen in portrait mode, there is no need to rotate the wallpaper, and the low-power HWC synthesis is selected; if the currently lit screen is a large screen in portrait mode, the wallpaper needs to be rotated, and the wallpaper height before rotation (for example, 2300 pixels) exceeds the maximum rotation height supported by the HWC hardware module (for example, 2176 pixels), and the GPU synthesis that supports more complex synthesis operations is selected. After rotating, scaling or cropping the wallpaper layer based on GPU synthesis and / or HWC synthesis, HWC uses hardware synthesis to synthesize the wallpaper layer with other layers into one frame of image and sends it to the small or large screen for display.

[0105] It should be noted that if the small screen is on, the small screen can be divided into two states: the small screen is in portrait state, and the angle between the long border of the small screen and the ground plane is greater than or equal to the angle between the short border of the small screen and the ground plane; or, the small screen is in landscape state, and the angle between the long border of the small screen and the ground plane is smaller than the angle between the short border of the small screen and the ground plane.

[0106] If the large screen is on, it can be divided into two states: the large screen is in the portrait state, the angle between the long frame of the large screen (parallel to the fold direction) and the ground plane is greater than or equal to the angle between the short frame of the large screen (perpendicular to the fold direction) and the ground plane, and the large screen is displayed in the first direction at this moment; or the large screen is in the landscape state, the angle between the long frame of the large screen (parallel to the fold direction) and the ground plane is less than the angle between the short frame of the large screen (perpendicular to the fold direction) and the ground plane, and the large screen is displayed in the second direction at this moment. The first direction and the second direction differ by 90°. In this way, whether the large screen is in the portrait state or the landscape state, from the user's perspective, the screen always maintains a positive display without rotation.

[0107] Below FIG. 2A to FIG. 2C Take the folding screen mobile phone shown as an example, combined with FIG. 4A to FIG. 4G , the relationship between small screen, large screen, wallpaper, GPU synthesis and HWC synthesis is illustrated with examples.

[0108] For example, the small and large screens of the foldable phone are of organic light-emitting diode (OLED) type. Regardless of the small or large screen, the most basic unit element that makes up the image is the pixel. The resolution of the screen is usually expressed as "row pixel value * column pixel value". For example, the resolution of a screen is m (height) * n (width), which means that the screen has m rows and n columns. It can be understood that there are m pixels in each column and n pixels in each row.

[0109] like Figure 4A As shown, the resolution of the small screen is 2300 (height) * 1176 (width) pixels, which means the small screen has 2300 rows and 1176 columns. A two-dimensional coordinate system XY is established with row 1 * column 1 as the origin. Figure 4A In the vertical installation mode shown, when the small screen is installed on the mobile phone body, the positive direction of the X axis of the two-dimensional coordinate system is along the first side of the mobile phone body to the right, and the positive direction of the Y axis of the two-dimensional coordinate system is along the second side of the mobile phone body downward, wherein the first side is perpendicular to the second side. The small screen transmits the pixel data of a frame of image to each pixel in the following scanning mode: taking row 1*column 1 as the starting point, scanning each pixel of the first row in the positive direction of the X axis in sequence until row 1*column 1176; after the transmission of the first row of pixel data is completed, taking row 2*column 1 as the starting point, scanning each pixel of the second row in the positive direction of the X axis in sequence until row 2*column 1176; ...; and so on, taking row 2300*column 1 as the starting point, scanning each pixel of the 2300th row in the positive direction of the X axis in sequence until row 2300*column 1176. It can be understood that when the installation direction of the small screen on the mobile phone body is changed (for example, the small screen is rotated 180° relative to the mobile phone body), the starting point and scanning direction of the small screen scanning will change with the mobile phone body as the reference.

[0110] like Figure 4B As shown in the figure, the resolution of the large screen is 2100 (height) * 2300 (width) pixels, which means that the large screen has 2100 rows and 2300 columns. The two-dimensional coordinate system XY is established with row 1 * column 1 as the origin. Figure 4BIn the horizontal installation method shown, when the large screen is installed on the mobile phone body, the positive direction of the X axis of the two-dimensional coordinate system is downward along the third side (i.e., the first side) of the mobile phone body, and the positive direction of the Y axis of the two-dimensional coordinate system is leftward along the fourth side (the fourth side is parallel to the second side) of the mobile phone body, wherein the third side is perpendicular to the fourth side. The large screen transmits pixel data of a frame of image to each pixel in the following scanning method: taking row 1*column 1 as the starting point, scanning each pixel of the first row in the positive direction of the X axis in sequence until row 1*column 2300; after the transmission of the first row of pixel data is completed, taking row 2*column 1 as the starting point, scanning each pixel of the second row in the positive direction of the X axis in sequence until row 2*column 2300; ...; and so on, taking row 2100*column 1 as the starting point, scanning each pixel of the 2100th row in the positive direction of the X axis in sequence until row 2100*column 2300. It can be understood that when the installation direction of the large screen on the mobile phone body is changed, taking the mobile phone body as a reference, the starting point and scanning direction of the large screen scanning will change.

[0111] According to the above two paragraphs, if the mobile phone body is used as a reference, then relative to Figure 4A The coordinate system of the small screen shown is Figure 4B The coordinate system of the large screen shown is rotated 90° clockwise, that is, the installation directions of the small screen and the large screen differ by 90°.

[0112] The orientation of wallpaper resources stored in the phone is fixed. Usually, the orientation of the wallpaper resources is consistent with the orientation of the small screen on the phone body, and is 90° or 270° clockwise from the orientation of the large screen on the phone body. Figure 4A Install the small screen on the mobile phone body in the vertical installation method shown in the figure. Figure 4B The horizontal installation method shown is to install a large screen on the mobile phone body. Figure 4C As shown, the direction of the wallpaper resource is consistent with the installation direction of the small screen on the mobile phone body, and the direction of the wallpaper resource is 90° clockwise from the installation direction of the large screen on the mobile phone body.

[0113] When the user holds the phone with both hands, the folding form of the phone can be changed by changing the folding angle, such as switching from the folded state to the unfolded state, or from the unfolded state to the folded state. If switching from the folded state to the unfolded state, the wallpaper is displayed on the small screen first, and then on the large screen. If switching from the unfolded state to the folded state, the wallpaper is displayed on the large screen first, and then on the small screen. It should be understood that during the switching process between the two folding forms, the user may prefer to see the wallpaper direction of the two screens remain consistent, such as Figure 4CAs shown, when the small screen and the large screen are in the vertical screen state, the phoenix displayed on the small screen and the phoenix displayed on the large screen both fly toward the upper right. Combined with the description of the above embodiment, since the direction of the wallpaper resource is consistent with the installation direction of the small screen on the mobile phone body, and inconsistent with the installation direction of the large screen on the mobile phone body, it is necessary to rotate the wallpaper resource first, and then send the rotated wallpaper to the large screen for display, so that from the user's visual perspective, the direction of the wallpaper displayed on the small screen and the large screen is consistent, thereby improving the user experience.

[0114] As an example, in Figure 4C On the basis of Figure 4D The following diagram shows the display being sent to the small screen when the phone is in a folded state and the small screen is in a vertical state. The orientation of the wallpaper resource is consistent with the orientation of the small screen on the phone body. There is no need to rotate the wallpaper and you can use HWC to synthesize it. The resolution of the wallpaper resource is 2300 (height) * 2100 (width) pixels, and the resolution of the small screen is 2300 (height) * 1176 (width) pixels. HWC can be used as follows: Figure 4D The dotted frame shown cuts out a partial image of 2300 (height)*1176 (width) from the wallpaper resource, and displays the partial image on the small screen.

[0115] As another example, in Figure 4C On the basis of Figure 4E The following diagram shows the display being sent to the small screen when the phone is in folded state and the small screen is in landscape mode. The orientation of the wallpaper resource is consistent with the orientation of the small screen on the phone body. There is no need to rotate the wallpaper. You can use HWC to synthesize it. The resolution of the wallpaper resource is 2300 (height) * 2100 (width) pixels, and the resolution of the small screen is 2300 (height) * 1176 (width) pixels. HWC can be used as follows: Figure 4E The dashed box shown cuts out a partial image of 2300 (height)*1176 (width) from the wallpaper resource, and sends the partial image to the small screen for display. It should be understood that no matter whether the small screen is in a vertical or horizontal state, the wallpaper is not rotated.

[0116] It should be noted that Figure 4D and Figure 4E The cropping operation of wallpaper resources is used as an example for illustration, which does not limit the present application. In actual implementation, the wallpaper resources can also be scaled according to a certain scaling ratio according to the resolution of the wallpaper and the small screen, or the cropping operation and the scaling operation can be performed simultaneously to obtain a wallpaper adapted to the resolution of the small screen.

[0117] As another example, in Figure 4C On the basis of Figure 4FA schematic diagram showing the display on the large screen when the mobile phone is in the unfolded state and the large screen is in the portrait state. The direction of the wallpaper resource is 90° clockwise from the installation direction of the large screen on the mobile phone body. The wallpaper needs to be rotated, and the GPU can be used for synthesis. The resolution of the wallpaper resource is 2300 (height) * 2100 (width) pixels, and the resolution of the large screen is 2100 (height) * 2300 (width) pixels. Since the height of the wallpaper resource is 2300 pixels, which exceeds the maximum rotation height supported by the HWC hardware module (for example, 2176 pixels), the GPU can rotate the wallpaper resource 90° counterclockwise, and then display the rotated wallpaper resource on the large screen.

[0118] It should be noted that the "resolution of the large screen is 2100 (height) * 2300 (width) pixels" in the embodiment of the present application is described with reference to the scanning direction of the large screen, which does not limit the present application. Figure 4B As shown, after the large screen is installed on the mobile phone body, the relative position relationship between the large screen and the mobile phone body is fixed. If the mobile phone body is used as a reference, 2300 pixels can be used as the height of the large screen and 2100 pixels can be used as the width of the small screen.

[0119] In the above embodiment, since the resolution of the rotated wallpaper is 2100 (height) * 2300 (width) pixels and the resolution of the large screen is 2100 (height) * 2300 (width) pixels, that is, the resolution of the rotated wallpaper is the same as the resolution of the large screen, there is no need to crop and / or scale the wallpaper. It should be understood that when the resolution of the rotated wallpaper is different from the resolution of the large screen, cropping and / or scaling operations can also be performed according to usage requirements.

[0120] in addition, Figure 4B , Figure 4C and Figure 4F The example is that the direction of the wallpaper resource is 90° clockwise from the direction of the large screen installed on the mobile phone body, which does not limit the present application. Figure 4G As shown in FIG. 1 , the direction of the wallpaper resource differs from the installation direction of the large screen on the mobile phone body by 270° clockwise (i.e., 90° counterclockwise), and the wallpaper height before rotation (e.g., 2300 pixels) exceeds the maximum rotation height supported by the HWC hardware module (e.g., 2176 pixels). The GPU can rotate the wallpaper resource 90° clockwise and then send the rotated wallpaper resource to the large screen for display. It should be noted that the following embodiments are all based on the following example. Figure 4B , Figure 4C and Figure 4F The large screen installation direction shown in the figure is used as an example for explanation. In actual implementation, you can also use Figure 4G The large screen installation direction is shown.

[0121] The above embodiments are combined with FIG. 4A to FIG. 4G, the relationship between small screen, large screen, wallpaper, GPU synthesis and HWC synthesis is illustrated. In actual implementation, the mobile phone screen will not only display the wallpaper layer, but also other layers. For example, the mobile phone screen will superimpose the desktop layer, video layer and status bar layer on the wallpaper layer in sequence, and synthesize multiple layers into one frame of image. It should be understood that in addition to these four types of layers, there may be other types of layers. The number and type of layers can be adjusted according to usage requirements, and the embodiments of the present application are not specifically limited.

[0122] The following is an example of how to combine multiple layers into one image when in the expanded state.

[0123] Figure 5A and Figure 5B A schematic diagram is shown of synthesizing multiple layers into one frame of image and sending it to a large screen for display.

[0124] like Figure 5A As shown, when the phone is in the unfolded state, SurfaceFlinger receives the synthesis tasks of the wallpaper layer, desktop layer, video layer and status bar layer in sequence. Among them, the synthesis task of the wallpaper layer specifies the overlay order 1, which means it is at the bottom layer; the synthesis task of the desktop layer specifies the overlay order 2, which means it is located in the upper layer of the bottom layer; the synthesis task of the video layer specifies the overlay order 3, which means it is located in the upper two layers of the bottom layer; the synthesis task of the status bar layer specifies the overlay order 4, which means it is located in the upper layer. That is, the overlay order of the layers from bottom to top is: wallpaper layer, desktop layer, video layer and status bar layer.

[0125] The desktop layer is the layer drawn by the desktop application, and the status bar layer is the layer drawn by the system user interface (UI). During the drawing process, the desktop application and the system UI have rotated the desktop layer and the status bar layer 90° counterclockwise, and no further rotation is required. In principle, SurfaceFlinger can give priority to assigning the synthesis tasks of the desktop layer and the status bar layer to HWC. However, since the wallpaper layer and the video layer are obtained by decoding the wallpaper resources and video resources through the video codec, and the video codec does not have the ability to rotate, the wallpaper layer and the video layer are layers that have not been rotated. In addition, the video layer needs to be rounded, and HWC synthesis does not support rounded corners, so SurfaceFlinger can assign the synthesis tasks of the wallpaper layer and the video layer to the GPU. In addition, when at least one layer is located between layer a and layer b, and layer a and layer b can only be synthesized by GPU, regardless of whether the at least one layer is suitable for HWC synthesis, SurfaceFlinger will assign the synthesis tasks of at least one layer, layer a and layer b to the GPU. Since the desktop layer is between the wallpaper layer and the video layer, SurfaceFlinger finally assigns the synthesis task of the desktop layer to the GPU instead of the HWC. Among them, the wallpaper resource can be a video resource composed of multiple pictures (i.e. dynamic wallpaper) or a single picture (i.e. static wallpaper).

[0126] The GPU rotates the wallpaper layer 90° counterclockwise, rotates the video layer 90° counterclockwise, and rounds the corners of the video layer. The GPU sequentially overlays the desktop layer and the rotated video layer on the rotated wallpaper layer to obtain a composite layer. Then, the GPU sends the composite layer to the HWC. The HWC can overlay the status bar layer on the composite layer, generate a frame of image to be displayed, and send the image to be displayed to the large screen. When the large screen is in portrait mode, the phoenix in the wallpaper displayed on the large screen flies toward the upper right (such as Figure 5B ), and the flying direction of the phoenix in the wallpaper displayed on the small screen (as shown Figure 4D In this way, although the installation directions of the large screen and the small screen are different, the wallpaper directions of the large screen and the small screen that the user finally sees are consistent, which is more in line with user habits.

[0127] The above embodiments are combined with Figure 5A and Figure 5B, an example is given of how to synthesize multiple layers into one frame of image when the folding screen device is in the unfolded state. In actual implementation, the large screen will use periodic tearing effect (TE) signals to continuously refresh the interface, and the TE signal can represent the refresh frequency of the large screen. After the large screen is powered on and initialized, the large screen begins to periodically send TE signals to SurfaceFlinger. The sending frequency of the TE signal determines the refresh frequency of the large screen. For example, when the refresh frame rate of the large screen is 120Hz, the sending period of the TE signal is 8.33 milliseconds. For another example, when the refresh frame rate of the large screen is 30Hz, the sending period of the TE signal is 33.33 milliseconds. After each TE signal is sent by the large screen, SurfaceFlinger performs a refresh action in response to the TE signal. Based on the TE signal, SurfaceFlinger also maintains the Vsync-SF (vertical synchronization (vertical synchronization, Vsync)-SurfaceFlinger) signal. Each time a Vsync-SF signal is obtained, a layer synthesis process is performed. Take obtaining a Vsync-SF signal as an example. When a Vsync-SF signal is obtained, SurfaceFlinger performs layer synthesis and rendering through the GPU or HWC to generate a new image to be displayed, and then sends it to the large screen for display.

[0128] It should be noted that this application involves two types of Vsync signals: Vsync-SF signal and Vsync-APP signal. Among them, Vsync-SF signal is used to control the synthesis speed of the layer, and Vsync-APP signal is used to control the speed of APP drawing layer. APP can request Vsync-APP signal from SurfaceFlinger according to demand.

[0129] Below is a large screen Figure 4F In the vertical screen state shown, SurfaceFlinger receives at least the synthesis task of the wallpaper layer and the video layer, for example, combined with Figure 6 , an example is given to illustrate the way of constantly refreshing the interface of the large screen. It should be understood that in actual implementation, in addition to the wallpaper layer and the video layer, it may also include a desktop layer, a status bar layer and / or a web page layer, etc., which is not limited in the embodiment of the present application.

[0130] For example, Figure 6 A schematic diagram is shown of synthesizing a wallpaper layer and a video layer into a multi-frame image and displaying the image on a large screen.

[0131] The frequency of window updates of each application may be different, which makes the picture sending cycle of each application different. For example, the wallpaper application may not refresh the wallpaper for tens of seconds, minutes or even hours, the video application window may refresh 40 to 60 frames in one second, the news window may refresh 120 frames in one second, and the game window may refresh 160 frames in one second. When the picture sending cycle of each application is different, SurfaceFlinger will calculate the refresh rate based on the minimum picture sending cycle, that is, the period of the Vsync-SF signal, to avoid frame loss caused by using a larger picture sending cycle.

[0132] For example, a video application can send a task to SurfaceFlinger every 33.33 milliseconds. Take task A1 as an example. The video application can notify the video codec to decode the video resource. The video resource stores the decoded resource a1 in the buffer and sends the buffer to SurfaceFlinger. SurfaceFlinger obtains resource a1 from the buffer. Although the height of resource a1 does not exceed the maximum rotation height supported by the HWC hardware module (for example, 2176 pixels), since the HWC hardware module does not support rounded corners, SurfaceFlinger hands task A1 to the GPU, which rotates and rounds the corners of resource a1 to obtain layer A1, and then sends layer A1 to the HWC.

[0133] The wallpaper application can send a new synthesis task to SurfaceFlinger every time the wallpaper is updated. For example, when the user changes the folding angle of the folding screen, the wallpaper update will be triggered, or when the user reselects the wallpaper through the settings application, the wallpaper update will be triggered. It can be seen that the wallpaper update is closely related to the user operation. If the wallpaper is not updated for a few seconds, minutes or even hours, the wallpaper application will not issue new tasks during this period. Take tasks B1 and B2 as examples. The wallpaper application can notify the video codec to decode the wallpaper resource; the video resource stores the decoded resource b1 in the buffer and sends the buffer to SurfaceFlinger; SurfaceFlinger obtains resource b1 from the buffer. Since the height of resource b1 exceeds the maximum rotation height supported by the HWC hardware module (such as 2176 pixels), SurfaceFlinger hands task B1 to the GPU, and the GPU rotates resource b1 to obtain layer B1. Before the next task B2 of task B1 arrives, in order to keep the wallpaper layer and the video layer at the same refresh rate, the GPU continuously rotates resource b1 to obtain layer B1, and sends the layer B1 obtained after each rotation to HWC.

[0134] HWC superimposes layer A1 on the first layer B1 to form frame 1, superimposes layer A2 on the second layer B1 to form frame 2, and superimposes layer A3 on the third layer B1 to form frame 3... In response to the received TE signal, HWC sends these images to the large screen in turn.

[0135] Combined with the description of the above embodiment, when the large screen is in portrait mode, if multiple frames of images are synthesized based on the same wallpaper resource, the GPU may need to rotate the same wallpaper resource multiple times. However, GPU synthesis has the disadvantage of high power consumption. Frequent rotation of wallpaper layers leads to high power consumption, high load, poor performance and severe heat generation.

[0136] In view of the above problems, an embodiment of the present application provides a layer synthesis method based on intelligent scene recognition. The method can be applied to a terminal device that is provided with at least one large screen, and the large screen is installed horizontally. In this method, when the large screen is in vertical screen mode, each layer to be synthesized is identified to determine whether the layer needs GPU preprocessing, such as wallpaper layer rotation. If GPU preprocessing is required, the preprocessed layer is rendered using the GPU, and the preprocessed layer is placed in a cache area. In this way, when there is a subsequent task of processing the same layer, the preprocessed layer can be directly obtained from the cache area, and the preprocessed layer can be sent to the HWC for hardware synthesis, thereby reducing the overall power consumption and load of the terminal device and improving the overall performance of the terminal device.

[0137] The above-mentioned terminal devices are not limited to foldable screen devices, but can also be large-screen devices such as tablet computers (PADs).

[0138] In order to better understand the layer synthesis method based on intelligent scene recognition provided by the embodiment of the present application, the architecture of the folding screen device is introduced below taking the folding screen device as an example.

[0139] For example, Figure 7 A schematic diagram of the architecture of a folding screen device provided in an embodiment of the present application is shown.

[0140] like Figure 7 As shown, the folding screen device can adopt a layered architecture to divide the software into several layers, each with clear roles and division of labor. The layers communicate through software interfaces. In some embodiments, the software layers of the software structure are divided from top to bottom into: application (APP) layer, application framework (FW) layer, system library (FWK LIB), hardware abstraction layer (HAL layer) and kernel layer. The above software architecture runs on the hardware layer, which may include screens, HWC hardware modules, GPUs, sensors, etc.

[0141] The application layer, also known as the application layer, can include a series of application packages, such as wallpaper applications, desktop applications, system UI, video applications, and browsers. Among them, the system UI is a native application of the Android system. When these application packages are run, they can access the various service modules provided by the application framework layer through the application programming interface (API) and execute corresponding intelligent services.

[0142] The application framework layer provides API and programming framework for applications. The application framework layer includes some predefined functions. Figure 7 As shown, the application framework layer may include an activity manager service (AMS), a window manager service (WMS), SurfaceFlinger, a screen management service, and a video codec (such as MediaCodec). Among them, AMS manages the life cycle of each application. WMS manages all windows in the system. The screen management service is responsible for calculating the folding angle of the folding screen device, the horizontal and vertical screen status of the folding screen device, etc. based on the data collected by the sensor using a preset algorithm. SurfaceFlinger is responsible for the synthesis and display of layer data. For example, SurfaceFlinger can provide layer information to the hardware composer (HWC) HAL, and the HWC HAL can decide whether to use the underlying hardware of the HWC or GPU synthesis based on the hardware performance. MediaCodec is an audio and video codec tool provided in the Android system. It mainly completes the encapsulation of the upper-level interface and provides it to developers for use. The encoding and decoding function is actually completed in the local layer service.

[0143] The system library can include multiple functional modules, such as surface manager, media library, two-dimensional (2D) graphics engine (e.g. SGL), three-dimensional (3D) graphics processing library (e.g. OpenGLES), etc. Among them, the surface manager is used to manage the display subsystem and provide fusion of 2D layers and 3D layers for multiple applications. The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 2D graphics engine is a drawing engine for 2D drawing. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis and layer processing, etc.

[0144] In the system library, Android Runtime includes the core library and the virtual machine. Android Runtime is responsible for the scheduling and management of the Android system. The core library consists of two parts: one is the function that the Java language needs to call, and the other is the core library of Android. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection and other functions.

[0145] The hardware abstraction layer has a standard interface implemented by hardware vendors. For example, the hardware abstraction layer may include HWC HAL and audio HAL. HWC HAL is responsible for determining whether the synthesis requirements of the layer to be synthesized exceed the capabilities of the HWC hardware module. It should be understood that the capabilities of HWC hardware modules manufactured by different manufacturers may vary.

[0146] The kernel layer is the layer between hardware and software and belongs to the bottom layer of the Android system. The kernel layer can contain various driver interfaces, such as display driver, audio driver, and sensor driver.

[0147] It should be noted that although the embodiments of the present application are described using the Android system as an example, its basic principles are also applicable to folding screen devices based on operating systems such as iOS or Windows.

[0148] Based on the above Figure 7 The system architecture of the folding screen device is shown below. The specific interaction process between the various functional structures of the folding screen device in the process of implementing the layer synthesis method of the present application is explained in detail in combination with the interaction process diagram.

[0149] Figure 8 1 is a flow chart of a layer synthesis method based on intelligent scene recognition provided by an embodiment of the present application. The method can be applied to a terminal device that is provided with at least one large screen, and the large screen is installed in a horizontal manner. When the large screen is in a vertical state, such as Figure 8 As shown, the method includes the following S1 to S5.

[0150] S1, SurfaceFlinger obtains layer 1 to be synthesized.

[0151] Among them, layer 1 is a layer corresponding to the window of any application, such as a wallpaper layer, a desktop layer, a status bar layer or a video layer.

[0152] In some embodiments, the metadata / resources of layer 1 can be stored in a cache. The metadata / resources of different layers are stored in different caches. For example, a desktop application can draw a desktop layer through a render thread and store the desktop layer in one cache; a system UI can draw a status bar layer through a render thread and store the status bar layer in another cache. In an embodiment of the present application, the cache used to store the metadata / resources of a layer can be an anonymous shared memory.

[0153] In one case, layer 1 is a desktop layer or a status bar layer, and the synthesis task of these layers is sent to SurfaceFlinger by the Vsync-APP signal.

[0154] Take the desktop layer as an example. The desktop application can notify SurfaceFlinger through WMS to create a desktop window, which is used to display the desktop. When the desktop changes, such as the position of the icon changes or the user clicks on an icon, the desktop application can execute the setView() method to add a view. In the process of the desktop application executing setView(), the requestLayout() method is first executed to re-layout, and then the addWindow() method is executed to add a window. The requestLayout() method is used to trigger the redrawing of the view, and the addWindow() method is used to add a window to the view. Among them, when executing the requestLayout() method, the desktop application will interact with SurfaceFlinger and request registration. After the registration is successful, SurfaceFlinger returns the Vsync-APP signal to the desktop application. Then, the desktop application draws the desktop layer through the render thread, stores the desktop layer in a buffer area, and calls a function to notify SurfaceFlinger of this buffer area. This buffer area contains the layer information of layer 1.

[0155] In another case, layer 1 is a wallpaper layer or a video layer, and these layer synthesis tasks are not sent to SurfaceFlinger based on the Vsync-APP signal.

[0156] Take the video layer as an example. The video application can notify the video codec to decode the video resource. The video resource stores the decoded resource in a buffer and calls a function to send the buffer to SurfaceFlinger. This buffer contains the layer information of layer 1.

[0157] Exemplarily, the above-mentioned layer information may include layer name, status bit, layer stacking order, and / or rotation parameters, etc.

[0158] The layer name can be used to indicate the name of layer 1, such as status bar layer, desktop layer, video layer, browser layer, wallpaper layer, etc.

[0159] The status bit is used to indicate whether layer 1 has been rotated. For example, the status bar layer and the desktop layer are layers drawn by the application and rotated, and the "rotated" status bit is used. For another example, the wallpaper layer and the video layer are layers read out by the video codec and have not been rotated, and the "not rotated" status bit is used.

[0160] The layer stacking order is used to indicate the stacking order of layer 1 in multiple layers. For example, the wallpaper layer is at the bottom layer of multiple layers, and the status bar layer is at the top layer of multiple layers.

[0161] The rotation parameter is used to indicate the rotation direction and angle of layer 1, such as 90° counterclockwise and 90° clockwise.

[0162] S2, SurfaceFlinger determines whether to preprocess layer 1.

[0163] As an example, SurfaceFlinger may include a layer identification module. The layer identification module may determine whether layer 1 needs to be preprocessed based on the state bit and cache area of ​​layer 1. The state bit is used to indicate whether application 1 has performed a rotation operation on layer 1. The cache area is used to store metadata / resources of layer 1.

[0164] Since different frames are stored in different buffers, each time an application sends a new buffer to SurfaceFlinger, the layer identification module can determine that a new frame (i.e., the first frame or a changed frame) has been received, and determine whether to perform preprocessing in combination with the status bit. In other cases, when the application does not send a new buffer to SurfaceFlinger, in order to synchronize the refresh of multiple layers, the previously received frames can continue to be sent to the layer identification module. The layer identification module can determine that the first frame and the changed frame have not been received in this round, and determine whether to perform preprocessing in combination with the status bit.

[0165] Among them, the first frame means that before SurfaceFlinger receives the synthesis task of layer 1, other layers of application 1 have not been stored in the cache area corresponding to application 1, that is, layer 1 is the first synthesis task of application 1 received by SurfaceFlinger. The change frame means that before SurfaceFlinger receives the synthesis task of layer 1, it has received the synthesis task of other layers of application 1, and other layers and layer 1 are stored in different cache areas.

[0166] Table 1 shows a correspondence between layer information, whether to perform preprocessing, and whether to use the preprocessing layer.

[0167] Table 1

[0168]

[0169]

[0170] S3, if the status bit of the layer information is rotated, SurfaceFlinger notifies HWC to use layer 1 to synthesize the image.

[0171] When the foldable screen device is in the unfolded state and the large screen is in the portrait state, if the status bit of layer 1 (such as the status bar layer and desktop layer shown in Table 1) is rotated, no matter whether it is the first frame or the change frame, there is no need to rotate the layer at this time. There is no problem that the height of the layer to be rotated exceeds the maximum rotation height supported by the HWC hardware module (such as 2176 pixels), and HWC synthesis can be used. In this case, SurfaceFlinger can send a synthesis request to HWC. HWC synthesizes layer 1 with other layers to generate a frame of image and sends it to the large screen for display.

[0172] S4, if the status bit is not rotated, and layer 1 belongs to the first frame or the change frame, then SurfaceFlinger notifies the GPU to preprocess layer 1, generate a preprocessed layer, and store the preprocessed layer in the cache.

[0173] When the folding screen device is in the unfolded state and the large screen is in the portrait state, if the status bit of layer 1 (such as the wallpaper layer shown in Table 1) is not rotated, and layer 1 belongs to the first frame or the change frame, it means that the pre-processed layer of layer 1 has not been stored before, and layer 1 needs to be rotated. The height of layer 1 (for example, 2300 pixels) exceeds the maximum rotation height supported by the HWC hardware module (for example, 2176 pixels), and GPU synthesis can be used. In this case, SurfaceFlinger sends a synthesis request to the GPU. The GPU rotates layer 1, generates a pre-processed layer, and stores the pre-processed layer in a buffer area (such as buffer area 1) for layer synthesis of the next frame. In addition, in this frame synthesis, the GPU can send the rotated pre-processed layer to the HWC, and the HWC synthesizes the rotated pre-processed layer with other layers to generate a frame of image and send it to the large screen for display.

[0174] For example, Fig. 9 The process of rendering out pre-processed layers using the GPU is shown.

[0175] like Fig. 9 As shown, application 1 first creates a canvas / drawing surface. In the Android system, each activity has an independent canvas (called a layer on the SurfaceFlinger side). No matter how complex the view structure of an activity is, it is ultimately rendered on the canvas of the activity to which it belongs. For example, the process of creating a surface for an application is as follows: WMS creates a connection to SurfaceFlinger for the application; the application executes the relayoutWindow() method to trigger WMS to create a SurfaceControl object; WMS obtains the surface through the SurfaceControl object.

[0176] After the surface creation is completed, the pre-processed layer can be rendered on the surface. The specific process of rendering the pre-processed layer on the surface is as follows: After the video codec decodes the wallpaper resource, the decoded wallpaper resource is stored in the layer buffer; WMS sets the rotation parameters for the wallpaper layer, such as the rotation direction and rotation angle; SurfaceFlinger can read the metadata / resources of the wallpaper layer from the buffer (referred to as wallpaper data for short), and then execute the DrawCall() instruction to call the image programming interface, such as the OpenGL interface or the Vulkan interface, to command the GPU to perform rendering operations. Among them, the DrawCall() instruction carries the rotation parameters. Then, the GPU can render the rotated wallpaper layer according to the rotation parameters.

[0177] S5, if the status bit is not rotated, and layer 1 does not belong to the first frame or the change frame, then SurfaceFlinger notifies HWC to use the previously stored pre-processed layer to synthesize the image.

[0178] When the foldable screen device is in the unfolded state and the large screen is in the portrait state, if the status bit of layer 1 (such as the wallpaper layer shown in Table 1) is not rotated, and layer 1 does not belong to the first frame or the change frame, it means that the preprocessed layer of layer 1 has been stored previously and the preprocessed layer can be used directly. In this case, SurfaceFlinger reads the preprocessed layer from the cache (such as cache 2) and then sends the preprocessed layer to the HWC. The HWC synthesizes the preprocessed layer with other layers to generate a frame of image and sends it to the large screen for display.

[0179] The cache area 1 in S4 and the cache area 2 in S5 may be the same cache.

[0180] In the above layer synthesis method, when the large screen is in the vertical screen state, the layer to be synthesized is identified to determine whether the layer needs GPU preprocessing, such as wallpaper layer rotation. If GPU preprocessing is required, the preprocessed layer is rendered using the GPU. In this way, when there is a subsequent task of processing the same layer, the preprocessed layer can be directly sent to the HWC for hardware synthesis, thereby reducing the power consumption of the whole machine and improving the performance of the whole machine.

[0181] It should be noted that the preprocessing operation of the above-mentioned layer synthesis method is illustrated by taking the rotating layer as an example, which does not limit the present application. In actual implementation, when the HWC hardware module is limited in its capabilities such as cutting corners, number of layers, conversion format, layer size, zoom ratio and window blur, and the synthesis requirements of the layer exceed the capabilities of the HWC hardware module, you can refer to the above-mentioned layer synthesis method to determine whether the layer needs to be preprocessed using the GPU. If GPU preprocessing is required, then use the GPU to render the preprocessed layer, so that when there is the same layer processing task later, the preprocessed layer can be sent to the HWC for hardware synthesis. You can refer to the description of the above-mentioned embodiment, which will not be repeated here.

[0182] The above embodiment takes a layer as an example to introduce the specific process of whether to pre-process a layer. Fig.10 , Fig.11A , Fig. 11B , introduces a layer synthesis method based on intelligent scene recognition.

[0183] Fig.10 A schematic diagram showing how to synthesize multiple layers into one frame of image based on a preprocessing layer.

[0184] When the large screen is in portrait mode, SurfaceFlinger first performs the synthesis task of the Nth frame, and then performs the synthesis task of the N+ith frame. Among them, the Nth frame and the N+ith frame are composed of a wallpaper layer, a desktop layer, a video layer, and a status bar layer, and the stacking order of these layers is: wallpaper layer, desktop layer, video layer, and status bar layer. It should be understood that in actual implementation, the Nth frame and the N+ith frame may also include more or fewer layers, such as replacing the video layer with a browser layer, etc., which is not limited in the embodiments of the present application.

[0185] For the synthesis task of the Nth frame: the height of the wallpaper resource (for example, 2300 pixels) exceeds the maximum rotation height supported by the HWC hardware module (for example, 2176 pixels), so SurfaceFlinger assigns the synthesis task of the wallpaper layer to the GPU. The GPU rotates the wallpaper layer 90° counterclockwise to obtain a pre-processed layer and puts the pre-processed layer into the cache.

[0186] For the synthesis task of the N+ith frame: If the wallpaper layer of the N+ith frame and the wallpaper layer of the Nth frame are the same wallpaper resource, SurfaceFlinger reads the pre-processed layer from the cache area, and then sends the pre-processed layer to the HWC. In addition, SurfaceFlinger sends the desktop layer and the status bar layer to the HWC. Combined with the description of the above embodiment, the video layer is read out by the video codec, and the video codec does not have the ability to rotate. The video layer is a layer that has not been rotated. In addition, the HWC does not support rounded corners, so SurfaceFlinger can assign the synthesis task of the video layer to the GPU. The GPU rotates the video layer 90° counterclockwise and cuts the rounded corners, and then sends the processed video layer to the HWC. The HWC superimposes the wallpaper layer, desktop layer, video layer, and status bar layer in sequence to obtain a frame of video image, and sends it to the large screen for display. When the large screen is in portrait mode, the phoenix displayed on the large screen flies toward the upper right (such as Figure 5B The flying direction of the phoenix shown on the small screen is Figure 4D In this way, although the installation directions of the large screen and the small screen are different, the wallpapers displayed on the large screen and the small screen that the user finally sees are in the same direction, which is more in line with the user's usage habits.

[0187] As an example, Fig.11A A schematic diagram is shown of synthesizing a video layer and a wallpaper layer into a multi-frame image based on a preprocessing layer.

[0188] like Fig.11AAs shown, the frequency of the video application updating the window is much greater than the frequency of the wallpaper application updating the window. In order to keep the wallpaper layer and the video layer at the same refresh frequency, SurfaceFlinger can synthesize the wallpaper layer based on the same wallpaper resource before the wallpaper application updates the window.

[0189] The synthesis process of frame 1 is as follows: the video application sends task A1 to SurfaceFlinger; because the video layer needs to be rotated and the corners need to be rounded, SurfaceFlinger hands task A1 to the GPU; the GPU rotates the video resource a1 obtained from the cache 90° counterclockwise and rounds the corners to obtain layer A1, and then sends layer A1 to HWC. At the same time, the wallpaper application sends task B1 to SurfaceFlinger; because the wallpaper layer needs to be rotated, and the height of the wallpaper layer exceeds the maximum rotation height supported by the HWC hardware module (for example, 2176 pixels), SurfaceFlinger hands task B1 to the GPU; the GPU rotates the video resource b1 obtained from the cache 90° counterclockwise to obtain layer B1, and then sends layer B1 to HWC. HWC superimposes layer A1 on layer B1 to form frame 1.

[0190] The synthesis process of frame 2 is as follows: the video application sends task A2 to SurfaceFlinger; because the video layer needs to be rotated and the corners need to be rounded, SurfaceFlinger hands task A2 to the GPU; the GPU rotates the video resource a2 obtained from the cache 90° counterclockwise and rounds the corners to obtain layer A2, and then sends layer A2 to HWC. At the same time, because the wallpaper application does not send new tasks to SurfaceFlinger, SurfaceFlinger can read the pre-processed layer B1 from the cache and then send the pre-processed layer B1 to HWC. HWC superimposes layer A2 on layer B1 to form frame 2.

[0191] As another example, Fig. 11B A schematic diagram is shown of synthesizing a status bar layer and a wallpaper layer into a multi-frame image based on a preprocessing layer.

[0192] When the device is running, the system UI and wallpaper application may update the window at different frequencies. For example, when the power level changes, the cellular network changes, or the wireless network changes, the system UI will notify SurfaceFlinger to update the status bar layer. For another example, when the wallpaper of the wallpaper application changes, the wallpaper application will notify SurfaceFlinger to update the wallpaper layer.

[0193] Assume that at a certain moment, the system UI calls the render thread to draw the rotated layer C1, and stores the layer C1 in the cache. SurfaceFlinger reads the rotated layer C1 from the cache and sends the rotated layer C1 to the HWC. At the same time, the wallpaper application sends task B1 to SurfaceFlinger; because the wallpaper layer needs to be rotated, and the height of the wallpaper layer exceeds the maximum rotation height supported by the HWC hardware module, SurfaceFlinger hands task B1 to the GPU; the GPU rotates the video resource b1 obtained from the cache counterclockwise by 90° to obtain the pre-processed layer B1, and sends the pre-processed layer B1 to the HWC. The HWC superimposes layer C1 on the pre-processed layer B1 to form frame 1.

[0194] The status bar is updated M seconds after this moment, and the wallpaper is updated N seconds after this moment, where N is greater than M. Before the Mth second, neither the wallpaper layer nor the status bar layer is updated, and the HWC continues to overlay layer C1 on the pre-processed layer B1 to form frame 2. After the Mth second, the status bar is updated, and the system UI calls the render thread to draw the rotated layer C2, and stores layer C2 in the cache. SurfaceFlinger reads the rotated layer C2 from the cache and sends the rotated layer C2 to the HWC. The HWC overlays layer C2 on the pre-processed layer B1 to form frame x. After the Nth second, the wallpaper layer is updated, and the wallpaper application sends task B2 to SurfaceFlinger. Since the wallpaper layer needs to be rotated and the height of the wallpaper layer exceeds the maximum rotation height supported by the HWC hardware module, SurfaceFlinger hands task B1 to the GPU; the GPU rotates the video resource b2 obtained from the cache counterclockwise by 90° to obtain the pre-processed layer B2, and sends the pre-processed layer B2 to the HWC.

[0195] Combined with the above embodiments Figure 5A , Figure 5B and Figure 6 If the description is Figure 5A , Figure 5B and Figure 6 In the layer synthesis scheme shown, multiple wallpaper layers are based on the same wallpaper resource. The GPU may need to rotate the same wallpaper resource multiple times. However, GPU synthesis has the disadvantage of high power consumption. Frequent rotation of wallpaper layers leads to high power consumption, poor performance and severe heat generation. Fig.11A and Fig. 11BIn the provided synthesis scheme, the GPU can use the pre-processed layer stored when synthesizing frame 1. For multiple frames located after frame 1, if the wallpaper layers of multiple frames and the wallpaper layer of frame 1 are based on the same wallpaper resource, there is no need to frequently rotate the wallpaper layer. Using the pre-processed layer reduces the power consumption and load of the entire machine and improves the performance of the entire machine.

[0196] exist Fig.10 , Fig.11A , Fig. 11B On the basis of Fig.12 FIG. 4 shows a schematic diagram of synthesizing the Nth frame image and the N+1th frame image. Fig.12 As shown, during the synthesis of the Nth frame image, the GPU rotates the wallpaper layer and sends the rotated wallpaper layer (pre-processed layer) to the cache. During the synthesis of the N+1th frame image, the wallpaper resource of the N+1th frame image is the same as the wallpaper resource of the Nth frame image, and the pre-processed layer can be directly used for synthesis.

[0197] The following is Figure 7 The functional module in the folding screen device shown is the execution body. Combined with the following examples 1 and 2, the specific method flow for synthesizing the Nth frame image and the N+1th frame image is introduced respectively.

[0198] Example 1

[0199] Fig.13 A flow chart of a method for synthesizing an Nth frame image based on a preprocessed layer is shown. The method can be applied to a folding screen device in an unfolded state. Fig.13 As shown, the method may include the following A1 to A16.

[0200] A1: The screen management service determines that the foldable screen device switches from the folded state to the unfolded state.

[0201] In some embodiments, sensors provided on the folding screen device can collect data such as the folding angle and acceleration of the folding screen device in real time, and send these data to the hardware configuration module of the folding screen device. The screen management service can obtain information such as the screen direction, window direction, and folding state from the hardware configuration module, and determine the current screen state based on this information, such as the folding screen device entering the unfolded state from the folded state. Then, the screen management service can notify the WMS of the current screen state.

[0202] For example, Fig.14AAs shown, at the initial moment, the foldable screen device is in a folded state, and the small screen displays wallpaper, desktop, video window, status bar, etc. When the user wants to use the large screen, he can hold the foldable screen device with both hands and bend it outward, so that the foldable screen device switches from the folded state to the unfolded state. The sensors provided on the foldable screen device can collect data such as folding angle and acceleration in real time, and send these data to the hardware configuration module of the foldable screen device. In this way, the screen management service can obtain information such as screen direction, window direction, and folding status from the hardware configuration module. After determining that the foldable screen device has switched from the folded state to the unfolded state based on this information, the screen management service can light up the large screen and notify the WMS. It should be noted that in the following example Fig.14A In the scenario example shown, after confirming to switch to the expanded state, no matter whether the large screen is in portrait or landscape mode at this moment, the folding screen device will default to portrait mode, and the wallpaper resource needs to be rotated.

[0203] It should be noted that Fig.14A The folding screen device is an inner folding screen mobile phone. This is used as an example for explanation, which does not limit the present application. The folding screen device can also be an outer folding screen mobile phone. Specifically, when the outer folding screen mobile phone is in the folded state, the small screen displays the wallpaper, and there is no need to rotate the wallpaper resource at this time; after switching from the folded state to the unfolded state, the large screen displays the wallpaper, and the wallpaper resource needs to be rotated at this time. The large screen and the small screen can use the same wallpaper resource.

[0204] In addition, the above A1 is explained by switching from a folded state to an unfolded state as a scenario example, which does not limit the present application. In other scenario examples, the image synthesis method provided in the embodiment of the present application can also be executed.

[0205] As another example scenario, Fig. 14B As shown, at the initial moment, the folding screen device is in the unfolded state, and the large screen displays wallpaper, desktop, video window and status bar in horizontal mode. If the user rotates the large screen clockwise or counterclockwise, the sensor set on the folding screen device can collect data such as folding angle and acceleration in real time, and send these data to the hardware configuration module of the folding screen device. In this way, the screen management service can obtain the screen direction from the hardware configuration module, such as switching from the horizontal screen state of the large screen to the vertical screen state of the large screen, and notify the WMS, so that the WMS determines the rotation parameters of the layer (such as 90° counterclockwise rotation) according to the screen direction, folding state and installation direction of the large screen, and executes the following A2-A16. It should be noted that before the user rotates the large screen clockwise or counterclockwise, the user can also trigger the folding screen device to turn on the horizontal and vertical screen functions by operating the setting application or control center. Among them, the horizontal and vertical screen functions allow the large screen to be displayed in a first direction when it is in a horizontal state, and in a second direction when it is in a vertical state, such as the first direction and the second direction are 90° or 270° clockwise.

[0206] As another example of a scenario, Fig. 14C As shown, the PAD is provided with a large screen, and the large screen is installed in a horizontal manner. When the large screen is in a horizontal state, the direction of the wallpaper is consistent with the direction of the large screen. In this case, there is no need to rotate the wallpaper. When the user rotates the large screen 90° clockwise, the large screen switches from a horizontal state to a vertical state. In order to make the wallpaper visually seen by the user consistent with the direction of the large screen, the screen management service of the PAD can calculate the rotation parameter as a counterclockwise rotation of 90° and execute the following A2-A16. The horizontal installation method of the large screen of the PAD is similar to that of the large screen of a folding screen device, and can refer to the above embodiment for details. Figure 4A-4G Description, no further elaboration is given here.

[0207] As another example of a scenario, Fig.14D As shown, at the initial moment, the folding screen device is in the unfolded state, and the large screen vertical screen displays wallpaper, desktop, video window and status bar, etc. The user holds the folding screen device with both hands and bends it inward, such as reducing the folding angle from 180° to 150°. The sensor set on the folding screen device can collect data such as folding angle and acceleration in real time, and send these data to the hardware configuration module of the folding screen device. In this way, the screen management service can obtain the folding angle change and screen direction from the hardware configuration module and notify the WMS. The embodiment of the present application also provides a method for changing dynamic wallpaper based on the folding angle: each folding angle corresponds to a frame of wallpaper resources, and when it is detected that the first folding angle D(A) changes to the second folding angle D(B), a frame of wallpaper resources corresponding to the second folding angle D(B) is determined. In this way, when the folding angle is changed in the large screen vertical state, the WMS will still calculate the rotation parameters of the layer (such as rotating 90° counterclockwise) and execute the following A2-A16.

[0208] The following takes the switching of a foldable screen device from a folded state to an unfolded state as an example, and describes a method for synthesizing the Nth frame image in sections A2-A16. The Nth frame image can be a static wallpaper or the last frame of a wallpaper video resource.

[0209] A2, WMS determines the rotation parameters of the layer according to the screen direction, folding state and installation direction of the large screen.

[0210] If the large screen follows Figure 4B If the wallpaper resource is installed on the phone body in the manner shown, the direction of the wallpaper resource is 90° clockwise from the direction of the large screen installed on the phone body. WMS determines that the large screen is in portrait mode based on the screen orientation and folding state. WMS calculates the rotation parameter of the layer to be 90° counterclockwise. Then, WMS notifies the wallpaper application, desktop application, video application, and system UI of the rotation parameter.

[0211] When the window of any application in the wallpaper application, desktop application, video application and system UI changes, the changed application can create a layer and hand over the synthesis task of the layer to SurfaceFlinger. The following A3-A5, A6-A8, A9-A10, A12-A15 respectively describe the rendering synthesis process of the wallpaper layer, desktop layer, video layer and status bar layer. It should be noted that the embodiment of the present application does not specifically limit the execution order of these steps.

[0212] A3, the wallpaper app notifies SurfaceFlinger to create a blank wallpaper layer / canvas.

[0213] A4, the wallpaper application notifies the video codec to decode the video resource. The video resource stores the decoded resource in a buffer (such as buffer b1), and calls a function to send buffer b1 to SurfaceFlinger.

[0214] The buffer b1 may include the layer name of the wallpaper layer, the status bit of the wallpaper layer, the layer stacking order of the wallpaper layer, and / or the rotation parameter layer information of the wallpaper layer.

[0215] A5, SurfaceFlinger determines whether to pre-process the wallpaper layer.

[0216] Whenever the application sends a new buffer area to SurfaceFlinger, the layer identification module can determine that a new frame has been received. Taking the received buffer b1 storing the first frame of the wallpaper resource and the status bit of the wallpaper layer as unrotated as an example, SurfaceFlinger can determine that the wallpaper layer needs to be preprocessed. SurfaceFlinger sends a synthesis request for the wallpaper layer to the GPU. The GPU rotates the wallpaper layer according to the rotation parameters, such as rotating it 90° counterclockwise, and renders the rotated wallpaper layer on the canvas using a line-by-line scanning method to obtain a preprocessed layer. On the one hand, the GPU can store the preprocessed layer in a buffer area (such as buffer area a) for use in the synthesis of the wallpaper layer in the N+1th frame. On the other hand, the GPU can execute the following A11 to use the preprocessed layer for the synthesis of the wallpaper layer in the Nth frame.

[0217] It should be noted that Example 1 involves two types of buffers: the first type of buffer is used to store the layer data to be synthesized, for example, buffer b1 stores the metadata of the wallpaper layer, buffer b2 stores the metadata of the desktop layer, buffer b3 stores the metadata of the video layer, and buffer b4 stores the metadata of the status bar layer; the second type of buffer is used to store the pre-processed layers that have been processed, for example, buffer a stores the wallpaper layer that has been rotated 90° counterclockwise. It should be understood that when the layers of multiple applications all need to be pre-processed, the pre-processed layers of each application can be stored in different buffers.

[0218] A6, when the desktop elements (such as icons) of the desktop application change, the desktop application sends a registration request to SurfaceFlinger. After the registration is successful, SurfaceFlinger starts to return the Vsync-APP signal to the desktop application. The wallpaper application notifies SurfaceFlinger to create a blank desktop layer.

[0219] A7, the desktop application calls the render thread to draw the desktop layer, and stores the drawn desktop layer in the buffer area (such as buffer area b2). That is, the desktop layer is a layer drawn according to the rotation parameters and rotated (such as 90° counterclockwise).

[0220] Then, the desktop application calls a function to send the buffer area b2 to SurfaceFlinger, where the buffer area b2 may include layer information such as the layer name of the desktop layer, the status bit of the desktop layer (rotated), and the stacking order of the desktop layer.

[0221] A8, SurfaceFlinger determines whether to pre-process the desktop layer based on the layer information of the desktop layer.

[0222] Whenever the application sends a new buffer to SurfaceFlinger, the layer identification module can determine that a new frame has been received. Since the desktop layer in buffer b2 is a rotated layer, SurfaceFlinger can determine that the wallpaper layer does not need to be preprocessed.

[0223] Referring to the description of the above embodiment, the desktop application has rotated the desktop layer 90° counterclockwise during the drawing process, and no further rotation is required. In principle, SurfaceFlinger can assign the synthesis task of the desktop layer to HWC. However, since the wallpaper layer located at the next layer of the desktop layer and the video layer located at the upper layer of the desktop layer are read out through the video codec, and the video codec does not have the ability to rotate, the wallpaper layer and the video layer are layers that have not been rotated. SurfaceFlinger can assign the synthesis task of the wallpaper layer, desktop layer and video layer to the GPU instead of to the HWC.

[0224] A9, the video application notifies SurfaceFlinger to create a blank video layer / canvas.

[0225] A10, the video application notifies the video codec to decode the video resource. The video resource stores the decoded resource in a buffer (such as buffer b3), and calls a function to send buffer b3 to SurfaceFlinger.

[0226] The buffer b3 may include the layer name of the video layer, the status bit of the video layer, the layer overlay order of the video layer, and / or the rotation parameter layer information of the video layer.

[0227] It should be noted that since each frame of the video application will be updated, SurfaceFlinger can determine whether to pre-process the video layer based on the layer name of the video layer, but directly synthesize the video frame.

[0228] For example, SurfaceFlinger requests the GPU to synthesize the video. The GPU rotates the video layer according to the rotation parameters, such as rotating it 90 degrees counterclockwise, and rounds the corners of the video layer to obtain the processed video layer. Then, the GPU can execute the following A11 to use the processed video layer for layer synthesis in the Nth frame.

[0229] A11, GPU synthesizes the wallpaper layer, desktop layer and video layer, and then sends the synthesized layer to HWC.

[0230] A12, when the status bar element of the system UI (such as the signal strength indicator or the remaining power indicator) changes, the system UI sends a registration request to SurfaceFlinger. After the registration is successful, SurfaceFlinger starts to return the Vsync-APP signal to the system UI.

[0231] A13, the system UI notifies SurfaceFlinger to create a blank status bar layer.

[0232] A14, the system UI calls the render thread to draw the status bar layer, and stores the drawn status bar layer in the buffer area (such as buffer area b4). That is, the status bar layer is a layer drawn according to the rotation parameters and rotated (such as 90° counterclockwise).

[0233] Then, the system UI calls a function to send the buffer area b4 to SurfaceFlinger, where the buffer area b4 may include layer information such as the layer name of the status bar layer, the status bit of the status bar layer (rotated), and the stacking order of the status bar layer.

[0234] A15, SurfaceFlinger determines whether to pre-process the status bar layer based on the layer information of the status bar layer.

[0235] Whenever the application sends a new buffer to SurfaceFlinger, the layer recognition module can determine that a new frame has been received. Since the status bar layer in buffer b4 is a rotated layer, SurfaceFlinger can determine that the status bar layer does not need to be preprocessed and sends the status bar layer to HWC.

[0236] Different from the desktop layer, the status bar layer of A15 is located above the wallpaper layer, status bar layer and video layer, so SurfaceFlinger can directly assign the synthesis task of the status bar layer to HWC.

[0237] A16, HWC synthesizes the synthesis layer and the status bar layer to obtain the Nth frame image, and sends the Nth frame image to the large screen for display.

[0238] It should be understood that when the wallpaper layer to be synthesized is the first frame or the change frame, SurfaceFlinger can notify the GPU to render the rotated wallpaper layer and send the rotated wallpaper layer to the cache. In this way, when there is a subsequent task of processing the same layer, the rotated wallpaper layer in the cache can be used directly, and the GPU does not have to rotate each wallpaper layer, thereby reducing the power consumption and load of the whole machine.

[0239] Example 2

[0240] When the folding state of the foldable screen device changes, or the horizontal or vertical state of the large screen changes, WMS can notify the wallpaper application, desktop application, and video application of the updated rotation parameters. Based on Example 1, when the window of an application changes, if the application does not obtain the updated rotation parameters, the application can still process the layer according to the rotation parameters obtained through Example 1.

[0241] The following example shows that after synthesizing the Nth frame image and before synthesizing the N+1th frame image, the windows of the wallpaper application and the desktop application do not change, while the windows of the video application and the status bar application change. Fig.15 The process of synthesizing the N+1th frame image based on the preprocessing layer is introduced.

[0242] like Fig.15 As shown, after A1-A16, the method may further include the following A17 to A22.

[0243] A17, SurfaceFlinger determines whether to preprocess the wallpaper layer.

[0244] Whenever the application sends a new buffer to SurfaceFlinger, the layer recognition module can determine that a new frame has been received. If the next wallpaper layer synthesis task is not received within the preset time after SurfaceFlinger receives the synthesis task of the wallpaper layer in the i-th frame image, SurfaceFlinger can send the wallpaper layer last obtained from the buffer b1 to the scene recognition module. Since the buffer of the wallpaper layer in A17 and the buffer of the wallpaper layer in A5 are both buffer b1, the wallpaper resource is not the first frame and the change frame. In addition, the status bit of the wallpaper layer is not rotated, so the scene recognition module recognizes that the N+1 frame can use the pre-processed layer and does not need GPU synthesis, thereby obtaining the pre-processed layer (that is, the rotated wallpaper layer) from buffer a, and then sending the pre-processed layer to the HWC.

[0245] A18, the desktop application determines whether to pre-process the desktop layer.

[0246] Whenever the application sends a new buffer to SurfaceFlinger, the layer recognition module can determine that a new frame has been received. If SurfaceFlinger does not receive the next desktop layer synthesis task within the preset time after receiving the synthesis task of the desktop layer in the i-th frame image, SurfaceFlinger can send the desktop layer obtained from the buffer b2 last time to the scene recognition module. Since the status bit of the desktop layer is rotated, SurfaceFlinger can determine that the desktop layer does not need to be preprocessed and sends the desktop layer to HWC.

[0247] It should be noted that, unlike the above-mentioned A8 in which SurfaceFlinger sends the desktop layer to the GPU, the A17 uses a pre-processing layer of wallpaper, and the GPU does not need to process the wallpaper layer. In this way, in the A18, the desktop layer between the wallpaper layer and the video layer does not need to be synthesized by the GPU, and SurfaceFlinger can directly send the desktop layer to the HWC.

[0248] A19, the video application notifies the video codec to decode the video resource. The video resource stores the decoded resource in a buffer (such as buffer b3') and calls a function to send buffer b3' to SurfaceFlinger. SurfaceFlinger requests the GPU to synthesize the video. The GPU rotates the video layer according to the rotation parameters, such as rotating it 90° counterclockwise, and rounds the corners of the video layer to obtain the processed video layer. The GPU sends the video layer to the HWC.

[0249] It should be noted that the buffer b3' and the buffer b3 of A10 are different buffer areas. For example, the buffer b3 is used to store the first frame of the video, and the buffer b3' is used to store the second frame of the video.

[0250] For the specific implementation method of A19, please refer to the description of A10 above, which will not be repeated here.

[0251] A20, when the status bar element of the system UI changes again, the system UI calls the render thread again to draw the status bar layer, and stores the drawn status bar layer in the buffer area (such as buffer area b4'). That is, the status bar layer is a layer drawn according to the rotation parameters and rotated (such as 90° counterclockwise).

[0252] Then, the system UI calls the function to send buffer area b4' to SurfaceFlinger.

[0253] It should be noted that the buffer area b4' and the buffer area b4 of A14 are different cache areas.

[0254] A21, SurfaceFlinger determines whether to pre-process the status bar layer based on the layer information of the status bar layer.

[0255] Whenever the application sends a new buffer to SurfaceFlinger, the layer recognition module can determine that a new frame has been received. Since the status bar layer in buffer b4' is a rotated layer, SurfaceFlinger can determine that the status bar layer does not need to be preprocessed and sends the status bar layer to HWC.

[0256] A22, HWC synthesizes the preprocessing layer (i.e., wallpaper layer), desktop layer, video layer, and status bar layer to obtain the N+1th frame image, and sends the N+1th frame image to the large screen for display.

[0257] It should be understood that since the GPU sends the rotated wallpaper layer to cache a in advance, when there is a subsequent same layer processing task, the rotated wallpaper layer in cache a can be directly used to finally generate the N+1th frame image, thereby reducing the power consumption of the entire machine and improving the performance of the entire machine.

[0258] In the above-mentioned examples 1 and 2, the wallpaper image obtained from the buffer area b1 can be the last frame of a video resource. Usually, after the large screen is turned on, the large screen will play from the first frame of the video resource to the last frame of the video resource, and then the large screen will continue to display the last frame of the video resource. It should be understood that during the period from the first frame of the video resource to the last frame of the video resource, the wallpaper is updated frequently, and the next frame of each frame of the wallpaper may not require the use of a pre-processing layer; after the last frame of the video resource is played, the wallpaper remains basically unchanged, and a pre-processing layer may need to be used subsequently. In this regard, combined with Fig.16 , an embodiment of the present application provides another layer synthesis method.

[0259] B1. The wallpaper application notifies SurfaceFlinger to create a wallpaper layer.

[0260] B2, the wallpaper application notifies the video codec to decode the video resource.

[0261] Assume that the video resource of the wallpaper consists of M frames. When the large screen is turned on, the wallpaper application can notify the video codec to decode the video resource. The video codec first stores the decoded first frame in Buffer1, the decoded second frame in Buffer2, the decoded third frame in Buffer3, ..., and so on, and directly stores the last frame in BufferM. Then, the video codec sends each Buffer to SurfaceFlinger in sequence. Among them, each Buffer can contain the layer name of the wallpaper layer, the status bit of the wallpaper layer, the layer overlay order of the wallpaper layer, and / or the rotation parameter layer information of the wallpaper layer. Every time SurfaceFlinger obtains a Buffer (a Buffer is equivalent to a synthesis task), it can perform the following judgment steps:

[0262] SurfaceFlinger obtains the i-th synthesis task and starts timing. Since the status of the wallpaper resource of the i-th synthesis task is marked as unrotated, SurfaceFlinger determines that the wallpaper layer needs to be preprocessed. SurfaceFlinger sends a synthesis request for the wallpaper layer to the GPU. The GPU rotates the wallpaper layer according to the rotation parameters, such as rotating it 90° counterclockwise, to obtain the preprocessed layer of the i-th synthesis task. The GPU stores the preprocessed layer in the buffer a_i and sends the preprocessed layer of the i-th synthesis task to the HWC.

[0263] If SurfaceFlinger obtains the i+1th synthesis task before the timing reaches the maximum timing (for example, N Vsync-SF signal cycles, N is a positive integer), it can be determined that the wallpaper is updated frequently and the pre-processed layer in the cache a_i does not need to be used. On the one hand, the pre-processed layer in the cache a_i can be cleared. On the other hand, SurfaceFlinger starts to re-timing. Since the status of the wallpaper resource of the i+1th synthesis task is marked as unrotated, SurfaceFlinger determines that the wallpaper layer needs to be pre-processed. SurfaceFlinger sends a synthesis request for the wallpaper layer to the GPU. The GPU rotates the wallpaper layer according to the rotation parameters, such as rotating it 90° counterclockwise, to obtain the pre-processed layer of the i+1th synthesis task. Then, the GPU stores the pre-processed layer of the i+1th synthesis task in the cache a_i+1, and sends the pre-processed layer of the i+1th synthesis task to the HWC.

[0264] If SurfaceFlinger does not obtain the i+1th synthesis task before the timing reaches the maximum timing (such as N Vsync-SF signal cycles), it can be determined that the wallpaper layer has not been updated and the pre-processed layer in the buffer a_i needs to be used. SurfaceFlinger reads the pre-processed layer of the i-th synthesis task from the buffer a_i and sends the pre-processed layer to HWC.

[0265] It should be understood that when the wallpaper resource is a video resource, the wallpaper is updated frequently during the period from the first frame of the video resource to the last frame of the video resource. The GPU can temporarily store each frame of the wallpaper in the cache after rotating it. If no new task is received during N Vsync-SF signal cycles after a certain frame is played, it means that the last frame of the video resource has been played, and the pre-processing layer in the cache can be used.

[0266] In each of the above embodiments, the preprocessing operation performed by the GPU is to rotate the wallpaper. The wallpaper application may not refresh the wallpaper for tens of seconds, minutes or even hours. Therefore, after preprocessing the wallpaper once, the preprocessed wallpaper can be reused for a long time, and the power consumption benefit is relatively large. Based on the preprocessing principle, the present application also provides a generalized layer synthesis method: when multiple applications send images to SurfaceFlinger, and the refresh rates of the multiple applications are different, SurfaceFlinger determines the maximum refresh rate as the system frame rate. In the process of sending images by the first application, if the synthesis requirements of the first application layer exceed the capabilities of the HWC hardware module (such as cropping radius, number of layers, conversion format, layer size, zoom ratio and window blur, etc.), the GPU is used to preprocess the first application layer. Among them, the refresh rate of the first application is less than the system frame rate.

[0267] For example, Fig.17 A schematic diagram of a scenario of applying a preprocessing layer provided in an embodiment of the present application. The refresh rate of the video application is 30fps, and the refresh rate of the headline application is 120fps. In order to keep the refresh rates of various applications consistent, SurfaceFlinger determines 120fps as the system frame rate, so that both the video application and the headline application send images to SurfaceFlinger at a refresh rate of 120fps. In this case, every time the headline application refreshes three frames, such as frame M1, frame M2, and frame M3, the video application also refreshes three frames N1. Combined with the description of the above embodiment, since the video layer needs to be rotated by the GPU and the fillet is cut, if the three frames N1 are all rotated by the GPU and the fillet is cut, the power consumption of the whole machine will be high. Therefore, the layer synthesis method provided in the embodiment of the present application can be used to preprocess the first frame N1 to obtain a preprocessing layer, and then the second frame N1 and the third frame N1 pre-multiplex the preprocessing layer, thereby reducing the power consumption of the whole machine of the folding screen device and improving the performance of the whole machine.

[0268] For example, Fig.18 A schematic diagram of the hardware structure of a folding screen device 100 provided in an embodiment of the present application.

[0269] like Fig.18As shown, the folding screen device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a SIM card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, and a bone conduction sensor 180M, etc.

[0270] The processor 110 may include one or more processing units. For example, the processor 110 may include a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), an image signal processor (ISP), a neural-network processing unit (NPU), a modem, a data processing unit (DPU) and / or a baseband processor, etc.

[0271] The CPU is the final execution unit for information processing and program running. Its main tasks include processing instructions, executing operations, controlling time, and processing data. The CPU may include a controller, an arithmetic unit, a cache memory, and a bus for connecting these components. The controller is the nerve center and command center of the folding screen device 100. The controller can generate an operation control signal based on the instruction opcode and timing signal to complete the control of fetching and executing instructions. The arithmetic unit refers to a component that performs various arithmetic and logical operations. The cache memory can save instructions or data that the processor 110 has just used or recycled.

[0272] GPU, also known as display core, visual processor or display chip, is a microprocessor for graphics and image processing. GPU connects display screen 194, AP and CPU. GPU can be used to perform complex mathematical and geometric calculations, floating-point operations, parallel calculations, and can also be used for graphics rendering. GPU reduces the graphics card's dependence on the CPU, especially when processing 3D graphics. GPU uses core technologies such as cubic environment material mapping and vertex blending, so GPU performs some of the work originally done by the CPU.

[0273] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the folding screen device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system and an application required for at least one function. The data storage area can store configuration files of various applications, as well as data created during the use of the folding screen device 100.

[0274] The folding screen device 100 can realize audio functions through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, and an AP.

[0275] Display screen 194 includes Figure 2A-2C Or Figure 3A-3C Large and small screens shown.

[0276] It is understandable that the hardware structure illustrated in the embodiment of the present application does not constitute a specific limitation on the folding screen device 100. In other embodiments, the folding screen device 100 may include more or fewer components than shown, or combine certain components, or split certain components, or arrange the components differently. Fig.18 The components shown may be implemented in hardware, software or a combination of software and hardware.

[0277] An embodiment of the present application also provides a terminal device, such as a folding screen device, including a processor, the processor is coupled to a memory, and the processor is used to execute a computer program or instruction stored in the memory to implement the methods in the above embodiments.

[0278] The embodiment of the present application also provides a computer-readable storage medium, in which computer instructions are stored; when the computer-readable storage medium is run on a computer, the computer is caused to execute the method shown above. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that can be integrated with one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk or a tape), an optical medium or a semiconductor medium (e.g., a solid state drive (SSD)), etc.

[0279] An embodiment of the present application further provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer executes the methods in the above embodiments.

[0280] The embodiment of the present application also provides a chip, which is coupled to a memory, and the chip is used to read and execute a computer program or instruction stored in the memory to perform the methods in the above embodiments. The chip can be a general-purpose processor or a dedicated processor. It should be noted that the chip can be implemented using the following circuits or devices: one or more field programmable gate arrays (field programmable gate array, FPGA), programmable logic devices (programmable logic device, PLD), controllers, state machines, gate logic, discrete hardware components, any other suitable circuits, or any combination of circuits capable of performing the various functions described throughout this application.

[0281] The folding screen device, computer-readable storage medium, computer program product and chip provided in the above-mentioned embodiments of the present application are all used to execute the method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects corresponding to the method provided above, and will not be repeated here.

[0282] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0283] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0284] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0285] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0286] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A layer synthesis method, characterized in that: The method is applied to a terminal device, the terminal device includes a graphics processor GPU, a hardware synthesizer HWC, a large screen and a small screen, and the method includes: When the terminal device is at a first folding angle, obtaining a first wallpaper layer; Combining the first wallpaper layer, the first desktop layer and the first status bar layer into a first image; Displaying the first image on the small screen; In response to a user adjusting the terminal device from the first folding angle to a second folding angle, acquiring the first wallpaper layer, the second folding angle being greater than the first folding angle; Rotating the first wallpaper layer by the GPU to obtain a first preprocessing layer, wherein the height of the first wallpaper layer is greater than the maximum rotation height supported by the HWC; Storing the first preprocessed layer in a first buffer area; Combining the first preprocessing layer, the second desktop layer, and the second status bar layer into a second image; Displaying the second image on the large screen; Acquire the first preprocessing layer from the first buffer area; Combining the first preprocessing layer, the third desktop layer and the third status bar layer into a third image; The third image is displayed on the large screen.

2. The method according to claim 1, characterized in that The obtaining the first wallpaper layer includes: Acquire a first Vsync-SF signal; If the second buffer area has been acquired before the first Vsync-SF signal is acquired, acquiring the first wallpaper layer from the second buffer area; The acquiring the first preprocessing layer from the first buffer area includes: Obtaining a second Vsync-SF signal; If no other buffer area is obtained before the second Vsync-SF signal is obtained, the first pre-processed layer is obtained from the first buffer area.

3. The method according to claim 2, characterized in that After acquiring the second Vsync-SF signal, the method further includes: If the third buffer area is acquired before the second Vsync-SF signal is acquired, acquiring the second wallpaper layer from the third buffer area; Rotating the second wallpaper layer by the GPU to obtain a second preprocessing layer, wherein the height of the second wallpaper layer is greater than the maximum rotation height supported by the HWC; storing the second preprocessed layer in a fourth buffer area; Combining the second preprocessing layer, the fourth desktop layer and the fourth status bar layer into a fourth image; The fourth image is displayed on the large screen.

4. The method according to any one of claims 1 to 3, characterized in that The row scanning direction of the small screen is parallel to the first frame, and the column scanning direction of the small screen is parallel to the second frame; The row scanning direction of the large screen is parallel to the second frame, and the column scanning direction of the large screen is parallel to the first frame; The first frame and the second frame are two adjacent frames on the body of the terminal device, and the length of the first frame is smaller than the length of the second frame.

5. A layer synthesis method, characterized in that: The method is applied to a terminal device, the terminal device includes a GPU, a HWC and a display screen, and the method includes: Acquire a first wallpaper layer, where the height of the first wallpaper layer is greater than the maximum rotation height supported by the HWC; Rotating the first wallpaper layer by the GPU to obtain a first preprocessing layer; Combining the first preprocessing layer, the first desktop layer and the first status bar layer into a first image; Storing the first preprocessed layer in a first buffer area; displaying the first image on the display screen; Acquire the first preprocessing layer from the first buffer area; Combining the first preprocessing layer, the second desktop layer, and the second status bar layer into a second image; The second image is displayed on the display screen.

6. The method according to claim 5, characterized in that The step of obtaining the first wallpaper layer includes: Acquire a first Vsync-SF signal; If the second buffer area has been acquired before the first Vsync-SF signal is acquired, acquiring the first wallpaper layer from the second buffer area; The acquiring the first preprocessing layer from the first buffer area includes: Obtaining a second Vsync-SF signal; If no other buffer area is obtained before the second Vsync-SF signal is obtained, the first pre-processed layer is obtained from the first buffer area.

7. The method according to claim 6, characterized in that After acquiring the second Vsync-SF signal, the method further includes: If the third buffer area is acquired before the second Vsync-SF signal is acquired, acquiring the second wallpaper layer from the third buffer area; Rotating the second wallpaper layer by the GPU to obtain a second preprocessing layer, wherein the height of the second wallpaper layer is greater than the maximum rotation height supported by the HWC; storing the second preprocessed layer in a fourth buffer area; Combining the second preprocessing layer, the third desktop layer and the third status bar layer into a third image; The third image is displayed on the display screen.

8. The method according to any one of claims 5 to 7, characterized in that The row scanning direction of the display screen is parallel to the second frame, and the column scanning direction of the display screen is parallel to the first frame; The first frame and the second frame are two adjacent frames on the body of the terminal device, and the length of the first frame is smaller than the length of the second frame.

9. The method according to claim 6, characterized in that The acquiring the first wallpaper layer from the second buffer area includes: Acquire the first wallpaper layer and layer information of the first wallpaper layer from the second buffer area; wherein the layer information of the first wallpaper layer includes a state bit and a rotation parameter, the state bit is used to indicate that the first wallpaper layer has not been rotated, and the rotation parameter is used to indicate a rotation direction and a rotation angle of the first wallpaper layer; The step of rotating the first wallpaper layer by the GPU to obtain a first preprocessing layer includes: The GPU rotates the first wallpaper layer based on the state bit and the rotation parameter to obtain the first preprocessing layer.

10. The method according to claim 9, characterized in that Before obtaining the first wallpaper layer, the method further includes: When the preset conditions are met, determining the rotation parameters; Sending the rotation parameter to a wallpaper application of the terminal device; Generate a status bit through the wallpaper application, and store the status bit, the rotation parameter and the first wallpaper layer in the second buffer area; The preset condition includes any one of the following: The display screen switches from a horizontal screen state to a vertical screen state; The terminal device switches from a folded state to an unfolded state.

11. The method according to claim 10, characterized in that The method further comprises: In response to user operation, the horizontal and vertical screen functions are turned on; wherein, the horizontal and vertical screen functions allow display in a first direction when the terminal device is in the horizontal screen state, and display in a second direction when the terminal device is in the vertical screen state; the first direction differs from the second direction by 90° or 270° clockwise.

12. The method according to claim 11, characterized in that When the terminal device is in the horizontal screen state, the angle between the first frame of the terminal device body and the ground plane is greater than the angle between the second frame of the terminal device body and the ground plane; when the terminal device is in the vertical screen state, the angle between the first frame and the ground plane is less than or equal to the angle between the second frame and the ground plane; Wherein, the length of the first frame is smaller than the length of the second frame.

13. The method according to claim 9, characterized in that The layer information of the first wallpaper layer also includes a layer stacking order of the first wallpaper layer; The synthesizing the first preprocessing layer, the first desktop layer and the first status bar layer into the first image comprises: superimposing the first preprocessing layer, the first desktop layer and the first status bar layer according to the layer superposition order of the first wallpaper layer, the layer superposition order of the first desktop layer and the layer superposition order of the first status bar layer to obtain the first image; The synthesizing the first preprocessing layer, the second desktop layer and the second status bar layer into the second image includes: superimposing the first preprocessing layer, the second desktop layer and the second status bar layer according to the layer superposition order of the first wallpaper layer, the layer superposition order of the second desktop layer and the layer superposition order of the second status bar layer to obtain the second image.

14. A terminal device, characterized in that: comprising a processor, and a memory coupled to the processor; The memory stores instructions, and the processor calls the instructions so that the terminal device executes the layer synthesis method as described in any one of claims 1 to 13.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on a terminal device, the terminal device executes the layer synthesis method according to any one of claims 1 to 13.

Images