Layer synthesis method, electronic equipment and storage medium

Abstract

The invention provides a layer synthesis method, electronic equipment and a storage medium, and relates to the technical field of display. The method is applied to the electronic equipment and specifically comprises the steps of obtaining to-be-displayed data; when the to-be-displayed data meets a first condition, a layer in a UBWC format corresponding to the to-be-displayed data is generated, the first condition is that the layer of the to-be-displayed data is displayed after being converted, conversion operation processing is carried out on the layer in the UBWC format, a conversion layer in the UBWC format is obtained, and layer synthesis is carried out on the conversion layer in the UBWC format. When the layer of the to-be-displayed data needs to be converted and then is displayed, the format of the generated layer corresponding to the to-be-displayed data is controlled to be the UBWC format, so that the situation that an IWE module preprocesses the converted layer in the non-UBWC format is avoided, the continuous frame loss phenomenon caused by too long preprocessing time of the IWE module is avoided, and the display fluency of the electronic equipment is improved.

Description

Technical Field

[0001] The present application relates to the field of display technology, and in particular to a layer synthesis method, electronic device and storage medium. Background Art

[0002] With the rapid development of science and technology, the application of Universal Bandwidth Compression (UBWC) technology is becoming more and more extensive. Generally, in the decoding stage of image or video data, setting the output layer format to UBWC format can effectively improve the system memory throughput, thereby increasing the memory transmission speed and saving power consumption.

[0003] In the layer synthesis process, the display processing unit (DPU) supports the transformation operations such as rotation and scaling of the layers in UBWC format, but does not support the transformation operations such as rotation and scaling of the layers in non-UBWC format. In order to solve the problem of not supporting the transformation operations such as rotation and scaling of the layers in non-UBWC format, in the current technology, a new IWE module is added to the display processing unit to pre-process the transformation layers in non-UBWC format (for example, rotation layers, scaling layers, etc.), thereby making up for the insufficient processing capability of the display processing unit for the layers in non-UBWC format.

[0004] In the current technology, since the IWE module takes a long time to preprocess the transformation layer in non-UBWC format, it causes continuous frame loss of video / image data, resulting in obvious lag in the display of electronic devices. Summary of the invention

[0005] The present application provides a layer synthesis method, an electronic device and a storage medium, which avoids continuous frame loss of video / image data caused by the long time consumed by the IWE module to preprocess the transformation layer in a non-UBWC format, and improves the smoothness of the electronic device display.

[0006] In order to achieve the above objectives, this application adopts the following technical solutions:

[0007] In the first aspect, the present application provides a layer synthesis method, which is applied to electronic devices, and the method includes: obtaining data to be displayed; when the data to be displayed meets the first condition, generating a layer in UBWC format corresponding to the data to be displayed, the first condition is to transform the layer of the data to be displayed before display, perform transformation operation processing on the layer in UBWC format, obtain the transformed layer in UBWC format, and perform layer synthesis on the transformed layer in UBWC format. When it is necessary to transform the layer of the data to be displayed before display, control the layer format corresponding to the generated data to be displayed to be in UBWC format, thereby avoiding the IWE module from preprocessing the transformed layer in non-UBWC format, thereby avoiding the continuous frame loss phenomenon caused by the long preprocessing time of the IWE module, and improving the smoothness of the display of the electronic device.

[0008] In a possible implementation, whether the data to be displayed meets the first condition is determined based on the shape data of the layer corresponding to the data to be displayed, the shape data of the layer including the length of the layer and the width of the layer, and when the length of the layer corresponding to the data to be displayed is greater than the width of the layer corresponding to the data to be displayed, the data to be displayed meets the first condition. Determining whether a transformation operation is required based on the shape data of the layer of the data to be displayed itself can simplify the determination steps.

[0009] In a possible implementation, when the data to be displayed does not meet the first condition, a non-UBWC format layer corresponding to the data to be displayed is directly generated, and the non-UBWC format layer is synthesized. When the layer of the data to be displayed does not need to be transformed before display, the layer transformation operation is not performed, and a non-UBWC format layer is directly generated, and the non-UBWC format layer is synthesized. This also avoids the IWE module from preprocessing the non-UBWC format transformation layer, thereby avoiding the continuous frame loss phenomenon caused by the long preprocessing time of the IWE module, and improving the smoothness of the electronic device display.

[0010] In a possible implementation, after generating a non-UBWC format layer corresponding to the data to be displayed, image processing is performed on the non-UBWC format layer to obtain a non-UBWC format layer after image processing; and the non-UBWC format layer after image processing is layer synthesized. After the non-UBWC format layer is image processed, as long as no transformation operation is performed, the IWE module will not be introduced to pre-process the non-UBWC format transformation layer, and the IWE module pre-processing is still avoided.

[0011] In a possible implementation, whether the data to be displayed meets the first condition is determined based on shape data of the layer corresponding to the data to be displayed; the shape data of the layer includes: the length of the layer and the width of the layer; when the length of the layer corresponding to the data to be displayed is less than or equal to the width of the layer corresponding to the data to be displayed, the data to be displayed does not meet the first condition.

[0012] In a possible implementation, after obtaining the data to be displayed, a non-UBWC format layer corresponding to the data to be displayed is generated; image processing is performed on the non-UBWC format layer to obtain a non-UBWC format layer after image processing; format conversion is performed on the non-UBWC format layer after image processing to obtain a corresponding UBWC format layer after image processing; transformation operation processing is performed on the UBWC format layer after image processing to obtain a transformed layer in UBWC format after image processing; and layer synthesis is performed on the transformed layer in UBWC format after image processing. In the case where image processing and transformation operation processing cannot be avoided, the format of the layer is converted to avoid the preprocessing of the IWE module, thereby avoiding the continuous frame loss phenomenon caused by the long preprocessing time of the IWE module, and improving the smoothness of the display of the electronic device.

[0013] In a possible implementation, it is determined whether the application that obtains the data to be displayed is in a whitelist; when the application that obtains the data to be displayed is in the whitelist, it is determined whether the data to be displayed meets a first condition.

[0014] In a possible implementation, an electronic device includes a display processing unit DPU, and an IWE module is provided on the DPU; the operating frequency of the DPU is increased to increase the processing frequency of the IWE module, and when the application that obtains the data to be displayed is not in the whitelist, a non-UBWC format layer corresponding to the data to be displayed is generated; the non-UBWC format layer is transformed and processed to obtain a corresponding non-UBWC format transformation layer; the non-UBWC format transformation layer is preprocessed using the IWE module with increased processing frequency; and the preprocessed non-UBWC format transformation layer is layer synthesized. When the preprocessing of the IWE module cannot be avoided, the processing frequency of the IWE module is increased to shorten the time consumed by the IWE module preprocessing, thereby reducing the continuous frame loss phenomenon and improving the smoothness of the electronic device display.

[0015] In a second aspect, the present application provides an electronic device, which includes a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the processor executes the method of the first aspect above.

[0016] In a third aspect, the present application provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is executed, the method of the first aspect described above is implemented.

[0017] In a fourth aspect, the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the method of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram showing a pipeline process of an electronic device in the related art;

[0019] Figure 2A It is a simplified schematic diagram of the layer rotation synthesis process in the UBWC format in the related art;

[0020] Figure 2B It is a simplified schematic diagram of the non-UBWC format layer rotation synthesis process in the related art;

[0021] Figure 3 A schematic diagram of another display assembly line process of an electronic device in the related art;

[0022] Figure 4A An exemplary diagram of the composition of an electronic device provided in an embodiment of the present application;

[0023] Figure 4B An example diagram of the software structure of an electronic device provided in an embodiment of the present application;

[0024] Figure 5 A simplified schematic diagram of a display assembly line process of an electronic device provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of a process of layer synthesis method provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of a process flow of another layer synthesis method provided in an embodiment of the present application;

[0027] Figure 8 A schematic diagram of a process flow of another layer synthesis method provided in an embodiment of the present application;

[0028] Fig. 9 A flowchart of another layer synthesis method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The terms "first", "second", "third", etc. in the specification, claims and drawings of this application are used to distinguish different objects rather than to limit a specific order.

[0030] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0031] In order to make the description of the following embodiments clear and concise, a brief introduction to the related technology is first given:

[0032] UBWC (Universal Bandwidth Compression) technology: A predictive band compression technology that increases system memory throughput by minimizing data bandwidth, thereby increasing memory transfer speeds and saving power.

[0033] Layer (surface): Each application may correspond to one or more graphical interfaces, and each graphical interface can be called a layer. The display interface of an electronic device is usually composed of multiple layers. Each layer in the display interface has its position, size, and content to be displayed on the screen. During the running of the application, the position, size or displayed content of the surface corresponding to the application may change. Each layer has corresponding layer properties, which may include information such as the position and size of the corresponding layer.

[0034] Buffer queue: Each layer has a corresponding buffer queue, which is used to store the display data of the corresponding layer. The display data is used to indicate the display content of the corresponding layer, that is, the display data of the layer can be the image obtained after rendering the layer.

[0035] Frame buffer is a continuous physical memory reserved for GPU (graphics processing unit), DPU (display processing unit), etc. in embedded systems. The display reads data from the frame buffer through a dedicated bus and displays it on the screen.

[0036] The advantages of a method for synthesizing rotating layers in video data provided by an embodiment of the present application are compared and explained below in combination with the method for synthesizing rotating layers in video data in the current technology.

[0037] Before introducing the method for synthesizing rotating layers in video data, let's first introduce the display process of the electronic device's display screen (also known as the display pipeline of the display system of the electronic device). The display screen display process of the electronic device mainly includes three processes: application rendering, layer synthesis, and drive display. That is to say, for a certain frame of image, the display system needs to execute the application rendering process, layer synthesis process, and drive display process in sequence before it can be displayed on the display screen. Among them, the application rendering process refers to rendering the application layer according to the view update request. The layer synthesis process refers to synthesizing multiple layers rendered by the application rendering process to generate a synthesized layer, which is the frame image to be displayed. The drive display process refers to driving the display image to be displayed generated by the layer synthesis process for display processing and pushing it to the display screen.

[0038] In the current technology, in order to avoid stuttering and improve the visual effect performance of the layer, the electronic device is generally displayed based on the vertical synchronization (Vertical Synchronization, Vsync) signal, that is, the Vsync signal is used to synchronize the application rendering, layer synthesis, and drive display in the display pipeline. The Vsync signal is generated by the Surface flinger, that is, the Surface flinger is also used to generate and distribute the Vsync signal. Specifically, the Vsync signal is used to trigger the three processes of application rendering, layer synthesis, and drive display. That is, when the application of the application layer of the electronic device receives the Vsync signal, the application renders the image data through the view system; when the layer management module of the electronic device receives the Vsync signal, the layer synthesis of the rendered image data can be performed; when the display of the electronic device receives the Vsync signal, the display driver of the kernel layer can be called to display the synthesized image data. The electronic device can perform the three processes in the display pipeline in sequence according to the "rhythm" of the Vsync signal, but the application rendering, layer synthesis, and drive display are different. Among them, within a cycle of a Vsync signal, the electronic device performs one application rendering, one layer synthesis, and one drive display.

[0039] The software related to the layer synthesis process in the electronic device includes but is not limited to: a layer management module (Surface flinger). The hardware related to layer synthesis in the electronic device includes but is not limited to a GPU and a HWC. The GPU and the HWC are used to provide hardware support for the Surface flinger.

[0040] Layer management module (Surface flinger): It can also be called a layer deliverer. It is a service system that is mainly used to implement functions such as layer creation, control, and management. In the display system of an electronic device, Surface synthesis can be achieved through Surface flinger. For example, after the application is started, Surface flinger can create a layer for the application. During the operation of the terminal, Surface flinger can obtain the layers to be displayed of various applications running in the terminal, and synthesize the obtained layers through GPU and / or HWC.

[0041] GPU synthesis: GPU is a general-purpose image processing device. In addition to being used for layer synthesis, GPU is also used to complete other graphics processing tasks. GPU synthesis refers to the layer synthesis method that uses the GPU to synthesize layers.

[0042] HWC synthesis: HWC is a dedicated image processing device used for layer synthesis and display. HWC can be an independent device or integrated into a system-level chip. HWC synthesis refers to the layer synthesis method of synthesizing layers through HWC. In the HWC synthesis process, since each layer needs to occupy a transmission channel of HWC, and the transmission channels of HWC are limited, the number of layers that HWC supports synthesis is limited.

[0043] For ease of understanding, the following example illustrates a layer synthesis process, assuming that the interface to be displayed of the electronic device includes two layers, namely the top status bar A and the application interface B of the b application, and the number of layers that the HWC supports synthesis is greater than 2. When synthesizing the two layers included in the interface to be displayed, the Surface flinger sends the two layers of the top status bar A and the application interface B of the b application to the HWC, and the HWC synthesizes the top status bar A and the application interface B of the b application to obtain a synthesized layer, completing the layer synthesis process.

[0044] Specifically, the display pipeline of an electronic device is actually a kind of buffer flow. For example, the application starts to render the application according to the Vsync-app signal, and sends the rendered suffer to the Surface flinger for synthesis; the Surface flinger starts to take the corresponding suffer according to the Vsync-sf signal and synthesizes the layer through the HWC or GPU; the final synthesized layer is sent to the display through the display driver, and refreshed to the screen when the screen hardware TE signal arrives, completing the entire display pipeline process. In order to ensure the order of the display process, the current technology introduces the fence mechanism, that is, the fence mechanism is added to the buffer flow process. Specifically, in the display pipeline process, when performing the next operation, it is detected whether the fence is available. If it is available, the next operation is performed. If it is not available, wait until the fence is available before performing the next operation. Therefore, when driving the display, it is necessary to wait for the layer synthesis process to end and release the fence before executing. The fence mechanism can ensure the order of the application rendering process, layer synthesis process, and driver display process in the display pipeline (that is, in the order of application rendering-layer synthesis-driver display), and ensure the implementation of the display pipeline.

[0045] For ease of understanding, the following Figure 1 The display pipeline process of an electronic device is described by taking the electronic device displaying a video as an example, the displayed video is a 4-frame video, and the content displayed by the electronic device corresponds to frame 1, frame 2, frame 3 and frame 4 in sequence according to the playback time sequence.

[0046] Taking the display of frame 1 as an example, the application of the electronic device draws frame 1 through the view system and renders frame 1 through the image rendering system. After the application of frame 1 is finished drawing and rendering, the application of the electronic device sends the drawn and rendered frame 1 to the image synthesis system. The image synthesis system performs layer synthesis on the drawn and rendered frame 1 through HWC or GPU. After the frame 1 layer synthesis process is completed and the fence is released, the electronic device can refresh the synthesized frame 1 on the screen by calling the kernel layer display driver and waiting for the screen hardware TE signal to arrive. It should be noted that the display pipeline process of frames 2, 3 and 4 is similar to that of frame 1, which will not be repeated here.

[0047] In the current technology, due to the insufficient processing power of the display processing unit, it is impossible to use image / video processing algorithms (such as video enhancement algorithms, video slow motion algorithms, etc.) to directly process the UBWC format layer, and only the non-UBWC format layer can be processed. Therefore, the format of the layer that can be processed must be non-UBWC format, so the format of the layer after image processing must also be non-UBWC format. Therefore, the display processing unit (DPU) in the current technology has a good adaptability to UBWC format data. In the layer synthesis stage, the display processing unit supports the rotation, scaling and other operations of the UBWC format layer, but does not support the rotation, scaling and other operations of the non-UBWC format layer.

[0048] For ease of understanding, the following Figure 2A and Figure 2B Explanation is given below. A 90° rotation is used as an example.

[0049] like Figure 2A The schematic diagram of the layer synthesis process of the UBWC format rotation layer shown in the figure shows that the processing unit supports the rotation operation of the UBWC format layer. The rotation operation of the UBWC format layer can be realized without introducing preprocessing of other modules, and then the corresponding layer synthesis processing is performed through the HWC.

[0050] Due to the insufficient processing power of the display processing unit, it is unable to synthesize transformation layers in non-UBWC formats (for example, rotation layers, scaling layers, etc.). Transformation layers in non-UBWC formats need to be synthesized by the graphics processing unit (Graphic Processing Unit, GPU), which will bring additional power overhead. In order to avoid the additional power overhead, a new IWE module is added to the DPU to make up for the insufficient processing power of the DPU for transformation layers in specific formats (non-UBWC formats). That is, a new IWE module is added to the DPU to assist the DPU in processing rotation layers in non-UBWC formats. Among them, the IWE module is a data processing module, which is mainly used to process transformation layers in specific formats (for example, rotation layers in non-UBWC formats). Figure 2B The schematic diagram of the layer synthesis process of the non-UBWC format rotation layer shown in the figure shows that after the non-UBWC format layer is rotated, the IWE module is used to pre-process the non-UBWC format rotation layer, and then the pre-processed corresponding non-UBWC format rotation layer is synthesized through the HWC, which makes up for the insufficient processing capability of the DPU for the specific format layer (non-UBWC format) and avoids the additional power consumption loss caused by the layer synthesis processing through the GPU.

[0051] In the current technology, an IWE module is introduced on the DPU to support the synthesis processing of non-UBWC format transformation layers after rotation, scaling and other transformation operations on non-UBWC format layers, avoiding the additional power consumption loss caused by layer synthesis processing through the GPU. However, due to the insufficient processing power of the IWE module, the preprocessing time for the non-UBWC format transformation layers is too long, resulting in a long time consumed by the overall layer synthesis process, which causes the layer synthesis process to fail to release the fence on time, causing the display driver to wait for the fence to be released after the layer synthesis process ends, resulting in the display driver missing the screen hardware TE signal, that is, when the screen hardware TE signal arrives, it is impossible to refresh the synthesized layer to the screen by calling the display driver of the kernel layer, resulting in continuous frame loss in the video and obvious stuttering.

[0052] For ease of understanding, the following Figure 3 Let's take an example to illustrate the display pipeline process of another electronic device, and introduce the IWE module in the layer synthesis process to pre-process the non-UBWC format rotated layer. Take the electronic device displaying a video as an example, and play the video in full screen (that is, the original layer in the non-UBWC format will be rotated, mainly 90° rotation), the displayed video is a 4-frame video, and the content displayed by the electronic device corresponds to frame 1, frame 2, frame 3 and frame 4 in sequence according to the playback time sequence.

[0053] Taking the display of frame 1 as an example, the application of the electronic device draws frame 1 through the view system and renders frame 1 through the image rendering system. After the application of frame 1 is finished drawing and rendering, the application of the electronic device sends the drawn and rendered frame 1 to the image synthesis system. The image synthesis system preprocesses the non-UBWC format rotation layer through the IWE module. After the IWE module preprocesses, the layer synthesis is performed through the HWC. When the screen hardware TE signal has arrived and the synthesized frame 1 needs to be refreshed and displayed on the screen, the layer synthesis of frame 1 has not been completed. It is necessary to wait for the layer synthesis of frame 1 to be completed and release the fence, and wait for the next screen hardware TE signal to arrive before the synthesized frame 1 can be refreshed and displayed on the screen. However, when the next screen hardware TE signal arrives, the next frame (i.e., the synthesized frame 2) needs to be refreshed and displayed, resulting in frame loss. When the synthesized frame 1 is refreshed and displayed on the screen when the second screen hardware TE signal arrives, the display pipeline process of frame 2, frame 3 and frame 4 is performed in sequence, which will not be repeated here. Therefore, due to the insufficient processing capability of the introduced IWE module, the IWE module takes a long time to process the non-UBWC format rotated layers, resulting in continuous frame loss in the video, causing obvious stuttering and poor user experience.

[0054] The present application provides a layer synthesis method, which includes: obtaining data to be displayed; when the data to be displayed meets a first condition, generating a layer in UBWC format corresponding to the data to be displayed, the first condition is to transform the layer of the data to be displayed before display, perform a transformation operation on the layer in UBWC format to obtain a transformed layer in UBWC format, and perform layer synthesis on the transformed layer in UBWC format. When it is necessary to transform the layer of the data to be displayed before display, control the layer format corresponding to the generated data to be displayed to be in UBWC format, thereby avoiding the IWE module from preprocessing the transformed layer in non-UBWC format, thereby avoiding the continuous frame loss phenomenon caused by the long preprocessing time of the IWE module, and improving the smoothness of the display of the electronic device.

[0055] In some embodiments, the electronic device may be a mobile phone, a tablet computer, a desktop, a laptop, a notebook computer, an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a personal digital assistant (PDA), a wearable electronic device, a smart watch, etc. The present application does not impose any special restrictions on the specific form of the above electronic devices. In this embodiment, the structure of the electronic device may be as follows: Figure 4A As shown, Figure 4A A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0056] like Figure 4A As shown, the electronic device may include a processor 110, a display screen 120, a camera 130, an internal memory 140, and a sensor module 150, etc. The sensor module 150 may include a gyroscope sensor 150A, an acceleration sensor 150B, a touch sensor 150C, etc.

[0057] It is to be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0058] The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural-network processing unit (neural-network processing unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors. For example, in the present application, it is determined whether the data to be displayed meets the first condition. If the first condition is met, a layer in UBWC format corresponding to the data to be displayed is generated.

[0059] The controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0060] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0061] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0062] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 150C, the charger, the flash, the camera, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 150C through the I2C interface, so that the processor 110 communicates with the touch sensor 150C through the I2C bus interface to realize the touch function of the electronic device.

[0063] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 120 and the camera 130. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 130 communicate via the CSI interface to implement the shooting function of the electronic device. The processor 110 and the display screen 120 communicate via the DSI interface to implement the display function of the electronic device.

[0064] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 130, the display 120, the sensor module 150, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0065] It is understandable that the interface connection relationship between the modules illustrated in this embodiment is only a schematic illustration and does not constitute a structural limitation of the electronic device. In other embodiments of the present application, the electronic device may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0066] The electronic device implements the display function through a GPU, a display screen 120, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 120 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0067] The display screen 120 is used to display images, videos, etc. The display screen 120 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oled, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include 1 or N display screens 120, where N is a positive integer greater than 1.

[0068] A series of graphical user interfaces (GUIs) can be displayed on the display screen 120 of the electronic device, and these GUIs are the main screen of the electronic device. Generally speaking, the size of the display screen 120 of the electronic device is fixed, and only limited controls can be displayed on the display screen 120 of the electronic device. A control is a GUI element, which is a software component included in an application program, and controls all data processed by the application program and the interactive operations on these data. Users can interact with the control through direct manipulation to read or edit the relevant information of the application program. Generally speaking, a control can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. .

[0069] The electronic device can realize the shooting function through the ISP, the camera 130, the video codec, the GPU, the display screen 120 and the application processor.

[0070] The ISP is used to process the data fed back by the camera 130. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 130.

[0071] The camera 130 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device may include 1 or N cameras 130, where N is a positive integer greater than 1.

[0072] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals. For example, when an electronic device selects a frequency point, a digital signal processor is used to perform Fourier transform on the frequency point energy.

[0073] Video codecs are used to compress or decompress digital videos. Electronic devices can support one or more video codecs. In this way, electronic devices can play or record videos in multiple coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, UBWC, etc.

[0074] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and continuously self-learn. NPU can realize applications such as intelligent cognition of electronic devices, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0075] The internal memory 140 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 140. For example, in this embodiment, the processor 110 can determine whether the data to be displayed meets the first condition by executing the instructions stored in the internal memory 140, and when it is determined that the data to be displayed meets the first condition, a layer in UBWC format corresponding to the data to be displayed is generated. Among them, the storage program area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.). The storage data area can store data created during the use of the electronic device (such as audio data, a phone book, etc.). In addition, the internal memory 140 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in the internal memory 140, and / or the instructions stored in the memory provided in the processor.

[0076] The gyro sensor 150A can be used to determine the motion posture of the electronic device. In some embodiments, the angular velocity of the electronic device around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 150A. The gyro sensor 150A can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 150A detects the angle of the electronic device shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device through reverse movement to achieve anti-shake. The gyro sensor 150A can also be used for navigation and somatosensory game scenes.

[0077] The acceleration sensor 150B can detect the magnitude of the acceleration of the electronic device in all directions (generally three axes). When the electronic device is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0078] The touch sensor 150C is also called a "touch control device". The touch sensor 150C can be set on the display screen 120, and the touch sensor 150C and the display screen 120 form a touch screen, also called a "touch control screen". The touch sensor 150C is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 120. In other embodiments, the touch sensor 150C can also be set on the surface of the electronic device, which is different from the position of the display screen 120.

[0079] In addition, an operating system is running on the above components, such as the iOS operating system developed by Apple, the Android open source operating system developed by Google, and the Windows operating system developed by Microsoft. Application programs can be installed and run on the operating system.

[0080] The operating system of the electronic device may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device.

[0081] Figure 4B It is a software structure block diagram of the electronic device according to an embodiment of the present application.

[0082] The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: the application layer, the application framework layer (or framework layer), the Android runtime and system library, the hardware abstraction layer, and the kernel layer. For ease of explanation, Figure 4B The software structure diagram of the electronic device shown also includes Figure 4A The hardware structure in, such as display processing unit, screen, etc.

[0083] The application layer can include a series of application packages. Figure 4B As shown, the application package may include applications such as beauty camera, gallery, video, short message, etc.

[0084] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 4B As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0085] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0086] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0087] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0088] The phone manager is used to provide communication functions for electronic devices, such as the management of call status (including answering, hanging up, etc.).

[0089] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0090] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0091] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0092] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0093] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0094] The system library may include multiple functional modules, such as image synthesis system (surface flinger), surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0095] The image synthesis system is used to control image synthesis and generate vertical synchronization (Vsync) signals. Specifically, its main function is to receive graphics data buffers from multiple sources, synthesize them, and then send them to the display hardware (touch screen); at the same time, the image synthesis system is also responsible for error synchronization with the hardware HW-VSync signal and simulation of the software VSync signal, including SF-VSync and APP-VSync signals; the synthesizer flow control, signal simulator and connection manager collaborate to notify the VSync signal to the application and SurfaceFlinger itself.

[0096] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0097] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0098] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0099] A 2D graphics engine is a drawing engine for 2D drawings.

[0100] The hardware abstraction layer may include multiple library modules, such as hardware composer (HWC), camera library module, etc. The Android system may load the corresponding library module for the device hardware, thereby enabling the application framework layer to access the device hardware.

[0101] The hardware composer (HWC) mainly provides hardware support for the image synthesis system, supporting layer synthesis and display modules.

[0102] The kernel layer is the layer between hardware and software. The kernel layer includes at least the touch panel (TP) driver, display driver, Bluetooth driver, WIFI driver, keyboard driver, shared memory driver, ambient light sensor driver and camera driver.

[0103] The display driver can use the API to transmit instructions and data to the graphics processing unit (GPU) or dedicated graphics processing hardware and perform operations such as 3D rendering, video decoding, and GPU computing. The hardware may include display screens, touch hardware, processors, GPUs, etc.

[0104] It should be noted that although the embodiments of the present application are described using the Android system as an example, its basic principles are also applicable to electronic devices based on operating systems such as iOS and Windows.

[0105] Embodiment 1:

[0106] In order to compensate for the power consumption caused by the use of GPU by the DPU to process the non-UBWC format rotating layers, the IWE module is introduced on the DPU to pre-process the non-UBWC format rotating layers, and then synthesize them through HWC. Due to the insufficient processing power of the IWE module, the processing time of the non-UBWC format layers is long, which leads to continuous frame loss when the electronic device plays the video in full screen / plays the non-UBWC format rotating layers, causing a more obvious sense of frustration and a poor user experience.

[0107] The embodiment of the present application provides a layer synthesis method, which is for non-UBWC format layers, and performs transformation operations such as rotation and scaling on non-UBWC format layers. The processing performance of the IWE module is improved, thereby shortening the time consumed by the IWE module for preprocessing the non-UBWC format transformation layer, thereby improving the time consumed for layer synthesis of the non-UBWC format transformation layer, thereby improving the jamming problem caused by frame loss.

[0108] Specifically, by increasing the operating frequency of the DPU, the processing frequency of the IWE module is increased, and the time consumed by the IWE module to pre-process the transformation layer in the non-UBWC format is shortened by increasing the frequency. For example: In the current technology, the common operating frequencies of the DPU used in general electronic devices are 60Hz and 90Hz. In this scenario, increasing the operating frequency of the DPU to 120Hz can reduce the time consumed for pre-processing the transformation layer in the non-UBWC format from 9ms to 5ms, which can effectively reduce the continuous frame loss of the video, reduce the freeze of the video playback, and improve the user experience.

[0109] It should be noted that for the sake of convenience, the rotation layer is taken as an example. In addition, it can also be a layer processed by other graphic transformation operations, and this application does not make specific limitations.

[0110] For ease of understanding, the following Figure 5 The display pipeline process of an electronic device provided by an embodiment of the present application is described below. The DPU is frequency-increased accordingly to improve the processing performance of the IWE module. The electronic device plays a video in full screen as an example (that is, the layer will be rotated). The video is directly output as a non-UBWC format layer in the decoding stage, because the video / image processing algorithm can only perform image processing on non-UBWC format layers. The played video is a 4-frame video. According to the playback time sequence, the content displayed by the electronic device corresponds to frame 1, frame 2, frame 3 and frame 4, respectively.

[0111] Taking the display of frame 1 as an example, the application of the electronic device draws frame 1 through the view system and renders frame 1 through the image rendering system. After the application of frame 1 is finished drawing and rendering, the application of the electronic device sends the layer corresponding to the drawn and rendered frame 1 to the image synthesis system. The application of the electronic device performs image processing through a video / image processing algorithm or a third-party video / image processing algorithm, and the layer format corresponding to the drawn and rendered frame 1 is a non-UBWC format. The image synthesis system preprocesses the rotation layer in the non-UBWC format through the IWE module. After the IWE module preprocesses, the layer is synthesized through the HWC. After the frame 1 layer synthesis process is completed and the fence is released, the electronic device can refresh the synthesized frame 1 on the screen by calling the kernel layer display driver when the screen hardware TE signal arrives. Since the working frequency of the DPU is increased, the time consumed by the preprocessing of the IWE module is shortened. Therefore, when the screen hardware TE signal arrives and the synthesized frame 1 needs to be refreshed and displayed on the screen, the layer synthesis of frame 1 has been completed and the fence is released, and the frame 1 after the layer synthesis is completed is sent to the display driver of the kernel layer, and the frame 1 after the layer synthesis is refreshed and displayed on the screen when the display screen TE signal corresponding to frame 1 arrives. It should be noted that the display pipeline process of frame 2, frame 3 and frame 4 is similar to that of frame 1, which will not be repeated here.

[0112] The embodiment of the present application provides a layer synthesis method, which increases the operating frequency of the DPU and shortens the preprocessing time of the IWE module for the transformation layer in a non-UBWC format, thereby effectively improving the problem of continuous frame loss in the video and improving the problem of video playback freeze.

[0113] Embodiment 2:

[0114] By increasing the working frequency of the DPU, the preprocessing time of the IWE module for the transformed layers in non-UBWC format after the transformation is shortened, and the stuttering phenomenon caused by continuous frame loss in the video is improved. However, improving the working efficiency of the DPU cannot completely avoid the stuttering phenomenon caused by continuous frame loss in the video due to the long preprocessing time of the IWE module for the transformed layers in non-UBWC format.

[0115] With the rapid development of science and technology, image / video processing algorithms are mostly effective in scenarios where video to be played on electronic devices is processed. For example, the quality of the video to be played is enhanced, a beautification filter is added to the video to be played, etc. In order to meet the requirements of image enhancement and decode the output layer of the video to be played as a non-UBWC format layer, the non-UBWC format layer is rotated, scaled, and other transformation operations. The IWE module is used to pre-process the non-UBWC format transformation layer. Since the IWE module takes too long to pre-process the non-UBWC format transformation layer, it causes continuous frame loss and video freeze.

[0116] Combine the following Figure 6 , another layer synthesis method provided in an embodiment of the present application is introduced in detail.

[0117] S601: Obtain video data to be played.

[0118] The video data to be played may be downloaded from the network by an electronic device, or may be transmitted to the electronic device via an external memory, which is not specifically limited in this application.

[0119] S602: Determine whether the video data to be played needs to be played in rotation.

[0120] When the video data to be played needs to be rotated, for example, the video data to be played needs to be played in horizontal screen, it means that the corresponding layer of the decoded output of the video data to be played needs to be rotated before the corresponding layer synthesis process is performed.

[0121] When the video data to be played does not need to be rotated, for example, the video to be played is played in portrait mode, it means that the corresponding layer of the decoded output of the video data to be played does not need to be rotated, and the corresponding layer synthesis process is directly performed.

[0122] When the video data to be played needs to be played in rotation, S603 is performed; when the video data to be played does not need to be played in rotation, S606 is performed.

[0123] S603: Generate a corresponding UBWC format layer according to the video data to be played.

[0124] The video data to be played is decoded, and the corresponding application programming interface (Application Programming Interface, API) is called to output the corresponding UBWC format layer.

[0125] S604: Perform a rotation operation on the corresponding layer in UBWC format to obtain a corresponding rotated layer in UBWC format.

[0126] S605. Synthesize the corresponding UBWC format rotation layer through the hardware synthesizer. When the video data to be played needs to be rotated, if the generated layer format is not in UBWC format, after the non-UBWC format layer is transformed, it is necessary to use the IWE module to pre-process the non-UBWC format transformation layer before performing the layer synthesis process. In the current technology, the IWE module takes a long time to pre-process the non-UBWC format transformation layer, which causes continuous frame loss in the video and causes the video playback to freeze.

[0127] Therefore, no image processing is performed on the layer corresponding to the video data to be played, so a layer in UBWC format is generated according to the video data to be played, that is, the layer format of the decoded output of the video data to be played is guaranteed to be in UBWC format, and the DPU supports processing of the transformation layer in UBWC format, and there is no need to introduce the IWE module for preprocessing before the hardware synthesizer performs layer synthesis. Since no image processing is required for the video data to be played, a layer in UBWC format is generated according to the video to be played, and no image processing is performed on the corresponding layer in UBWC format, which will not affect the playback effect of the video to be played. After the transformation operation is performed on the layer in UBWC format, the layer synthesis process is directly performed, and there is no need to use the IWE module for preprocessing, thereby avoiding the preprocessing of the IWE module, thereby directly avoiding the problem of continuous frame loss of the video due to the long time consumed by the IWE module preprocessing.

[0128] S606: Generate a corresponding non-UBWC format layer according to the video data to be played.

[0129] The video data to be played is decoded, and the corresponding application programming interface (Application Programming Interface, API) is called to output the corresponding non-UBWC format layer.

[0130] S607 : Perform image processing on the corresponding non-UBWC format layer to obtain the corresponding non-UBWC format layer after image processing.

[0131] S608 , synthesizing the corresponding non-UBWC format layer after the image processing by a hardware synthesizer.

[0132] When the video data to be played does not need to be rotated, no matter whether the decoded layer format is UBWC format or non-UBWC format, it does not need to be preprocessed using the IWE module, so the long time consumed by the IWE module for preprocessing will not cause continuous video frame loss, thereby causing video playback jams. Therefore, the video data to be played can be image processed, such as video enhancement processing, video filter beautification processing, etc.

[0133] In one possible implementation, the following is performed: Figure 7 The process of a layer synthesis method is shown.

[0134] S701: Obtain video data to be played.

[0135] The video data to be played may be downloaded from the network by an electronic device, or may be transmitted to the electronic device via an external memory, which is not specifically limited in this application.

[0136] S702: Determine whether the layer needs to be rotated based on the shape data of the layer corresponding to the video to be played.

[0137] If the layer is a rectangular layer, determine whether the width of the corresponding layer is greater than the height. If the width of the corresponding layer is greater than the height, it means that the layer is a rotating layer, that is, the video to be played needs to be rotated; if the width of the corresponding layer is not greater than the height, it means that the layer is not a rotating layer, that is, the video to be played does not need to be rotated.

[0138] If yes, proceed to S703; if no, proceed to S706.

[0139] S703: Decode the video data to be played and output the corresponding layer in UBWC format.

[0140] When the layer corresponding to the video data to be played is a rotating layer, no image processing is performed on the video data to be played, and the layer format of the decoded output of the video to be played is kept in UBWC format. The DPU supports layer synthesis of rotating layers in UBWC format, and there is no need to introduce the IWE module for preprocessing. The layer synthesis process is directly performed, avoiding the preprocessing of the IWE module, thereby directly avoiding the problem of continuous video frame loss due to the long time consumed by the IWE module preprocessing.

[0141] S704 : Perform a rotation operation on the corresponding layer in UBWC format to obtain a corresponding rotated layer in UBWC format.

[0142] Since no image processing is allowed, the layer format of the decoded output is guaranteed to be UBWC format. After the UBWC format layer is rotated, there is no need to introduce the IWE module for preprocessing.

[0143] S705 : Decode the video data to be played, and output the corresponding layer in non-UBWC format.

[0144] When the layer corresponding to the video data to be played is not a rotating layer, image processing (for example: video enhancement processing) is allowed for the video data to be played, so the video data to be played is decoded and output as a layer in a non-UBWC format. Since the layer corresponding to the video to be played is not a rotating layer, the decoded output layer does not need to be rotated, and no matter whether the decoded layer format is UBWC format or non-UBWC format, it does not need to be preprocessed using the IWE module, so as not to cause continuous frame loss due to the long time consumed by the IWE module for preprocessing, thereby causing the video playback to freeze. However, in order to ensure that the corresponding image processing can be performed, the decoding and output of the video data to be played is controlled to correspond to the non-UBWC format layer.

[0145] S706: Perform image processing on the corresponding non-UBWC format layer to obtain the corresponding non-UBWC format layer after image processing.

[0146] S707. Perform layer synthesis through a hardware synthesizer.

[0147] A layer synthesis method provided by an embodiment of the present application is aimed at the scenario of using image processing to play the video to be played, and determines whether the video data to be played needs to be rotated. When the video data to be played does not need to be rotated, image processing is allowed for the video data to be played; when the video data to be played needs to be rotated, no image processing is allowed for the video to be played. When the video data to be played does not need to be rotated, no matter whether the layer format of the decoded output is UBWC format or non-UBWC format, the IWE module will not be referenced for preprocessing; when the video data to be played needs to be rotated, for the rotating layer in non-UBWC format, the IWE module needs to be referenced for preprocessing, so no image processing is allowed, and the decoded output is a layer in UBWC format. After the layer in UBWC format is rotated, the layer synthesis is directly performed without the need for preprocessing by the IWE module. It can be seen from this that by isolating the video playback scene, the preprocessing of the IWE module is avoided, thereby avoiding the continuous frame loss caused by the long time consumed by the IWE module preprocessing, and improving the smoothness of video playback.

[0148] Embodiment three:

[0149] For the video playback scenario of the electronic device, it can be selectively determined whether to perform image processing on the video to be played. In addition, there is a scenario in which the electronic device cannot avoid the use of image processing, and the layer transformation operation must be performed in this scenario, that is, it must involve IWE module preprocessing, for example: the preview scenario of the camera filter / special effect. Since the installation direction of the camera of the electronic device and the screen display direction of the electronic device are different, if the image taken by the camera is to be previewed in real time, it must involve the corresponding layer rotation. At the same time, the preview of the camera filter / special effect must be accompanied by image processing. Therefore, for such scenarios, it is impossible to avoid the layer transformation operation, and it is also impossible to avoid image processing. For such scenarios, the embodiment of the present application provides another layer synthesis method.

[0150] Combine the following Figure 8 , another layer synthesis method provided in an embodiment of the present application is introduced in detail.

[0151] Generate a layer in a non-UBWC format according to input data (data to be displayed), perform image processing on the layer in the non-UBWC format to obtain a layer in a non-UBWC format after image processing, and perform format conversion before subjecting the layer in the non-UBWC format after image processing to a layer in a UBWC format after image processing; perform transformation operations such as rotation and scaling on the layer in the UBWC format after image processing to obtain a transformed layer in a UBWC format after image processing; the image synthesis system directly uses HWC to synthesize the rotated layer in the UBWC format to complete the layer synthesis process. Before the layer synthesis process is performed, the non-UBWC format layer format output by image processing is converted into a UBWC format layer, and then the UBWC format layer is rotated, scaled and other transformation operations are performed, and finally the transformed layer in UBWC format is synthesized. Since the format is converted to UBWC format, the synthesis of the rotated layer does not need to be pre-processed using the IWE module, thereby avoiding the long processing time caused by the insufficient performance of the IWE module, resulting in continuous frame loss.

[0152] In the scenario where image processing cannot be avoided and layer transformation operations must exist, the output non-UBWC format layer is transformed into a UBWC format layer, and then the UBWC format layer is transformed and then the transformed layer in UBWC format is synthesized. The use of the IWE module for preprocessing is avoided, and the format conversion is located in the application side process, which will not affect the time consumed by the layer synthesis process. Since the IWE module is avoided for preprocessing, the continuous frame loss caused by the long time consumed by the insufficient preprocessing performance of the IWE module is avoided, and the smoothness of the electronic device display is improved.

[0153] Embodiment 4:

[0154] For the layer synthesis of non-UBWC format rotation layers, considering various application scenarios, in order to solve the continuous frame loss problem caused by the long time consumed by the IWE module preprocessing, combined with Fig. 9 Another layer synthesis method provided in an embodiment of the present application is introduced in detail.

[0155] S901: Obtain data to be displayed.

[0156] It may be generated by an application or downloaded from the Internet. This application does not specifically limit the source of the data to be displayed.

[0157] S902: Determine whether the application that obtains the data to be displayed is in the whitelist.

[0158] The image processing algorithm can be applied to multiple applications, and an application in the whitelist indicates that the application uses the image processing algorithm. For example, if the whitelist of the image enhancement algorithm is application A and application B, it means that application A and application B run the image enhancement algorithm.

[0159] If yes, proceed to S903; if no, proceed to S908

[0160] S903: Determine whether the data to be displayed meets the first condition.

[0161] The first condition is to transform the layer of the data to be displayed before displaying it, that is, to determine whether the data to be displayed needs to be transformed before displaying it, and the transformation operation includes: rotation operation, scaling operation, etc.

[0162] Specifically, whether the data to be displayed meets the first condition is determined based on the shape data of the layer corresponding to the data to be displayed, and the shape data of the layer includes the length of the layer and the width of the layer. When the length of the layer corresponding to the data to be displayed is greater than the width of the layer corresponding to the data to be displayed, the data to be displayed meets the first condition; when the length of the layer corresponding to the data to be displayed is less than or equal to the width of the layer corresponding to the data to be displayed, the data to be displayed does not meet the first condition.

[0163] If yes, proceed to S904; if no, proceed to S906.

[0164] S904: Generate a layer in UBWC format corresponding to the data to be displayed.

[0165] S905 , performing a transformation operation on the layer in the UBWC format to obtain a transformed layer in the UBWC format.

[0166] When the layer of the data to be displayed needs to be transformed before display, the layer format corresponding to the generated data to be displayed is controlled to be in UBWC format, thereby avoiding the IWE module from preprocessing the transformed layer in a non-UBWC format, thereby avoiding the continuous frame loss phenomenon caused by the long preprocessing time of the IWE module, and improving the smoothness of the display of the electronic device.

[0167] S906: Generate a layer in a non-UBWC format corresponding to the data to be displayed.

[0168] S907 , performing image processing on the layer in the non-UBWC format to obtain the layer in the non-UBWC format after image processing.

[0169] When the layer of the data to be displayed does not need to be transformed before display, the layer transformation operation is not performed, and the layer in non-UBWC format is directly generated. After image processing is performed on the layer in non-UBWC format, layer synthesis is performed on the layer in non-UBWC format after image processing. This also avoids the IWE module from preprocessing the transformation layer in non-UBWC format, thereby avoiding the continuous frame loss caused by the long preprocessing time of the IWE module, and improving the smoothness of the display of electronic equipment.

[0170] S908: Generate a layer in a non-UBWC format corresponding to the data to be displayed.

[0171] S909: Perform transformation operation processing on the layer in the non-UBWC format to obtain a corresponding transformation layer in the non-UBWC format.

[0172] S910: Preprocess the transformation layer in a non-UBWC format by using the IWE module that increases the processing frequency.

[0173] The IWE module with increased processing frequency is used to preprocess the transformation layer in non-UBWC format; the preprocessed transformation layer in non-UBWC format is synthesized. When the preprocessing of the IWE module cannot be avoided, the processing frequency of the IWE module is increased to shorten the time consumed by the IWE module preprocessing, thereby reducing the continuous frame loss phenomenon and improving the smoothness of the electronic device display.

[0174] S911. Perform layer synthesis.

[0175] In a possible implementation, after acquiring the data to be displayed, a non-UBWC format layer corresponding to the data to be displayed is generated; image processing is performed on the non-UBWC format layer to obtain a non-UBWC format layer after image processing; format conversion is performed on the non-UBWC format layer after image processing to obtain a corresponding UBWC format layer after image processing; transformation operation is performed on the UBWC format layer after image processing to obtain a transformed layer in the UBWC format after image processing; and layer synthesis is performed on the transformed layer in the UBWC format after image processing.

[0176] In the case where image processing and transformation operations cannot be avoided, the format of the layer is converted to avoid the preprocessing of the IWE module, thereby reducing the continuous frame loss caused by the long preprocessing time of the IWE module and improving the smoothness of the electronic device display.

[0177] The embodiment of the present application provides a layer synthesis method, which, under certain conditions, avoids the preprocessing of the IWE module, thereby avoiding the continuous frame loss phenomenon caused by the long preprocessing time of the IWE module, thereby improving the smoothness of the electronic device display. In the case where the preprocessing of the IWE module cannot be avoided, the processing efficiency of the IWE module is improved, thereby reducing the time consumed by the preprocessing of the IWE module, and improving the smoothness of the electronic device display.

[0178] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is executed, the electronic device implements the above-mentioned functions or steps.

[0179] An embodiment of the present application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, enables an electronic device to implement the above-mentioned functions or steps.

[0180] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A layer synthesis method, characterized in that: Applied to electronic equipment, the method comprises: Get the data to be displayed; When the data to be displayed meets a first condition, a layer in UBWC format corresponding to the data to be displayed is generated; the first condition is that the layer of the data to be displayed is transformed before display; the transforming of the layer of the data to be displayed includes: rotating or scaling the layer of the data to be displayed; Performing a transformation operation on the layer in the UBWC format to obtain a transformation layer in the UBWC format; The transformation layer in the UBWC format is layer-synthesized.

2. The method according to claim 1, characterized in that The data to be displayed meets the first condition, including: According to the shape data of the layer corresponding to the data to be displayed, judging whether the data to be displayed meets the first condition; the shape data of the layer includes: the length of the layer and the width of the layer; When the length of the layer corresponding to the data to be displayed is greater than the width of the layer corresponding to the data to be displayed, the data to be displayed meets the first condition.

3. The method according to claim 1, characterized in that The method further comprises: When the data to be displayed does not satisfy the first condition, generating a layer in a non-UBWC format corresponding to the data to be displayed; The non-UBWC format layer is layer-synthesized.

4. The method according to claim 3, characterized in that After generating the layer in the non-UBWC format corresponding to the data to be displayed, the method further includes: Performing image processing on the non-UBWC format layer to obtain a non-UBWC format layer after image processing; The synthesizing the non-UBWC format layers includes: synthesizing the non-UBWC format layers after the image processing.

5. The method according to claim 3 or 4, characterized in that: The data to be displayed does not satisfy the first condition, including: According to the shape data of the layer corresponding to the data to be displayed, judging whether the data to be displayed meets the first condition; the shape data of the layer includes: the length of the layer and the width of the layer; When the length of the layer corresponding to the data to be displayed is less than or equal to the width of the layer corresponding to the data to be displayed, the data to be displayed does not meet the first condition.

6. The method according to claim 1, characterized in that After obtaining the data to be displayed, the method further includes: Generating a layer in a non-UBWC format corresponding to the data to be displayed; Performing image processing on the non-UBWC format layer to obtain a non-UBWC format layer after image processing; Performing format conversion on the non-UBWC format layer after the image processing to obtain a corresponding UBWC format layer after the image processing; Performing a transformation operation on the layer in the UBWC format after the image processing to obtain a transformation layer in the UBWC format after the image processing; The transformed layers in the UBWC format after the image processing are layer-synthesized.

7. The method according to claim 1, characterized in that After obtaining the data to be displayed, the method further includes: Determine whether the application that obtains the data to be displayed is in the whitelist; When the application for acquiring the data to be displayed is in the whitelist, it is determined whether the data to be displayed meets the first condition.

8. The method according to claim 7, characterized in that The electronic device comprises a display processing unit DPU, and the DPU is provided with an IWE module; Increasing the operating frequency of the DPU to increase the processing frequency of the IWE module, the method further comprising: When the application for acquiring the data to be displayed is not in the whitelist, generating a layer in a non-UBWC format corresponding to the data to be displayed; Performing a transformation operation on the non-UBWC format layer to obtain a corresponding non-UBWC format transformation layer; Preprocessing the non-UBWC format transformation layer using the IWE module that increases the processing frequency; The pre-processed non-UBWC format transformation layer is layer-composited.

9. An electronic device, characterized in that: including a processor and a memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program or instruction, and when the computer program or instruction is executed, the method according to any one of claims 1 to 8 is implemented.

Images