A system on chip (SoC), a rendering method of a user interface, and an electronic device

Abstract

A system on chip (SoC), a rendering method of a user interface and an electronic device, the SoC comprising a graphic processing unit (GPU) and a display subsystem; wherein the GPU is configured to perform image rendering to obtain at least one layer; the display subsystem is configured to generate a control, and superimpose the control and the at least one layer to obtain a first image, the first image being displayed on a display as a user interface, and the control is configured to realize human-computer interaction. Thus, the control is generated by the display subsystem instead of the GPU, thereby reducing the calculation workload of the GPU, reducing the power consumption of the GPU, and improving the rendering efficiency of the user interface. Moreover, the requirement of the calculation power of the GPU can be reduced, as long as the calculation power of the GPU meets the calculation power required for image rendering, the rendering efficiency of the user interface will not be affected, thereby preventing the user interface from appearing to be stuck or not smooth, and ensuring the experience of the user using the electronic device.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a system-on-a-chip (SoC), a user interface rendering method, and an electronic device. Background Technology

[0002] With the development of terminal (or electronic device) technology, users can interact with terminals through the terminal's user interface (UI) (which can be simply referred to as human-computer interaction). The content of this user interface that enables human-computer interaction generally includes the display screen shown to the user (such as a portrait) and controls for user operation (such as mobile phone navigation keys). Therefore, the rendering of the user interface can be divided into image rendering for the display screen and control rendering for the controls.

[0003] Currently, user interfaces are generally generated by graphics processing units (GPUs). This means the GPU performs not only image rendering but also control rendering. Because both image and control rendering rely on the GPU, its computing power can easily affect the rendering efficiency of the user interface, potentially leading to stuttering and choppy visuals, thus impacting the user experience. Furthermore, the GPU executes a large number of instructions during image and control rendering, which can cause the device to overheat due to increased GPU power consumption. Summary of the Invention

[0004] This application provides a system-on-a-chip (SoC), a user interface rendering method, and an electronic device to reduce the computational workload of the GPU, lower the GPU's power consumption, and improve the rendering efficiency of the user interface.

[0005] In a first aspect, embodiments of this application provide a system-on-a-chip (SoC) applied to an electronic device. The SoC includes a graphics processing unit (GPU) and a display subsystem. The GPU is used to perform image rendering to obtain at least one layer. The display subsystem is used to generate controls and overlay the controls and at least one layer to obtain a first image. The first image is displayed on a display as a user interface, and the controls are used to implement human-computer interaction.

[0006] In the aforementioned SoC, image rendering is performed by the GPU, and control rendering is performed by the display subsystem. Image rendering refers to generating at least one layer required for the display screen, while control rendering refers to generating controls. The display subsystem then overlays the controls and at least one layer to obtain the first image to be displayed; that is, the first image contains controls. Since the first image is displayed as the user interface, it can also be said that the user interface contains controls, thus enabling human-computer interaction. Therefore, this embodiment of the application renders user interface controls through the display subsystem, rather than the GPU, thereby reducing the GPU's computational workload, lowering GPU power consumption, improving user interface rendering efficiency, and preventing overheating issues in electronic devices. Furthermore, it reduces the GPU's computing power requirements; as long as the GPU's computing power is sufficient for image rendering, it will not affect the user interface rendering efficiency, thus preventing screen stuttering and unresponsiveness, and ensuring a good user experience.

[0007] Optionally, the aforementioned SoC also includes a central processing unit (CPU), which is connected to the GPU and the display subsystem. The CPU sends a first signal to the GPU, instructing it to perform image rendering. This causes the GPU to render the image, obtaining at least one layer of a first image, and storing the at least one layer of the first image in a preset buffer. Then, the GPU sends corresponding response information back to the CPU, indicating that image rendering is complete. After receiving the response information, the CPU sends a second signal to the display subsystem, instructing it to generate a first image containing controls. This causes the display subsystem to retrieve at least one layer from the preset buffer.

[0008] One possible implementation is that the display subsystem is specifically used to: generate controls based on control properties, which include at least one of the following: the position of the control, the transparency of the control, the contrast of the control with at least one layer, and the shape and size of the control.

[0009] In this implementation, control properties can be configured by the system or by the user; this embodiment of the application does not impose any restrictions. Optionally, control properties can be associated with the application type.

[0010] One possible implementation is that the control's position includes a first position in landscape mode and a second position in portrait mode.

[0011] In one possible implementation, when overlaying controls and at least one layer, the display subsystem is also used to: blur the image of a target area of ​​at least one layer, the target area including the position of the control.

[0012] One possible implementation is that the control is displayed on the screen in real time as a user interface, and the control includes at least one of the following: an AI function entry point and a jump interface button.

[0013] One possible implementation is that the display subsystem includes a layer overlay and a control drawer; wherein the layer overlay is used to overlay at least one layer to obtain a first layer; the control drawer is used to draw controls and overlay the controls and the first layer to obtain a first image.

[0014] Secondly, embodiments of this application provide an electronic device, which includes a System-on-Chip (SoC) and a memory as described in the first aspect above. The SoC's image processor (GPU) is used for image rendering to obtain at least one layer; the SoC's display subsystem is used to generate controls and overlay the controls and at least one layer to obtain a first image. The first image is displayed on a display as a user interface, and the controls are used to implement human-computer interaction. The memory is used to store at least one layer, the controls, and the first image.

[0015] Thirdly, embodiments of this application provide a user interface rendering method, which is applied to the System-on-a-Chip (SoC) described in the first aspect above. The method includes: a graphics processing unit (GPU) within the SoC performing image rendering to obtain at least one layer; a display subsystem within the SoC generating controls and overlaying the controls and the at least one layer to obtain a first image; wherein the first image is displayed on a display as a user interface, and the controls are used to implement human-computer interaction.

[0016] One possible implementation involves generating a control, which includes: generating the control based on control properties, the control properties including at least one of the following: the position of the control, the transparency of the control, the contrast between the control and the at least one layer, and the shape and size of the control.

[0017] One possible implementation, when overlaying controls and at least one layer, further includes: blurring a target area of ​​at least one layer, the target area including the position of the control.

[0018] Fourthly, embodiments of this application provide an apparatus comprising: one or more processors configured to perform the method described in any one of the third aspects above.

[0019] In one possible implementation, the apparatus further includes one or more memories; wherein the one or more memories store one or more programs that, when executed by the one or more processors, cause the apparatus to perform the method described in any one of the third aspects above.

[0020] Fifthly, a non-volatile computer-readable storage medium is provided, the non-volatile computer-readable storage medium including a program that, when the program is run on a device, causes the device to perform the method described in any one of the third aspects.

[0021] Sixthly, embodiments of this application provide a computer program product that, when run on a device, causes the device to perform the method described in any one of the third aspects above.

[0022] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0023] Figure 1 A schematic diagram of a control in a mobile phone user interface provided in an embodiment of this application;

[0024] Figure 2 A schematic diagram illustrating the generation of a user interface provided in an embodiment of this application;

[0025] Figure 3 This is a schematic diagram of the application interface of a game application provided in an embodiment of this application;

[0026] Figure 4 This is a schematic diagram of the internal hardware structure of an electronic device provided in an embodiment of this application;

[0027] Figure 5 This is a schematic diagram of a layered software structure of an electronic device provided in an embodiment of this application;

[0028] Figure 6 This application provides a schematic diagram of the structure of a System-on-Chip (SoC) according to an embodiment of the present application.

[0029] Figure 7 A flowchart illustrating a user interface rendering method provided in an embodiment of this application;

[0030] Figure 8 A schematic diagram of the structure of a display subsystem (DSS) provided in an embodiment of this application;

[0031] Figure 9 A flowchart illustrating another user interface rendering method provided in this application embodiment;

[0032] Figure 10 This is a flowchart illustrating another user interface rendering method provided in an embodiment of this application. Detailed Implementation

[0033] To better illustrate the content provided in the embodiments of this application, the concepts and terms involved in the embodiments of this application will be briefly explained first.

[0034] (1) User Interface (UI)

[0035] The user interface (UI) is the medium through which electronic devices interact with users (human-computer interaction). Taking a mobile phone as an example, the UI can include the phone's system interface and the application interfaces (apps) on the phone. The system interface displays the application icons, while the application interfaces display the application content (such as the content of a video application). Generally, based on user actions, users can directly jump from the application interface to an interface within the system interface; this interface is usually called the main interface.

[0036] (2) Layers

[0037] Layers consist of multiple pixels (the number is generally related to the screen resolution of the terminal), and typically, the size of a layer is the same as the size of the terminal screen. Optionally, multiple layers can be stacked to form a complete image. A layer can contain text, images, tables, plugins, and can also nest other layers.

[0038] (3) Image rendering

[0039] Image rendering refers to the process of converting a graphical interface description structure or rendering instructions into a two-dimensional image.

[0040] With the development of terminal (i.e. electronic device) technology, users can interact with electronic devices through their user interfaces. The content of such user interfaces typically includes the screen displayed to the user (such as a portrait of a person in a video application; to distinguish it from controls, we will refer to it as the screen) and controls for user operation (such as navigation keys on a mobile phone).

[0041] Taking mobile phones as an example, Figure 1 This is a schematic diagram of controls in a mobile phone user interface provided in an embodiment of this application. (Reference) Figure 1 , Figure 1 Figure a shows the phone's system interface, which includes icons for multiple applications, as well as navigation keys 1 (which can be called the multitasking key), 2 (which can be called the home key), and 3 (which can be called the back key). When navigation key 1 is triggered, the phone performs the operation of "bringing up to the interface of the app accessed within the preset time up to the current moment"; when navigation key 2 is triggered, the phone performs the operation of "jumping to the home screen"; and when navigation key 3 is triggered, the phone performs the operation of "returning to the previous screen".

[0042] Figure 1 Image b shows the image interface of the Picture app on a phone in portrait mode, with the navigation bar located at the bottom of the screen. Figure 1 Image c shows the paused video interface of a video application in landscape mode, with a navigation bar at the bottom. When the navigation bar is triggered, the Artificial Intelligence (AI) assistant is activated, and the phone displays AI function entry points. There can be one or more AI function entry points, and AI functions include, but are not limited to, at least one of the following: scanning, search, and voice input functions.

[0043] In some embodiments, the controls in the user interface can also serve as entry points for AI functions.

[0044] For example, users can activate the control by clicking, double-clicking, swiping (such as swiping left, right, up, or down), long-pressing, large-area gestures, and hovering touch.

[0045] In related technologies, in order to better display controls in the user interface, controls generally have the following display effects:

[0046] 1. The area where the control is located in the user interface has contrast with other areas of the user interface (or the displayed screen); where contrast can be understood as the degree of color difference.

[0047] 2. The control has transparency, meaning that the transparency of the control needs to be set.

[0048] 3. The brightness of the controls in the user interface is greater than the brightness of the screen displayed in the user interface, that is, the saturation of the controls needs to be increased.

[0049] In some embodiments, the displayed screen is generally obtained by overlaying at least one layer, and controls are generated and added to the displayed screen based on at least one layer. Therefore, the rendering of the user interface can be divided into image rendering for the displayed screen (to obtain at least one layer) and control rendering. Image rendering refers to generating at least one rendered layer required for the displayed screen, while control rendering refers to generating controls, or drawing controls. Here, the calculation process for image rendering is not limited in this embodiment; the process for generating controls is described below.

[0050] Taking a navigation bar as an example, the process of generating a navigation bar may include the following steps:

[0051] S2.1: Overlay all the layers (obtained after image rendering) required to generate the display screen to obtain the overlaid layer A.

[0052] S2.2: In layer A, calculate the average color of the target area, which is the area in layer A covered by the navigation bar.

[0053] S2.3: Calculate the color of the navigation bar based on the average color. This application embodiment does not limit the calculation method of the navigation bar color.

[0054] S2.4: Draw a navigation bar with transparency based on the color of the navigation bar.

[0055] After generating the controls, the user interface can be generated based on layer A and the navigation bar. Figure 2 This is a schematic diagram illustrating the generation of a user interface according to an embodiment of this application. (Reference) Figure 2 By overlaying layer A and the navigation bar, an image is created to represent the user interface, incorporating the navigation bar effect. It can be understood that over a period of time, the user interface is generally formed by sequentially displaying multiple consecutive frames of images; that is, each frame serves as the user interface. Assuming that the user interface consistently contains controls throughout this period, it means that each frame contains controls.

[0056] In some embodiments, the above description can apply to an application interface. The following uses a mobile phone as an example to illustrate the process of generating an image displayed as a user interface.

[0057] The mobile phone application, for a frame of image to be displayed as the application interface (generally the application's interface), submits a rendering task to the rendering service (RS) through the application programming interface (API) of the navigation bar provided by the mobile phone system (OS). The rendering task is used to instruct the execution of the above S2.1-S2.4 process and to generate the frame image. The rendering task includes the data required to execute the above S2.1-S2.4, such as the layers of the frame image (generally multiple layers).

[0058] After receiving a rendering task, the rendering service executes the process described in S2.1-S2.4 based on the data included in the rendering task to generate controls, and then overlays the layers and controls to obtain this frame image, and then stores this frame image in the frame buffer.

[0059] The rendering service calls the display service, which then retrieves the frame image from the buffer and sends it to the phone screen so that the phone screen can display the frame image.

[0060] To better illustrate the process of generating images for display as a user interface, we will further describe it using different scenarios below.

[0061] Scenario 1: Video application playing video.

[0062] In this scenario, the image needed to generate the control could be the frame where the video is paused, or it could be any frame of the video; there is no restriction. Because the video is pre-encoded and its images do not contain controls, the phone needs to decode the video to obtain the image layer corresponding to the video frame (for clarity, this can be called the video layer). Optionally, for the paused video frame, the GPU can be used for image rendering to obtain other layers of that image (for clarity, this can be called UI layers, such as progress bars or pause indicators).

[0063] Taking a paused image as an example, the video application submits a rendering task to the rendering service for that frame. The rendering service then retrieves the video layer and UI layer of that frame from the corresponding buffer, and generates controls based on the video and UI layers described in steps S2.1-S2.4. Finally, the rendering service overlays the layers and controls of that frame to regenerate a new image containing the controls. Optionally, the video decoder (Vdec) in the phone decodes the video to obtain the image layers and stores them in the corresponding buffer.

[0064] Scenario 2: Displaying the application interface in a custom-drawn interface application (such as a game application).

[0065] In this scenario, taking mobile games as an example, Figure 3 This is a schematic diagram of the application interface of a game application provided in an embodiment of this application. (Reference) Figure 3 To enable users to activate the AI ​​assistant via the navigation bar when entering, exiting, or pausing the game, the rendering service generates an image containing the navigation bar after receiving a rendering task from the game application (assuming the rendering task is triggered when the game is paused). It's important to note that mobile game applications typically use 3D rendering engines such as Unity or Unreal Engine, rather than directly using the UI libraries provided by the mobile system. Therefore, the GPU is required for image rendering to obtain the image layers corresponding to the game screen (for clarity, we can refer to this as game screen layers).

[0066] Taking a paused game frame as an example, the game application submits a rendering task to the rendering service for that frame. The rendering service can then generate a navigation bar based on the game screen layer of that frame, executing steps S2.1-S2.4 above. Finally, it overlays the game screen layer and the navigation bar to regenerate the frame containing the navigation bar (e.g.,...). Figure 3 (Image containing a navigation bar).

[0067] In the above process, rendering services are generally performed by the graphics processing unit (GPU) in the phone's System on Chip (SoC). That is, the GPU not only performs image rendering but also control rendering. Because image and control rendering rely on the GPU, the GPU's computing power can easily affect the efficiency of generating images displayed in the user interface, i.e., it affects the rendering efficiency of the user interface. This can lead to stuttering and unresponsiveness in the user interface, negatively impacting the user experience of using the electronic device.

[0068] Furthermore, as seen in Scenario 1, when the GPU generates controls for each frame of image, it needs to retrieve the image layer to be processed from the cache. This undoubtedly increases the GPU's bandwidth and power consumption, leading to complex and time-consuming user interface rendering processes, thus reducing the rendering efficiency of the user interface. Scenario 2 shows that the GPU needs to perform image rendering and control generation, which significantly increases its power consumption. This could potentially cause electronic devices to overheat due to increased GPU power consumption.

[0069] To this end, this application provides a system-on-a-chip (SoC), a user interface rendering method, and an electronic device. The SoC includes a display subsystem (DSS), through which controls are generated, thereby reducing the computational workload of the GPU, reducing the power consumption of the GPU, and improving the rendering efficiency of the user interface.

[0070] It should be understood that in the embodiments of this application, the character " / " generally indicates that the preceding and following objects are in an "or" relationship. Words such as "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance, nor should they be construed as indicating or implying order.

[0071] In some embodiments of this application, the electronic device generally has a camera function, as well as other functions such as a personal digital assistant and / or music player function. For example, the electronic device may be a mobile phone, a tablet computer, or a wearable device with wireless communication capabilities (such as a smartwatch). Exemplary embodiments of the electronic device are equipped with operating systems including, but not limited to, iOS, Android, Microsoft, or other operating systems. The aforementioned electronic device may also be other electronic devices such as a camera with data processing capabilities; this application does not limit the specific form of the electronic device.

[0072] Figure 4 This is a schematic diagram of the internal hardware structure of an electronic device provided in an embodiment of this application. (See also...) Figure 4 The electronic device can execute the methods provided in the embodiments of this application. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0073] Processor 110 may include one or more processing units. For example, processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, a display processing unit (DPU), and / or a neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors. In some embodiments, electronic device 100 may also include one or more processors 110. The processor is the nerve center and command center of electronic device 100. The processor can generate operation control signals according to instruction opcodes and timing signals to control instruction fetching and execution. Processor 110 may also include memory for storing instructions and data.

[0074] In some embodiments, the processor 110 may include one or more interfaces. These interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc. The USB interface 130 is a USB standard-compliant interface, specifically a Mini USB interface, a Micro USB interface, a USB Type-C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used for data transfer between the electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback.

[0075] The sensor module 180 may include one or more of the following: a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and a gravity sensor. Optionally, in this embodiment, the sensor can detect the unfolding angle of the foldable display screen, so that the processor 110 can trigger the execution of user interface animations based on changes in the unfolding angle. Optionally, the sensor used to detect the unfolding angle of the foldable display screen may include a Hall effect sensor, a load sensor, an infrared sensor, a pressure sensor, or an electromagnetic sensor, etc. Optionally, in this embodiment, the sensor (such as a gravity sensor) can also detect the usage mode of the foldable display screen (e.g., including a vertical split-screen usage mode and a horizontal split-screen usage mode).

[0076] The charging management module 140 receives charging input from the charger. The power management module 141 connects to the battery 142, and the charging management module 140 connects to the processor 110. The wireless communication function of the electronic device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor, etc.

[0077] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0078] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110.

[0079] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0080] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLAN) (such as Wi-Fi), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR). The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signal, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0081] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. Electronic device 100 can implement shooting functions through ISP, camera 193, video codec, GPU, display screen 194, and application processor, etc.

[0082] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0083] The electronic device 100 can realize display functions through a GPU, a display screen 194, and an application processor (AP). The display screen 194 is used to display images, videos, etc. In this embodiment, the display screen 194 is a foldable display screen.

[0084] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0085] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0086] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses a layered architecture software system (such as Android). Taking a system as an example, the software structure of an electronic device is illustrated.

[0087] Figure 5 This is a schematic diagram of a layered software structure for an electronic device provided in an embodiment of this application. (Reference) Figure 5 A layered architecture divides the software system of an electronic device into several layers, each with its own role and function. Layers communicate with each other through software interfaces. Furthermore, the system structure is divided into four layers: applications 210, application framework 220, Driver Development Kit (DDK) layer / Hardware Abstraction Layer (HAL) 230, and driver layer 240. It should be understood that... Figure 5 A hardware layer 250 for electronic devices was added on top of the software system.

[0088] It should be understood that Figure 5 The modules included in each layer shown are those involved in the embodiments of this application. The modules included in each layer below do not constitute a limitation on the structure of the electronic device and the hierarchy of module deployment (example illustration). Figure 5 The modules shown can be deployed individually, or several modules can be deployed together. Figure 5 The module division and module names shown are just one example.

[0089] The application layer may include a series of application packages. The application layer may include applications such as camera, contacts, call logs, gallery, calendar, calling (video and / or voice calls), map, navigation, Bluetooth, music, video, and SMS. Applications may include system applications and third-party applications. Only the first application 211 and the second application 212 are shown as examples in the figure. For example, the first application 211 may be a game application, and the second application 212 may be an application assistant. An application assistant is client management software that can be installed on an electronic device. The application assistant provided in this embodiment of the application has the function of enabling scenario-based resource control.

[0090] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications within the application layer. The application framework layer can include some predefined functions. (See reference...) Figure 5 The application framework layer may include: a rendering service module 221, a window manager service (WMS) module 222, and other modules (such as a display service module and a Bluetooth module). Among them, the rendering service module 221 is used to render images to obtain layers, and the window manager service module 222 is used to manage the user interface.

[0091] The DDK layer / HAL230 is an interface layer located between the operating system kernel and the hardware circuitry, used to abstract the hardware. The DDK layer / HAL230 may include: GPU DDK231, HAL232 of the display subsystem, and other DDK / HALs.

[0092] The driver layer 240 may include system runtime and system libraries, as well as functional modules from the kernel layer. (See reference...) Figure 5 The driver layer 240 may include drivers for driving hardware operations. For example, a display driver 241, a sensor driver 242, a GPU driver 243, etc., but this application embodiment does not limit this.

[0093] Hardware layer 250 includes the hardware in electronic devices. This is understandable. Figure 5 The hardware layer shown in the following embodiments includes a display screen 251, a memory 252, a system-on-a-chip (SoC) 253, etc. The memory 252 stores the instructions and data required for the SoC 253 to run, as well as the data generated during the operation of the SoC 253.

[0094] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the software structure of electronic devices.

[0095] Furthermore, Figure 6 This is a schematic diagram of a System-on-Chip (SoC) provided as an embodiment of this application. (Reference) Figure 6 A System-on-a-Chip (SoC) can include: a central processing unit (CPU), a GPU, a display subsystem (DSS), a video decoder (Vdec), a system cache (such as CPU cache), and random access memory (RAM). An SoC can connect to external memory (also called internal memory or main memory) and other hardware units (such as a video encoder and baseband processor). The hardware units within an SoC (such as CPU, GPU, DSS, and Vdec) are connected via a bus. This bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The bold line indicates that there is only one bus or one type of bus.

[0096] The CPU can be used to run applications and, based on the application's rendering requests, send image rendering tasks to the GPU and control rendering tasks to the DSS.

[0097] The GPU is used for image rendering, obtaining image layers and storing the image layers in a preset frame buffer. The frame buffer is generally located in memory, that is, the frame buffer is a part of the memory space.

[0098] An image can have one or more layers, and the number of layers is not limited in this embodiment.

[0099] The video decoder is used to decode video and obtain image layers.

[0100] A Display Controller (DSS) is a hardware unit that performs display processing within a chip, also known as a display processor. In this embodiment, the DSS is used to generate controls and overlay controls and layers (including layers obtained from GPU row image rendering and / or layers obtained from video decoding by a video decoder) to obtain the image to be displayed (i.e., the image used as the user interface).

[0101] The system cache is used to store data and instructions that are frequently accessed by the CPU.

[0102] Memory is used to store the instructions and data required to perform image rendering and control rendering tasks, such as video images and layers of images to be generated.

[0103] Taking the layers obtained from GPU image rendering as an example, the user interface rendering process can include the following:

[0104] The CPU receives a rendering request initiated by the application and sends a first signal to the GPU. This rendering request is used to request the rendering of the application's interface, which contains controls. The first signal is used to indicate that image rendering should be performed.

[0105] The GPU receives and responds to the first signal, performs image rendering to obtain at least one layer of the image, and stores the rendered at least one layer of the image in the frame buffer. Then, the GPU sends corresponding response information back to the CPU, which indicates that image rendering is complete.

[0106] After receiving the response information, the CPU sends a second signal to the DSS, which instructs the generation of an image containing controls.

[0107] After receiving the second signal, the DSS retrieves at least one layer of the image from the frame buffer and generates controls. It then overlays the controls and at least one layer to obtain the image displayed as the user interface. Finally, the DSS sends the image to a display (such as the screen of an electronic device) so that the display shows the image.

[0108] In this embodiment, the DSS can generate controls in the following ways. The controls include at least one of the following: an AI function entry point and a jump interface button. The jump interface button can be... Figure 1 For a detailed description of the navigation keys or navigation bar shown above, please refer to the above. Figure 1 The contents of this application will not be repeated here.

[0109] Method 1 for generating controls: Generate controls directly based on control properties; where control properties may include: control position (such as the coordinates of the control's center position, the coordinates of the control's vertex position), control transparency, control color, control shape and size, and control brightness.

[0110] In Method 1, control properties can be user-configurable. This means the generated controls are immutable, such as their color. Therefore, this method can be executed only once, and the resulting controls are applied to every image used as a user interface.

[0111] Method 2 for generating controls: Generate controls based on some or all layers of at least one layer of the image, and configured control properties; wherein, control properties may include: the position of the control, the transparency of the control, the contrast of the control with at least one layer, and the shape and size of the control.

[0112] Optionally, taking all layers in at least one layer as an example, the control is generated by referring to the process S2.1-S2.4 above. It can be understood that in method 2, for each image, a corresponding control is generated based on the contrast of all layers. This ensures that the contrast of the control in each image is fixed compared to other layers, thus better displaying the control. Below, this application embodiment describes the user interface rendering method using method 2.

[0113] based on Figure 6 To better illustrate the functions of DSS, the following flowchart describes the process of generating the user interface using DSS. Figure 7 This is a flowchart illustrating a user interface rendering method provided in an embodiment of this application. (Reference) Figure 7 The process includes:

[0114] Step 701: The image processor (GPU) renders the image to obtain at least one layer. This at least one layer comprises all the layers required to generate the first image.

[0115] In this process, the first image refers to any frame image that serves as the user interface. Image rendering can be implemented using techniques and algorithms known in the art, and this application embodiment is not limited thereto.

[0116] Step 702: The GPU stores at least one layer in the cache / memory.

[0117] Step 703: The display subsystem retrieves at least one layer from the cache / memory.

[0118] In this step, the at least one layer may include the video layer obtained by video decoding by the video decoder in scenario 1 above.

[0119] Step 704: The display subsystem generates controls and overlays the controls and at least one layer to obtain a first image; wherein the first image is displayed on the monitor as a user interface, and the controls are used to implement human-computer interaction.

[0120] After obtaining the first image, the display subsystem will send the first image to the display so that the display can show the first image.

[0121] In one possible implementation, for scenario 1 above, the display subsystem can retrieve at least one layer (i.e., the video layer in scenario 1) from the cache / memory, and then execute step 704 to obtain a second image, which refers to an image containing controls and the video layer, such as a frame image when the video is playing.

[0122] Furthermore, Figure 8 This is a schematic diagram of the structure of a display subsystem (DSS) provided in an embodiment of this application. (Reference) Figure 8 The DSS may include the following hardware: a layer overlay unit, a control drawer, a data reader, an image transmitter, and other layer processors. These other layer processors include, but are not limited to, a High Dynamic Range Imaging (HDRI or HDR) unit, a layer scaler, and a color space converter (CSC). It should be noted that the types and number of hardware components included in the DSS in this embodiment are not limited. The functions of each hardware component in the DSS are described below.

[0123] A layer overlay operator is used to overlay at least one layer of a first image to obtain a first layer (e.g., ...). Figure 2 Layer A shown in the image.

[0124] The control drawer is used to draw the control based on its properties and the contrast of the first layer, and then overlay the first layer and the control to obtain the first image.

[0125] A data reader is used to receive a second signal and, in response to the second signal, acquire at least one layer of the first image, and then send the at least one layer to the layer overlay unit. Optionally, the data reader can be a Direct Memory Access (DMA) device to ensure efficient data reading.

[0126] An image transmitter is a display interface on the SoC side, used to send the first image to the display interface on the monitor. The display interface includes, but is not limited to: Mobile Industry Processor Interface (MIPI), Display Serial Interface (DSI) defined by MIPI, Display Port (DP) defined by Vesa, and High Definition Multimedia Interface (HDMI), etc., and this application embodiment is not limited thereto. Based on this, after receiving the first image through the corresponding display interface, the monitor displays the first image.

[0127] HDR is used to process HDR effects for at least one layer.

[0128] Scaler is used to handle scaling of at least one layer.

[0129] CSC is used to handle gamut conversion of at least one layer.

[0130] Optionally, the DSS may also include a post-rendering processor for processing the first image. For example, the post-rendering processor includes a colorimeter and a color temperature controller; wherein the colorimeter is used to adjust the color accuracy of the first image, and the color temperature controller is used to adjust the color temperature of the first image.

[0131] It is understood that the DSS may also include more hardware for processing images, and this application embodiment does not limit this.

[0132] based on Figure 8 The following describes the interaction methods of various hardware components in DSS in further detail. Figure 7 The process. Figure 9 This is a flowchart illustrating another user interface rendering method provided in an embodiment of this application. (See reference...) Figure 9 The process includes:

[0133] Step 901: The data reader sends at least one layer of the first image to the layer overlay.

[0134] In this step, at least one layer of the first image can be obtained after image rendering by the GPU in the SoC. For example, the GPU obtains a Render Tree, which includes layer information of the image to be displayed (taking the first image as an example), and then the GPU performs image rendering based on the Render Tree to obtain at least one layer. Alternatively, at least one layer can be obtained after video decoding by the video decoder in the SoC. Or, at least one image includes a layer obtained after image rendering by the GPU and a layer obtained after video decoding by the video decoder.

[0135] Then, the GPU can traverse all nodes in the rendering tree except for those corresponding to the control information. These other nodes record the layer information of the first image. The GPU then renders the image from these other nodes to obtain at least one layer of the first image. The GPU then stores at least one layer of the first image in one or more preset buffers (for ease of distinction, we can call them layer buffers).

[0136] In summary, before performing this step, the data reader receives a second signal from the CPU, and then reads at least one layer of the first image from the layer buffer. After reading at least one layer of the first image, it sends at least one layer of the first image to the layer overlay unit.

[0137] Step 902: The layer overlay tool overlays at least one layer to obtain the first layer.

[0138] In this step, the layer overlay can overlay at least one layer in a bottom-to-top order to obtain a first overlay layer. It should be noted that this embodiment does not limit the layer overlay method; the layer overlay can also be called an overlay device.

[0139] In one possible implementation, the layer overlay overlays a portion of at least one layer to obtain a first layer.

[0140] In one possible implementation, the layer overlay operator overlays all layers in at least one layer to obtain a first layer. It should be noted that this embodiment is described using this implementation as an example.

[0141] Step 903: The layer overlay sends the first layer to the control drawer.

[0142] Step 904: The control painter generates controls and overlays the controls and the first layer to obtain the first image.

[0143] Referring to method 2 above, the control drawer generates a control based on the first layer and control attributes; wherein, the control attributes may include: the position of the control, the transparency of the control, the contrast between the control and at least one layer (i.e., the first layer), and the shape and size of the control. For example, the control attributes may also include brightness, but this embodiment of the application does not limit this.

[0144] In one possible implementation, the shape of the control can be a right rectangle, a rounded rectangle, a circle, or a heart shape, and the size of the control can be preset by the user; this application embodiment does not limit this.

[0145] In this embodiment, the control attributes can be pre-configured to the DSS. Optionally, the control attributes can be system-configured or user-configured. In one possible implementation, the control attributes are associated with the application type; for example, for a game application, the control's position is indicated at the top of the application interface, and for a video application, the control's position is indicated at the bottom of the application interface.

[0146] In one possible implementation, the control renderer is also configured with the display state of the monitor, which can include landscape and portrait modes. The landscape mode is associated with a first position of the control, and the portrait mode is associated with a second position. In other words, the control's position includes a first position in landscape mode and a second position in portrait mode. For example, the first position indicates that the control is located on the longer side of the phone screen, and the second position indicates that the control is located on the shorter side of the phone screen.

[0147] In other words, the position of a control can be determined by the control renderer based on the screen display state. Specifically, before overlaying the control and at least one layer, if the control renderer determines that the screen is in landscape mode, it determines the control's position as the first position; or, if the control renderer determines that the screen is in portrait mode, it determines the control's position as the second position.

[0148] It can be understood that the position of a control is used to indicate its location within the first layer, such as... Figure 2 The location of the navigation code. Therefore, the control renderer can determine the target area of ​​the control in the first layer based on the control's position, shape, and size. In other words, the target area in the first layer includes the control's position.

[0149] After identifying the target area in the first layer, the control renderer can calculate the average color of the target area in the first layer. Optionally, the average color can be calculated based on the RGB values ​​of each pixel in the target area.

[0150] After determining the average color, the control renderer can calculate the control's color based on the average color of the target area in the first layer. Optionally, the control renderer can invert the color of the target area in the first layer to obtain the control's color. Inversion refers to completely reversing the color, making the original color its opposite. For example, red becomes green, blue becomes orange, and black becomes white, used to enhance the contrast between the control and the first layer. Optionally, the control renderer can calculate the control's color based on the average color of the target area in the first layer and the contrast between the control and at least one layer in the control properties.

[0151] In summary, the control renderer can draw the controls corresponding to the first layer based on the control's transparency, color, shape, and size. Then, the control renderer overlays (or blends) the controls onto the target area in the first layer to obtain the first image. It should be noted that the blending method between the controls and the first layer can be preset, and this embodiment does not limit it.

[0152] In one possible implementation, when the control renderer overlays a control onto a target area in at least one layer, the control renderer can also perform image blurring on the target area in the first layer so that the control can display the blurred content of the layers below. The image blurring techniques (also known as image blurring algorithms) include, but are not limited to, Gaussian blur, radial blur, and directional blur, and are not limited here.

[0153] In one possible implementation, when the control renderer overlays the control onto the target area in at least one layer, the control renderer can also determine the brightness of the control based on the brightness of the monitor, so as to increase the brightness of the control.

[0154] Optionally, the control drawer can also be called a control calculator or an AI Bar calculator.

[0155] Step 905: The image sender receives the first image sent by the control drawer.

[0156] Step 906: The image transmitter sends the first image to the display interface of the monitor.

[0157] In summary, this application embodiment generates controls through the Display Subsystem (DSS), thereby reducing the workload and power consumption of the GPU, improving the rendering efficiency of the user interface, and preventing overheating issues in electronic devices. Furthermore, it reduces the GPU's computing power requirements; as long as the GPU's computing power is sufficient for image rendering, it will not affect the rendering efficiency of the user interface, thus preventing screen stuttering and unresponsiveness, and ensuring a smooth user experience.

[0158] In addition, the Display Subsystem (DSS) can perform other processing on the layers. For example, before overlaying at least one layer, it can scale at least one layer, complete HDR effects, and perform color gamut conversion. After obtaining the first image, it can adjust the color accuracy and color temperature of the first image.

[0159] It's important to note that the hardware components in a DSS can be understood as intellectual property cores (IP cores) in an integrated circuit. Furthermore, the operation of these hardware components is software-driven and configured. After receiving and responding to instructions, the data reader in the DSS reads data (including at least one layer), and then sends this data to other hardware components within the DSS. These other hardware components then process the data accordingly to obtain the image. In other words, the other hardware components in the DSS do not need to receive and respond to instructions; they only need to process the data. The advantage of this approach is that it reduces the consumption of communication resources and improves computational efficiency.

[0160] Based on the same inventive concept Figure 10 This is a flowchart illustrating another user interface rendering method provided in an embodiment of this application. (See reference...) Figure 10 The process includes:

[0161] Step 1001: The graphics processing unit (GPU) within the SoC performs image rendering to obtain at least one layer.

[0162] Step 1002: The display subsystem within the SoC generates controls and overlays the controls and at least one layer to obtain a first image; wherein the first image is displayed on the display as a user interface, and the controls are used to implement human-computer interaction.

[0163] One possible implementation involves generating the control, which includes: generating the control based on control properties, the control properties including at least one of the following: the position of the control, the transparency of the control, the contrast of the control with at least one layer, and the shape and size of the control.

[0164] One possible implementation, when overlaying controls and at least one layer, further includes: blurring a target area of ​​at least one layer, the target area including the position of the control.

[0165] also, Figure 10 Other implementations of the method shown can be found in the above. Figure 9 The relevant descriptions in the document will not be repeated here.

[0166] Based on the above embodiments, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program (also referred to as code or instructions), which, when executed by a computer, causes the computer to perform the methods described in the embodiments of this application.

[0167] Based on the above embodiments, this application also provides a chip. The chip is used to read a computer program stored in a memory and implement the methods described in the embodiments of this application.

[0168] Based on the above embodiments, this application provides a chip system including a processor for supporting a computer device in implementing the methods described in the embodiments of this application. In one possible embodiment, the chip system further includes a memory for storing programs and data necessary for the computer device. The chip system may include one or more chips, or it may include chips and other discrete devices.

[0169] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0170] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0171] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0172] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0173] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope of protection of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A System-on-Chip (SoC), characterized in that, include: A graphics processing unit (GPU) is used to render images to obtain at least one layer. A display subsystem is used to generate controls and overlay the controls and at least one layer to obtain a first image; wherein the first image is displayed on the display as a user interface, and the controls are used to implement human-computer interaction.

2. The SoC according to claim 1, characterized in that, The display subsystem is specifically used for: The control is generated based on control properties, which include at least one of the following: the position of the control, the transparency of the control, the contrast between the control and the at least one layer, and the shape and size of the control.

3. The SoC according to claim 2, characterized in that, The position of the control includes a first position in landscape mode and a second position in portrait mode.

4. The SoC according to any one of claims 1-3, characterized in that, When the control and the at least one layer are superimposed, the display subsystem is further configured to: Image blurring is applied to the target area of ​​the at least one layer, the target area including the position of the control.

5. The SoC according to any one of claims 1-4, characterized in that, The control is displayed on the monitor in real time as the user interface, and the control includes at least one of the following: an AI function entry point and a jump interface button.

6. The SoC according to any one of claims 1-5, characterized in that, The display subsystem includes: A layer overlay tool is used to overlay the at least one layer to obtain a first layer; A control drawer is used to draw the control and overlay the control and the first layer to obtain the first image.

7. An electronic device, characterized in that, Includes the system-on-a-chip (SoC) and memory as described in any one of claims 1-6.

8. A method for rendering a user interface, characterized in that, The method is applied to a system-on-a-chip (SoC), and the method includes: The graphics processing unit (GPU) within the SoC performs image rendering to obtain at least one layer; The display subsystem within the SoC generates controls and overlays the controls and the at least one layer to obtain a first image; wherein the first image is displayed on the display as a user interface, and the controls are used to implement human-computer interaction.

9. The method according to claim 8, characterized in that, The generated control includes: The control is generated based on control properties, which include at least one of the following: the position of the control, the transparency of the control, the contrast between the control and the at least one layer, and the shape and size of the control.

10. The method according to claim 8 or 9, characterized in that, When overlaying the control and the at least one layer, the method further includes: Image blurring is applied to the target area of ​​the at least one layer, the target area including the position of the control.

11. A readable storage medium, characterized in that, Includes a computer program that, when run on a system-on-a-chip (SoC), causes the SoC to perform the method as described in any one of claims 8-10.

Images