Display method and display system based on operating system

Abstract

The present application discloses a display method and a display system based on an operating system. The operating system includes an application layer, a hardware abstraction layer and a hardware driver layer. The hardware abstraction layer is configured with a fast display management module and a synthetic display module. The display method includes: an application of the application layer sends a data display instruction and display data to the fast display management module of the hardware abstraction layer, and the fast display management module is used to provide an interface for a variety of applications that require fast display; the fast display management module responds to the data display instruction and sends the data display instruction and display data to the synthetic display module of the hardware abstraction layer; the synthetic display module responds to the data display instruction and calls the driver hardware of the hardware driver layer to perform hardware display. The display method of the present application can achieve fast display.

Description

Technical Field

[0001] The present invention relates to the field of system architecture, and in particular to a display method and a display system of an operating system. Background Art

[0002] With the widespread use of electronic devices in people's daily lives, the display of electronic devices is becoming more and more important. In many scenarios, there are high demands for display, such as fast display, multi-layer simultaneous display, and multi-screen display.

[0003] Current solutions either focus on fast display but have simple functions and cannot meet various display requirements; or they provide diversified displays but sacrifice a lot of startup time and cannot achieve fast display. Summary of the invention

[0004] The present application provides a display method and a display system based on an operating system to solve the problem of insufficient display function in the prior art.

[0005] To solve the above technical problems, the present application proposes a display method based on an operating system, wherein the operating system includes an application layer, a hardware abstraction layer and a hardware driver layer, wherein the hardware abstraction layer is configured with a fast display management module and a synthetic display module, and the display method includes: an application program of the application layer sends a data display instruction and display data to the fast display management module of the hardware abstraction layer, wherein the fast display management module is used to provide an interface for multiple application programs; the fast display management module responds to the data display instruction and sends the data display instruction and display data to the synthetic display module of the hardware abstraction layer; the synthetic display module responds to the data display instruction and calls the driver hardware of the hardware driver layer for hardware display.

[0006] To solve the above technical problems, the present application proposes a display system, including an application layer; a hardware abstraction layer, the hardware abstraction layer is configured with a fast display management module and a synthetic display module; the fast display management module is used to provide an interface for a variety of applications that require fast display; a hardware driver layer; wherein the application of the application layer sends data display instructions and display data to the fast display management module; the fast display management module is used to respond to data display instructions and send data display instructions and display data to the synthetic display module; the synthetic display module is used to respond to data display instructions and call the driver hardware of the hardware driver layer for hardware display.

[0007] The operating system of the present application includes an application layer, a hardware abstraction layer, and a hardware driver layer. In the present application, a fast display management module and a synthetic display module are configured in the hardware abstraction layer. The fast display management module can provide interfaces for a variety of applications. The application of the application layer sends display instructions and display data directly to the fast display management module, which is forwarded by the fast display management module to the synthetic display module, and the synthetic display module then calls the hardware driver layer for display. In the present application, a fast display interface is specially set up so that the application layer can directly transmit data with the lower hardware abstraction layer, which can improve the display speed; and the fast display interface set up can provide interfaces for a variety of applications, thereby realizing a variety of displays. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a flowchart of the first embodiment of the display method based on the operating system of the present application;

[0009] Figure 2 It is the architecture diagram of the display system of this application;

[0010] Figure 3 yes Figure 1 A schematic diagram of a specific flow chart of a first embodiment of the display method is shown;

[0011] Figure 4 It is a flowchart of the second embodiment of the display method based on the operating system of the present application;

[0012] Figure 5 yes Figure 2 The architecture diagram of the composite display module in the display system shown;

[0013] Figure 6 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0014] Figure 7 It is a structural diagram of an embodiment of the computer storage medium of the present application. DETAILED DESCRIPTION

[0015] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be appreciated that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the art without making creative work are within the scope of protection of the present application.

[0016] This application is used to implement the display in the operating system. The operating system architecture includes the application layer (App), application framework layer (Framework), hardware abstraction layer (HAL) and hardware driver layer (Device Driver) from top to bottom. The process of the operating system to implement the display mainly lies in the transmission process of data and instructions between the various layers in the system architecture. For each layer of the operating system, the bottom layer is biased towards hardware, with faster startup speed, and the high layer is biased towards software, with more diversified functions.

[0017] Based on this operating system, this application implements the display method, please refer to Figure 1 , Figure 1 It is a flowchart of the first embodiment of the display method based on the operating system of the present application. The operating system for implementing the display method is the display system proposed in the present application. In the display system of this embodiment, a fast display management module (Fast Display Service) and a composite display module (Composer) are configured in the hardware abstraction layer (HAL).

[0018] The composite display module (Composer) provides an external interface, but it mainly serves the application framework layer (Framework). For the application, it is very complicated to call. Therefore, a fast display management module (Fast Display Service) is designed in this embodiment. As an external interface module, it is used to provide an interface for multiple applications. The application can directly interact with the fast display management module (Fast Display Service), that is, bypassing the application framework layer (Framework), thereby improving the data transmission speed and then improving the display efficiency.

[0019] The display method of this embodiment specifically includes the following steps.

[0020] S11: The application program of the application program layer sends a data display instruction and display data to the fast display management module of the hardware abstraction layer.

[0021] The application program of this embodiment is a program that needs to be displayed, and can be applied to various scenes that need to be displayed. Taking the application of intelligent driving as an example, the application program can be an intelligent driving application program, a reversing image program, a fatigue driving warning system (DMS), an panoramic monitoring image system (AVM), etc.

[0022] The process of realizing display includes control flow and data flow, which respectively involve the flow of data display instructions and display data. In this step, the application sends the data display instruction to the Fast Display Service, that is, calls the Fast Display Service to start the display function. The application also transmits the display data to the Fast Display Service.

[0023] In this step, the sending of the data display instruction and the display data may be sequential and may be implemented in different processes.

[0024] S12: The fast display management module responds to the data display instruction and sends the data display instruction and the display data to the synthetic display module of the hardware abstraction layer.

[0025] The fast display management module (Fast Display Service) responds to the data display instruction, that is, after being called to start executing the display function, it sends the data display instruction to the synthetic display module (Composer), that is, calls the synthetic display module (Composer) and sends the display task to the synthetic display module (Composer).

[0026] And the fast display management module (Fast Display Service) sends the display data to the composite display module (Composer).

[0027] S13: The synthesis display module responds to the data display instruction and calls the driver hardware of the hardware driver layer to perform hardware display.

[0028] The composite display module (Composer) can directly call the driver hardware in the hardware driver layer (Device Driver) to display the display data for hardware display. Specifically, the processor can be called to realize the display, and the processor is, for example, DPU, GPU, etc.

[0029] This embodiment configures a fast display management module (Fast DisplayService) in the hardware abstraction layer (HAL) to provide interfaces for various application programs, so that display data can be quickly transmitted to the composite display module (Composer) to achieve display. This embodiment improves the display speed.

[0030] The fast display management module (Fast Display Service) of this embodiment specifically includes an external interface (HidlInterface) and an implementation module (Hidl Implementation), and the implementation module (Hidl Implementation) includes a Display Surface submodule, a BufferQueue submodule, and a HwcHal submodule. For the specific architecture of the operating system of this embodiment, please refer to Figure 2 , Figure 2 It is an architectural diagram of the display system of this application.

[0031] Among them, the external interface (Hidl Interface) encapsulates multiple external interfaces for multiple applications to call, and is used to create a fast display buffer, transfer display data, and set display parameters.

[0032] The implementation module (Hidl Implementation) is used to receive the display data and display parameters transmitted by the application, and call the composite display module (Composer) to realize the display.

[0033] The Display Surface submodule is used to enable the application to obtain an empty display buffer and receive the display buffer passed by the application; it is used to receive and process the application settings or obtain display parameters; it is also used to perform corresponding processing for the show and hide of the application.

[0034] The BufferQueue submodule is responsible for the allocation and management of display buffers. GraphicBuffer dynamically loads the vendor's gralloc module to implement the allocation and release of display buffers. Since the application and the composite display module (Composer) use the display buffer in different processes, in order to reduce data copying in the display buffer and improve data transmission efficiency, the BufferQueue submodule uses slot indexes to bind different display buffers, and only the slot indexes are transmitted between different processes without transferring the buffer itself.

[0035] The HwcHal submodule is used to implement unified processing of front-end data, call the standard hidl interface of the back-end composite display module (Composer), and pass the display data to the composite display module (Composer) for processing.

[0036] Based on this, it can be seen that the specific process of the above display method embodiment can be referred to Figure 3 , Figure 3 yes Figure 1The specific flow chart of the first embodiment of the display method is as follows.

[0037] S21: The application sends a data display instruction to the fast display management module.

[0038] The application first sends a data display instruction to the fast display management module (Fast Display Service). If there are multiple applications that need to be displayed, multiple data display instructions are sent accordingly.

[0039] S22: The fast display management module creates a display buffer in response to the data display instruction.

[0040] The external interface (Hidl Interface) in the fast display management module (Fast Display Service) creates a display buffer. Corresponding to multiple data display instructions, multiple display buffers can be created. In order to improve data transmission efficiency, an index directory is established for the display buffer, such as a slot directory. In the subsequent display buffer data transmission, only the index directory needs to be transmitted.

[0041] S23: The application obtains the display buffer, and fills the display buffer with display data, and sends the display buffer filled with display data to the fast display management module.

[0042] As mentioned above, after the application fills the display data into the display buffer, the application may transmit the index directory to the fast display management module (Fast Display Service) so that the fast display management module (Fast Display Service) can obtain the display buffer filled with the display data.

[0043] S24: The rapid display management module sends the data display instruction and the display data to the synthetic display module.

[0044] Similarly, the fast display management module (Fast Display Service) can directly send the data display instruction and display data to the composite display module (Composer), and also send the index directory to the composite display module (Composer) so that it can obtain the display data.

[0045] S25: The synthesis display module responds to the data display instruction and calls the driver hardware of the hardware driver layer to perform hardware display.

[0046] This embodiment also provides an architecture implementation of a fast display management module (Fast Display Service), as well as a specific control flow and data transfer process based on the architecture. The fast display management module (Fast Display Service) is applied to the display of specific data content, such as the display of video streams.

[0047] This application can also further realize the integrated display of the UI effect and data of the application. In the UI interface, operations such as switching views and configuring display angles can be realized to improve the display effect. Figure 4 , Figure 4 It is a flowchart of the second embodiment of the display method based on the operating system of the present application.

[0048] The application framework layer (Framework) of the operating system of this embodiment is configured with a SurfaceFlinger module, which is used for sending and displaying the UI. The display method of this embodiment is used to realize the sending and displaying of UI data and the sending and displaying of display data in the first embodiment above, and specifically includes the following steps.

[0049] S31: The application sends a data display instruction and display data to the fast display management module, and the application sends a UI display instruction and UI data to the SurfaceFlinger module.

[0050] In this step, the application not only has the display requirements of video data but also the display requirements of UI data. For the display data, as in the first embodiment, the display is realized by calling the Fast Display Service module; for the UI data, the display is realized by calling the SurfaceFlinger module. The display data and UI data are also transmitted to these two modules respectively.

[0051] S32: The fast display management module sends the data display instruction and the display data to the synthetic display module of the hardware abstraction layer.

[0052] This step is similar to the related steps in the first embodiment and will not be described in detail.

[0053] S33: The SurfaceFlinger module sends the UI display instruction and UI data to the synthetic display module.

[0054] The SurfaceFlinger module calls the Composer module to display the UI data. Since the Composer module needs to synthesize the display data and the UI data, when the SurfaceFlinger module transmits data to the Composer module, it is necessary to add an application identifier to the UI data. This can be an identifier agreed with the application or an identifier of the application itself. This is to facilitate the Composer module to identify and better overlay the UI data on the display data.

[0055] S34: The synthesis display module synthesizes the display data and the UI data and then calls the driver hardware to perform hardware display.

[0056] Since the present embodiment can realize the display of multiple applications, for accurate synthesis, the synthesis display module (Composer) synthesizes the display data and UI data having the same application identifier.

[0057] The composite display module (Composer) needs to receive data transmitted by two channels respectively, so the architecture of the composite display module (Composer) is optimized in this embodiment. Figure 5 , Figure 5 yes Figure 2 The architecture diagram of the composite display module in the display system is shown.

[0058] The composite display module (Composer) specifically includes a ComposerClient submodule and a HwComposer submodule.

[0059] The ComposerClient submodule is used to create a Client. When creating a new Client, there is no need to destroy the previous client, allowing the creation of multiple clients.

[0060] When the HwComposer submodule returns the callback to the Client, it does not return it only to the Surfaceflinger module. In this embodiment, the callback is distributed to each client separately so that each client can receive the corresponding callback.

[0061] The HwComposer submodule includes the HwComposer library implemented by various vendors based on their hardware capabilities, which can handle the display requirements of two clients from the Surfaceflinger module and the Fast Display Service module. It processes the display of the Fast Display Service module before and after the Surfaceflinger module is started, and after receiving the display requirements sent by the two clients, it reasonably allocates resources according to the hardware capabilities for synthesis and display.

[0062] In the display process of this step S34, the ComposerClient submodule creates two Clients, the two Clients correspond to the Fast Display Management Module (Fast Display Service) and the SurfaceFlinger module respectively, and the two Clients send the display data and UI data to the HwComposer submodule respectively.

[0063] The HwComposer submodule returns callbacks to the two Clients respectively, synthesizes the display data and the UI data, and calls the driver hardware for display.

[0064] In this embodiment, a fast display management module (Fast Display Service) is added to the hardware abstraction layer (HAL), which provides a simple and easy-to-use external interface. The application layer (App) only needs to communicate with the fast display management module (Fast Display Service).

[0065] In addition, the present application also modifies the composite display module (Composer) to support multiple clients for display. The Fast Display Service and the SurfaceFlinger module of the application framework layer (Framework) are two clients that transmit display data and UI data to the composite display module (Composer) respectively. The composite display module (Composer) composites the display data and UI data for display.

[0066] It realizes the rapid display of display data and seamless overlay with UI data, improving display speed and diversity.

[0067] The above method is applied to an electronic device, and its logical process is represented by a computer program. When a computer program is implemented in the form of software and sold or used as an independent product, it can be stored in a storage medium that can be read by an electronic device.

[0068] Therefore, the present application also provides an electronic device, please refer to Figure 6 , Figure 6 2 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 200 of the present embodiment includes a processor 21 and a memory 22. The processor 21 is coupled to the memory 22, and the memory 22 stores a computer program. The processor 21 can load and execute the computer program to implement the above display method.

[0069] The electronic device of this embodiment is based on an operating system, which can be applied to intelligent driving, and can be a car computer to realize the display of automatic parking, DMS, AVM and other intelligent driving.

[0070] The present invention also provides a computer storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented. Figure 7 , Figure 7 It is a schematic diagram of the structure of an embodiment of a computer storage medium of the present invention. The computer storage medium 300 contains program data that can be executed to implement the method of the above embodiment. The computer storage medium can be, for example, a USB flash drive, a CD, a server, etc.

[0071] The above description is only an implementation mode of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A display method based on an operating system, characterized in that: The operating system includes an application layer, a hardware abstraction layer and a hardware driver layer, the hardware abstraction layer is configured with a fast display management module and a synthetic display module, and the display method includes: The application program of the application program layer sends data display instructions and display data to the fast display management module of the hardware abstraction layer, and the fast display management module is used to provide an interface for various application programs that require fast display; The fast display management module sends the data display instruction and the display data to the synthesis display module of the hardware abstraction layer in response to the data display instruction; The synthetic display module responds to the data display instruction by calling the driver hardware of the hardware driver layer to perform hardware display; The operating system further includes an application framework layer, and the application framework layer is configured with a SurfaceFlinger module; the display method further includes: The application sends UI display instructions and UI data to the SurfaceFlinger module; The SurfaceFlinger module sends the UI display instruction and UI data to the synthetic display module; The synthesis display module synthesizes the display data and the UI data and then calls the driving hardware to perform hardware display; The synthesis display module includes a ComposerClient submodule and a HwComposer submodule. The synthesis display module synthesizes the display data and the UI data and then calls the driver hardware for hardware display, including: The ComposerClient submodule creates at least two Clients respectively, the fast display management module and the SurfaceFlinger module correspond to one Client respectively, and the two Clients corresponding to the fast display management module and the SurfaceFlinger module send display data and UI data to the HwComposer submodule respectively; The HwComposer submodule returns callbacks to the two Clients corresponding to the fast display management module and the SurfaceFlinger module respectively, synthesizes the display data and UI data, and calls the driver hardware for display.

2. The display method according to claim 1, characterized in that: The application program of the application program layer sends data display instructions and display data to the fast display management module of the hardware abstraction layer; The fast display management module sends the data display instruction and the display data to the synthesis display module of the hardware abstraction layer in response to the data display instruction, including: The application sends a data display instruction to the fast display management module; The fast display management module creates a display buffer in response to the data display instruction; The application program obtains the display buffer, and fills the display buffer with the display data, so as to send the display buffer filled with the display data to the fast display management module; The fast display management module sends the data display instruction and the display data to the synthetic display module.

3. The display method according to claim 2, characterized in that: The fast display management module includes an external interface and an implementation module, and the implementation module includes a Display Surface submodule, a BufferQueue submodule and a HwcHal submodule; The external interface is used to create the display buffer; The Display Surface submodule is used to receive a display buffer filled with the display data; The BufferQueue submodule is used to allocate display buffer for the application; The HwcHal submodule is used to send the display data to the synthetic display module.

4. The display method according to claim 1, characterized in that: The application program of the application program layer sends data display instructions and display data to the fast display management module of the hardware abstraction layer; The fast display management module sends the data display instruction and the display data to the synthesis display module of the hardware abstraction layer in response to the data display instruction, including: The application sends a data display instruction to the fast display management module; The fast display management module creates a display buffer in response to the data display instruction and establishes an index directory for the display buffer; The application obtains the index directory to obtain the display buffer, and fills the display buffer with the display data; The application sends the index directory to the fast display management module, so that the fast display management module obtains a display buffer filled with the display data; The fast display management module sends the index directory to the synthesis display module, so that the synthesis display module obtains the display data.

5. The display method according to claim 1, characterized in that: The synthesis display module synthesizes the display data and the UI data and then calls the driving hardware to perform hardware display, including: The synthesis display module synthesizes the display data and UI data having the same application program identifier, and calls the driving hardware to perform hardware display.

6. The display method according to claim 1, characterized in that: The applications include smart driving applications.

7. A display system, characterized in that: The display system comprises: Application layer; A hardware abstraction layer, wherein the hardware abstraction layer is configured with a fast display management module and a synthetic display module; the fast display management module is used to provide an interface for a variety of application programs that require fast display; Hardware driver layer; The application program of the application program layer sends a data display instruction and display data to the fast display management module; the fast display management module is used to respond to the data display instruction and send the data display instruction and display data to the synthetic display module; the synthetic display module is used to respond to the data display instruction and call the driver hardware of the hardware driver layer to perform hardware display; The display system further includes an application framework layer, and the application framework layer is configured with a SurfaceFlinger module; the application sends a UI display instruction and UI data to the SurfaceFlinger module; the SurfaceFlinger module sends the UI display instruction and UI data to the synthetic display module; the synthetic display module synthesizes the display data and the UI data and then calls the driver hardware for hardware display; The synthesis display module includes a ComposerClient submodule and a HwComposer submodule. The synthesis display module synthesizes the display data and the UI data and then calls the driver hardware for hardware display, including: The ComposerClient submodule creates at least two Clients respectively, the fast display management module and the SurfaceFlinger module correspond to one Client respectively, and the two Clients corresponding to the fast display management module and the SurfaceFlinger module send display data and UI data to the HwComposer submodule respectively; The HwComposer submodule returns callbacks to the two Clients corresponding to the fast display management module and the SurfaceFlinger module respectively, synthesizes the display data and UI data, and calls the driver hardware for display.

8. The display system according to claim 7, characterized in that: The fast display management module includes: An external interface, the external interface is used to create a display buffer, An implementation module, the implementation module comprising: The Display Surface submodule is used to receive a display buffer filled with the display data; The BufferQueue submodule is used to allocate a display buffer for the application; the application is used to fill the display data into the display buffer; The HwcHal submodule is used to send the display data to the synthetic display module.

Images