Multi-screen management method and device, electronic equipment and system

Abstract

The present invention discloses a multi-screen management method and device, an electronic device and a system, the multi-screen management method comprising: selecting any screen in an open state from multiple currently connected screens; binding the screen to a rendering thread, and locking the critical area of ​​the rendering thread accessing the screen dictionary; searching the screen information of the screen in the screen dictionary, and unlocking the critical area of ​​the rendering thread accessing the screen dictionary; rendering the screen using the screen information of the screen, displaying the rendered content, and returning to the step of selecting any screen in an open state from multiple currently connected screens. The present invention reduces code complexity and hardware resource occupation by using a single thread to implement multi-screen rendering, and at the same time, protects public data by using a lock mechanism, i.e., the critical area of ​​the thread accessing the screen dictionary.

Description

Technical Field

[0001] The present invention relates to the field of screen display technology, and in particular to a multi-screen management method and device, electronic equipment and system. Background Art

[0002] Due to the development of in-vehicle navigation systems, more and more manufacturers have proposed the demand for multi-screen and diversified displays, such as using the dashboard as an additional screen to display navigation information, adding HUD to display AR navigation, etc. Figure 1 As shown, in order to achieve this requirement, different screen apps are usually developed independently to manage the screens, and data is transmitted and synchronized through inter-process communication. For example, when the main screen needs to display content, a main screen app needs to be developed to manage the main screen; when the dashboard needs to display navigation information, a dashboard app needs to be developed to manage the dashboard; when the HUD needs to display AR navigation information, a HUD app needs to be developed to manage the HUD. Then, the app corresponding to each screen will be used to transmit the content of the screen to shared memory or socket cross-process communication and synchronize the data of each screen.

[0003] However, using this multi-threaded method to manage the corresponding screens requires each screen to be rendered by a corresponding rendering thread, which not only consumes a lot of resources but also increases the complexity of the code. At the same time, since public data may be used between different screen apps, when one screen app operates on the public data, if another screen app operates on the public data at the same time, it may cause errors in the public data. Summary of the invention

[0004] Therefore, in order to solve the problem that the current multi-threaded method is used to render different screen contents separately, which increases code complexity and makes maintenance difficult, the present invention provides a multi-screen management method and device, electronic device and system.

[0005] To achieve the above object, the present invention adopts the following technical solution:

[0006] An embodiment of the present invention provides a multi-screen management method, including: selecting any screen in an open state from multiple screens currently connected; binding the screen to a rendering thread, and locking a critical area for the rendering thread to access a screen dictionary; searching for screen information of the screen in the screen dictionary, and unlocking a critical area for the rendering thread to access the screen dictionary; rendering the screen using the screen information of the screen, displaying the rendered content, and returning to the step of selecting any screen in an open state from multiple screens currently connected.

[0007] Optionally, the screen dictionary is acquired in the following manner: acquiring a plurality of currently connected screens and window information of the plurality of screens; and establishing the screen dictionary according to a mapping relationship between the plurality of screens and the window information of the plurality of screens.

[0008] Optionally, establishing the screen dictionary based on the mapping relationship between the multiple screens and the window information of the multiple screens includes: establishing, for any one of the multiple screens currently connected, a mapping relationship between the screen and the window information of the screen; and traversing the multiple screens to obtain the mapping relationship between the multiple screens and the window information of the multiple screens to form the screen dictionary.

[0009] Optionally, the multi-screen management method also includes: when a screen is turned on, obtaining the window information of the screen through the UI thread, and locking the critical area for the UI thread to access the screen dictionary; establishing a mapping relationship between the screen and the window information of the screen, adding the mapping relationship to the screen dictionary, and unlocking the critical area for the UI thread to access the screen dictionary.

[0010] Optionally, the multi-screen management method also includes: when a screen is closed or deleted, locking the critical area of ​​the screen dictionary accessed by the UI thread; storing the mapping relationship between the screen and the window information of the screen in a recycling linked list, and deleting the mapping relationship between the screen and the window information of the screen in the screen dictionary; when the screen is finished being used by the rendering thread, deleting the mapping relationship between the screen and the window information of the screen in the recycling linked list, and unlocking the critical area of ​​the screen dictionary accessed by the UI thread.

[0011] Optionally, the screen is rendered using the screen information of the screen, and the rendered content is displayed, including: binding the screen to a canvas, calling the drawing method of the canvas to render the screen according to the screen information of the screen; determining whether the rendered content is a video stream, and when the rendered content is a video stream, transmitting and displaying the video stream using an encoder; when the rendered content is not a video stream, displaying the rendered content.

[0012] According to the second aspect, an embodiment of the present invention also discloses a multi-screen management method device, including: a selection module, used to select any screen in an open state from multiple screens currently connected; a locking module, used to bind the screen to a rendering thread, and lock the critical area of ​​the rendering thread accessing the screen dictionary; an unlocking module, used to search the screen information of the screen in the screen dictionary, and unlock the critical area of ​​the rendering thread accessing the screen dictionary; a rendering module, used to render the screen using the screen information of the screen, display the rendered content, and return to the step of selecting any screen in an open state from multiple screens currently connected.

[0013] According to the third aspect, an embodiment of the present invention also discloses an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor executes the steps of the multi-screen management method as described in the first aspect or any optional implementation manner of the first aspect.

[0014] According to a fourth aspect, an embodiment of the present invention further discloses a multi-screen management system, which includes: the multi-screen management device described in the second aspect and a plurality of screens.

[0015] According to a fourth aspect, an embodiment of the present invention further discloses a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the multi-screen management method as described in the first aspect or any optional implementation manner of the first aspect are implemented.

[0016] The technical solution of the present invention has the following advantages:

[0017] The present invention selects any screen in an open state from the currently connected multiple screens, binds the screen to a rendering thread, and locks the critical area of ​​the rendering thread accessing the screen dictionary, then searches for the screen information of the screen in the screen dictionary, unlocks the critical area of ​​the rendering thread accessing the screen dictionary, and then renders the screen using the screen information of the screen, displays the rendered content, and finally completes the rendering of all the currently connected multiple screens. The present invention realizes multi-screen rendering by adopting a single thread, reduces code complexity, reduces hardware resource occupation, and realizes protection of public data by utilizing a lock mechanism, i.e., the critical area of ​​the thread accessing the screen dictionary. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A schematic diagram of the current multi-screen management method;

[0020] Figure 2 A schematic diagram of a multi-screen management method according to an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of the structure of a multi-screen management system in an embodiment of the present invention;

[0022] Figure 4 A schematic diagram of a flow chart of a multi-screen management method according to an embodiment of the present invention;

[0023] Figure 5 A schematic diagram of a flow chart of a multi-screen management method according to an embodiment of the present invention;

[0024] Figure 6 A schematic diagram of a flow chart of a multi-screen management method according to an embodiment of the present invention;

[0025] Figure 7 A schematic diagram of the structure of a multi-screen management device in an embodiment of the present invention;

[0026] Figure 8 Schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] In the description of the present invention, it should be noted that the term "and / and / or" used in the present specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0029] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0030] Due to the development of in-vehicle navigation systems, more and more manufacturers have proposed the demand for multi-screen and diversified displays, for example, using the dashboard as an additional screen to display navigation information, adding HUD to display AR navigation, etc. Figure 1 As shown, in order to simultaneously control the above-mentioned multiple screens, a separate APP is developed for each display screen to control the corresponding screen. For example, the main screen of the vehicle cab is controlled by the main screen APP, the screen of the instrument panel is controlled by the instrument panel APP, and the HUD is controlled by the HUD APP. However, in this method of controlling separate screens by separate APPs, each screen needs to be rendered by a corresponding rendering thread, and a separate data communication protocol needs to be formulated between APPs on different screens to carry out data communication between APPs on different screens. In addition, since public data may be used between APPs on different screens, when one screen APP operates on the public data, if another screen APP operates on the public data at the same time, errors may occur in the public data.

[0031] In view of the above problems, the embodiment of the present invention discloses a multi-screen management method, such as Figure 2 As shown, the method comprises the following steps:

[0032] Step 201: Select any screen that is in an open state from multiple screens that are currently connected.

[0033] As a specific implementation method, Figure 3 As shown, the present invention is applied to a multi-screen management system, which includes: a multi-screen management device and multiple screens. Among them, screen 0 is the main screen of the vehicle, screen 1 is the instrument panel screen, and screen k is the HUD. When the screen is displayed, the multi-screen management device can output the rendered content to screen 0 through HDMI, the screen management device can transmit the rendered content to screen 1 through wifi projection for output display, and the screen management device can transmit the rendered content to screen k in the form of H264 video stream for output display.

[0034] When the multi-screen management device is started, any screen in the turned-on state will be selected from the multiple screens currently connected.

[0035] Step 202: Bind the screen to the rendering thread and lock the critical section for the rendering thread to access the screen dictionary.

[0036] As a specific implementation method, Figure 3As shown, the multi-screen management device of the multi-screen management system is also used to interact with the rendering thread. After selecting any screen that is in the turned-on state from the multiple screens currently connected, the screen is bound to the rendering thread by binding the screen ID to the rendering thread, and the critical section for the rendering thread to access the screen dictionary is locked. When the rendering thread renders the screen, by locking the critical section for the rendering thread to access the screen dictionary, other threads such as the UI thread need to wait, which can prevent the rendering thread and the UI thread from operating the screen dictionary at the same time and avoid errors in the screen dictionary data.

[0037] The screen dictionary is obtained by obtaining the currently connected multiple screens and the window information of the multiple screens, and establishing the screen dictionary according to the mapping relationship between the multiple screens and the window information of the multiple screens.

[0038] Specifically, according to the mapping relationship between multiple screens and the window information of multiple screens, the process of establishing the screen dictionary is: for any screen among the multiple screens currently connected, a mapping relationship between the screen and the window information of the screen is established, and the multiple screens are traversed to obtain the mapping relationship between the multiple screens and the window information of the multiple screens to form a screen dictionary.

[0039] Among them, the window information of the screen includes but is not limited to: the pixel output mode of the window after being marked and rendered, the storage of the underlying window handle obtained through the window display interface, the height value when the screen is rendered, the width value when the screen is rendered, and a pointer to the window object.

[0040] Step 203: Search the screen dictionary for the screen information of the screen, and unlock the critical section for the rendering thread to access the screen dictionary.

[0041] Specifically, the screen information corresponding to the screen is found by querying the screen ID in the screen dictionary, and the critical section of the currently locked rendering thread to access the screen dictionary is unlocked. Because the screen dictionary, a public data, is no longer needed at this time, it can be handed over to other threads for use, which is convenient for other threads to use in a timely manner.

[0042] Step 204, rendering the screen using the screen information of the screen, displaying the rendered content, and returning to the step of selecting any screen in an open state from the multiple screens currently connected.

[0043] Specifically, the screen can be bound to the canvas, and the drawing method of the canvas can be called to render the screen according to the screen information of the screen, and it can be determined whether the rendered content is a video stream. When the rendered content is a video stream, the encoder is used to transmit and display the video stream; when the rendered content is not a video stream, the rendered content is displayed.

[0044] As a specific implementation method, the screen can be bound to the rendering environment by calling the GL function, and then the corresponding rendering method is called to render the screen. Finally, the rendered content is output to the target screen, and it is determined whether the rendered content at this time is a video stream. When the current rendering content is a video stream, the rendering content can be output while being downloaded. The H264 encoder can be called to process the rendering content, and then the processed rendering content is sent to the target screen, which displays the rendering content.

[0045] As another optional implementation of the present invention, when a screen connected to a multi-screen management device is turned on, the window information of the screen can be obtained through the UI thread, and the critical area for the UI thread to access the screen dictionary can be locked, and a mapping relationship between the screen and the window information of the screen can be established, and the mapping relationship can be added to the screen dictionary, and the critical area for the UI thread to access the screen dictionary can be unlocked.

[0046] Specifically, Figure 4 As shown, when a screen connected to the multi-screen management device is turned on, the multi-screen management system will trigger the window display interface. By obtaining the underlying window handle of the screen, the ID of the screen and the window information of the screen can be obtained, and the critical area of ​​the screen dictionary accessed by the UI thread is locked to prevent the rendering thread from accessing the critical area of ​​the screen dictionary at this time, and then a mapping relationship between the screen ID and the window information of the screen is established, and the mapping relationship is added to the screen dictionary, thereby unlocking the critical area of ​​the screen dictionary for the UI thread to access.

[0047] As another optional implementation of the present invention, when a screen is closed or deleted, the critical section for the UI thread to access the screen dictionary is locked, the mapping relationship between the screen and the window information of the screen is stored in a recycling linked list, and the mapping relationship between the screen and the window information of the screen is deleted from the screen dictionary. When the screen is used up by the rendering thread, the mapping relationship between the screen and the window information of the screen is deleted from the recycling linked list, and the critical section for the UI thread to access the screen dictionary is unlocked.

[0048] Specifically, Figure 5As shown, when a screen connected to the multi-screen management device is closed or deleted, the multi-screen management system triggers the window hiding interface and locks the critical area of ​​the UI thread accessing the screen dictionary to prevent the rendering thread from accessing the critical area of ​​the screen dictionary at this time, and then finds the window information of the screen corresponding to the screen ID in the screen dictionary according to the screen ID, and then stores the mapping relationship between the screen ID and the window information of the screen in the recycling list, and deletes the mapping relationship between the screen ID and the window information of the screen in the screen dictionary. This process of transferring the mapping relationship between the screen ID and the window information of the screen from the screen dictionary to the recycling list can avoid abnormal states such as black screen or flowery screen caused by the rendering process still using the window when the screen is closed or deleted. Finally, unlock the critical area of ​​the UI thread accessing the screen dictionary.

[0049] The present invention can manage multiple screens in a componentized manner by utilizing a screen dictionary, and can open, close or delete screens at any time according to actual needs, thereby increasing the flexibility of multi-screen management. In addition, by transferring the mapping relationship between screens and window information of the screens from the screen dictionary to the recycling linked list, abnormal states such as a black screen or a distorted screen caused by the immediate release of window resources can be effectively avoided. Specific embodiment:

[0051] Exemplarily, as an optional embodiment of the present invention, Figure 6 As shown, when the multi-screen management system is turned on, any screen in the turned-on state is selected from the currently connected multiple screens, and the rendering thread is bound to the currently selected screen by calling the canvas selection method, thereby locking the critical area for the rendering thread to access the screen dictionary, and looking up the window information of the screen in the screen dictionary. When the screen ID does not exist, it returns to selecting any screen in the turned-on state from the current multiple screens. Until all the current multiple screens are traversed, the resource cleanup method is called again to clean up the resources, and then any screen in the turned-on state is selected again from the current multiple screens.

[0052] When the window object of the screen exists, the mapping relationship between the current screen ID and the window information of the screen is retrieved, and the critical area of ​​the rendering thread accessing the screen dictionary is unlocked, and then the GL function is called to bind the screen to the drawing environment, and then the screen is rendered by calling the drawing method corresponding to the screen, and the drawn content is displayed on the screen through the screen rendering method. Finally, by judging whether the rendered content is a video stream, if the rendered content is a video stream, the rendered content is processed by calling the H264 encoder and then sent to the screen for display. If the rendered content is not a video stream, the rendered content is directly sent to the screen for display.

[0053] The present invention realizes multi-screen rendering by adopting a single thread, thereby reducing code complexity and hardware resource occupation, and at the same time realizes protection of public data by utilizing a lock mechanism, namely, a critical area of ​​a thread accessing a screen dictionary.

[0054] The present invention also provides a multi-screen management device, such as Figure 7 As shown, the device comprises:

[0055] A selection module 71 is used to select any screen in an open state from the multiple screens currently connected, and the details are described in step 201;

[0056] A locking module 72 is used to bind the screen to the rendering thread and lock the critical section of the rendering thread accessing the screen dictionary. For details, refer to step 202.

[0057] Unlocking module 73, used for searching the screen information of the screen in the screen dictionary, and unlocking the critical section for the rendering thread to access the screen dictionary, for details, refer to step 203;

[0058] The rendering module 74 is used to render the screen using the screen information of the screen, display the rendered content, and return to the step of selecting any screen in the open state from the multiple screens currently connected. For details, please refer to step 204.

[0059] The embodiment of the present invention further provides an electronic device, such as Figure 8 As shown, the electronic device may include a processor 801 and a memory 802, wherein the processor 801 and the memory 802 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0060] The processor 801 may be a central processing unit (CPU). The processor 801 may also be other general-purpose processors, digital signal processors (DAP), application-specific integrated circuits (AAIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0061] The memory 802 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the key shielding method of the multi-screen management method device in the embodiment of the present invention. The processor 801 executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory 802, that is, realizing the multi-screen management method in the above method embodiment.

[0062] The memory 802 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created by the processor 801, etc. In addition, the memory 802 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 802 may optionally include a memory remotely arranged relative to the processor 801, and these remote memories may be connected to the processor 801 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0063] The one or more modules are stored in the memory 802, and when executed by the processor 801, the following is performed: Figure 1-6 The multi-screen management method in the illustrated embodiment.

[0064] For details of the above electronic equipment, please refer to Figure 1-6 The corresponding related descriptions and effects in the illustrated embodiments can be understood and will not be repeated here.

[0065] The present invention also provides a multi-screen management system, such as Figure 3 As shown, the system includes: a multi-screen management device and multiple screens.

[0066] Specifically, screen 0 is the main screen of the vehicle, screen 1 is the dashboard screen, and screen k is the HUD. When the screens are displayed, the multi-screen management device can output and display the rendered content to screen 0 via HDMI, the screen management device can transmit the rendered content to screen 1 for output display via wifi projection, and the screen management device can transmit the rendered content to screen k for output display in the form of an H264 video stream.

[0067] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FlaAhMemory), a hard disk (Hard DiAk Drive, abbreviated: HDD) or a solid-state drive (ASD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0068] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A multi-screen management method, characterized in that: include: Select any open screen from the currently connected multiple screens; Bind the screen to the rendering thread and lock the critical section of the rendering thread's access to the screen dictionary; Searching for screen information of the screen in the screen dictionary, and unlocking a critical section for the rendering thread to access the screen dictionary; Rendering the screen using the screen information of the screen, displaying the rendered content, and returning to the step of selecting any screen in an open state from the multiple screens currently connected; The screen dictionary is obtained as follows: Obtaining the currently connected multiple screens and the window information of the multiple screens; For any screen among the multiple screens currently connected, a mapping relationship between the screen and the window information of the screen is established; Traversing the multiple screens, obtaining mapping relationships between the multiple screens and the window information of the multiple screens to form the screen dictionary; The window information of the screen includes but is not limited to: the pixel output mode of the window after being marked and rendered, the storage of the underlying window handle obtained through the window display interface, the height value when the screen is rendered, the width value when the screen is rendered, and a pointer to the window object.

2. The method according to claim 1, characterized in that Also includes: When a screen is turned on, the window information of the screen is obtained through the UI thread, and the critical area for the UI thread to access the screen dictionary is locked; A mapping relationship between the screen and the window information of the screen is established, the mapping relationship is added to the screen dictionary, and the critical section for the UI thread to access the screen dictionary is unlocked.

3. The method according to claim 1, characterized in that Also includes: When a screen is closed or deleted, the critical area of ​​the UI thread accessing the screen dictionary is locked; storing the mapping relationship between the screen and the window information of the screen in a recycling linked list, and deleting the mapping relationship between the screen and the window information of the screen in the screen dictionary; When the screen is completely used by the rendering thread, the mapping relationship between the screen and the window information of the screen is deleted in the recycling linked list, and the critical section for the UI thread to access the screen dictionary is unlocked.

4. The method according to claim 1, characterized in that The screen information of the screen is used to render the screen, and the rendered content is displayed, including: Binding the screen to the canvas, calling the drawing method of the canvas, and rendering the screen according to the screen information of the screen; Determine whether the rendered content is a video stream. When the rendered content is a video stream, using an encoder to transmit and display the video stream; When the rendered content is not a video stream, the rendered content is displayed.

5. A multi-screen management device, characterized in that: include: A selection module is used to select any screen that is turned on from multiple screens that are currently connected; A locking module, used to bind the screen to the rendering thread and lock the critical section of the rendering thread accessing the screen dictionary; An unlocking module, used for searching the screen information of the screen in the screen dictionary, and unlocking the critical section for the rendering thread to access the screen dictionary; A rendering module, used to render the screen using the screen information of the screen, display the rendered content, and return to the step of selecting any screen in an open state from the multiple screens currently connected; The screen dictionary is obtained as follows: Obtaining the currently connected multiple screens and the window information of the multiple screens; For any screen among the multiple screens currently connected, a mapping relationship between the screen and the window information of the screen is established; Traversing the multiple screens, obtaining mapping relationships between the multiple screens and the window information of the multiple screens to form the screen dictionary; The window information of the screen includes but is not limited to: the pixel output mode of the window after being marked and rendered, the storage of the underlying window handle obtained through the window display interface, the height value when the screen is rendered, the width value when the screen is rendered, and a pointer to the window object.

6. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the multi-screen management method according to any one of claims 1 to 4 by executing the computer instructions.

7. A multi-screen management system, characterized in that: include: The multi-screen management device and multiple screens as described in claim 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multi-screen management method according to any one of claims 1 to 4 are implemented.

Images