High-safety and high-real-time multi-screen interaction method

Abstract

The invention discloses a high-safety and high-real-time multi-screen interaction method, relates to the technical field of multi-screen display, and aims to solve the problems that convenience and smoothness are not enough during use, and the user experience is undoubtedly reduced if jamming, delay or dislocation occurs. The multi-machine cooperation process is simplified, and the control flow is simplified. The display content and the instruction are interacted between the multiple UOSs and the SOS in a secure memory sharing mode, zero copy of the memory is achieved in the whole process, the updating speed of the display content and the response speed of the interaction instruction are increased, and the real-time performance of the system is improved. According to the method, the memory management technology provided by the SMMU, microkernel fine-grained permission division and other mechanisms are utilized, isolation and sharing of the memory are achieved, the UOS can read data in the shared memory, but software and hardware in the SOS cannot be monitored and controlled accordingly, and the security of the system is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of multi-screen display technology, and in particular to a high-security and high-real-time multi-screen interaction method. Background Art

[0002] Multi-screen interactive technology is a technology that enables multiple screens to collaborate and interact with each other. Through multi-screen interactive technology, users can share information, transfer data and control operations on different screens, thus achieving a richer and more convenient interactive experience. For example, in a meeting, multi-screen interactive technology allows participants to view meeting materials, share ideas, and conduct instant discussions and collaborations through their respective mobile devices or screens in real time. In educational scenarios, teachers can use multi-screen interactive technology to better present teaching content and stimulate students' interest and participation. In the smart cockpit, the driver can easily obtain vehicle status and navigation information through the multi-screen interactive interface, and interact with the smart cockpit system more naturally and conveniently. Multi-screen interactive technology has made significant progress in implementation. It can connect various screens through the network, and realize data sharing, operation control and information exchange between different screens through multi-screen interactive software applications. Virtualization technology can also be used, with each virtual machine controlling one screen, and multi-screen interaction can be achieved between different virtual machines through data sharing, collaborative work, etc.

[0003] However, the existing multi-screen interactive technology mainly focuses on the realization of its functions, that is, ensuring seamless connection and content sharing between different devices. Although these advances have achieved remarkable results in providing users with a convenient experience, they often lack sufficient attention in terms of security and real-time performance. In order to achieve the purpose of multi-screen interaction, the system needs to adopt technical means to support data transmission, sharing and communication between devices. The lack of security in existing technologies makes it easy for criminals to attack the system, which may not only lead to the leakage and abuse of user data, but also cause the loss of device control, thereby threatening the user's information security and device security. Real-time performance is also a key issue that multi-screen interactive technology needs to solve. Especially for the virtual machine and the display screen are bound, and multi-screen interaction is achieved by virtual machine communication or system communication. Due to the long communication time during the interaction process, it will cause excessive data transmission delay or inaccurate synchronization, which will undoubtedly greatly reduce the convenience and fluency of use. If there is a freeze, delay or misalignment, it will undoubtedly lead to a decline in user experience.

[0004] Therefore, a highly secure and real-time multi-screen interaction method is needed. Summary of the invention

[0005] In order to solve all or part of the above-mentioned problems, the purpose of the present invention is to provide a high-security and high-real-time multi-screen interaction method, so as to solve the multi-screen interaction realized by binding a virtual machine with a display screen and adopting communication between virtual machines or communication between systems. Since the communication in the interactive process takes a long time, it will cause excessive data transmission delay or inaccurate synchronization, which will undoubtedly greatly reduce the convenience and fluency of use. If freezes, delays or misalignments occur, it will undoubtedly lead to a decline in user experience.

[0006] To achieve the above object, the present invention provides the following technical solution: a high-security and high-real-time multi-screen interaction method, characterized in that it includes the following steps:

[0007] S1: The system creates multiple virtual machines, each of which is connected to a separate screen, and the operating system in the virtual machine controls the display content of the screen;

[0008] S2: The virtual machine SOS acts as the master controller of the screen display. SOS is responsible for determining the display content of all screens and receiving the user's interactive commands and processing and providing feedback. Another virtual machine acts as UOS. The display screen connected to the UOS virtual machine acts as a slave of the display. The UOS virtual machine does not control the display content of its screen, but only reads the display content from SOS and displays the received video signal on its own screen. The screen connected to UOS is responsible for displaying the corresponding content according to the control of SOS, receiving the user's interactive commands and forwarding them to SOS for processing, and displaying the corresponding content according to the processing results of SOS.

[0009] Furthermore, when SOS in S2 is started, SOS is responsible for the creation, management and shutdown of UOS. When UOS is connected to the screen, Display Framework will manage the information of this screen. When there is no user interaction, the SOS kernel obtains the display information processed by GPU. First, Display Framework will transfer the video information and the number of screens to KDE, split the display information, and determine the content that each screen should display. Then, the display content is stored in the corresponding memory space. The display content of the local screen is stored in the local memory, and the content of the non-local screen is stored in the corresponding secure shared memory. UOS reads the secure shared memory by the Virtio Wayland module. A secure shared memory can be divided between SOS and UOS to realize fast data exchange between the two ends. The creation, destruction and security of this memory are all completed with the assistance of Micro-KenelHypervisor. The Hypervisor controls the read and write permissions to ensure that only the corresponding UOS can read and write data from it, thereby ensuring its security, obtaining display information, and presenting it on its own screen.

[0010] Furthermore, in S2, when user interaction occurs in SOS, SOS directly responds to and processes the interaction, and writes the content that should be displayed on the UOS screen into the shared memory. UOS reads the display content from the shared memory and presents it on its own screen.

[0011] Furthermore, in S2, when user interaction occurs in the UOS, the interaction instruction is received by the client program of the application layer in the UOS, and is transmitted to the server program in the SOS for processing through communication between virtual machines.

[0012] Furthermore, when a new screen is added in S2, the virtual machine manager will create a new virtual machine connected to the new screen and allocate shared memory for exchanging display information. The Display Framework will record the updated screen information. After receiving a new video signal, KDE will split the screen according to the new number of screens.

[0013] Furthermore, when reducing the screen in S2, the virtual machine manager T-Hyper will reclaim the shared memory, and the Display Framework will record the changed screen information and make corresponding changes when performing the split-screen processing.

[0014] Furthermore, in S2, the screen is split and the window layout is organized through the Display Framework and KDE, and then the rendering instructions are processed by the graphics API. The processed instructions are passed to the GPU and Display Driver via the Vhost Adapter Layer for image processing to generate display information, and then passed to the user layer via the Vhost Adapter Layer. At the user layer, the display information is divided into control flow information and data flow information.

[0015] Furthermore, the control flow is completely managed by SOS, including receiving, processing and responding to user interaction commands. At the same time, the control flow is deeply integrated with the upper-level KDE multi-window mechanism to split the display information and determine the display content distributed to each screen.

[0016] Furthermore, the data flow uses the secure shared memory technology provided by SOS to store the display content of each screen into the shared memory, and uses the system memory management unit SMMU for memory isolation. The SMMU divides a fixed size of memory from the memory as the shared memory between SOS and UOS for transmitting display information between the two.

[0017] The virtual machine manager Micro-KenelHypervisor will reclaim the shared memory. The virtual machine manager controls the read and write access rights of the memory, ensuring that only the corresponding UOS can read the memory.

[0018] Furthermore, memory isolation is performed between the SOS and the UOS through SMMU.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] 1. A high-security and high-real-time multi-screen interactive method proposed by the present invention, in which an SOS virtual machine is responsible for image processing and interactive instruction processing, which simplifies the process of multi-machine collaboration and the control process. In the existing multi-screen interactive technology based on the macro kernel, it is assumed that the UOS controls a display screen alone. If the application in the UOS wants to control the display content on this display screen, then the hardware control information and video stream data must be forwarded multiple times by the virtualization layer before they can be presented on the display screen. This process is relatively complicated and time-consuming, which will cause the screen to freeze and be unsmooth, affecting the user experience. In addition, under the virtualization architecture of the macro kernel, such a complex process cannot guarantee the real-time performance of the video signal. Therefore, this patent adopts a microkernel architecture, and uses its memory management mechanism to create a shared memory between SOS and UOS. SOS writes the video signal into the shared memory, and UOS reads the video signal from the shared memory and displays it, so that the speed will be very fast and real-time performance can be guaranteed. The memory management mechanism of the microkernel provides a security mechanism for this shared memory.

[0021] 2. The present invention proposes a high-security and high-real-time multi-screen interactive method. Multiple UOS and SOS interact with each other through secure shared memory to display content and instructions. The whole process has zero memory copy, which improves the update speed of display content and the response speed of interactive instructions, and improves the real-time performance of the system. The memory management technology provided by SMMU, the fine-grained permission division of the microkernel and other mechanisms are used to achieve memory isolation and sharing. Sharing refers to the shared memory between SOS and UOS, which is used to transmit video stream information. Isolation refers to the use of the memory management mechanism of the microkernel to isolate this shared memory from other applications and other virtual machines to avoid information leakage. UOS can read the data in the shared memory, but cannot monitor and control the software and hardware in SOS, which greatly improves the security of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a schematic diagram of the overall architecture of the high-security and high-real-time multi-screen interactive method of the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] like Figure 1 As shown, a high-security and high-real-time multi-screen interaction method includes the following steps:

[0025] 1. The system creates multiple virtual machines, each of which is connected to a separate screen, and the operating system in the virtual machine controls the display content of the screen. The system uses a virtual machine manager based on a microkernel architecture to complete virtual machine management, control and communication.

[0026] One of the virtual machines is used as a service operating system (SOS) as the main controller of the screen display, that is, the virtual machine running the Server OS system in the attached figure. The Server OS system and the subsequent Android system here are both references, just for the convenience of explanation, and they can be replaced by other operating systems. The other two virtual machines are used as user operating systems (UOS, User Operating System), SOS: Server OS, service virtual machine, a virtual machine that plays a management role in the entire architecture, UOS: User OS, user virtual machine, running various applications and services, there is only one SOS in the entire architecture, there can be multiple UOS, UOS can be created and closed as needed, SOS is always in operation.

[0027] SOS and UOS are both virtual machines, each of which can run different systems. The systems they run are not limited to Android, but can also run other systems. When running the Android system, the display screens connected to the two UOS virtual machines, Android-IVI1 and Android-IVI2, serve as display slaves. Android-IVI1 and Android-IVI2 do not control the display content of their screens. They only read the display content from SOS and display the received video signals on their own screens.

[0028] 2. When running the Android system, SOS is responsible for determining the display content of all screens. It not only controls the content of its own screen, but also determines the display content of the screens connected to UOS such as Android-IVI1 and Android-IVI2. SOS is also responsible for receiving user interaction commands and processing and feedback. The screen connected to UOS is responsible for displaying the corresponding content according to the control of SOS, receiving user interaction commands and forwarding them to SOS for processing, and displaying the corresponding content according to the processing results of SOS.

[0029] 3. When running the Android system, in SOS, the GPU and Display Driver of the kernel layer perform image processing to generate display information, which is then passed to the user layer via the Vhost Adapter Layer. In the user layer, the display information is divided into control flow information and display flow information. The control flow is completely managed by SOS, including receiving, processing and responding to user interaction commands. At the same time, the control flow is deeply integrated with the upper-layer KDE multi-window mechanism, that is, the multi-window mechanism is called to split the display information and decide the display content to be distributed to each screen. The data flow uses the secure shared memory technology provided by SOS to store the display content of each screen in the shared memory, and uses the system memory management unit SMMU (System Memory Management Unit) for memory isolation.

[0030] 4. When running the Android system, SMMU will divide a fixed-size memory from the main memory as the shared memory between SOS and UOS to transfer display information between the two. There is a shared memory between Server OS and Android-IVI1, there is a shared memory between Server OS and Android-IVI2, and there is no shared memory between Android-IVI1 and Android-IVI2.

[0031] 5. SOS writes the content that should be displayed on the UOS screen into the shared memory. UOS reads the display content from the shared memory and presents it on its own screen. SOS can read and write shared memory, while UOS can only read shared memory. SOS and UOS use SMMU for memory isolation, so that UOS can read the display information written by SOS in the shared memory, but cannot access the hardware, thus ensuring security. Since the system uses a virtual machine manager based on the microkernel architecture, it provides a fine-grained permission division mechanism, which can set permission levels and permission lists for each service, and can explicitly control which services can use which functions. This mechanism can be used to regulate and restrict the access behavior of SOS and UOS to shared memory. If you need to describe the specific content of the isolation mechanism, you can refer to the mechanism of fine-grained permission division of SMMU and microkernel.

[0032] In addition, UOS can directly display information from shared memory, and no longer needs to copy display data from one virtual machine to another in various ways as in the traditional way. The whole process is zero-copy, which improves real-time performance.

[0033] 6. When SOS is started, the Display Framework has learned how many screens there are currently. When the system is running normally and there is no user interaction, the SOS kernel obtains the display information processed by the GPU. First, the Display Framework will cooperate with KDE to split the display information and determine what content should be displayed on each screen. The display content is then stored in the corresponding memory space. The display content of the local screen is stored in the local memory, and the content of the non-local screen is stored in the corresponding secure shared memory. UOS reads the secure shared memory through the Virtio Wayland module, obtains the display information, and presents it on its own screen.

[0034] 7. When user interaction occurs in SOS, SOS directly responds to and processes the interaction, and presents the processing results to multiple screens in the manner of step 6.

[0035] 8. When user interaction occurs in UOS, the interactive instructions will be sent to SOS through the communication between virtual machines. There are many solutions for the "communication between virtual machines" mentioned here. This communication method is not limited in the present invention, and any method can be used. For example, a client can be installed in SOS and UOS to send instructions in UOS to SOS. After SOS receives the interactive instructions, it will respond and process them immediately, and present the processing results to multiple screens in the manner of step 7. UOS only reads display information from shared memory and cannot perform more operations. This design can ensure that UOS cannot steal more information in SOS by contributing memory, nor can it hack SOS through shared memory.

[0036] 9. When the system adds or subtracts screens, the operation may be manual, or some screens may become disabled due to a malfunction, and the system will handle the situation accordingly. If a new screen is added, the virtual machine manager will create a new virtual machine to connect to the new screen, and allocate shared memory for exchanging display information. The DisplayFramework will record the updated screen information, and when new display content is received, it will process the display content in a new way. If the screen is reduced, the virtual machine manager Micro-Kenel Hypervisor will reclaim the shared memory, and the Display Framework will record the changed screen information and make corresponding changes when performing split-screen processing.

[0037] It should be noted that, in the description of the present application, it should be understood that the terms "length", "thickness", "inside", "outside", "axial", "radial", etc., indicating orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0038] In addition, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises", or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article, or apparatus.

[0039] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A highly secure and real-time multi-screen interactive method, characterized in that: The steps include: S1: The system creates multiple virtual machines, each of which is connected to a separate screen, and the operating system in the virtual machine controls the display content of the screen; S2: The virtual machine SOS acts as the master controller of the screen display. SOS is responsible for determining the display content of all screens and receiving the user's interactive commands and processing and providing feedback. Another virtual machine acts as UOS. The display screen connected to the UOS virtual machine acts as a slave of the display. The UOS virtual machine does not control the display content of its screen, but only reads the display content from SOS and displays the received video signal on its own screen. The screen connected to UOS is responsible for displaying the corresponding content according to the control of SOS, receiving the user's interactive commands and forwarding them to SOS for processing, and displaying the corresponding content according to the processing results of SOS.

2. A highly secure and highly real-time multi-screen interactive method as claimed in claim 1, characterized in that: When SOS in S2 is started, SOS is responsible for the creation, management and shutdown of UOS. When UOS is connected to the screen, Display Framework will manage the information of this screen. When there is no user interaction, the SOS kernel obtains the display information processed by GPU. First, DisplayFramework will transfer the video information and the number of screens to KDE, split the display information, and decide what content should be displayed on each screen. Then, the display content is stored in the corresponding memory space. The display content of the local screen is stored in the local memory, and the content of the non-local screen is stored in the corresponding secure shared memory. UOS reads the secure shared memory through the Virtio Wayland module, obtains the display information, and presents it on its own screen.

3. A highly secure and real-time multi-screen interactive method as claimed in claim 1, characterized in that: In S2, when user interaction occurs in SOS, SOS directly responds to and processes the interaction, and writes the content that should be displayed on the UOS screen into the shared memory. UOS reads the display content from the shared memory and presents it on its own screen.

4. A highly secure and highly real-time multi-screen interactive method as claimed in claim 1, characterized in that: When user interaction occurs in UOS in S2, the interaction instruction is received by the client program of the application layer in UOS, and is passed to the server program in SOS for processing through communication between virtual machines.

5. A high-security and high-real-time multi-screen interactive method as claimed in claim 1, characterized in that: When a new screen is added in S2, the virtual machine manager will create a new virtual machine to connect to the new screen and allocate shared memory for exchanging display information. The Display Framework will record the updated screen information. After receiving a new video signal, KDE will split the screen according to the new number of screens.

6. A highly secure and real-time multi-screen interactive method as claimed in claim 1, characterized in that: When reducing the screen in S2, the virtual machine manager will reclaim the shared memory, and the Display Framework will record the changed screen information and make corresponding changes when performing the split-screen processing.

7. A high-security and high-real-time multi-screen interactive method as claimed in claim 1, characterized in that: In the S2, the display framework and KDE are used to split the screen and organize the window layout. Then, the graphics API processes the rendering instructions. The processed instructions are passed to the GPU and the display driver via the Vhost Adapter Layer for image processing to generate display information, which is then passed to the user layer via the Vhost Adapter Layer. At the user layer, the display information is divided into control flow information and data flow information.

8. A high-security and high-real-time multi-screen interactive method as claimed in claim 7, characterized in that: The control flow is completely managed by SOS, including receiving, processing and responding to user interaction commands. At the same time, the control flow is deeply integrated with the upper-level KDE multi-window mechanism to split the display information and determine the display content distributed to each screen.

9. A high-security and high-real-time multi-screen interactive method as claimed in claim 7, characterized in that: The data flow uses the secure shared memory technology provided by SOS to store the display content of each screen in the shared memory, and uses the system memory management unit SMMU for memory isolation. The SMMU divides a fixed size of memory from the memory as the shared memory between SOS and UOS for transmitting display information between the two. The virtual machine manager Micro-KenelHypervisor will reclaim the shared memory. The virtual machine manager controls the read and write access rights of the memory, ensuring that only the corresponding UOS can read the memory.

10. A high-security and high-real-time multi-screen interactive method as claimed in claim 9, characterized in that: The SOS and UOS are memory isolated through SMMU.

Images