Low-resource real-time multi-screen synchronization and adaptive interactive display frame

Abstract

The invention provides a low-resource real-time multi-screen synchronous and self-adaptive interactive display frame, which adopts center driving display, has low requirements on the display capability of a display control terminal, has low requirements on a terminal processor and a memory, greatly reduces the equipment cost, is suitable for embedded equipment, supports heterogeneous terminals with different resolutions, and has high practicability. The method does not need to customize and modify applications for equipment, ensures the consistency of display contents, ensures that the contents are still clearly displayed after being zoomed through an interpolation adaptive algorithm, is compatible with various screen sizes, and is low in software and hardware maintenance cost.

Description

Technical Field

[0001] The present invention relates to the field of embedded device display and interaction technology, and in particular to a low-resource real-time multi-screen synchronization and adaptive interactive display framework. Background Art

[0002] In embedded device environments, memory and processing power are often limited. As the number of display devices increases and the resolution improves, the demand for multi-screen synchronization is increasing. Especially in scenarios such as education and training, command and control, and industrial site monitoring, the diversity of devices (large differences in resolution) further increases the complexity. This has prompted an urgent need for an efficient and lightweight synchronous display framework. In embedded systems, screen display and human-computer interaction are key components. However, existing technologies face challenges in terms of resource usage, real-time performance, and scalability. The current mainstream technical frameworks can be divided into the following categories:

[0003] 1. Point-to-point transmission technology:

[0004] Point-to-point transmission is the most common terminal communication method, which realizes data exchange by connecting the server and the terminal separately. The advantage of this method is that the architecture is simple and easy to implement. However, when the number of terminals increases, the server needs to maintain a large number of connections, resulting in a sharp increase in bandwidth usage and computing resource requirements, making it difficult to meet the needs of multi-terminal synchronization. Especially in large-scale display or multi-screen systems, it may affect real-time performance and the overall performance of the system.

[0005] 2. Broadcast distribution technology:

[0006] Broadcast technology achieves synchronous display by sending data to all devices in the network at once. This method is efficient in a single network scenario, but it leads to serious bandwidth waste due to the lack of selectivity of broadcast data. In addition, broadcast technology has high requirements on the receiving and parsing capabilities of the terminal, and is not suitable for resource-constrained embedded devices.

[0007] 3. Terminal graphics solutions and solutions based on high-resource devices:

[0008] Some technical solutions run independent graphics applications on display terminals and rely on high-performance servers and terminal devices to achieve multi-screen display synchronization. These solutions usually use complex graphics rendering and transmission algorithms, such as high-definition streaming media encoding and decoding, and have high requirements on the display terminal resource capabilities and complex technologies, such as the need to run graphics frameworks, transmit large amounts of information, and synchronize local processing and other display terminals. However, this high resource dependence makes it difficult to promote and apply such solutions in low-cost, low-resource embedded environments.

[0009] Therefore, current technologies have significant deficiencies in the following aspects:

[0010] 1. High resource requirements: Existing technical frameworks have high requirements for the device's processor, memory, display capabilities and network bandwidth, which are difficult for embedded devices to bear. In particular, the display terminal runs a graphics framework, which has high synchronization costs and high resource requirements. High-resource architecture solutions cannot be extended to embedded systems with limited memory and insufficient processor performance.

[0011] 2. Poor real-time performance: Point-to-point transmission solutions have high latency in multi-terminal scenarios and cannot achieve fast response. They are difficult to expand. Performance degrades rapidly in multi-terminal environments.

[0012] 3. Insufficient adaptability: Traditional frameworks cannot provide adaptive display for different screen resolutions and cannot adapt to various heterogeneous display terminals and resolutions, resulting in inconsistent user experience and poor scalability.

[0013] 4. Complex synchronization and interaction: Multi-terminal display and control devices need to synchronize display graphics devices and respond to control instructions, and update display graphics. The technology and process are complex, and the accuracy of display data synchronization is difficult to guarantee, especially in the case of network jitter. The graphic interface is not synchronized and the display quality is poor. Summary of the invention

[0014] The technical problem to be solved by the technical solution of the present invention is:

[0015] 1. How to achieve synchronous display and real-time interaction of multiple screen terminals in a low-resource manner when embedded device resources are limited.

[0016] 2. How to reduce bandwidth usage and server computing overhead through efficient data transmission protocols (such as multicast) and synchronization frameworks. How to reduce bandwidth usage while meeting the needs of multi-screen real-time synchronization?

[0017] 3. How to adapt to screen displays of different sizes and resolutions, and adaptively display graphics to ensure clear graphic content and consistent layout.

[0018] 4. How to improve the real-time nature of interaction, reduce response delays during human-computer interaction, and increase the heterogeneous capabilities and scalability of display terminals.

[0019] The technical solution of the present invention provides a low-resource real-time multi-screen synchronization and adaptive interactive display framework, including a display control terminal, a central server module and a communication transmission module. The display control terminal and the central server module establish a control multicast group and a graphic multicast group through the communication transmission module;

[0020] The display and control terminal in use captures the user input events and screen coordinate position requests and assembles them into a communication protocol format and sends them to the communication transmission module along the control multicast group;

[0021] The communication transmission module adopts UDP multicast mode to parse user input events and screen coordinate position requests into data and control instructions and transmit them to the central server module along the control multicast group;

[0022] The central server module obtains data and control instructions, manages the legitimacy of multiple display and control terminals with the help of the communication transmission module, runs the graphics framework to perform graphical interface and rendering according to the data and control instructions for the display and control terminals in use that have passed the verification, and generates new graphics in response to the processing. According to the graphic display strategy, the graphics content is read and encapsulated into graphics data in the video memory, and the graphics data in the video memory is sent to the communication transmission module along the graphics multicast group according to the protocol rules;

[0023] The communication transmission module uses UDP multicast to transmit the graphics data in the display memory along the graphics multicast group to the display control terminal corresponding to the control instruction;

[0024] The display control terminal that receives the video memory graphic data verifies and analyzes the video memory graphic data, adapts the analyzed video memory graphic data to screens with different resolutions to realize adaptive graphics and flushes them into the video memory for display.

[0025] Preferably, the display and control terminal verifies the central server module, and if the verification is successful, receives the display memory graphic data transmitted by the central server module through the communication transmission module.

[0026] Preferably, the central server reads the video memory address mapped with the video memory graphic data in segments, and sends it to the display control terminal corresponding to the control instruction through multicast multipoint communication protocol encapsulation.

[0027] Preferably, the legality management step includes:

[0028] Establish multicast groups for device management in multiple display and control terminals;

[0029] The central server joins the device management multicast group and runs the receiving task;

[0030] When the display and control terminal is turned on, it declares to join the device management multicast group and sends the key string to the central server through the communication transmission module. After the central server authenticates according to the announced password, it will include the legitimate display and control devices under management and only receive control instructions from legitimate display and control devices.

[0031] Preferably, the graphics size is dynamically adjusted by interpolation according to the data packet offset and the native resolution, and the graphics are scaled up to ensure display efficiency and quality, so as to achieve the adaptation of the parsed graphics data in the video memory to screens with different resolutions.

[0032] The technical solution of the present invention proposes a low-resource real-time multi-screen synchronization and adaptive interactive display framework, which adopts central drive display, has low requirements on the display capability of the display control terminal, and has low requirements on the terminal processor and memory, greatly reduces the device cost, is suitable for use in embedded devices, and supports heterogeneous terminals with different resolutions. There is no need to customize and modify applications for the devices, ensuring the consistency of displayed content. The interpolation adaptive algorithm ensures that the content is still clearly displayed after scaling. It is compatible with various screen sizes and has low software and hardware maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of a center-driven synchronous display in a low-resource real-time multi-screen synchronization and adaptive interactive display framework provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0035] The embodiment of the present invention provides a low-resource real-time multi-screen synchronization and adaptive interactive display framework, which involves low-resource configuration requirements of display and control terminals, plug-and-play discovery mechanism, center-driven multi-screen synchronization and interaction technology, adaptive graphics resolution display capability, etc., including:

[0036] 1. Central server module

[0037] The central server is the core of the entire system, responsible for graphics generation, data distribution and event processing. It includes the following sub-modules:

[0038] 1) Terminal management: Perform legal management on the discovery, joining, authentication, and leaving process cycles of display and control terminals to achieve plug-and-play elastic expansion.

[0039] 2) Graphics Generator: Runs the graphics framework, implements the human-computer interaction interface and content, and is responsible for generating the graphics interface and rendering according to the default resolution.

[0040] 3) Virtual device driver: implement virtual keyboard and mouse driver, receive human-computer interaction events (mouse, keyboard, touch, etc.) uploaded by the display and control terminal, respond and process to generate new graphics.

[0041] 4) Graphics data distribution module: reads and encapsulates graphics content according to the graphics display strategy, and sends graphics data to the video memory according to the protocol rules.

[0042] 5) Control instruction distribution module: reads human-computer interaction control instructions for display and control terminal devices that meet the identity authentication requirements.

[0043] 2. Display and control terminal

[0044] The display and control terminal is responsible for receiving and displaying graphic data, capturing user operation events and feeding them back to the central server. The main sub-modules are as follows:

[0045] 1) Device management module: announce discovery protocol and display and control device information, authentication.

[0046] 2) Data receiver: The receiving server verifies and analyzes the graphic data packets sent by multicast.

[0047] 3) Graphics adapter: adapts the parsed graphics data to screens of different resolutions to achieve adaptive graphics and flushes them into video memory for display.

[0048] 4) Event capture module: captures user input events (such as touch, keyboard, mouse) and screen coordinate position requests and assembles them into a communication protocol format and sends them to the server via the network.

[0049] 3. Communication transmission module

[0050] UDP multicast is used for data and control command transmission, supporting one-to-many efficient data distribution and command feedback, reducing bandwidth overhead for repeated transmission, and ensuring the reliability and sequence of event transmission. The sub-modules are as follows:

[0051] 1) Implement efficient distribution of graphic display data based on multicast groups and UDP communication protocol.

[0052] 2) Implement plug-and-play protocol distribution for devices based on multicast groups and UDP communication protocols.

[0053] 3) Realize human-computer interaction event communication based on multicast group and UDP communication protocol.

[0054] The schematic diagram of center drive synchronous display is as follows Figure 1 shown.

[0055] The embodiments of the present invention include the following technical implementations:

[0056] 1. Center-driven graphics generation and multi-screen display

[0057] The center-driven multi-screen display synchronization technology mentioned in the present invention is different from the local display of traditional graphics and the distributed display overlay technology based on the network. The technical focus is that the graphics application software only runs on the central server and is completely generated by the central server, and the display control terminal does not run any graphics generation software locally. The central server runs a graphics framework, such as QT, GTK, etc. The GPU renders according to the default resolution such as 1024*768, and generates a graphical interface and display content according to the user-designed application, and then stores the rendered graphical interface in the video memory of the central server, and reads it through the video memory mapping interface for subsequent applications. The graphics distribution application of the central server reads the video memory address of the generated image mapping in segments through the multicast multipoint communication protocol, and sends the read graphics information data to the display control terminal registered to the multicast group according to the protocol package. The display terminal does not need to run the graphics framework, as long as the received image is scaled and adapted to the local resolution according to the algorithm, and written to the local video memory for display. It greatly reduces the resource requirements and equipment costs of the display terminal such as memory, graphics card, and calculation. The graphical interface upgrade only needs to be upgraded on the central server, which greatly reduces the upgrade and maintenance costs.

[0058] 2. Plug-and-play discovery and registration of display and control terminals

[0059] For security reasons, a security discovery and control mechanism is designed between the multi-display and control terminals and the central server. The registration and management of the display and control terminals is realized through the device management multicast group. The central server joins the device management multicast group and runs the receiving task. When the display and control terminal is turned on, it declares to join the device management multicast group and sends the key string to the central server. After the central server authenticates according to the announced password, it will include the legal display and control devices into the management. Therefore, the display and control terminal can achieve plug-and-play without configuring any parameters.

[0060] 3. Human-computer interaction technology

[0061] The display and control end is usually connected to a keyboard, mouse or touch device to realize graphical human-computer interaction. The focus of centrally driven human-computer interaction and display is on the collection of display and control end data and the response to server-side events. The central server realizes the integration and conversion of remote human-computer interaction events to the local graphics framework through the virtual keyboard and mouse driver. After the display and control end device collects the user's human-computer interaction events such as keyboard key values, mouse event coordinates, and touch device event coordinates, it encapsulates them into a pre-defined communication protocol format and sends the events to the control multicast group. The central server runs the control event receiving program to restore the received interactive events, and uses the virtual mouse and keyboard driver to write the restored keyboard and mouse events into the virtual keyboard and mouse driver, and send them to the local graphics framework of the central server. The graphics framework responds to events and location requirements, generates a new graphical interface, refreshes it to the local video memory, and then sends it to the display and control terminal through the sending program.

[0062] 4. Multi-screen low-resource graphics transmission and display technology

[0063] The traditional multi-screen synchronization technology will increase the demand for resources sharply as the number of display and control terminals increases, and puts forward higher configuration requirements for data communication and central servers. The present invention adopts multicast technology to transmit graphics and images, which can realize real-time communication and new image display for display and control terminals in the multicast group, reduce the demand for communication resources to the extreme, and realize resource overhead that is independent of the number of display and control terminals.

[0064] Since the graphics images are transmitted through the technology of video memory data acquisition, the characteristics of multicast UDP transmission are utilized here, and the direct video memory mapping technology is adopted. After mapping, the graphics data is efficiently read through the driver interface. The video memory fragment number transmission technology is adopted to segment the video memory and transmit the graphics information less than one Ethernet MTU at a time, which effectively avoids the situation where the large packet UDP transmission cannot be reassembled due to the loss of fragments, and is discarded by the network protocol stack as a whole. The receiving end receives the transmission of the segment number lower than the MTU in segments, writes the content to the local video memory display according to the offset in the data packet, and renders it to the screen through the display driver of the display control terminal. This solution can realize the refresh of local image data, and even if some non-critical position numbered graphics are lost, they can still be displayed correctly. Further optimization measures are adopted to maintain a double buffer in the terminal memory, one for the current display and one for receiving new graphics to avoid screen tearing.

[0065] 5. Resolution Adaptive Technology

[0066] The display and control terminal dynamically adjusts the image size and scales the image according to the data packet offset sent by the server and the local resolution through algorithms such as interpolation, ensuring display efficiency and quality. It realizes local adaptive display of human-computer interaction interface and content based on standard graphics with fixed resolution (such as 1024×768).

[0067] The implementation process of the embodiment of the present invention is as follows:

[0068] 1) The central server starts initialization, runs the device management submodule, joins the graphic data communication multicast group, controls the interactive instruction multicast group, runs the authentication module, starts the virtual keyboard and mouse driver, runs the application, and renders and generates graphics;

[0069] 2) The device management submodule of the display and control terminal is started, joins the default control interaction multicast group, joins the graphic data communication multicast group, runs the local graphic adaptation task and the control instruction collection task, allocates memory and maps it to the video memory of the local display device;

[0070] 3) The display and control terminal sends device announcement information to the default control interaction multicast group in accordance with the device discovery protocol format, and the information includes an authentication password;

[0071] 4) The central server authenticates the received declaration protocol, and the IP address of the display and control terminal that passes the authentication will be added to the permission control list;

[0072] 5) The central server graphics generation submodule renders graphics according to the human-computer interaction interface designed by the user application at a standard resolution of 1024*768, and outputs the graphics data to the video memory of the central server through mapping technology;

[0073] 6) The graphics data distribution module of the central server reads and numbers the graphics data in the video memory in a real-time loop with a length less than the MTU of an Ethernet IP data packet, and encapsulates the corresponding numbered data according to the designed communication protocol format, and sends it to the graphics data communication multicast group. After the current round of video memory graphics data is sent, a special frame with a sending offset of 0x80000000 is added as the end frame to notify the display control terminal that the current round of video memory graphics data has been sent;

[0074] 7) The data receiver of the display and control terminal receives the multicast graphic data and copies the graphic data to the mapped memory address in sequence. When it detects the special frame with a sending offset of 0x80000000, it considers that the data sent this time is complete, executes the graphic adaptive algorithm, and scales the graphic data with a standard resolution of 1024*768 to the local resolution such as 1920*1080, and renders it to the local video memory.

[0075] 8) The display and control terminal displays the latest human-computer interaction graphic data generated by the central server;

[0076] 9) The user operates the display and control terminal through keyboard, mouse, touch and other methods. The event capture module of the display and control terminal captures the keyboard, mouse, touch and other events of the display and control device, and encapsulates the event, position coordinates and other information into the format specified by the control protocol, and sends it to the control interaction instruction multicast group;

[0077] 10) After receiving the control command sent by the display control, the control command receiving module of the central server verifies the source address of the command, discards the control command sent by the illegal device, writes the event type and location information of the control command of the legal device into the virtual keyboard and mouse driver of the central server through conversion, and sends it to the graphic display framework;

[0078] 11) The graphics display framework of the central server receives events and position information sent by the virtual keyboard and mouse driver, and generates new human-computer interaction interface graphics information in response;

[0079] 12) The central server goes to step 5 to start updating the current round of graphics memory information. This process repeats itself until the central server is shut down or stops working.

[0080] 13) Multiple display and control terminals perform the joining steps according to step 3;

[0081] 14) The display and control terminal sends a departure announcement protocol and leaves the framework.

[0082] The beneficial effects of the embodiments of the present invention are as follows:

[0083] 1. Low resource usage: The center drives the display, which has low requirements on the display capability of the display and control terminal, and low requirements on the terminal processor and memory, which greatly reduces the equipment cost and is suitable for use in embedded devices.

[0084] 2. Efficient data transmission: Both graphics and control use multicast protocols to reduce repeated data transmission, simple software technology, and save bandwidth.

[0085] 3. Adaptive display: Centrally driven, supports heterogeneous terminals with different resolutions, no need to customize and modify applications for devices, ensures consistency of displayed content, and interpolation adaptive algorithm ensures that content is still clearly displayed after scaling. It is compatible with various screen sizes and has low software and hardware maintenance costs.

[0086] 4. Real-time interaction: The center centrally manages control instructions and generates graphics technology, transmits a small amount of data, avoids fragmentation and retransmission, reduces interaction delays, and improves user experience.

[0087] 5. Strong scalability: Plug-and-play configuration technology supports multi-terminal concurrent operation without significantly increasing server load. It can support multiple terminals concurrently, and the server performance and resource occupancy are stable.

Claims

1. A low-resource real-time multi-screen synchronization and adaptive interactive display framework, characterized in that: It includes a display and control terminal, a central server module and a communication transmission module. The display and control terminal and the central server module establish a control multicast group and a graphic multicast group through the communication transmission module. The display and control terminal in use captures the user input events and screen coordinate position requests and assembles them into a communication protocol format and sends them to the communication transmission module along the control multicast group; The communication transmission module adopts UDP multicast mode to parse user input events and screen coordinate position requests into data and control instructions and transmit them to the central server module along the control multicast group; The central server module obtains data and control instructions, manages the legitimacy of multiple display and control terminals with the help of the communication transmission module, runs the graphics framework to perform graphical interface and rendering according to the data and control instructions for the display and control terminals in use that have passed the verification, and generates new graphics in response to the processing. According to the graphic display strategy, the graphics content is read and encapsulated into graphics data in the video memory, and the graphics data in the video memory is sent to the communication transmission module along the graphics multicast group according to the protocol rules; The communication transmission module uses UDP multicast to transmit the graphics data in the display memory along the graphics multicast group to the display control terminal corresponding to the control instruction; The display control terminal that receives the video memory graphic data verifies and analyzes the video memory graphic data, adapts the analyzed video memory graphic data to screens with different resolutions to realize adaptive graphics and flushes them into the video memory for display.

2. A low-resource real-time multi-screen synchronization and adaptive interactive display framework as claimed in claim 1, characterized in that: The display control terminal verifies the central server module, and if the verification is successful, receives the display memory graphic data transmitted by the central server module through the communication transmission module.

3. A low-resource real-time multi-screen synchronization and adaptive interactive display framework as claimed in claim 1, characterized in that: The central server reads the video memory address mapped with the video memory graphic data in segments, and sends it to the display control terminal corresponding to the control instruction through multicast multipoint communication protocol encapsulation.

4. A low-resource real-time multi-screen synchronization and adaptive interactive display framework as claimed in claim 1, characterized in that: The legality management steps include: Establish multicast groups for device management in multiple display and control terminals; The central server joins the device management multicast group and runs the receiving task; When the display and control terminal is turned on, it declares to join the device management multicast group and sends the key string to the central server through the communication transmission module. After the central server authenticates according to the announced password, it will bring the legal display and control device under management.

5. A low-resource real-time multi-screen synchronization and adaptive interactive display framework as claimed in claim 1, characterized in that: According to the data packet offset and the native resolution, the graphic size is dynamically adjusted through interpolation, and the graphic is scaled up to ensure display efficiency and quality, so as to achieve the adaptation of the parsed video memory graphic data to screens with different resolutions.

Images