Double-touch-screen interaction system and method based on protocol conversion and multiple logic terminals

Abstract

The invention relates to a double-touch-screen interaction system and method based on protocol conversion and multiple logic terminals, and the system comprises a hardware layer, a driving layer, a system service layer and an application layer. Wherein the hardware layer adopts a uniprocessor platform, the uniprocessor platform is connected with two touch screens, and the touch screens transmit touch signals to the uniprocessor platform through serial ports; the driving layer is provided with a protocol conversion module, and the protocol conversion module is used for converting a touch signal into a system standard input event; the system service layer is provided with a multi-logic terminal isolation module, the multi-logic terminal isolation module creates an independent logic terminal for each touch screen, and physical equipment isolation is realized through a udev binding strategy; and the application layer adopts a multi-instance architecture to realize single-processor double-screen independent operation. According to the invention, the development complexity and the hardware cost can be reduced.

Description

Technical Field

[0001] This invention relates to the field of embedded systems and touch screen interaction technology, and in particular to a dual touch screen interaction system and method based on protocol conversion and multiple logic terminals. Background Technology

[0002] In the field of avionics, the demand for independent dual-screen operation is becoming increasingly prominent, but existing technical solutions still have significant shortcomings in adapting to complex hardware environments and high reliability requirements. Traditional methods typically rely on multi-processor architectures or mirror / extended modes to achieve multi-screen control. The former significantly increases system complexity and cost due to the need for independent hardware units, while the latter is difficult to meet the needs of independent dual-screen interaction due to limitations in operational logic (such as the inability to truly isolate input / output). In addition, the widespread use of non-standard touch devices (such as aviation-specific displays based on the RS422 protocol) forces developers to deeply couple hardware protocol parsing logic (such as coordinate calibration and gesture mapping) at the application layer, resulting in code redundancy, poor cross-device compatibility, and high system maintenance costs.

[0003] While existing development tools (such as SCADE) offer good cross-platform compatibility, their model-based programming paradigm still has limitations in supporting complex desktop applications (such as dynamic layer overlays and multi-control interactions). Developers need to invest a significant amount of time adapting to hardware characteristics and interaction logic, resulting in a high learning curve. On the other hand, while the operating system's native multi-seat mechanism can achieve user-level isolation, its design is primarily geared towards remote terminal scenarios and is not optimized for local dual-screen hardware binding. It cannot automatically associate physical devices (such as dynamically binding a specific touchscreen to a monitor) and lacks resource scheduling support for high real-time tasks, making it difficult to directly apply to aviation-grade interaction scenarios. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a dual touch screen interactive system and method based on protocol conversion and multiple logical terminals, thereby reducing development complexity and hardware costs.

[0005] The technical solution adopted by the present invention to solve its technical problem is: to provide a dual touch screen interactive system based on protocol conversion and multiple logical terminals, including a hardware layer, a driver layer, a system service layer and an application layer;

[0006] The hardware layer adopts a single-processor platform, which is connected to two touch screens. The touch screens transmit touch signals to the single-processor platform via a serial port, and the single-processor platform transmits the touch signals to the driver layer.

[0007] The driver layer is equipped with a protocol conversion module, which is used to convert the touch signal into a system standard input event and transmit the system standard input event to the system service layer.

[0008] The system service layer deploys a multi-logical terminal isolation module. This module creates an independent logical terminal session for each touchscreen and implements physical device isolation through a udev binding strategy. The physical device isolation includes input isolation and output isolation. Input isolation binds the touchscreen to the logical terminal to ensure that touch events are routed only to the corresponding logical terminal. Output isolation binds the physical display pipeline to the corresponding virtual seat (HDMI-0 to seat0, HDMI-1 to seat1) based on DRM to achieve physical isolation of the graphics output channels of each logical terminal.

[0009] The application layer deploys a collaborative operation architecture, which starts multiple application instances in an operating system isolation session, and each instance achieves cross-terminal collaboration through the IPC communication mechanism.

[0010] The single-processor platform connects to two touchscreens via dual video output interfaces.

[0011] The protocol conversion module performs transparent parsing of the touch signal protocol based on the Serio bus, including data frame verification, coordinate calibration, and gesture type mapping.

[0012] The multi-logic terminal isolation module creates an independent logical terminal for each touchscreen based on the Linux multi-user session mechanism.

[0013] The application layer utilizes system semaphores to ensure atomic access to critical data among multiple instances; it combines LocalSocket communication to achieve cross-process state synchronization; it implements the last-write priority principle based on timestamps to synchronize shared data; and each instance has an independent metadata storage path to achieve isolation of non-shared data.

[0014] The application layer is also used to capture heartbeat abnormal events reported by the driver layer. When the signal loss of the touch screen is detected, the instance function of the inactive touch screen is automatically downgraded to the lowest function display window, while retaining data synchronization and heartbeat monitoring tasks. When a heartbeat recovery event is captured, the function recovery is triggered.

[0015] The technical solution adopted by this invention to solve its technical problem is: to provide a dual-touchscreen interaction method based on protocol conversion and multiple logical terminals, using the above-mentioned dual-touchscreen interaction system based on protocol conversion and multiple logical terminals, including the following steps:

[0016] The driver layer parses the protocol of the touch signal in real time and converts the touch signal into the system's standard input event;

[0017] The system service layer multi-logical terminal isolation module creates two independent logical terminals and configures rule files to associate each logical terminal with an independent input / output device;

[0018] The application layer uses two user login systems, loads the application to start two session programs, and each user instance outputs the screen to the corresponding display screen according to its corresponding logical terminal configuration, and receives and responds to its own touch input events.

[0019] The application layer uses LocalServer / LocalSocket to pass instructions between instances, combines system semaphores to avoid data contention, and uses database sharding to isolate metadata storage paths, preserving each instance's independent operation.

[0020] The driver layer monitors the touchscreen's heartbeat and reports heartbeat loss and heartbeat recovery events. The application layer uses QSocketNotifier to listen for input events. When a heartbeat loss event is detected, it triggers function degradation, suspends unnecessary tasks, and automatically switches the main interface. When a heartbeat recovery event is detected, it triggers function recovery.

[0021] Beneficial effects

[0022] By adopting the above-mentioned technical solution, the present invention has the following advantages and positive effects compared with the prior art:

[0023] This invention adopts an integrated design that drives dual screens with a single processor, replacing the traditional multi-processor or dedicated graphics card solution. By reducing the number of hardware components and communication bus dependence, it reduces system complexity and manufacturing costs. Based on logical terminal isolation, a single processor can efficiently carry the dual-screen task load, maximizing the utilization of hardware resources.

[0024] This invention achieves transparent conversion from non-standard RS422 protocol to standard input events through the driver layer, eliminating the need for application layer developers to write hardware protocol parsing code (traditional solutions require developing a large amount of customized code for each screen). This allows interface development to be completed entirely based on general frameworks such as QT, significantly shortening the development cycle. The modular architecture design further supports dynamic loading and cross-platform reuse of functional controls, improving code maintainability and device compatibility.

[0025] This invention features a physical-level isolation design for the input / output channels (based on multi-logic terminal modification), combined with a kernel-level event distribution mechanism, which can meet the requirements of independent response and low-latency processing of dual-screen touch events. Through heartbeat monitoring, the system can quickly activate a function degradation mode when a single-screen anomaly is detected, ensuring the continuous operation of core functions.

[0026] This invention employs a hybrid synchronization strategy based on system semaphores and inter-process communication (IPC) to achieve atomic control and real-time state synchronization of critical data operations, effectively mitigating data contention risks. This design ensures the flexibility of independent dual-screen operation while maintaining global data consistency and enhancing system fault tolerance. Attached Figure Description

[0027] Figure 1 This is an architecture diagram of a dual-touchscreen interactive system based on protocol conversion and multiple logical terminals according to an embodiment of the present invention.

[0028] Figure 2 This is a flowchart illustrating the conversion of touch signals in an embodiment of the present invention;

[0029] Figure 3 This is a flowchart illustrating the data synchronization and mutual exclusion design in an embodiment of the present invention.

[0030] Figure 4 This is an example diagram of fault detection and function degradation in an embodiment of the present invention. Detailed Implementation

[0031] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0032] The embodiments of the present invention relate to a dual-touchscreen interactive system based on protocol conversion and multiple logical terminals. The system is suitable for dual-operation terminal scenarios with high reliability requirements (such as aircraft cockpits and industrial control consoles). By abstracting the driver layer protocol and isolating multiple logical terminals, it enables a single processor to drive independent operation of the two screens. Furthermore, it achieves functional degradation and data sharing after a single screen fails based on shared locks and IPC communication.

[0033] like Figure 1 As shown, the dual-touchscreen interactive system based on protocol conversion and multiple logical terminals includes a hardware layer, a driver layer, a system service layer, and an application layer.

[0034] The hardware layer adopts a single-processor platform, which is connected to two touch screens. The touch screens transmit touch signals to the single-processor platform via a serial port, and the single-processor platform transmits the touch signals to the driver layer.

[0035] In this embodiment, an embedded processor (Phytium 2000 / 4) that supports dual video output is selected as the single-processor platform, and two non-standard RS422 touch screens are connected via HDMI and DP interfaces respectively.

[0036] Two RS422 touchscreens are connected to the processor via serial ports ( / dev / ttyG0, / dev / ttyG3), with a baud rate configured at 115200bps. Their data frame format is defined as follows:

[0037] [Frame Header][Package Length][Command Code][First Point State][X Coordinate][Y Coordinate]...[Fifth State][X Coordinate][Y Coordinate][Checksum]

[0038] Therefore, this implementation method adopts an integrated design that drives dual screens with a single processor, replacing the traditional multi-processor or dedicated graphics card solution. By reducing the number of hardware components and communication bus dependence, it reduces system complexity and manufacturing costs. Based on logical terminal isolation, a single processor can efficiently carry the dual-screen task load, maximizing the utilization of hardware resources.

[0039] The driver layer is equipped with a protocol conversion module. The protocol conversion module converts the protocol of the touch signal into a standard event that conforms to the Linux input subsystem (evdev) through the serialtouch driver (based on the Serio bus), and transmits the system standard input event to the system service layer to realize the reporting of coordinates and gesture status.

[0040] This implementation achieves transparent parsing of the RS422 protocol at the operating system kernel level, including data frame verification, coordinate calibration (adapting to the screen's physical resolution), and gesture type mapping (such as pressing and swiping), generating standard evdev events (ABS_X / Y, BTN_TOUCH), allowing the application layer to directly call the general input interface without being aware of hardware differences. Figure 2 As shown, the specific implementation is as follows:

[0041] Driver registration: Register the serialtouch driver in the kernel and associate it with the RS422 serial port device.

[0042]

[0043]

[0044] Data parsing: The RS422 data frame is parsed in the serialtouch_interrupt interrupt. For touch data, the coordinates and pressure values ​​are extracted, and standard input events are generated after calibration.

[0045] input_report_abs(dev,ABS_X,calibrated_x);

[0046] input_report_abs(dev,ABS_Y,calibrated_y);

[0047] input_report_key(dev,BTN_TOUCH,pressure>0?1:0);

[0048] input_sync(dev);

[0049] It's easy to see that this implementation method achieves transparent conversion from non-standard RS422 protocol to standard input events through the driver layer, eliminating the need for application layer developers to write hardware protocol parsing code (traditional solutions require developing a large amount of customized code for each screen). This allows interface development to be completed entirely based on general frameworks such as QT, significantly shortening the development cycle. The modular architecture design further supports dynamic loading and cross-platform reuse of functional controls, improving code maintainability and device compatibility.

[0050] The system service layer deploys a multi-logical terminal isolation module. This module, based on the Linux multi-seat mechanism, creates independent logical terminals (seat0, seat1) for each touchscreen and implements physical device isolation through udev binding policies. Physical device isolation includes input isolation and output isolation. Input isolation binds the RS422 touchscreen to the logical terminal, ensuring that touch events are routed only to the corresponding logical terminal. Output isolation uses DRM to bind the physical display pipeline to the corresponding virtual seats (e.g., HDMI-0 bound to seat0, HDMI-1 bound to seat1) to achieve physical isolation of the image output channels of each logical terminal.

[0051] In this implementation, the udev binding strategy is as follows: Create a rule file / etc / udev / rules.d / 99-seat.rules to dynamically associate the device with the logical terminal:

[0052] bash

[0053] #Pair HDMI-1 and event4 to seat0

[0054] SUBSYSTEM=="drm",KERNEL=="card0-HDMI-A-1",ENV{ID_SEAT}="seat0"

[0055] SUBSYSTEM=="input",KERNEL=="event4",ENV{ID_SEAT}="seat0"

[0056] #Connect HDMI-2 and event5 to seat1

[0057] SUBSYSTEM=="drm",KERNEL=="card0-HDMI-A-2",ENV{ID_SEAT}="seat1"

[0058] SUBSYSTEM=="input",KERNEL=="event5",ENV{ID_SEAT}="seat1"

[0059] This application layer is developed based on the QT framework and deploys a cooperating operating system. Multiple application instances are launched within isolated sessions of the cooperating operating system. These instances achieve cross-terminal collaboration through IPC communication. In other words, this implementation uses two users to log into the system, load applications, and launch two session programs. Each user instance independently receives and processes its own touchscreen input according to its terminal logic configuration and outputs the screen to an independent display terminal, achieving independent dual-screen operation. It utilizes system semaphores (QSystemSemaphore) to ensure atomic access to critical data between multiple instances, combines LocalSocket communication to achieve cross-process state synchronization, and implements a last-write-first principle based on timestamps to synchronize shared data. Simultaneously, each instance has an independent metadata storage path (SQLite sharding) to isolate non-shared data. These synchronization strategies collectively achieve mutual exclusion of operations and data synchronization. Figure 3 As shown, the implementation of this application layer is as follows:

[0060] Semaphore initialization:

[0061] / / Create a semaphore

[0062] QSystemSemaphore semaphore("flight_plan_sem",1,QSystemSemaphore::open);

[0063] / / Attempting to acquire a semaphore

[0064] if(!semaphore.tryAcquire());

[0065] {

[0066] ShowMsg(“In use…”); / / Notify me that the device is in use

[0067] return;

[0068] }

[0069] / / Mutual exclusion operations, data modification, etc.

[0070] DoEdit();

[0071] semaphore.release();

[0072] LocalSocket communication:

[0073] QLocalServer server;

[0074] server.listen("sync_channel");

[0075] connect(&server,&QLocalServer::newConnection,[]()

[0076] {

[0077] QLocalSocket*socket=server.nextPendingConnection();

[0078] socket->write(updated_data); / / Broadcast data update

[0079] });

[0080] In this embodiment, the driver layer is also used to monitor the touchscreen's heartbeat and report heartbeat loss and heartbeat recovery events. For heartbeat data, the timer is reset upon receiving a heartbeat; if no heartbeat is received for three consecutive times, the timer times out, generating a heartbeat loss event (KEY_HEARTBEAT_LOST). Upon heartbeat recovery (when a heartbeat is received and the previous state was heartbeat loss), a heartbeat recovery event (KEY_HEARTBEAT_RESTORE) is generated. The processing method is as follows:

[0081] Timeout handling:

[0082] if(!ht_lost)

[0083] {

[0084] input_report_key(dev,KEY_HEARTBEAT_LOST,1);

[0085] input_sync(dev);

[0086] input_report_key(dev,KEY_HEARTBEAT_LOST,0);

[0087] input_sync(dev);

[0088] }

[0089] mod_timer(); / / Keeps it active and continuously monitors

[0090] Heartbeat processing:

[0091] if(check_success)

[0092] {

[0093] if(ht_lost)

[0094] {

[0095] input_report_key(dev,KEY_HEARTBEAT_RESTORE,1);

[0096] input_sync(dev);

[0097] input_report_key(dev,KEY_HEARTBEAT_RESTORE,0);

[0098] input_sync(dev);

[0099] }

[0100] mod_timer(); / / Reset the timer

[0101] }

[0102] The application layer also listens for heartbeat anomaly events (i.e., heartbeat loss events and heartbeat recovery events) reported by the driver layer via QSocketNotifier. When a signal loss is detected from the touchscreen, the instance of the inactive touchscreen is automatically downgraded to the lowest function display window, while retaining data synchronization and heartbeat monitoring tasks. Function recovery is triggered upon capturing a heartbeat recovery event. For example... Figure 4 As shown, the processing method is as follows:

[0103] if (event->type == EV_KEY)

[0104] {

[0105] if(event->code==KEY_HEARTBEAT_LOST&&event->value==1)

[0106] {

[0107] EnterDegradedMode(); / / Enable feature degradation

[0108] }

[0109] else if(event->code==KEY_HEARTBEAT_RESTORE&&event->value==1)

[0110] {

[0111] RestoreNormalMode(); / / Restores normal mode

[0112] }

[0113] }

[0114] It is easy to see that the physical isolation design of the input / output channels in this implementation (based on multi-logic terminal modification), combined with the kernel-level event distribution mechanism, can meet the requirements of independent response and low-latency processing of dual-screen touch events. Through heartbeat monitoring, the system can quickly activate a function degradation mode when a single-screen anomaly is detected, ensuring the continuous operation of core functions. Simultaneously, this implementation uses a hybrid synchronization strategy based on system semaphores and inter-process communication (IPC) to achieve atomic control and real-time state synchronization of critical data operations, effectively avoiding data contention risks. This design ensures the flexibility of independent dual-screen operation while maintaining global data consistency and improving system fault tolerance.

[0115] This embodiment also provides a dual-touchscreen interaction method based on protocol conversion and multiple logical terminals, which adopts the above-mentioned dual-touchscreen interaction system based on protocol conversion and multiple logical terminals, and includes the following steps:

[0116] Step 1: The driver layer parses the touch signal protocol in real time and converts the touch signal into a system standard input event; specifically, this includes: parsing the coordinates and gesture information in the data frame; calibrating the coordinates to the screen object resolution and generating standard input events (ABS_X, ABS_Y, BIN_TOUCH).

[0117] Step 2: The system service layer multi-logical terminal isolation module creates two independent logical terminals and configures rule files to associate each logical terminal with an independent input / output device; specifically, it includes binding the touch device (dev / input / eventX) to the logical terminal; and binding the display device ( / sys / class / drm / card0-HDMI-AY) to the corresponding logical terminal.

[0118] Step 3: The application layer uses two user login systems, loads the application, and starts two session programs. Each user instance outputs its screen to the corresponding display screen according to its corresponding logical terminal configuration, and receives and responds to its own touch input events. The application layer starts an independent software instance for each logical terminal and implements the following synchronization and conflict resolution mechanisms:

[0119] (3.1) Mutual exclusion operation: restricting atomic access to critical data through system semaphores;

[0120] (3.2) Parallel operation: Based on timestamps, the last write priority principle is implemented, and data is overwritten according to the last time to resolve data conflict issues;

[0121] (3.3) Independent operation: The use of database sharding (SQlite) isolates the metadata storage path of each instance, allowing instances to independently read and write non-shared data;

[0122] (3.4) Data synchronization: The server name is dynamically identified based on the user (Seat) ID. It acts as both a server and a client, and exchanges data based on bidirectional communication between QLocalServer and QLocalSocket to meet specific data synchronization needs.

[0123] This application layer can also perform cross-process data management, specifically by using LocalServer / LocalSocket to pass instructions between instances, using system semaphores to avoid data contention, using database sharding to isolate metadata storage paths, and retaining each instance's independent operation.

[0124] In this implementation, the driver layer parses the touch signal protocol in real time and converts the touch signal into standard system input events. Simultaneously, it monitors the touchscreen's heartbeat, reporting heartbeat loss and heartbeat recovery events. The application layer uses QSocketNotifier to listen for input events. Upon detecting a heartbeat loss event, it triggers a function degradation mechanism, pausing unnecessary tasks and automatically switching to the main interface. Upon detecting a heartbeat recovery event, it triggers a function recovery mechanism.

[0125] It is easy to see that this invention converts non-standard RS422 touch signals into standard system input events at the driver layer and utilizes multi-logic terminal isolation technology to achieve independent operation of dual screens driven by a single processor. At the application layer, based on system semaphores, inter-process communication, a timestamp-based last-write priority principle, and independent metadata storage in a distributed database, mutual exclusion and data synchronization between multiple instances of the same application are achieved, reducing development complexity and hardware costs. This makes it suitable for scenarios such as aircraft cockpits and industrial control.

Claims

1. A dual-touchscreen interactive system based on protocol conversion and multiple logical terminals, characterized in that, It includes the hardware layer, driver layer, system service layer, and application layer; The hardware layer adopts a single-processor platform, which is connected to two touch screens. The touch screens transmit touch signals to the single-processor platform via a serial port, and the single-processor platform transmits the touch signals to the driver layer. The driver layer is equipped with a protocol conversion module, which is used to convert the touch signal into a system standard input event and transmit the system standard input event to the system service layer. The system service layer deploys a multi-logical terminal isolation module. The multi-logical terminal isolation module creates an independent logical terminal session for each touch screen and implements physical device isolation through udev binding strategy. The physical device isolation includes input isolation and output isolation. Input isolation binds the touch screen to the logical terminal to ensure that touch events are only routed to the corresponding logical terminal. Output isolation is based on DRM to bind the physical display pipeline to the corresponding virtual seat to achieve physical isolation of the image output channels of each logical terminal. The application layer deploys a collaborative operating system, and multiple application instances are launched in the isolated session of the collaborative operating system. Each application instance achieves cross-terminal collaboration through the IPC communication mechanism.

2. The dual-touchscreen interactive system based on protocol conversion and multiple logical terminals according to claim 1, characterized in that, The single-processor platform connects to two touchscreens via dual video output interfaces.

3. The dual touchscreen interactive system based on protocol conversion and multiple logical terminals according to claim 1, characterized in that, The protocol conversion module performs transparent parsing of the touch signal protocol based on the Serio bus, including data frame verification, coordinate calibration, and gesture type mapping.

4. The dual touchscreen interactive system based on protocol conversion and multiple logical terminals according to claim 1, characterized in that, The multi-logic terminal isolation module creates an independent logical terminal for each touchscreen based on the Linux multi-user session mechanism.

5. The dual touchscreen interactive system based on protocol conversion and multiple logical terminals according to claim 1, characterized in that, The application layer utilizes system semaphores to ensure atomic access to critical data among multiple instances; Cross-process state synchronization is achieved by combining LocalSocket communication; the last-write priority principle is implemented based on timestamps to synchronize shared data; each instance has an independent metadata storage path to achieve isolation of non-shared data.

6. The dual-touchscreen interactive system based on protocol conversion and multiple logical terminals according to claim 1, characterized in that, The application layer is also used to capture heartbeat abnormal events reported by the driver layer. When the signal loss of the touch screen is detected, the instance function of the inactive touch screen is automatically downgraded to the lowest function display window, while retaining data synchronization and heartbeat monitoring tasks. When a heartbeat recovery event is captured, the function recovery is triggered.

7. A dual-touchscreen interaction method based on protocol conversion and multiple logical terminals, characterized in that, The dual-touchscreen interactive system based on protocol conversion and multiple logical terminals as described in any one of claims 1-6 includes the following steps: The driver layer parses the protocol of the touch signal in real time and converts the touch signal into the system's standard input event; The system service layer multi-logical terminal isolation module creates two independent logical terminals and configures rule files to associate each logical terminal with an independent input / output device; The application layer uses two user login systems, loads the application to start two session programs, and each user instance outputs the screen to the corresponding display screen according to its corresponding logical terminal configuration, and receives and responds to its own touch input events.

8. The dual-touchscreen interaction method based on protocol conversion and multiple logical terminals according to claim 7, characterized in that, The application layer uses LocalServer / LocalSocket to pass instructions between instances, combines system semaphores to avoid data contention, and uses database sharding to isolate metadata storage paths, preserving each instance's independent operation.

9. The dual-touchscreen interaction method based on protocol conversion and multiple logical terminals according to claim 7, characterized in that, The driver layer monitors the touchscreen's heartbeat and reports heartbeat loss events and heartbeat recovery events. The application layer uses QSocketNotifier to listen for input events. When a heartbeat loss event is detected, it triggers a function degradation, suspends unnecessary tasks, and automatically switches to the main interface. When a heartbeat recovery event is detected, it triggers a function recovery.

Images