Multi-window interaction optimization method for containerized swan gap application

Abstract

The invention discloses a multi-window interaction optimization method for containerized swan application, which comprises the following steps of: when the containerized swan application is started, creating an independent host side window at a host side, storing an initial state into a first window state table, and adding a binding relationship among the swan application, the host side window and input equipment into a first binding table; a differentiated event receiving rule is configured to start a result receiving service, a container side starts an event receiving service and a result feedback service to mark an application as a ready state, a host side generates an original input event when a user operates an input device, a first window state table is updated, and an event processing rule is adjusted; performing differential parameter preprocessing on the original input event to obtain a unified window event, performing differential event processing on the unified window event by the event receiving service to generate an event processing result, sending the event processing result to a host side, and performing differential interaction response by the result receiving service according to an application type to complete interaction.

Description

Technical Field

[0001] This invention belongs to the field of computer software development technology, specifically relating to a multi-window interaction optimization method for containerized HarmonyOS applications. Background Technology

[0002] With the widespread adoption of container technology in cross-system application deployments, when OpenHarmony runs on a Linux system via containers, input event interaction becomes a key bottleneck affecting user experience, especially when OpenHarmony applications are bound to independent Linux windows. However, existing technologies suffer from the following main drawbacks: First, the granularity of user input collection is insufficient. Current solutions only collect basic event parameters (such as touch coordinates and keyboard key codes), failing to capture pen tilt angles, pressure sensitivity, game controller joystick displacement, and the semantic intent of voice commands, leading to a degraded interactive experience in professional scenarios (such as digital tablet drawing and AAA games). Second, Linux-side window event processing is crude, lacking differentiated preprocessing logic designed for the one-to-one binding of applications and windows. For example, game windows require low latency but share the same event queue as office windows, while drawing windows require high-precision pressure sensitivity but are forcibly compressed to levels 0 to 1024. Furthermore, the event reception rules are not dynamically adjusted when the window state changes, resulting in incorrect event delivery. Third, semantic loss occurs in OpenHarmony application event processing due to differences in the definitions of event parameters between Linux and OpenHarmony. The existing static mapping table cannot adapt to hot-swapping of devices in real time (such as inserting a new graphics tablet), resulting in semantic loss during application processing (e.g., the pressure sensitivity value 2048 is mistakenly mapped to 512 on the Linux side); fourth, the feedback loop is broken. After the OpenHarmony application processes events (such as game character movement or drawing lines), there is a delay (>50ms) in the feedback to the Linux side window, and there is a lack of real-time synchronization of event processing status (e.g., the application freezes without notifying the Linux side to stop sending events), resulting in asynchrony between user operation and window response; fifth, multi-screen scenario adaptation fails. When the Linux system is connected to multiple monitors (main screen for office work, secondary screen for gaming), the existing solution cannot distinguish the event target based on the screen to which the window belongs, resulting in the secondary screen game controller event being mistakenly sent to the main screen office window, and the event coordinates are not recalibrated when dragging the window across screens.

[0003] In summary, OpenHarmony still suffers from problems such as low accuracy of cross-system event targeting, poor semantic consistency, lag in focus and window synchronization, and disordered multi-screen interaction when running in containers on Linux systems. Summary of the Invention

[0004] In view of this, the present invention provides a multi-window interaction optimization method for containerized HarmonyOS applications, which realizes end-to-end interaction of containerized HarmonyOS applications from user input to interface presentation.

[0005] This invention provides a method for optimizing multi-window interaction in containerized HarmonyOS applications, specifically including the following steps:

[0006] When a containerized HarmonyOS application starts, the host side creates an independent host-side window for the application, assigns a host-side window ID, saves the obtained initial state to the first window state table, adds the binding relationship between the HarmonyOS application, the host-side window, and the input device to the first binding table, including the application type, input device path, and allowed event types, configures differentiated event receiving rules according to the application type and initial window state, starts the result receiving service, and then sends the HarmonyOS application ID, host-side window ID, binding relationship, and event receiving rules to the container side. The container side starts the event receiving service and result feedback service, and marks the HarmonyOS application as ready.

[0007] When a user interacts with an input device, the host side generates a raw input event, including the input device path, event type, raw parameters, and timestamp. It queries the first binding table, writes the host-side window ID into the raw input event, records the source screen ID, converts the screen coordinates to relative coordinates, writes them into the raw input event, updates the first window state table based on the current window state, adjusts the event handling rules, preprocesses the raw input event with differentiated parameters based on the application type, and then encapsulates it into a unified window event, including the event ID, target host-side window ID, target window's home screen ID, event type, priority, event generation timestamp, and application type differentiated parameters. The unified window event is then sent to the container side.

[0008] The application's event receiving service performs differentiated event processing on the unified window event according to the application type, generates the event processing result, and sends it to the host side; the host side's result receiving service parses the event processing result, and if the processing status is failure, it prompts an error and retryes; if the processing status is crash, it triggers the HarmonyOS application to restart automatically; if the processing status is success, it executes differentiated interactive response according to the application type to complete the interaction.

[0009] Furthermore, the method of configuring differentiated event receiving rules based on application type and initial window state is as follows: If the application type is a game application, then configure a low-latency event queue, enable gamepad joystick dead zone filtering and button combo detection, and reserve a vibration feedback interface; if the application type is a drawing application, then configure a first-level parameter mapping rule, preset pen pressure sensitivity scaling formula and coordinate transformation logic; if the application type is an office application, then configure a keyboard key code mapping table and voice semantic filtering rules, and preset a system conflict shortcut key blocking strategy.

[0010] Furthermore, the host side generates original input events by collecting different full parameters for different input devices, and constructing original input events from the full parameters. Specifically, for touch devices, precision coordinates, touch pressure, and multi-finger IDs are collected, with a fixed sampling rate of 120Hz; for digital pen devices, coordinates, original pressure sensitivity, tilt angle, rotation angle, and side button status are collected; for game controllers, joystick axis displacement, button status, and LED status are collected, with a reserved vibration feedback interface; for keyboard devices, key codes, modifier key status, and keyboard layout are collected, supporting automatic layout reloading after hot-swapping; for voice devices, voice streams are collected, speech-to-text conversion and semantic intent extraction are completed, and confidence levels are recorded.

[0011] Furthermore, the event handling rules are adjusted according to the current window state as follows: if the current window state is a minimized window, the wake-up event is retained, and the remaining events are temporarily stored in the wake-up queue; if the current window state is a semi-transparent window, the sliding and dragging events are discarded, and only the click and key events with a confidence greater than the threshold are retained; if the current window state is a fully obscured window, only the application-related global shortcut key events are retained, and the remaining events are directly discarded; if the current window state is a focused window, the preprocessing priority is increased.

[0012] Furthermore, the method of performing differentiated parameter preprocessing on the original input event according to the application type is as follows: if the application type is a game application, then joystick dead zone filtering and combo detection are performed, and a vibration intensity field is reserved; if the application type is a drawing application, then pressure sensitivity parameter scaling, coordinate transformation and sampling rate adaptation are performed; if the application type is an office application, then key code conversion, setting confidence level voice result filtering and system conflict shortcut key blocking are performed.

[0013] Furthermore, the event receiving service of the application performs differentiated event processing on the unified window event according to the application type as follows: If the application type is a game application, the controller parameters are parsed and converted into character control commands, combo logic is processed, vibration feedback commands are generated based on the game scene, the application status is detected by heartbeat, and a crash status notification is sent to the host side when there is no response; If the application type is a drawing application, the pen parameters are parsed and converted into pen stroke parameters, the vector drawing engine is called to complete the drawing, abnormal parameters are automatically corrected and a warning is sent to the host side; If the application type is an office application, keyboard and voice events are parsed, text input, document operations and shortcut key responses are completed, conflicting shortcut key prompts are handled, and the voice input text format is corrected.

[0014] Furthermore, the method for executing differentiated interactive responses based on application type is as follows:

[0015] If the application type is a game application, the window screen will be updated based on the processing results, the controller will be vibrated, and the window refresh frame rate will be adjusted synchronously. If the application type is a drawing application, the line rendering will be completed based on the drawing parameters, the window canvas will be refreshed, and the pressure sensitivity and tilt angle parameters will be visualized. If the application type is an office application, text insertion, cursor synchronization, and operation result prompts will be completed.

[0016] Furthermore, the container-side launch of the event receiving service and result feedback service is as follows: the container launches the event receiving service for the HarmonyOS application, binds and sets a port so that it only receives event data from that port; the result feedback service is launched, and a set port is bound to send the processing results back to the host side.

[0017] Furthermore, the host side pre-configures a dedicated transmission channel for windows in multi-screen scenarios. After the host side and container side complete the connectivity verification of the dedicated channel for HarmonyOS applications, the HarmonyOS application is then marked as ready.

[0018] Furthermore, the host side queries the first binding table based on the input device path in the original input event. If there is no corresponding binding relationship in the first binding table, the input event is marked as illegal, discarded directly, and logged. Otherwise, the event type is checked to see if it is an allowed event type. Unallowed events are temporarily stored in the host side window's dedicated low-priority queue, while allowed events have the host side window ID in the binding relationship written into the target window ID field of the original input event.

[0019] Beneficial effects:

[0020] When a HarmonyOS application starts, the host side creates a dedicated independent window for it, assigns a globally unique window ID, and completes the initial registration of the window state table with the binding tables of the HarmonyOS application, the host window, and the input device. Differentiated event receiving rules are configured according to the application type, and the result receiving service is started. Simultaneously, core configuration information is synchronized to the container side, completing the startup of event receiving and result feedback services, and mapping configuration of dedicated communication ports and multi-screen dedicated transmission channels. After dual-end connectivity verification, the application is marked as input-ready. When a user operates an input device, the host side collects all parameters to generate raw input events. Based on the binding table, it performs dual verification of device validity and event type permissions, and binds the target window ID to the valid event to complete the initial... Initial orientation; the host side updates the window state in real time and dynamically adjusts the event handling rules. Based on the application type, it performs differentiated parameter preprocessing on the original events and encapsulates them into unified window events. Then, it completes hierarchical and precise routing based on the real-time window state. After receiving the event, the container-side application performs a second check on the consistency of the window ID and the integrity of the event. After the check passes, it performs differentiated event handling according to the application type. After the processing is completed, it generates an event handling result with traceability information and sends it back to the host side. After the host side completes the result validity check, it performs actions such as discarding invalid results, retrying failed events, and automatically restarting the application if it crashes, depending on the processing status. If the processing is successful, it performs differentiated window interaction response, thus fully realizing the end-to-end interaction from user input to interface presentation. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a multi-window interaction optimization method for containerized HarmonyOS applications provided by the present invention. Detailed Implementation

[0022] The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0023] This invention provides a multi-window interaction optimization method for containerized HarmonyOS applications. The core idea is as follows: When a HarmonyOS application starts, the host side creates a dedicated independent window, assigns a globally unique window ID, and completes the initial registration of the window state table with the binding tables of the HarmonyOS application, host window, and input device. Differentiated event receiving rules are configured according to the application type, and the result receiving service is started. Simultaneously, core configuration information is synchronized to the container side, completing the startup of event receiving and result feedback services, and mapping configuration of dedicated communication ports and multi-screen dedicated transmission channels. After dual-end connectivity verification, the application is marked as input-ready. When a user operates an input device, the host side collects all parameters to generate raw input events. Based on the binding table, dual verification of device validity and event type permissions is performed. Legitimate events are bound to the target window ID to complete initial routing; the host side updates the window state in real time and dynamically adjusts the event handling rules, performs differentiated parameter preprocessing on the original events according to the application type and encapsulates them into unified window events, and then completes hierarchical and precise routing based on the real-time window state; after receiving the event, the container-side application completes a second check for window ID consistency and event integrity. After the check passes, it performs differentiated event handling according to the application type. After the handling is completed, it generates an event handling result with traceability information and sends it back to the host side; after the host side completes the result validity check, it performs invalid result discarding, retrying the handling failure event, and automatically restarting the application crash according to the handling status. If the handling is successful, it performs differentiated window interaction response, thus fully realizing the end-to-end interaction from user input to interface presentation.

[0024] This invention provides a multi-window interaction optimization method for containerized HarmonyOS applications, the processing flow of which is as follows: Figure 1 As shown, the specific steps include:

[0025] Step 1: When the containerized HarmonyOS application starts, the host side creates a dedicated independent host-side window for the application based on the window system, assigns a globally unique host-side window ID, and obtains the initial state parameters of the host-side window, including the screen ID, window coordinates, width and height, transparency, focus state, etc., and saves the initial window state to the first window state table to complete the initialization of the window state. The host side establishes the binding relationship between the HarmonyOS application, the host-side window, and the input device and adds it to the first binding table to complete the registration of the global binding relationship. The first binding table includes the host-side window ID, HarmonyOS application ID, application type, input device path, and allowed event types. The host side configures differentiated event receiving rules for the host-side window according to the application type of the HarmonyOS application and the initial window state of the corresponding host-side window, and starts the result receiving service. The host side sends the HarmonyOS application ID, the independent host-side window ID, the binding relationship, and the event receiving rules to the container side. The container starts the event receiving service for the HarmonyOS application, binds and sets the port to only receive event data from that port, and starts the result feedback service. The bound and set port is used to send the processing results back to the host side.

[0026] The host side completes the mapping configuration between the communication port and the host side window, pre-configures a screen-specific transmission channel for the window in the multi-screen scenario, and after the host side and the container side complete the connectivity verification of the HarmonyOS application-specific channel, the HarmonyOS application is marked as ready to receive and process user input events.

[0027] Furthermore, based on the application type of the HarmonyOS application and the initial window state of the corresponding host-side window, the host side configures differentiated event receiving rules for the host-side window, specifically as follows:

[0028] For game applications, configure a low-latency event queue, enable gamepad dead zone filtering and button double-click detection, and reserve a vibration feedback interface; for drawing applications, configure the highest precision first parameter mapping rule, preset pen pressure sensitivity scaling formula and coordinate transformation logic, and support high sampling rate adaptation; for office applications, configure a keyboard keycode mapping table, voice semantic filtering rules, and preset system conflict shortcut key blocking strategy.

[0029] Step 2: When a user operates an input device, the host side collects all parameters through the device-specific interface to generate raw input events for different input devices. The raw input events include the input device path, event type, raw parameters, and timestamp. The host side queries the first binding table based on the input device path in the raw input event. If there is no corresponding binding relationship in the first binding table, the input event is marked as illegal, discarded, and logged. Otherwise, the event type is checked to see if it is an allowed event type. Unallowed events are temporarily stored in the host side window's dedicated low-priority queue. For allowed events, the host side window ID in the binding relationship is written into the target window ID field of the raw input event to complete the initial orientation of the event. The source screen ID of the event is recorded, the absolute screen coordinates are converted to the relative coordinates of the corresponding display, and written into the raw input event to avoid cross-screen coordinate confusion.

[0030] The input devices include touch devices, digital pen devices, game controllers, and voice devices. Different input devices collect different full parameters. Specifically: touch devices collect precision coordinates, touch pressure, and multi-finger IDs, with a fixed sampling rate of 120Hz; digital pen devices collect coordinates, original pressure sensitivity, tilt angle, rotation angle, and side button status; game controllers collect joystick axis displacement, button status, and LED status, and reserve a vibration feedback interface; keyboard devices collect key codes, modifier key status, and keyboard layout, supporting automatic layout reloading after hot-swapping; and voice devices collect speech streams, completing speech-to-text conversion and semantic intent extraction during the collection process, and recording confidence levels.

[0031] Step 3: The host side determines the application type of the HarmonyOS application corresponding to the relevant host side window based on the original input event, obtains the current window state of the host side window, updates the first window state table with the current window state, adjusts the event processing rules according to the current window state, and performs differential parameter preprocessing on the original input event according to the application type to obtain the intermediate input event; the intermediate input event is encapsulated into a structure with unified semantics according to the window type and recorded as a unified window event. The unified window event includes event ID, target host side window ID, target window screen ID, event type, priority, event generation timestamp, and application type differential parameters. The unified window event corresponds one-to-one with the target host side window ID to avoid multi-window confusion. The event ID contains the window identifier to facilitate subsequent tracing and retransmission.

[0032] Before routing, the window state is checked again. Non-wake-up events of minimized windows are temporarily stored in the wake-up queue and routed after the window is restored. Events of the focused window are sent directly to the event receiving service of the corresponding HarmonyOS application on the container side. High-priority events of out-of-focus windows are forcibly routed and the HarmonyOS application is notified to handle the out-of-focus status synchronously. Low-priority events are temporarily stored in the focus waiting queue and routed after the window gains focus. In multi-screen scenarios, events of secondary screen windows are transmitted through a dedicated screen channel to avoid competing for bandwidth with the main screen events. After routing is completed, unified window events are written to the routing log for subsequent tracing, verification and retransmission.

[0033] Furthermore, the method for adjusting event handling rules based on the current window state is as follows:

[0034] If the current window is minimized, only the wake-up event is retained, and the rest are temporarily stored in the wake-up queue. If the current window is semi-transparent, scrolling and dragging events are discarded, and only high-confidence click and key events are retained. If the current window is completely obscured, only application-specific global shortcut events are retained, and the rest are discarded. If the current window is focused, the preprocessing priority is increased, and the processing latency is controlled within 5ms.

[0035] Furthermore, the method for preprocessing the original input events with differentiated parameters based on the application type is as follows:

[0036] If the application type is a game application, then joystick dead zone filtering and combo detection are performed, and a vibration intensity field is reserved to ensure low latency processing; if the application type is a drawing application, then pressure sensitivity parameter scaling, coordinate transformation and sampling rate adaptation are performed to ensure that the parameters comply with the OpenHarmony specification; if the application type is an office application, then key code conversion, low confidence voice result filtering and system conflict shortcut key blocking are performed to ensure compatibility in office scenarios.

[0037] Among them, the application type differentiated parameters include game window parameters, drawing window parameters, office window parameters, and general parameters. Game window parameters include controller X-axis displacement, controller Y-axis displacement, game key codes, whether to perform a continuous click, and vibration intensity. Drawing window parameters include window relative X-coordinate, window relative Y-coordinate, pressure sensitivity after adaptation, pen X-axis tilt angle, pen Y-axis tilt angle, and pen rotation angle. Office window parameters include OpenHarmony key codes, whether to use modifier keys, speech-to-text results, and voice intent. General parameters include touch, mouse, trackball, and other parameters.

[0038] Step 4: After the event receiving service of the HarmonyOS application on the container side reads the unified window event, it verifies whether the target host-side window ID in the event is consistent with the host-side window ID bound to the HarmonyOS application. If they are inconsistent, a rejection message is sent to request the host side to reroute. If they are consistent, the event integrity verification fails and a retransmission request is sent. If the verification passes, differentiated event processing is performed according to the application type. After the event processing is completed, the event processing result is generated, which includes the corresponding event ID, application ID, window ID, processing status, processing completion timestamp, and result data differentiated according to the application type. The HarmonyOS application sends the event processing result to the host side.

[0039] Furthermore, the container-side implementation of differentiated event handling based on application type is as follows:

[0040] If the application type is a game application, it parses the gamepad parameters and converts them into character control commands, handles combo logic, generates vibration feedback commands based on the game scene, detects the application status through heartbeat, and immediately sends a crash status notification to the host side when it freezes; if the application type is a drawing application, it parses the pen parameters and converts them into pen stroke parameters (including line width, transparency, and pen stroke type), calls the vector drawing engine to complete the drawing, automatically corrects abnormal parameters, and sends a warning to the host side; if the application type is an office application, it parses keyboard and voice events, completes text input, document operations, and shortcut key responses, handles conflicting shortcut key prompts, and automatically corrects the text format of voice input.

[0041] Step 5: After receiving the event processing result, the result receiving service of the corresponding host-side window verifies whether the event ID exists in the routing log. If it does not exist, it is marked as an invalid result and discarded directly. Otherwise, it parses the processing status in the event processing result. If the processing status is failure, it prompts an error and retryes. If the processing status is crash, it triggers the HarmonyOS application to restart automatically. If the processing status is success, it executes a differentiated interactive response according to the application type to complete the interaction.

[0042] Furthermore, the method for executing differentiated interactive responses based on application type is as follows:

[0043] For game applications, the window screen is updated based on the processing results, triggering controller vibration and adjusting the window refresh frame rate to avoid screen tearing. For drawing applications, line rendering is completed based on drawing parameters, the window canvas is refreshed, and pressure sensitivity and tilt angle parameters are visualized. For office applications, text insertion, cursor synchronization, and operation result prompts are completed, with optional voice input confirmation. The general response is to adjust the window size and switch window focus based on application commands, and update the window status table synchronously.

[0044] Furthermore, for multi-screen scenarios, the secondary screen window updates the screen through a dedicated rendering channel, and the coordinates and screen affiliation are calibrated in real time during cross-screen operations, ensuring synchronization of dual-screen displays through latency compensation.

[0045] Example:

[0046] This embodiment employs a multi-window interaction optimization method for containerized HarmonyOS applications provided by this invention. Addressing the core scenario where each OpenHarmony application corresponds to an independent Linux window between the OpenHarmony container and the Linux host system, it provides directional interaction for multimodal input events such as touch, keyboard, pen, game controller, voice, and trackball. It is suitable for scenarios such as containerized application compatibility, hybrid system office work, and distributed terminal collaboration, and particularly solves problems such as low cross-system event targeting accuracy, poor semantic consistency, focus and window synchronization lag, and multi-screen interaction errors. The specific process includes:

[0047] S1. Refined collection and device binding verification of user multimodal input events: For the user input process, the parameter collection of multimodal devices is refined, and illegal input is filtered by combining the binding relationship between application, window and device to ensure that the input information is complete and directed to the associated device of the corresponding Linux window.

[0048] S1.1 Full acquisition of parameters from multimodal input devices.

[0049] For different input devices, use device-specific interfaces to collect refined parameters to ensure no information is missed:

[0050] Touch devices: Coordinates are collected using libinput_event_touch_get_x and libinput_event_touch_get_y from the Linux libinput library (accuracy 0.1px), touch pressure is collected using libinput_event_touch_get_pressure (0 to 255), and multi-finger IDs are collected using libinput_event_touch_get_slot (0 to 10). The sampling rate is set to 120Hz (to avoid loss of high-frequency operations).

[0051] Pen input device: Coordinates are acquired via libinput_event_tablet_tool_get_x and libinput_event_tablet_tool_get_y, raw pressure sensitivity (0 to 1024) is acquired via libinput_event_tablet_tool_get_pressure, tilt angle (-90° to 90°) is acquired via libinput_event_tablet_tool_get_tilt_x and libinput_event_tablet_tool_get_tilt_y, and rotation angle (0° to 360°) is acquired via libinput_event_tablet_tool_get_rotation. The pen side button status (single and double clicks) is also recorded.

[0052] Gamepad: The joystick axis displacement (X, Y axis, -32768 to 32767) is collected via libinput_event_gamepad_get_axis_value, the button state (such as A, B, X and Y buttons being pressed and released) is collected via libinput_event_gamepad_get_button_state, and the controller LED state is obtained via libinput_device_get_leds. A vibration feedback control interface (libinput_event_gamepad_send_rumble) is also reserved.

[0053] Keyboard device: Key codes (such as LinuxKEY_A, KEY_ENTER) are collected via libinput_event_keyboard_get_key, modifiers (Ctrl, Shift, Alt) are collected via libinput_event_keyboard_get_modifiers, and keyboard layout (such as QWERTY, AZERTY) is obtained via libinput_device_get_keyboard_layout. Automatic layout reloading is supported after hot-plugging.

[0054] Voice device: Acquire speech streams (sampling rate 16kHz, bit depth 16bit) through Linux PulseAudio, call libfst to perform speech-to-text conversion (recognition accuracy ≥98%), then extract semantic intent (such as OPEN_FILE, SAVE_FILE, INPUT_TEXT) through the bert-base-chinese model, and record speech confidence (0 to 100).

[0055] After data collection, a RawInputEvent structure is generated, which includes the input device path (e.g., / sys / devices / pci / 0000:01:00.0 / usb1 / 1-1 / input / input15), event type, raw parameters, and data collection timestamp (nanosecond level, based on CLOCK_MONOTONIC_RAW) to ensure that each input event is traceable.

[0056] S1.2 Validation based on application, window and device binding.

[0057] The application, window, and device binding table (AppWinDevBindTable) is preloaded. This table is generated by the Linux-side binding manager when the application starts. Each record contains:

[0058] typedef struct {

[0059] uint32_t lnx_win_id; / / Linux standalone window ID (e.g., 0x00020005)

[0060] char oh_app_id

[128] ; / / Corresponding OpenHarmony application ID (e.g., com.example.sketchpad)

[0061] enum AppType app_type; / / Application type (GAME / DRAW / OFFICE / OTHER)

[0062] char bound_dev_paths[8]

[256] ; / / Paths to bound input devices (maximum 8)

[0063] enum EventType allowed_types

[16] ; / / Allowed event types

[0064] } AppWinDevBindEntry;

[0065] The verification logic is as follows:

[0066] Extract the input device path from RawInputEvent, query AppWinDevBindTable, and determine whether the device is in the bound_dev_paths of a certain window;

[0067] If no bound window is found (e.g., unbound handle input): mark the event as illegal, log it (format: [illegal event] input device path:%s, event type:%d, timestamp:%llu), or discard it directly;

[0068] If a bound window is found (such as a drawing window bound to a graphics tablet): further determine whether the event type is in the allowed_types of the window (such as allowing PEN and TOUCH events and disabling JOYSTICK events for the drawing window). If not allowed, mark it as low priority to be processed and temporarily store it in the window's dedicated low priority queue.

[0069] If a bound window is found and the event type allows it: extract the lnx_win_id of the window and write it into the target_lnx_win_id field of RawInputEvent to prepare for subsequent targeted preprocessing.

[0070] Example: When a user inputs using a graphics tablet bound to a drawing window (lnx_win_id=0x00020005), the validation passes and target_lnx_win_id is set to 0x00020005; if an unbound game controller is used for input, the validation fails and the event is discarded.

[0071] S1.3 Initial calibration of input coordinates in multi-screen scenarios.

[0072] If a Linux system is connected to multiple monitors (e.g., main screen0: 1920×1080, secondary screen1: 2560×1440), the resolution and position offset of each monitor can be obtained through xrandr (e.g., the top left corner coordinates of the secondary screen1 are (1920,0)).

[0073] When collecting screen coordinates of touch, pen, and mouse events, the screen ID of the event source is recorded synchronously (e.g., by associating wl_seat with libinput_event_get_seat, and then obtaining the screen ID through wl_seat_get_pointer).

[0074] Convert the screen coordinates to display relative coordinates (e.g., from the main screen coordinates (1000, 500) to the screen0 relative coordinates (1000, 500), and from the secondary screen coordinates (2000, 500) to the screen1 relative coordinates (80, 500)), and write them to the screen_relative_x and screen_relative_y fields of RawInputEvent to avoid coordinate errors during subsequent cross-screen processing.

[0075] S2, Linux-side window-level event preprocessing and semantic modeling: For the Linux-side window input receiving stage, differentiated preprocessing is performed based on window type and state to build a unified semantic model containing window identifiers, ensuring that event parameters match window requirements and preparing for targeted routing.

[0076] S2.1 Adaptation of window state acquisition and event reception rules.

[0077] Collect fine-grained state data for each independent Linux window and add it to the WindowStateTable (a hash table with the key lnx_win_id):

[0078] typedef struct {

[0079] uint32_t lnx_win_id; / / Linux window ID

[0080] int screen_id; / / The ID of the screen to which the device belongs (screen0 / screen1)

[0081] int x, y; / / Coordinates of the top-left corner of the window (relative to the screen)

[0082] int w, h; / / Window width / height (pixels)

[0083] uint8_t alpha; / / Opacity (0 to 100, 0 for fully transparent)

[0084] bool has_focus; / / Whether the object has focus (true / false)

[0085] bool is_minimized; / / Whether to minimize (true / false)

[0086] bool is_occluded; / / Whether the image is occluded (true / false, determined by Z-axis hierarchy)

[0087] uint64_t last_interact_ts; / / Last interaction timestamp (nanoseconds)

[0088] } WindowStateEntry;

[0089] Dynamically adjust event receiving rules based on window status:

[0090] If is_minimized=true: only wake-up events (such as mouse click on taskbar icons, keyboard shortcut Alt+Tab) are retained, and other events (such as keyboard input, pen input) are temporarily stored in the wake-up queue and processed after the window is restored;

[0091] If the window alpha is less than 30 (semi-transparent): discard sliding and dragging events (to avoid accidental operation of the lower window), and only retain click and key events. Click events must satisfy the condition that the coordinates are within the window range and the confidence level is greater than 80% (judged by a coordinate matching algorithm).

[0092] If the window is_occluded=true (completely occluded): discard all touch, pen, and mouse events, retain only global shortcut events (such as Ctrl+S for saving), and verify whether the shortcut belongs to the application (e.g., Ctrl+S is valid in the office window, but invalid in the game window).

[0093] If the window has_focus=true (has focus): Increase the priority of event preprocessing (e.g., shorten the processing delay to within 5ms) to ensure timely interactive response.

[0094] Example: When a semi-transparent drawing window (alpha=20) receives pen input, if the event type is swipe, it is discarded directly; if it is a click and the coordinates are within the range of (100,200) to (800,600) in the window, the event is retained and marked as high priority.

[0095] S2.2, Event parameter differentiation preprocessing.

[0096] Different preprocessing logic is used for windows of different application types to ensure that parameters match application requirements:

[0097] Game window:

[0098] Gamepad joystick axis displacement: The original value (-32768 to 32767) is filtered for dead zones according to game requirements (e.g., set to 0 within ±5% of the center to avoid joystick drift), and then mapped to the in-game character movement speed (e.g., X-axis displacement from 10000 to movement speed of 5m / s).

[0099] Key events: Enable combo detection (e.g., pressing the A key 3 times in a row within 1 second is considered a combo) and record the key interval (less than 100ms is considered a rapid combo).

[0100] Vibration feedback: Reserve the rumble_intensity field (0 to 100), and adjust it later based on feedback from OpenHarmony applications (e.g., set to 80 during collisions and 30 during acceleration).

[0101] Drawing window:

[0102] Pen pressure sensitivity: Scale the original pressure sensitivity on the Linux side (0 to 1024) according to OpenHarmony requirements (formula: oh_pressure = linux_pressure × 4, such as 512 to 2048), retain the original tilt angle (-90° to 90°), and convert the rotation angle to the drawing angle (0° to 360°).

[0103] Touch coordinates: Convert the screen relative coordinates to window relative coordinates (formula: win_x = screen_relative_x - window_x, win_y = screen_relative_y - window_y), such as screen relative coordinates (200, 300), and the top left corner of the window (100, 200) to the window relative coordinates (100, 100);

[0104] Sampling rate adaptation: If the drawing application supports a 200Hz sampling rate, increase the pen input sampling rate from 120Hz to 200Hz (achieved through libinput_device_set_sample_rate) to ensure smooth lines;

[0105] Office window:

[0106] Keyboard events: Convert Linux key codes to OpenHarmony key codes (e.g., KEY_BACKSPACE to KEYCODE_DEL, KEY_ENTER to KEYCODE_ENTER), and support dynamic loading of keyboard layouts (e.g., when switching to the AZERTY layout, KEY_Q is mapped to KEY_A).

[0107] Voice events: Filter out speech-to-text results with low confidence (less than 60), convert the text corresponding to the semantic intent INPUT_TEXT to UTF-8 encoding, and remove special characters (such as newline characters and tab characters).

[0108] Keyboard shortcut blocking: Blocks keyboard shortcuts that conflict with the system (such as Ctrl+Alt+T being ineffective in office windows, to avoid triggering the Linux terminal).

[0109] After preprocessing, a PreprocessedEvent is generated, which includes lnx_win_id, preprocessing parameters, priority (HIGH, MID, LOW), and processing timestamp, ensuring that the parameters can be directly used for semantic modeling.

[0110] S2.3 Unified semantic modeling including window identifiers.

[0111] Define a UnifiedWindowEvent structure to encapsulate the preprocessed parameters according to the window type and explicitly associate it with the target Linux window. The structure is as follows:

[0112] typedef struct {

[0113] uint64_t event_id; / / Event ID (window ID + sequence number: (lnx_win_id<<32) seq)

[0114] uint32_t target_lnx_win_id; / / Target Linux window ID (key related field)

[0115] int target_screen_id; / / The screen ID to which the target window belongs

[0116] enum EventType type; / / Event type (PEN / JOYSTICK / KEYBOARD / VOICE, etc.)

[0117] enum EventPriority priority; / / Priority (HIGH / MID / LOW)

[0118] uint64_t timestamp; / / Event generation timestamp (nanoseconds)

[0119] / / Parameters for differentiating by window type union

[0120] union {

[0121] / / Game window parameters

[0122] struct {

[0123] int32_t joy_x; / / Handle X-axis displacement (after dead zone filtering, -32768 to 32767)

[0124] int32_t joy_y; / / Handle Y-axis displacement (after dead zone filtering, from -32768 to 32767)

[0125] uint32_t game_key; / / Game key code (e.g., GAME_KEY_A / GAME_KEY_B)

[0126] bool is_rapid_fire; / / Whether to perform a rapid fire (true / false)

[0127] uint8_t rumble_int; / / Vibration intensity (0 to 100, reserved)

[0128] } game_param;

[0129] / / Drawing window parameters

[0130] struct {

[0131] int32_t pen_x; / / Window's relative X coordinate (from 0 to window width)

[0132] int32_t pen_y; / / Window's Y-coordinate (from 0 to window height)

[0133] uint16_t pen_pressure; / / Adjusted pressure sensitivity (0 to 4096)

[0134] int8_t pen_tilt_x; / / Pen tilt angle on the X-axis (-90 to 90)

[0135] int8_t pen_tilt_y; / / Pen Y-axis tilt angle (-90 to 90)

[0136] uint8_t pen_rot; / / Pen rotation angle (0 to 360 degrees)

[0137] } draw_param;

[0138] / / Office window parameters

[0139] struct {

[0140] uint32_t oh_keycode; / / OpenHarmony keycode (e.g., KEYCODE_DEL)

[0141] bool is_modifier; / / Whether to modify the key (true / false)

[0142] char voice_text

[512] ; / / Speech-to-text result (UTF-8)

[0143] enum VoiceIntent voice_intent; / / Voice intent

[0144] } office_param;

[0145] / / General parameters (touch / mouse / trackball)...

[0146] param;

[0147] UnifiedWindowEvent;

[0148] When performing semantic modeling, ensure that:

[0149] Each event's target_lnx_win_id uniquely corresponds to a Linux window, avoiding confusion between multiple windows;

[0150] The parameter units are consistent with the requirements of OpenHarmony applications (e.g., pressure sensitivity 0 to 4096, coordinates 0 to window size).

[0151] The event ID contains the window identifier, which facilitates subsequent tracing and retransmission by window (e.g., event_id=0x0002000500000001 represents the first event of window 0x00020005).

[0152] S2.4, Application-to-window binding-based directed routing.

[0153] Before routing, query WindowStateTable and AppWinDevBindTable to ensure that events are directed to the correct window:

[0154] Find the window status by target_lnx_win_id. If the window is_minimized=true and the event is not a wake-up event, add the event to the wake-up queue and route it after the window is restored.

[0155] If the window has_focus=true, the event is sent directly to the event receiving port of the corresponding OpenHarmony application on the container side (via vsock, port number = 1024 + lnx_win_id, ensuring that each window has an independent port to avoid conflicts).

[0156] If the window has_focus=false (out of focus), determine if the event is a high priority event (such as a game controller event in a game window): if so, force routing (the OpenHarmony application needs to be notified synchronously to handle the event in the out-of-focus state); otherwise, add it to the focus waiting queue and route it after the window regains focus.

[0157] In a multi-screen scenario, if the target window of the event is on the secondary screen (target_screen_id=1), it is transmitted through the screen-specific channel of vsock (channel ID=screen_id) to avoid competing for bandwidth with the main screen event.

[0158] After routing, a RouteLog is recorded, including event_id, target_lnx_win_id, routing time, and transmission channel, which facilitates subsequent troubleshooting.

[0159] S3, OpenHarmony application event reception and semantic processing: For the input stage of OpenHarmony application processing, the event reception, parsing and processing logic on the application side is refined to ensure that the processing results match the application type and solve the problem of parameter semantic loss.

[0160] S3.1 Application-side event reception and validity verification.

[0161] When each application within the OpenHarmony container starts, it automatically creates an event receiving service, binds to the vsock port (1024 and lnx_win_id), and only receives UnifiedWindowEvents from that port.

[0162] After receiving the event, first verify whether the lnx_win_id of event_id is consistent with the lnx_win_id bound to the application (by querying OHAppWinMap, which is maintained by the container-side binding synchronization service, with the key OH_AppID and the value lnx_win_id).

[0163] If there is a discrepancy (e.g., application A receives an event from application B), send an EventReject message to the Linux side to mark the event as an illegal target and request rerouting;

[0164] If they match, verify the integrity of the event using CRC32 (verification fields include event_id, param, and timestamp). If the verification fails, send an EventNack message to trigger retransmission on the Linux side (the number of retransmissions shall not exceed 3, with intervals of 10ms, 20ms, and 40ms respectively).

[0165] After successful verification, the type and param of the UnifiedWindowEvent are parsed, and the corresponding event handling module is called according to the application type.

[0166] S3.2, Application type differentiated event handling logic.

[0167] Game applications (such as racing games):

[0168] Parse game_param: Extract the X and Y axis displacements of the controller (joy_x and joy_y) and convert them into in-game character control commands (e.g., joy_x>10000 converts to turning right, joy_y<-10000 converts to acceleration).

[0169] Handling combo events: If is_rapid_fire=true, trigger the weapon rapid-fire logic (e.g., fire 10 bullets per second) and record the number of combos (used for in-game combo reward calculation).

[0170] Vibration feedback control: Adjust rumble_int according to the game scene (such as collision, acceleration) (e.g., set to 80 when colliding, and 0 when driving smoothly), and feed back to the Linux side through EventFeedback message to trigger the gamepad vibration;

[0171] State synchronization: If the application freezes (detected by heartbeat; if no heartbeat is sent within 1 second, it is considered frozen), immediately send AppStateMsg (state=CRASHED) to the Linux side to request that event sending be stopped.

[0172] Drawing applications (such as sketching applications):

[0173] Parsing `draw_param`: Extracting pen coordinates (pen_x and pen_y), pressure sensitivity (pen_pressure), and tilt angle (pen_tilt_x and pen_tilt_y), and converting them into pen stroke parameters for the drawing engine:

[0174] Line width: width = pen_pressure / 10 (e.g., pressure sensitivity 2048 is converted to a line width of 204.8px);

[0175] Line opacity: alpha = (pen_tilt_x + 90) / 180 * 100 (e.g., 0° tilt angle converts to 50% opacity);

[0176] Pen stroke type: Select according to pen_rot (e.g., 0° converts to a round pen, 90° converts to a flat pen);

[0177] Drawing logic processing: Calls the application's built-in vector drawing engine to draw lines based on stroke parameters, and supports real-time preview (the canvas is refreshed once every 10 events).

[0178] Pressure sensitivity error correction: If pen_pressure exceeds the application's supported range (e.g., greater than 4096), it will be automatically clipped to 4096, and a ParamWarning message will be sent to the Linux side to indicate that the pressure sensitivity parameter is out of range.

[0179] Office applications (such as document editing):

[0180] Parse office_param:

[0181] Keyboard events: Map oh_keycode to in-application key logic (e.g., KEYCODE_DEL is converted to delete the character after the cursor, KEYCODE_ENTER is converted to newline), and support modifier key combinations (e.g., Ctrl+S is converted to save document).

[0182] Voice events: If voice_intent=INPUT_TEXT, insert voice_text at the document cursor position and automatically handle line breaks; if voice_intent=SAVE_FILE, call the application's save interface to trigger document saving.

[0183] Text format processing: Performs format validation on voice input text (such as removing extra spaces and correcting punctuation), and supports automatic error correction (such as converting input errors to input errors);

[0184] Keyboard shortcut conflict handling: If a shortcut key that is blocked on the Linux side (such as Ctrl+Alt+T) is triggered in the application, the user will be prompted that the shortcut key is not available on the Linux system and to use an alternative shortcut key (such as Ctrl+Shift+T).

[0185] S3.3 Processing result encapsulation and state synchronization.

[0186] After the application processes the event, it generates an EventProcessResult, which contains the following core information:

[0187] typedef struct {

[0188] uint64_t event_id; / / The ID of the corresponding input event

[0189] char oh_app_id

[128] ; / / OpenHarmony application ID

[0190] uint32_t lnx_win_id; / / Associate Linux window ID

[0191] enum ProcessState state; / / Processing status (SUCCESS / FAIL / CRASHED)

[0192] uint64_t process_ts; / / Processing completion timestamp (nanoseconds)

[0193] / / Processing result data (by application type)

[0194] union {

[0195] / / Game application results

[0196] struct {

[0197] char game_cmd

[64] ; / / Game control commands (such as "TURN_RIGHT" / "ACCELERATE")

[0198] uint8_t rumble_int; / / Vibration intensity feedback (0 to 100)

[0199] int32_t role_pos_x; / / Character's X-coordinate (in-game coordinates)

[0200] int32_t role_pos_y; / / Character's Y-coordinate (in-game coordinates)

[0201] } game_result;

[0202] / / Drawing application results

[0203] struct {

[0204] char draw_action

[32] ; / / Drawing action (e.g., "DRAW_LINE" / "ERASE")

[0205] int32_t line_start_x; / / Line start point X (relative coordinates within the window)

[0206] int32_t line_start_y; / / Line start point Y (relative coordinates within the window)

[0207] int32_t line_end_x; / / Line end point X (relative to the window coordinates)

[0208] int32_t line_end_y; / / Line end point Y (relative coordinates within the window)

[0209] uint32_t line_color; / / Line color (ARGB format)

[0210] uint16_t line_width; / / Line width (px)

[0211] } draw_result;

[0212] / / Office Application Results

[0213] struct {

[0214] char office_action

[32] ; / / Office action (such as "INSERT_TEXT" / "SAVE_DOCUMENT")

[0215] char inserted_text

[512] ; / / Inserted text (if intended as INPUT_TEXT)

[0216] uint32_t cursor_pos; / / Cursor position (document character index)

[0217] bool save_success; / / Whether the save was successful (true / false)

[0218] } office_result;

[0219] } result;

[0220] EventProcessResult;

[0221] If the processing status is FAIL (e.g., parameter error), add error_msg (e.g., pressure sensitivity value out of range) to the result to facilitate troubleshooting on the Linux side; if the status is CRASHED, synchronously send AppStateMsg to the Linux side to trigger the application restart logic.

[0222] S3.4 Processing result feedback and status synchronization.

[0223] The application sends the EventProcessResult to the corresponding window's result receiving service on the Linux side via the vsock result feedback channel (port number 2048 + lnx_win_id):

[0224] Feedback frequency: Ordinary events (such as keyboard input) are fed back in real time (within 10ms after processing), while high-frequency events (such as pen input and gamepad input) are fed back in summary every 10 frames (to avoid excessive bandwidth usage);

[0225] State synchronization: Send AppStateMsg (state=RUNNING, IDLE, CRASHED) every 500ms. The Linux side adjusts the event sending strategy according to the state (e.g., reduce the event sending frequency when state=IDLE, and stop sending when state=CRASHED).

[0226] Anomaly Handling: If the feedback channel is interrupted (e.g., due to network fluctuations), enable result caching (cache up to 100 results), and send them in batches after the channel is restored to avoid result loss;

[0227] S4. Processing result feedback and Linux-side window interaction response: For the process of feeding the results back to the Linux window, the logic of receiving, parsing and responding to the results on the Linux side is refined to build a closed loop of processing, feedback and response, and ensure that user operation and window display are synchronized.

[0228] S4.1 Result reception and parsing on the Linux side.

[0229] On the Linux side, a service receives the startup result of each window, bound to a vsock port (2048 + lnx_win_id), and only receives EventProcessResult corresponding to lnx_win_id.

[0230] After receiving the result, first verify whether the event_id exists in the RouteLog (to ensure that the result corresponds to the routed event). If it does not exist, mark the result as invalid, discard it, and record it in the log.

[0231] Parse ProcessState:

[0232] If state=SUCCESS, then extract the result data according to the application type and call the response processing module corresponding to the window;

[0233] If state=FAIL, then extract error_msg, display event handling failure: %s (e.g., pressure sensitivity value is out of range) in the window, and send an EventRetry message to the application side to request reprocessing (maximum of 2 retries).

[0234] If state=CRASHED, the application restart process is immediately triggered (the OpenHarmony application is restarted via the container management tool), and the window displays "Application has restarted, data is being restored".

[0235] Results archiving: Associate the EventProcessResult with the corresponding RouteLog and archive it to ResultArchive (retain for 24 hours for troubleshooting and user behavior analysis).

[0236] S4.2, Window type differentiated interactive response.

[0237] Based on the application type and processing result, the Linux side window executes differentiated responses to ensure that the response is consistent with the user's expected actions:

[0238] Game window response:

[0239] Parse game_result: Update the window display based on game_cmd (e.g., convert TURN_RIGHT to rotate the game character model, ACCELERATE to increase the character's movement speed), and update the character coordinates (role_pos_x and role_pos_y) simultaneously.

[0240] Vibration feedback trigger: The libinput_event_gamepad_send_rumble interface is called according to rumble_int to control the controller vibration (e.g., strong vibration is triggered when rumble_int=80, lasting for 100ms).

[0241] Frame rate synchronization: If the game application reports a current frame rate of 50fps, adjust the window refresh rate to 50Hz (set the CRTC refresh rate via drmModeSetCrtc) to avoid screen tearing.

[0242] Drawing window response:

[0243] Parse draw_result: Based on draw_action and line parameters (line_start_x, line_start_y, line_end_x, line_end_y, line_color, line_width), call the Cairo drawing library to draw lines on the window canvas (supports anti-aliasing and stroke smoothing);

[0244] Real-time preview: The window refreshes once every 10 plotting results are received (to avoid lag caused by frequent refreshes). If the user stops inputting (no new results within 500ms), a full refresh is performed immediately.

[0245] Pressure sensitivity visualization: The current pen pressure value (pen_pressure) and tilt angle (pen_tilt_x, pen_tilt_y) are displayed in the lower right corner of the window to help users adjust the drawing pressure.

[0246] Office window response:

[0247] Parse office_result: Perform the operation based on office_action (e.g., INSERT_TEXT is converted to insert inserted_text at the cursor position, SAVE_DOCUMENT is converted to display a save success message if save_success=true);

[0248] Cursor synchronization: Update the text cursor position in the window based on cursor_pos (e.g., cursor_pos=100 means moving the cursor to the 100th character in the document).

[0249] Voice feedback: If the event is voice input and voice_intent=INPUT_TEXT, the inserted text will be read aloud by the Linuxespeak tool (optional function, which users can turn off) to confirm that the input is correct.

[0250] General window response:

[0251] If the processing result includes a window resizing instruction (such as requiring the window to change from 800×600 to 1024×768 after application processing), call xdg_surface_set_window_geometry to adjust the window size and update the WindowStateTable synchronously.

[0252] If the processing result includes a focus request (such as an application pop-up modal window requesting focus), call wl_seat_set_focus to set the window as focus and simultaneously notify other windows to lose focus.

[0253] S4.3, Response synchronization in multi-screen scenarios.

[0254] If the target window is on a secondary screen (target_screen_id=1), the response processing should be as follows:

[0255] The window display is updated by calling the dedicated rendering channel for the secondary screen (via DRM's crtc_id=screen_id) to avoid competing with the main screen for GPU resources;

[0256] If the processing result includes cross-screen movement instructions (such as a game character moving from the secondary screen to the main screen), the window's belonging screen is adjusted synchronously (target_screen_id=0), and the event coordinates are recalibrated (such as the secondary screen character coordinates (2560,720) being converted to the main screen coordinates (1920,720)).

[0257] Cross-screen response latency compensation: Due to the display latency difference between the secondary screen and the primary screen (approximately 5ms), the response execution timing is adjusted based on the difference between process_ts and the current time (e.g., the secondary screen response is executed after a 5ms delay to ensure synchronized display between the two screens).

[0258] S4.4 Feedback loop optimization and user experience enhancement.

[0259] To reduce the perceived latency between user actions and window responses, the following optimization measures are implemented:

[0260] Pre-rendering: If the Linux side predicts the application processing result (such as the game window predicting the character's movement direction based on historical controller input), the response effect is pre-rendered in the window (such as a semi-transparent preview of the character's movement path). After the application provides feedback, the effect is corrected to reduce perceived latency (from 50ms to within 20ms).

[0261] Operation confirmation: For critical operations (such as deleting documents in the office window), after the application processes the results, a confirmation dialog box will pop up, and the user will confirm before the response is executed to avoid accidental operation;

[0262] Error message: If the application processing delay is greater than 100ms (calculated by process_ts-route_ts), an animation will be displayed in the window indicating that the operation has been accepted, thus avoiding duplicate operations;

[0263] History: Records the processing results of the last 100 events in the window event log (e.g., 2025-10-22 14:30:01 The controller event was successfully processed, and the character moved to (1200,800)). Users can view it using shortcut keys (e.g., Ctrl+Shift+L) for easy problem location.

[0264] S5 provides end-to-end status monitoring and fault tolerance, ensuring the stability of user input, Linux window reception, OpenHarmony processing, and result feedback, and resolving anomalies in multiple scenarios.

[0265] S5.1, Full-process status monitoring.

[0266] Deploy cross-system event monitoring services to collect real-time status data at each stage:

[0267] Input acquisition and monitoring: The event acquisition frequency of each device is counted every 100ms (e.g., 120Hz for touch and 200Hz for pen). If the frequency is lower than the preset value (e.g., less than 60Hz for touch), a device performance warning is sent to the Linux side to prompt the user to check the device connection.

[0268] Route monitoring: Statistically analyze the event routing success rate (number of successful routes and total number of routes) for each window. If the success rate is less than 95%, analyze the RouteLog to locate the problem (such as port conflict or abnormal window state) and automatically repair it (such as restarting the vsock port).

[0269] Processing monitoring: Statistical analysis of application processing latency (process_ts-receive_ts). If the latency is greater than 100ms and lasts for 5 seconds, send an application performance warning to the application side and recommend disabling unnecessary functions (such as reducing the graphics quality of game windows).

[0270] Feedback monitoring: Statistical analysis of the success rate of feedback results (number of successful feedbacks and total number of processes). If the success rate is less than 98%, check the status of the vsock channel and restart the abnormal channel.

[0271] The monitoring data is displayed in real time on the Linux system monitoring panel, and users can view the event processing status of each window (such as the plotting window: 120Hz acquisition, 100% routing success rate, and 30ms processing delay).

[0272] S5.2, Multi-scenario fault tolerance.

[0273] Device hot-swap fault tolerance:

[0274] When an input device is unplugged (such as a handle), the Linux-side device monitoring service immediately sends a DeviceUnplugMsg to the application side of the corresponding window. The application stops processing the device event and displays in the window that the device has been disconnected.

[0275] When a new device is inserted (such as a graphics tablet), parameter collection and binding verification are automatically triggered, AppWinDevBindTable is updated, and the data is synchronized to the application side, so that the new device can be supported without restarting the application.

[0276] Network fluctuation tolerance:

[0277] When the vsock channel is interrupted, the Linux side caches the events in the channel recovery queue (up to 1000 events). After the channel is restored, the events are re-routed according to the timestamp order.

[0278] If the application does not receive an event, it sends an event request to the Linux side. The Linux side then retransmits the unacknowledged event according to the RouteLog (up to 3 retransmissions).

[0279] Application crash tolerance:

[0280] If the OpenHarmony application crashes (state=CRASHED), the Linux container management service will automatically restart the application and restore the event handling state before the crash (by reading the result of the most recent successful processing through ResultArchive).

[0281] During the restart, newly received events are added to the restart wait queue and routed in order after the application restarts.

[0282] Multi-screen switching fault tolerance:

[0283] When a window is dragged from the main screen to the secondary screen, the Linux side updates the screen_id and coordinates of the WindowStateTable in real time, and simultaneously notifies the application side to update the screen-specific channel for receiving events.

[0284] If event routing is interrupted during the switching process, the event is added to the switching waiting queue. After the switching is completed, the routing is re-established and the coordinates are calibrated (e.g., the main screen coordinates (1920, 500) are converted to the secondary screen coordinates (0, 500)).

[0285] S5.3 Performance Optimization and Resource Management.

[0286] Event queue management: The size of the event queue for each window is dynamically adjusted according to the application type (256 queues for game windows, 128 queues for drawing windows, and 64 queues for office windows) to avoid queue overflow; low-priority events (such as low-confidence voice events in office windows) are automatically discarded when the queue is full.

[0287] Bandwidth allocation: vsock bandwidth is allocated according to window priority (30% for game windows, 25% for drawing windows, 20% for office windows, and 25% for other windows) to avoid a single window from monopolizing bandwidth;

[0288] Memory release: After the event processing is completed and the feedback is successful, release the memory of RawInputEvent, PreprocessedEvent, and UnifiedWindowEvent in a timely manner (using OpenHarmony's ArkMalloc and Linux's free) to avoid memory leaks.

[0289] Experimental verification shows that this embodiment has the following beneficial effects compared to existing technologies: Significantly improved input orientation accuracy; each OpenHarmony application only receives events from its corresponding Linux window, with a cross-window error rate of less than 0.1%, and event orientation accuracy reaching 99.7% in multi-screen scenarios, meeting the needs of professional scenarios (such as multi-screen games and cross-screen drawing); optimized semantic consistency and processing quality; the semantic loss rate of parameters such as pen pressure sensitivity and controller displacement is reduced to 0; the matching degree between application processing results and user operation expectations reaches 99%; game controller event latency is less than 10ms; and the smoothness of drawing lines is improved by 40%; enhanced user experience and operation synchronization; and improved closed-loop latency from processing and feedback to response. With latency less than 50ms, users are unaware of differences in operation and response. Window response is synchronized during multi-screen switching. Abnormal scenarios (device plugging / unplugging, application crashes) are automatically repaired without user intervention. System stability and resource utilization are improved. Full-process monitoring and fault tolerance mechanisms reduce the system failure rate to below 2%. Event queues and dynamic bandwidth allocation reduce memory and bandwidth usage by 30%, preventing single windows from competing for resources. It has broad compatibility and scalability, adapting to OpenHarmony 3.2+ and Linux 5.15+, supporting Wayland and X11 window systems, and compatible with multimodal devices such as touch, pen, gamepad, keyboard, and voice. New devices can be supported without modifying the core code.

[0290] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for optimizing multi-window interaction in containerized HarmonyOS applications, characterized in that, Specifically, the following steps are included: When a containerized HarmonyOS application starts, the host side creates an independent host-side window for the application, assigns a host-side window ID, saves the obtained initial state to the first window state table, adds the binding relationship between the HarmonyOS application, the host-side window, and the input device to the first binding table, including the application type, input device path, and allowed event types, configures differentiated event receiving rules according to the application type and initial window state, starts the result receiving service, and then sends the HarmonyOS application ID, host-side window ID, binding relationship, and event receiving rules to the container side. The container side starts the event receiving service and result feedback service, and marks the HarmonyOS application as ready. When a user interacts with an input device, the host side generates a raw input event, including the input device path, event type, raw parameters, and timestamp. It queries the first binding table, writes the host-side window ID into the raw input event, records the source screen ID, converts the screen coordinates to relative coordinates, writes them into the raw input event, updates the first window state table based on the current window state, adjusts the event handling rules, preprocesses the raw input event with differentiated parameters based on the application type, and then encapsulates it into a unified window event, including the event ID, target host-side window ID, target window's home screen ID, event type, priority, event generation timestamp, and application type differentiated parameters. The unified window event is then sent to the container side. The application's event receiving service performs differentiated event processing on the unified window event according to the application type, generates the event processing result, and sends it to the host side; the host side's result receiving service parses the event processing result, and if the processing status is failure, it prompts an error and retryes; if the processing status is crash, it triggers the HarmonyOS application to restart automatically; if the processing status is success, it executes differentiated interactive response according to the application type to complete the interaction.

2. The multi-window interaction optimization method according to claim 1, characterized in that, The method of configuring differentiated event receiving rules based on application type and initial window state is as follows: If the application type is a game application, configure a low-latency event queue, enable gamepad joystick dead zone filtering and button combo detection, and reserve a vibration feedback interface; if the application type is a drawing application, configure first-level parameter mapping rules, preset pen pressure sensitivity scaling formula and coordinate transformation logic; if the application type is an office application, configure a keyboard key code mapping table and voice semantic filtering rules, and preset system conflict shortcut key blocking strategy.

3. The multi-window interaction optimization method according to claim 1, characterized in that, The host side generates raw input events by collecting different full parameters for different input devices, and constructing raw input events from these full parameters. Specifically: for touch devices, it collects precision coordinates, touch pressure, and multi-finger IDs, with a fixed sampling rate of 120Hz; for digital pen devices, it collects coordinates, raw pressure sensitivity, tilt angle, rotation angle, and side button status; for game controllers, it collects joystick axis displacement, button status, and LED status, and reserves a vibration feedback interface; for keyboard devices, it collects key codes, modifier key status, and keyboard layout, supporting automatic layout reloading after hot-swapping; for voice devices, it collects the speech stream, completes speech-to-text conversion and semantic intent extraction, and records the confidence level.

4. The multi-window interaction optimization method according to claim 1, characterized in that, The event handling rules are adjusted based on the current window state as follows: If the current window state is a minimized window, wake-up events are retained, and other events are temporarily stored in the wake-up queue; if the current window state is a semi-transparent window, swipe and drag events are discarded, and only click and key events with a confidence level greater than the threshold are retained; if the current window state is a fully obscured window, only application-related global shortcut events are retained, and other events are discarded directly; if the current window state is a focused window, the preprocessing priority is increased.

5. The multi-window interaction optimization method according to claim 1, characterized in that, The method of preprocessing the original input events with differentiated parameters according to the application type is as follows: if the application type is a game application, then joystick dead zone filtering and combo detection are performed, and a vibration intensity field is reserved; if the application type is a drawing application, then pressure sensitivity parameter scaling, coordinate transformation and sampling rate adaptation are performed; if the application type is an office application, then key code conversion, setting confidence level voice result filtering and system conflict shortcut key blocking are performed.

6. The multi-window interaction optimization method according to claim 1, characterized in that, The event receiving service of the application performs differentiated event processing on the unified window events according to the application type as follows: If the application type is a game application, the gamepad parameters are parsed and converted into character control commands, combo logic is processed, vibration feedback commands are generated based on the game scene, the application status is detected by heartbeat, and a crash status notification is sent to the host side when there is no response; If the application type is a drawing application, the pen parameters are parsed and converted into pen stroke parameters, the vector drawing engine is called to complete the drawing, abnormal parameters are automatically corrected and a warning is sent to the host side; If the application type is an office application, keyboard and voice events are parsed, text input, document operations and shortcut key responses are completed, conflicting shortcut key prompts are handled, and the text format of voice input is corrected.

7. The multi-window interaction optimization method according to claim 1, characterized in that, The method for executing differentiated interactive responses based on application type is as follows: If the application type is a game application, the window screen will be updated based on the processing results, the controller will be vibrated, and the window refresh frame rate will be adjusted synchronously. If the application type is a drawing application, the line rendering will be completed based on the drawing parameters, the window canvas will be refreshed, and the pressure sensitivity and tilt angle parameters will be visualized. If the application type is an office application, text insertion, cursor synchronization, and operation result prompts will be completed.

8. The multi-window interaction optimization method according to claim 1, characterized in that, The container-side event receiving service and result feedback service are started in the following way: the container starts the event receiving service for the HarmonyOS application, and binds and sets a port so that it only receives event data from that port; Start the result feedback service and bind the configured port to send the processing results back to the host side.

9. The multi-window interaction optimization method according to claim 1, characterized in that, The host side pre-configures a dedicated transmission channel for windows in multi-screen scenarios. After the host side and container side complete the connectivity verification of the dedicated channel for HarmonyOS applications, the HarmonyOS application is marked as ready.

10. The multi-window interaction optimization method according to claim 1, characterized in that, The host side queries the first binding table based on the input device path in the original input event. If there is no corresponding binding relationship in the first binding table, the input event is marked as illegal, discarded directly, and logged. Otherwise, the event type is checked to see if it is an allowed event type. Unallowed events are temporarily stored in the host side window's dedicated low-priority queue, while allowed events have the host side window ID in the binding relationship written into the target window ID field of the original input event.

Images