Input device and signal unified management and synchronization system

Abstract

The invention relates to the technical field of electric digital data processing, and particularly discloses an input device and signal unified management and synchronization system. The system comprises an equipment abstraction and registration module, a unified signal acquisition and preprocessing module, an intention understanding and fusion decision module, a conflict detection and resolution module and a synchronous execution and feedback module. The method comprises the steps of acquiring and preprocessing equipment signals based on a time sequence scheduling strategy, analyzing a user operation intention through a multi-mode intention recognition model, detecting and eliminating signal conflicts in real time, finally converting the user operation intention into a uniform event format to be distributed and executed, and providing consistent feedback, so that cooperative work and signal synchronization of multiple input equipment are realized; and the interaction accuracy and the user experience are improved.

Description

Technical Field

[0001] This invention belongs to the field of electronic digital data processing technology, specifically relating to a unified management and synchronization system for input devices and signals. Background Technology

[0002] In the field of computer human-computer interaction, input devices serve as a crucial interface for information transmission and operational control between users and computing systems. Their performance and collaborative efficiency directly impact user experience and system efficiency. With the increasing complexity and diversity of interaction scenarios, single input methods are no longer sufficient to meet user needs, making multimodal and multi-device collaborative input an important development direction.

[0003] The unified management and synchronization system for input devices and signals aims to achieve centralized control and signal coordination of various input devices, ensuring that users receive consistent and seamless operational feedback in different interactive scenarios. This system needs to process input data from heterogeneous devices such as mice, keyboards, touchpads, gesture sensors, and eye trackers, and perform unified parsing and response at the system level.

[0004] In existing technologies, the signal acquisition and reporting mechanisms of each input device operate independently and in parallel, lacking effective collaborative management and conflict resolution strategies. This leads to problems such as input signal conflicts, user intent recognition errors, and inconsistent operation logic when multiple devices are working simultaneously. For example, if a user accidentally touches the mouse while performing precise operations using a touchpad, it will cause the cursor to jump unexpectedly; in immersive interactive environments such as virtual reality, gesture input and controller button input may produce logical contradictions, affecting the consistency of interaction and system reliability.

[0005] These problems severely restrict the accuracy and user experience of multi-device collaborative input, and there is an urgent need for a system solution that can achieve unified management of input devices and signal synchronization. Summary of the Invention

[0006] This invention provides a unified management and synchronization system for input devices and signals to solve the problems of input signal conflict, user intent recognition deviation and inconsistent operation logic caused by the independent and parallel operation of multiple input device signal acquisition and reporting mechanisms in the prior art.

[0007] This invention provides a unified management and synchronization system for input devices and signals. The system includes a device abstraction and registration module, a unified signal acquisition and preprocessing module, an intent understanding and fusion decision-making module, a conflict detection and resolution module, and a synchronization execution and feedback module.

[0008] The device abstraction and registration module is used to provide a unified abstract description and dynamic registration management for various input devices connected to the system. Its abstract description includes device type identifier, signal feature vector, sampling rate parameter and device priority weight.

[0009] The unified signal acquisition and preprocessing module is used to poll and acquire raw signal data from each registered input device according to a preset timing scheduling strategy, and to perform timestamp alignment, data format standardization and noise filtering on the acquired raw signal data to generate standardized signal data frames.

[0010] The intent understanding and fusion decision module receives standardized signal data frames from the unified signal acquisition and preprocessing module, and analyzes and fuses the current user's operation intent based on the preset multimodal intent recognition model, outputting the dominant operation command at the current moment.

[0011] The conflict detection and resolution module is used to monitor signal flows from different input devices in real time. When a conflict is detected in the spatiotemporal or logical dimension, it arbitrates according to the preset conflict resolution rule set and generates a conflict-free final operation command.

[0012] The synchronous execution and feedback module is used to receive the main operation instructions from the intent understanding and fusion decision module or the final operation instructions from the conflict detection and resolution module, and convert them into a unified event format that can be recognized by the underlying operating system for distribution and execution, while providing consistent visual, auditory or tactile feedback to the user.

[0013] Furthermore, the specific implementation process of the device abstraction and registration module is as follows: During system initialization, all available input devices are automatically scanned and identified; a unique device type identifier is assigned to each type of input device, which is a 32-bit integer value; the inherent attributes of each input device are extracted, including but not limited to signal dimension, data precision, and maximum sampling frequency, and a signal feature vector of the device is generated accordingly; based on the ergonomic importance and interaction frequency of the device, a device priority weight between 1 and 10 is set for each device; after completing the above abstraction description, the device information is written to the system device registry, and a mapping relationship between device handles and physical channels is established.

[0014] Furthermore, the timing scheduling strategy of the unified signal acquisition and preprocessing module adopts a dynamic time-slice round-robin algorithm based on device priority weights; Devices with higher priority weights receive more frequent data collection opportunities; The timestamp alignment process uses a hardware clock source as a reference to add timestamps accurate to the millisecond level to all acquired signal data; The data format standardization process transforms different data formats from different devices, such as mouse displacement vectors, keyboard key codes, multi-touch coordinates of touchpads, and key point coordinate sequences of gesture sensors, into a standard data frame structure containing three parts: timestamp, device identifier, and data payload. Noise filtering is performed by selecting appropriate digital filters based on the device type. A Kalman filter is used to smooth the mouse displacement data, and a median filter is used to remove jitter from the touchpad coordinate data.

[0015] Furthermore, the construction and operation process of the multimodal intent recognition model in the intent understanding and fusion decision module is as follows: This model is a multi-task learning model based on deep neural networks, and its input layer receives standardized signal data frames from different input devices; The model internally includes a feature extraction subnetwork, a context encoding subnetwork, and an intent classification subnetwork; the feature extraction subnetwork uses a dedicated convolutional neural network or recurrent neural network structure to extract features for different types of input signals; The context encoding subnetwork concatenates the extracted features with the current system state context information and calculates the contribution weight of each signal source to the current intent through a self-attention mechanism. The intent classification subnetwork outputs the most likely dominant operation intent category of the current user and its corresponding confidence score based on the weighted and fused feature vector; When the confidence score is greater than the preset threshold of 0.85, the dominant operation intention is adopted as the dominant operation instruction at the current moment.

[0016] Furthermore, the conflict detection and resolution module performs conflict detection in three dimensions: spatial conflict detection determines whether the cursor positioning or operation area generated by different input devices overlaps and whether the instructions are contradictory; The time-dimensional conflict detection determines whether there are semantically mutually exclusive operation commands from different devices within a preset 200ms time window; The logical dimension conflict detection determines whether the combined operation instructions violate the system's preset business logic constraints; the conflict resolution rule set includes a series of priority rules, context adaptation rules, and user habit learning rules; Priority rules stipulate that when a spatial or temporal conflict is detected, signals from devices with higher priority weights take precedence; context-adaptive rules stipulate that in specific application scenarios, such as document editing mode, keyboard input has higher priority than mouse input. User habit learning rules record users' historical conflict resolution choices and use machine learning algorithms to dynamically adjust the parameters of conflict resolution strategies.

[0017] Furthermore, the synchronous execution and feedback module encapsulates the received operation instructions into a unified input event structure, which includes event type, event source device, event coordinates, event parameters, and event timestamp; The encapsulated events are injected through the low-level input event interface provided by the operating system; The synchronous execution and feedback module simultaneously monitors the system's display output and audio output status to ensure that the execution results of user operation commands can provide consistent and perceptible feedback to the user within 50ms through changes in interface elements, system sound prompts, or vibrations from force feedback devices.

[0018] Furthermore, the system also includes a user configuration and adaptive learning module; the user configuration and adaptive learning module provides a graphical interface that allows users to manually adjust device priority weights and conflict resolution rules; Meanwhile, this module continuously collects user interaction data and conflict resolution preferences during actual use, and uses incremental learning algorithms to update the model parameters in the intent understanding and fusion decision module and the user habit learning rules in the conflict detection and resolution module, so that the system can gradually adapt to the user's personalized operation mode.

[0019] Furthermore, the system adopts a layered processing model in its architecture, which includes a device driver layer, a signal processing layer, and a decision application layer; The device driver layer is responsible for direct communication with physical input devices and the most basic data acquisition. The signal processing layer contains the core logic of a unified signal acquisition and preprocessing module and a conflict detection and resolution module, and is responsible for the initial processing of signals and conflict management. The decision application layer includes an intent understanding and fusion decision module and a synchronous execution and feedback module, which are responsible for high-level intent recognition and system-level instruction execution.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention achieves unified and standardized management of heterogeneous input devices through a device abstraction and registration module, eliminating interface incompatibility issues caused by device differences and laying a solid foundation for subsequent unified signal processing.

[0021] 2. This invention ensures the consistency of multiple input signals in terms of time and format by unifying the timing scheduling and data standardization of the signal acquisition and preprocessing modules, effectively avoiding parsing errors caused by signal asynchrony or format confusion.

[0022] 3. This invention, through its multimodal intent recognition model of intent understanding and fusion decision module, can comprehensively analyze signals from different input devices, accurately infer the user's true operational intent, and improve the accuracy of intent recognition in complex interaction scenarios.

[0023] 4. This invention, through multi-dimensional conflict detection and intelligent resolution rules of the conflict detection and resolution module, can promptly detect and arbitrate conflicts between input signals, fundamentally solving the common signal interference and logical contradiction problems when multiple devices operate in parallel, and ensuring the determinism and reliability of system operation.

[0024] 5. This invention ensures that user operation commands are executed accurately and promptly through a synchronous execution and feedback module, and provides consistent feedback across modalities, greatly enhancing the coherence of the interaction process and the user experience.

[0025] 6. This invention endows the system with personalized adaptability through user configuration and adaptive learning modules, enabling the system to continuously optimize its behavior according to user habits, thereby providing more intelligent and tailored interactive support. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the overall technical solution architecture of the unified management and synchronization system for input devices and signals proposed in this invention; Figure 2 This is a schematic diagram of the core principle framework of the intent understanding and fusion decision module in this invention; Figure 3 This is a logical flowchart of the unified signal acquisition and preprocessing module in this invention; Figure 4 This is a schematic diagram illustrating the multi-dimensional conflict detection and resolution principle of the conflict detection and resolution module in this invention; Figure 5 This is a schematic diagram of the multi-level interaction relationships and data flow of the system hierarchical processing model in this invention. Detailed Implementation

[0027] This embodiment details the specific implementation scheme of the unified management and synchronization system for input devices and signals. Please refer to the appendix. Figure 1 The system's overall architecture includes a device abstraction and registration module, a unified signal acquisition and preprocessing module, an intent understanding and fusion decision-making module, a conflict detection and resolution module, a synchronous execution and feedback module, and a user configuration and adaptive learning module. These modules are organized according to a strict hierarchical processing model, including a device driver layer, a signal processing layer, and a decision application layer, ensuring a complete data flow from acquisition to execution.

[0028] The device abstraction and registration module is responsible for providing a unified abstract description and dynamic registration management for all input devices connected to the system. During system initialization, this module automatically scans all available input devices in the current hardware environment. The scanning process enumerates the list of input devices connected to the host by calling the operating system's underlying application programming interface, including but not limited to Universal Serial Bus (USB) HMI devices, Bluetooth HMI devices, and integrated touch input devices.

[0029] For each identified physical device, the module assigns a globally unique 32-bit integer value as a device type identifier. This identifier follows a predefined device classification tree, where the high 16 bits represent the major device category (e.g., pointer device, keyboard device, touch device, gesture sensing device), and the low 16 bits represent the sub-category or manufacturer-specific model. After assigning the identifier, the module further extracts the inherent attribute set of the input device.

[0030] These attributes include signal dimension, which is the number of data components output by the device per sample. For example, a traditional mouse outputs a 2D displacement vector, a keyboard outputs a 1D key value code, a high-end touchpad outputs 5D multi-touch coordinates, and a gesture sensor outputs a 45-dimensional sequence containing the coordinates of 15 key points. Attributes also include data precision, which is the number of significant bits or resolution for each data component. For example, mouse displacement precision is 400 dots per inch, keyboard key value codes are 8-bit unsigned integers, and touchpad coordinate precision is 0.01mm.

[0031] The attributes also include the maximum sampling frequency, which is the highest data reporting rate supported by the device hardware, such as 125Hz or 1000Hz. Based on these inherent attributes, the module calculates and generates the device's signal feature vector. This vector is a normalized array of values, the dimension of which is equal to the maximum signal dimension supported by the system. Each element value is filled in after scaling according to the actual device attributes and the system's preset baseline value, and unused dimensions are filled with 0.

[0032] Subsequently, the module assigns a device priority weight between 1 and 10 to each device based on the device's ergonomic importance and historical interaction frequency.

[0033] Importance assessment is based on the device’s role in core interactive tasks. For example, in graphical interface operations, the priority weight of the mouse is usually set to 8, the keyboard to 7, the touchpad to 6, and the gesture sensor to 5.

[0034] Interaction frequency is monitored by a background thread that counts the number of signal triggers per unit time and dynamically adjusts their weights. After completing the above abstract description, the module writes complete device information, including device type identifier, signal feature vector, sampling rate parameter, device priority weight, device physical address, device driver handle, etc., into a persistent device registry located in system memory.

[0035] At the same time, the module establishes a mapping relationship between device handles and physical channels. This mapping relationship is stored in a hash table, with the device handle as the key and the physical channel identifier as the value, ensuring that the subsequent signal acquisition module can accurately access the corresponding hardware data stream through the device handle.

[0036] The unified signal acquisition and preprocessing module is responsible for polling and acquiring the raw signal data from each registered input device according to a preset timing scheduling strategy, and performing a series of preprocessing operations. Please refer to the appendix. Figure 3 The core of this module is the timing scheduler.

[0037] The timing scheduler employs a dynamic time-slice round-robin algorithm based on device priority weights. This algorithm maintains a queue of ready devices, which are arranged in descending order of their priority weights.

[0038] At the start of each scheduling cycle, the timing scheduler takes the highest priority device from the head of the queue and allocates a time slice for acquiring the signal of that device.

[0039] The length of the time slice is directly proportional to the device priority weight. The specific calculation formula is: the base time slice length multiplied by the device priority weight and then divided by the sum of the priority weights of all active devices.

[0040] This allows high-priority devices to obtain more acquisition opportunities and data bandwidth per unit of time. The acquisition process is implemented by calling blocking or non-blocking read functions provided by the device driver layer.

[0041] For each batch of raw signal data acquired, the module immediately performs timestamp alignment.

[0042] Timestamp alignment is based on the system’s high-precision hardware clock source, which is typically the processor’s timestamp counter or a dedicated high-precision timer chip with an accuracy of microseconds.

[0043] The module attaches an absolute timestamp accurate to the millisecond to each acquired data sample, calculated from system startup. This ensures that the timing relationships between signals from multiple devices can be accurately reconstructed in subsequent processing.

[0044] Subsequently, the module performs a data format standardization process. This process converts heterogeneous data formats from different input devices into a standard data frame structure defined internally by the system. The standard data frame is a fixed-length binary structure containing three fixed fields: a timestamp field, occupying 8 bytes, storing the millisecond-level timestamp of the acquisition time; a device identifier field, occupying 4 bytes, storing the device type identifier that generated the data; and a data payload field, occupying 64 bytes, storing the specific signal data.

[0045] The conversion rules are predefined for different device types: For mouse devices, the raw displacement increment data is converted into two 32-bit signed integers, representing the displacement on the X-axis and Y-axis respectively, and filled into the first 8 bytes of the data payload field; For keyboard devices, the key value code and key state are converted into a 32-bit unsigned integer, where the lower 16 bits represent the key value code and the higher 16 bits represent the pressed or released state, and are filled into the first 4 bytes of the data payload field. For touchpad devices, the multi-touch coordinate sequence is converted into a series of 32-bit floating-point pairs, each pair representing the normalized coordinates of a touch point, with a maximum of 5 touch points supported, which are then filled into the data payload field in sequence. For gesture sensors, the sequence of joint coordinates is converted into a series of 32-bit floating-point numbers, representing the three-dimensional coordinates of each joint, and then filled into the data payload field.

[0046] After format conversion, the module performs noise filtering on the standardized signal data frames. The filtering algorithm is dynamically selected based on the device type.

[0047] For mouse displacement data, a discrete Kalman filter is used for smoothing. The Kalman filter is based on a uniform motion model, whose state vector includes position and velocity.

[0048] The prediction phase estimates the current position based on the state at the previous moment, and the update phase uses the current observation value to correct the estimated value, thereby effectively suppressing the noise caused by optical sensor jitter.

[0049] The filter parameters, including process noise covariance and observation noise covariance, are dynamically adjusted based on the mouse's nominal accuracy and actual movement speed.

[0050] For touchpad coordinate data, a window-valued filter is used to remove jitter. The filter maintains a sliding window of length 5, storing the coordinate data of the most recent 5 sampling periods.

[0051] Each time new coordinate data arrives, the filter sorts the X and Y coordinates of all data within the window and takes the median as the filtered output coordinate of the current frame.

[0052] For keyboard data, complex filtering is usually not performed; only simple debouncing is done, which means ignoring minute changes in the state within 2ms after the physical key contact stabilizes.

[0053] All processed, standardized signal data frames are placed into a thread-safe multi-producer single-consumer queue for consumption by subsequent modules.

[0054] The intent understanding and fusion decision-making module receives standardized signal data frames from the unified signal acquisition and preprocessing module, and performs parsing and fusion decision-making based on the multimodal intent recognition model to determine the current user's operational intent. Please refer to the appendix. Figure 2 This multimodal intent recognition model is a multi-task learning model based on deep neural networks. Its computational graph consists of an input layer, a feature extraction subnetwork, a context encoding subnetwork, and an intent classification subnetwork connected sequentially. The input layer is responsible for receiving and buffering standardized signal data frames from different input devices.

[0055] The buffer maintains a 200ms time window, within which data frames are ordered by their timestamps. The model triggers forward inference computation every 50ms. The feature extraction subnetwork contains multiple parallel branch networks, each specializing in processing a specific type of input signal. For mouse displacement vector sequences, a one-dimensional convolutional neural network is used for feature extraction.

[0056] This convolutional neural network consists of three convolutional layers, each using 64 kernels of size 3, with a stride of 1 and padding of 1. Each convolutional layer is followed by a modified linear unit activation function and a max-pooling layer, ultimately converting the variable-length shift sequence into a 256-dimensional feature vector. For keyboard key value sequences, a gated recurrent unit network is used for feature extraction.

[0057] The network contains two layers of gated recurrent units and a hidden layer dimension of 128. It can capture the temporal dependencies of key press sequences, and the hidden state of the last time step is used as a 256-dimensional vector representation of keyboard features.

[0058] For multi-touch coordinate sequences on a touchpad, a graph convolutional network is used for feature extraction. This network treats each touch point as a node in a graph, with node features being coordinates and touch area, and edge features being the distance between touch points. Neighborhood information is aggregated through three layers of graph convolution operations, and finally, a 256-dimensional vector representation of the touch features is obtained through global average pooling.

[0059] For the sequence of hand gesture sensor joint coordinates, a spatiotemporal graph convolutional network is used for feature extraction.

[0060] This network models the human skeleton as a spatiotemporal graph, capturing the spatial relationships between joints through graph convolution and the temporal dynamics of movements through temporal convolution. The final output is a 256-dimensional vector representation of the gesture features.

[0061] The context encoding subnetwork receives feature vectors from all feature extraction branches and concatenates them with the current system state context information.

[0062] The system state context information is a 128-dimensional vector that encodes information such as the identifier of the currently active application window, the system input focus position, and a summary of the user operation history within the last 3 seconds.

[0063] The concatenated high-dimensional feature vector is first reduced in dimensionality by a fully connected layer, and then input into a self-attention layer based on a scaled dot product attention mechanism.

[0064] The self-attention layer calculates the contribution weights of different input signal source features to the current intent.

[0065] Specifically, given the query matrix, key matrix, and value matrix, the self-attention output is calculated using the following formula: , The query matrix, key matrix, and value matrix are all obtained from the concatenated feature vectors through different linear transformations. It is the dimension of the key vector. The probability distribution output by the function represents the contribution weights of each signal source. The intent classification subnetwork performs the final intent classification based on the weighted fused feature vector.

[0066] This subnetwork is a 3-layer fully connected neural network with hidden layer dimensions of 512 and 256 respectively. The output layer dimension equals the number of operation intent categories supported by the system, such as cursor movement, text input, object selection, object dragging, scaling, rotation, menu navigation, etc., totaling 15 categories. The output layer uses the softmax activation function and outputs a probability distribution vector, where the category corresponding to the maximum value is the most likely dominant operation intent of the current user, and its probability value is the confidence score.

[0067] When the confidence score exceeds a preset threshold of 0.85, the dominant operation intent is adopted as the dominant operation command for the current moment. The dominant operation command is a structure containing the intent category, confidence score, associated target coordinates, associated parameter list, and timestamp. This command is sent to the synchronization execution and feedback module or the conflict detection and resolution module for further processing.

[0068] The collision detection and resolution module monitors signal flows from different input devices in real time and arbitrates when a signal collision is detected. Please refer to the appendix. Figure 4 The conflict detection mechanism of this module operates in parallel across three dimensions: space, time, and logic.

[0069] The spatial dimension conflict detection unit is responsible for determining whether there is overlap or contradiction in the cursor positioning or operation area generated by different input devices.

[0070] This unit maintains a system-wide virtual cursor position mapping table. When a movement command is received from a pointer device or touch device, this unit calculates the overlap between the command target position and other currently active operating areas.

[0071] If two devices are detected attempting to move the cursor to different positions on the screen that are more than 100 pixels apart almost simultaneously, or if one device attempts to select an interface element while another attempts to deselect it, a spatial conflict is identified. The temporal conflict detection unit maintains a 200ms sliding window storing all recent operation commands from different devices. This unit analyzes the temporal sequence of commands within the window to determine if there are semantically mutually exclusive operation command pairs. For example, if a Ctrl+C copy command from the keyboard and a right-click delete command from the mouse are detected within 200ms, since copying and deleting are mutually exclusive operations, a temporal conflict is identified.

[0072] The logical dimension conflict detection unit loads the system's preset business logic constraint rule base. This rule base defines the combinations of operations that are not allowed to occur simultaneously in a specific application context.

[0073] For example, performing text input and graphic drawing operations simultaneously in a document editor may violate constraints; performing playback and fast forward operations simultaneously in a video player may violate constraints.

[0074] This unit matches the currently received operation instructions with the current system state. If any constraint rules are violated, it is determined to be a logical dimension conflict. Once a conflict is detected in any dimension, the conflict resolution engine is immediately activated.

[0075] The conflict resolution engine loads a preset set of conflict resolution rules.

[0076] This rule set contains three categories of rules: priority rules, context-adaptive rules, and user habit learning rules. Priority rules are the basic rules, which stipulate that when a spatial or temporal conflict is detected, the priority weights of the devices that generated the conflict signals are compared, and the signal from the device with the higher weight is adopted first.

[0077] Context-adaptive rules cover exceptions to priority rules in specific scenarios.

[0078] For example, in document editing mode, the rule set explicitly prioritizes keyboard input over mouse input temporarily to ensure smooth text input. User habit learning rules are dynamically adjusted through an online learning mechanism. This mechanism records the results of each manual selection made by the user in their history when encountering a conflict; these results serve as training samples for training a logistic regression classifier.

[0079] The classifier's features include conflict type, device combination, application context, and time. A well-trained classifier can predict a user's preference choices in similar conflict scenarios and dynamically adjust the parameters of the conflict resolution strategy accordingly, such as fine-tuning the effective priority of specific devices in a specific context.

[0080] The resolution engine applies these rules comprehensively to generate a conflict-free final operation instruction. This instruction will override any potentially conflicting instructions and will be sent to the synchronous execution and feedback module.

[0081] The synchronous execution and feedback module is responsible for converting received operation instructions into a unified event format recognizable by the underlying operating system and distributing them for execution, while providing consistent feedback. This module maintains an event wrapper.

[0082] The event encapsulator encapsulates the dominant operation instructions from the intent understanding and fusion decision module or the final operation instructions from the conflict detection and resolution module into a unified input event structure.

[0083] The structure is a fixed-length binary format containing the following fields: Event Type, a 4-byte unsigned integer encoding the type of operation, such as mouse movement, mouse click, keyboard press, touch start, etc.; Event Source Device, a 4-byte unsigned integer, which is the device type identifier that generated the original signal; Event Coordinates, two 4-byte floating-point numbers representing the screen normalized coordinates of the event occurrence; Event Parameters, a 32-byte flexible field used to store event-specific parameters, such as key codes, scroll amounts, pressure values, etc.; Event Timestamp, an 8-byte unsigned integer recording the absolute time of the event occurrence.

[0084] After encapsulation, the module writes the event structure into the system's raw input stream through the low-level input event injection interface provided by the operating system.

[0085] On Windows systems, this is typically implemented via the SendInput application programming interface; on Unix-like systems, it may be achieved by emulating the / dev / input / eventX device node. The injection process ensures that the event is recognized by the system as genuine input from the physical device, thereby triggering the corresponding application response. Simultaneously, the synchronous execution and feedback module initiates a feedback coordination mechanism.

[0086] This mechanism includes a feedback listener that monitors changes in the system's display and audio output states in real time. The feedback listener detects the execution results of user commands by hooking into the system's graphics rendering calls and audio playback buffers.

[0087] For example, after a cursor movement command is executed, the listener detects whether the cursor position on the interface is updated to the target position within 50ms; after a button click command is executed, the listener detects whether the button's visual state changes to a pressed state and whether the associated system sound is triggered. If successful execution is detected, the module does not perform any additional intervention.

[0088] If an execution delay is detected or the expected feedback is not generated, the module will trigger the auxiliary feedback channel.

[0089] These channels include: generating a brief vibration via a force feedback device, with the vibration pattern associated with the operation type; playing a brief, non-intrusive confirmation sound via the system audio device; and, in extreme cases, briefly displaying a semi-transparent visual cue icon in the corner of the screen.

[0090] All these feedback measures are designed to ensure that the user receives consistent and perceptible confirmation within 50ms that their action has been received and processed by the system through at least one modality, whether visual, auditory, or tactile.

[0091] The user configuration and adaptive learning module provides the system with personalized configuration and continuous optimization capabilities. This module offers a graphical user interface.

[0092] The interface mainly consists of two configuration panels: a device priority configuration panel and a conflict rule configuration panel. In the device priority configuration panel, users can see a list of all currently registered input devices, including their device type identifier and current priority weight. Users can manually adjust the priority weight of any device within the range of 1 to 10 by dragging a slider or directly entering a value. The adjustment takes effect immediately and is persistently saved to the user's configuration file.

[0093] In the conflict rule configuration panel, users can see a list of predefined conflict resolution rules, each rule describing the conflict scenario and resolution action.

[0094] Users can enable or disable specific rules, or modify rule parameters, such as modifying the device priority coverage value for a specific application scenario in a context-adaptive rule.

[0095] Besides manual configuration, the core functionality of this module is adaptive learning. The module runs a data collection agent in the background that continuously collects user interaction data anonymously during actual use.

[0096] The collected data includes: standardized signal data frame sequences, dominant operation commands and their confidence levels output by the intent understanding and fusion decision module, conflict instances detected by the conflict detection and resolution module and the final resolution schemes, and operation records of users manually adjusting configurations through the graphical user interface.

[0097] After being anonymized and aggregated, this data is used to incrementally update the system model. For the multimodal intent recognition model in the intent understanding and fusion decision module, online distillation technology is used for incremental learning.

[0098] Every 24 hours, the system uses high-confidence samples collected in the past 24 hours to form a small training set. The consistency between the model prediction and the subsequent actions of real users is used as a soft label to fine-tune the original model, update the weight parameters of its fully connected layer, and set the learning rate to 0.001.

[0099] For the user habit learning rules in the conflict detection and resolution module, the underlying logistic regression classifier is updated online using the stochastic gradient descent algorithm.

[0100] Whenever a user manually resolves a conflict, the feature vector of that conflict and the user's choice result are used as a new training sample, which is immediately used to update the weight vector of the classifier, with the learning rate set to 0.01.

[0101] Through this continuous, data-driven optimization, the system can gradually adapt to the user's unique operating rhythm, preferences, and habits, making intent recognition more accurate and conflict resolution more in line with user expectations.

[0102] The system architecture strictly adheres to a layered processing model. Please refer to the appendix. Figure 5 The model clearly defines the responsibilities and data interaction boundaries of the device driver layer, signal processing layer, and decision application layer.

[0103] The device driver layer is the lowest layer of the system, interacting directly with the firmware and operating system kernel drivers of the physical input devices. This layer contains a series of device-specific driver adapters, such as mouse driver adapters, keyboard driver adapters, touchpad driver adapters, gesture sensor driver adapters, etc.

[0104] Each adapter is responsible for initializing the corresponding hardware device, configuring its operating mode, and reading raw, unprocessed device data via polling or interrupt.

[0105] The read data is encapsulated into internal data packets within the driver layer, containing the raw data payload and device physical identifier, and then passed up to the signal processing layer through a standardized driver interface. The signal processing layer is the system's intermediate layer; it receives the raw data packets from the device driver layer. This layer primarily contains the core logic of the unified signal acquisition and preprocessing module and the core logic of the conflict detection and resolution module.

[0106] The unified signal acquisition and preprocessing module completes the timing scheduling, acquisition, timestamp alignment, format standardization and noise filtering of signals at the signal processing layer, and produces standardized signal data frames.

[0107] The conflict detection and resolution module performs preliminary, rule-based, multi-dimensional conflict detection and resolution on standardized signal data frames at the signal processing layer.

[0108] The data processed by the signal processing layer, namely the collision-free standardized signal data frames or the final operation instructions after resolution, is transmitted to the decision application layer through the inter-layer communication interface.

[0109] The decision application layer is the highest layer of the system, containing the intent understanding and fusion decision module, as well as the synchronous execution and feedback module. This layer receives refined data from the signal processing layer and performs high-level semantic understanding and decision-making.

[0110] The intent understanding and fusion decision module runs a complex multimodal intent recognition model at the decision application layer to infer user intent.

[0111] The synchronous execution and feedback module converts high-level instructions into system-level events at the decision application layer and injects them into the system, while managing cross-modal user feedback.

[0112] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0113] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An input device and signal unified management and synchronization system, characterized by, The application relates to a multi-modal input device conflict resolution system and method. The device abstraction and registration module is used for uniformly abstractly describing and dynamically registering and managing various input devices accessed in a system, the abstract description including a device type identifier, a signal characteristic vector, a sampling rate parameter and a device priority weight; The unified signal acquisition and preprocessing module is used for polling and acquiring original signal data of each registered input device according to a preset timing scheduling strategy, and performing timestamp alignment, data format standardization and noise filtering processing on the acquired original signal data to generate a standardized signal data frame; The intention understanding and fusion decision module is used for receiving the standardized signal data frame from the unified signal acquisition and preprocessing module, and analyzing and fusing the current user operation intention based on a preset multi-modal intention recognition model to output a dominant operation instruction at the current moment; The conflict detection and resolution module is used for monitoring the signal flow from different input devices in real time, arbitrating according to a preset conflict resolution rule set when a conflict exists in the signal in the time and space dimensions or the logic dimension, and generating a final operation instruction without conflict; The synchronous execution and feedback module is used for receiving the dominant operation instruction from the intention understanding and fusion decision module or the final operation instruction from the conflict detection and resolution module, converting the operation instruction into a unified event format recognized by a bottom operation system to perform distribution and execution, and providing consistent visual, audible or tactile feedback to the user.

2. The input device and signal unified management and synchronization system of claim 1, wherein, The specific implementation process of the device abstraction and registration module comprises the following steps: automatically scanning and identifying all available input devices during system initialization; allocating a unique integer value as a device type identifier for each type of input device; extracting the inherent attributes of each input device to generate a signal characteristic vector; setting a device priority weight for each device according to the importance of human-machine engineering and the interaction frequency of the device; 3. The input device and signal unified management and synchronization system of claim 1, wherein, writing the device information into a system device registry and establishing a mapping relationship between the device handle and the physical channel. The timing scheduling strategy of the unified signal acquisition and preprocessing module adopts a dynamic time slice round-robin algorithm based on the device priority weight; The timestamp alignment process uses a hardware clock source as a reference and adds a time stamp with millisecond accuracy to all acquired signal data; The data format standardization process uniformly converts the data formats of different devices into a standard data frame structure containing a time stamp, a device identifier and data load; 4. The input device and signal unified management and synchronization system of claim 3, wherein, The noise filtering processing selects a corresponding digital filter according to the device type. The noise filtering processing comprises the following steps: using a Kalman filter to smooth the mouse displacement data; using a median filter to remove jitter from the touchpad coordinate data; 5. The input device and signal unified management and synchronization system of claim 1, wherein, only performing simple jitter removal processing on the keyboard data. The multi-modal intention recognition model is a multi-task learning model based on a deep neural network, and the construction and running process comprises the following steps: an input layer receives the standardized signal data frame from different input devices; a feature extraction subnetwork uses a dedicated convolutional neural network or recurrent neural network structure to extract features for different types of input signals; a context encoding subnetwork splices the extracted features and the current system state context information and calculates the contribution weight of each signal source to the current intention through a self-attention mechanism; The intention classification subnetwork outputs the dominant operation intention category and corresponding confidence score of the current user according to the weighted fused feature vector.

6. The input device and signal unified management and synchronization system of claim 5, wherein, The intention understanding and fusion decision module is further configured to adopt the dominant operation intention as the dominant operation instruction at the current time when the confidence score is greater than a preset threshold of 0.

85. The dominant operation instruction includes an intention category, a confidence score, associated target coordinates, an associated parameter list, and a timestamp.

7. The input device and signal unified management and synchronization system of claim 1, wherein, The conflict detection and resolution module detects conflicts from three dimensions: spatial dimension conflict detection determines whether the cursor positioning or operation area generated by different input devices overlap and the instructions contradict each other; time dimension conflict detection determines whether there are semantically exclusive operation instructions from different devices within a preset time window; logical dimension conflict detection determines whether the combined operation instruction violates the system's preset business logic constraints.

8. The input device and signal unified management and synchronization system of claim 7, wherein, The conflict resolution rule set includes priority rules, context adaptability rules, and user habit learning rules; The priority rules stipulate that when spatial or temporal conflicts are detected, the signals of the device with high priority weight are given priority; The context adaptability rules stipulate that in certain application scenarios, keyboard input has higher priority than mouse input; The user habit learning rules dynamically adjust the parameters of the conflict resolution strategy by recording the user's historical conflict resolution choices.

9. The input device and signal unified management and synchronization system of claim 1, wherein, The synchronization execution and feedback module encapsulates the received operation instructions into a unified input event structure, which includes event type, event source device, event coordinates, event parameters, and event timestamp; The encapsulated event is injected through the underlying input event interface provided by the operating system; Meanwhile, the system display output and audio output states are monitored to ensure that the user is given consistent and perceptible feedback on the execution results of the operation instructions.

10. The input device and signal unified management and synchronization system of claim 1, wherein, It also includes a user configuration and adaptive learning module; The user configuration and adaptive learning module provides a graphical interface that allows users to manually adjust the device priority weight and conflict resolution rules; At the same time, it continuously collects user interaction data and conflict resolution preferences during actual use, and updates the model parameters in the intention understanding and fusion decision module and the user habit learning rules in the conflict detection and resolution module using incremental learning algorithms.

Images