User-oriented interaction information extension method

Abstract

The invention relates to the technical field of intelligent peripherals, in particular to a user-oriented interaction information expansion method, which comprises the following steps that: each keycap acquires displacement and pressure, generates an event packet, uploads the event packet through an optical wireless link, and obtains a clock and supplies power; constructing a keycap graph by a master controller, obtaining an intention vector through graph convolution and pooling, inputting the intention vector into a cyclic network to generate a pixel matrix and a phase matrix, and downlink after Bernoulli sparse coding; performing key cap side orthogonal matching pursuit decoding and four-round gradient optimization, and outputting a differential pixel set, a differential phase set and a synchronization mark; the master control refreshes the digital micromirror array and the ultrasonic phased array in a differential manner, locks a frame-level time sequence through a synchronization pulse, compensates link delay and hardware drift in real time based on Kalman filtering, and keeps an optical-acoustic alignment error to be less than 100 microseconds; degradation monitoring enables a link to be automatically switched to a pre-carved character layer and to be quickly recovered when the link is abnormal, and a keyboard interaction information expansion scheme with low delay and high consistency is provided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent peripherals, and in particular to a user-oriented interactive information expansion method. BACKGROUND

[0002] Under the tide of panoramic office and immersive entertainment, interactive information expansion has become an important direction for keyboard upgrade. If menus, shortcuts, and status prompts can be dynamically presented around the keycap, and at the same time, tactile feedback can be output, users can complete complex operations without frequently checking the main screen, greatly improving input efficiency and immersion.

[0003] Existing solutions mostly embed micro-OLED or electrophoretic screens on the top of the keycap and transmit frame data through a wire; some high-end products further add a vibration motor to realize tactile feedback. However, due to reasons such as space occupation of wired arrangement, single-frame refresh limited by USB bandwidth, and no time sequence coupling between vibration and image, the overall delay is usually higher than 20 milliseconds, and the maintenance is complex and energy consumption is high. Especially in the e-sports and accessibility scenarios, the mismatch between the two domains of image and tactile feedback will lead to mis-touch and visual fatigue, limiting the industry popularization of keyboard screens. SUMMARY

[0004] To solve the problems of the prior art, the present application provides a user-oriented interactive information expansion method. The present application transmits keycap displacement and pressure events over an optical link, constructs a space-time graph to obtain an intention vector through graph convolution, generates a pixel-phase mapping using a recurrent network, and then jointly encodes and transmits the mapping set to each keycap. The keycap side decodes and outputs the differential pixel set, the differential phase set, and the synchronization marker through gradient optimization.

[0005] A user-oriented interactive information expansion method, comprising the following steps:

[0006] Collect displacement signals and pressure signals at the keycap, and after preprocessing, package them into event packets containing keycap identification, displacement, pressure, and timestamp. Transmit the event packets to the master controller through an optical wireless communication link, and supply power and synchronize the clock to the keycap through the optical wireless communication link;

[0007] The master controller aggregates the event packets in chronological order into an event stream, generates a keycap graph based on the spatial adjacency of the keycaps and the co-trigger relationship within the set time window, maps the event stream into node features, and inputs them into a graph convolutional neural network for inference. Obtain the intention vector through pooling;

[0008] Input the intention vector into a recurrent neural network to generate a pixel matrix and a phase matrix to form a mapping set. Implement sparse joint encoding on the mapping set and transmit it to each keycap. Decode at the keycap side and output the differential pixel set, the differential phase set, and the synchronization marker through gradient optimization;

[0009] The master controller drives the digital micromirror array according to the differential pixel set, drives the ultrasonic phased array according to the differential phase set, and embeds a synchronization marker; monitors the optical wireless communication link delay and driving offset, and synchronously corrects the optical modulation phase, scanning timing and phase reference according to the state estimation result.

[0010] Preferably, the event package records the keycap identification, displacement, pressure value and timestamp in fixed byte order, and performs two steps of data cleaning, i.e. low-pass filtering and outlier rejection, according to the preset threshold before uploading.

[0011] Preferably, the keycap graph contains first-type edges and second-type edges, wherein the first-type edges are established according to the mutual adjacent relationship of the physical center points of the keycaps, and the second-type edges are established according to the continuous co-triggering relationship within a set time window, and the weights of the two types of edges are written into a weighted adjacency matrix.

[0012] Preferably, the graph convolutional neural network includes two convolutional layers, the convolution kernel performs matrix multiplication on the normalized adjacency matrix, and the convolution output is input into the next convolutional layer after linear rectifier activation.

[0013] Preferably, the pooling operation performs global average operation on all node convolution outputs, and the obtained average vector is the intent vector.

[0014] Preferably, when implementing sparse joint coding on the mapping set, a Bernoulli measurement matrix is used for random subsampling compression, and the pixel matrix and the phase matrix are recovered on the keycap side through an orthogonal matching pursuit algorithm.

[0015] Preferably, the pixel matrix and the phase matrix after decoding are subjected to four times of fixed-point gradient descent on the keycap side, the pixel value and the phase value are adjusted simultaneously in each update to reduce the reconstruction error, and the differential pixel set, the differential phase set and the synchronization marker are output.

[0016] Preferably, the refresh timing of the digital micromirror array and the ultrasonic phased array is locked to a reference clock provided by the master controller through a frame synchronization signal.

[0017] Preferably, the master controller constructs a three-dimensional state vector using the optical wireless communication link delay, the digital micromirror array time base offset and the ultrasonic phased array phase offset, iteratively calculates the compensation amount through Kalman filtering, and adjusts the optical modulation phase, the line scanning timing and the phase reference in real time.

[0018] Preferably, when the bit error rate of the optical wireless communication link continuously exceeds a preset threshold for a preset duration, the master controller stops driving the digital micromirror array and the ultrasonic phased array and switches the keycap display to a pre-engraved character layer; when the bit error rate continuously falls below the preset threshold for a preset duration, the master controller restores the previously saved driving state and re-drives the digital micromirror array and the ultrasonic phased array.

[0019] Compared with the prior art, the application has the advantages and beneficial effects that:

[0020] Through the optical wireless communication bidirectional link, power supply without wiring, clock synchronization and data uploading are realized, the structure thickness is reduced and the reliability is improved.

[0021] Through the graph convolution-cyclic network association reasoning, millisecond-level intent recognition of complex shortcut key combinations is realized.

[0022] Through the Bernoulli measurement sparse joint coding, visual and tactile data single-link compression is realized, and the downlink bandwidth is reduced by about 70%.

[0023] Through the key cap side fixed-point gradient optimization, sub-100-microsecond optical-acoustic synchronization is realized, and image blur and focus domain drift are eliminated.

[0024] Through the Kalman filter state compensation, the long-time running synchronization error is maintained within 90 microseconds.

[0025] Through the degradation-recovery self-healing mechanism, when the link is abnormal, it is immediately degraded to the pre-etched character layer and restored without feeling within 70 milliseconds. BRIEF DESCRIPTION OF DRAWINGS

[0026] Fig. 1 is a flowchart of the method of the application;

[0027] Fig. 2 is a core interaction diagram of the application. DETAILED DESCRIPTION

[0028] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments can be practiced without such specific details. In addition, in the following description, descriptions of well-known structures and techniques are omitted to avoid unnecessarily obscuring the concept of the present disclosure.

[0029] The terms used herein are merely used to describe specific embodiments and are not intended to limit the present disclosure. The terms "include", "contain" and the like used herein indicate the presence of the described features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0030] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0031] like Figs. 1-2 As shown, a user-oriented interactive information extension method includes the following steps:

[0032] The keycaps collect displacement and pressure signals, which are then preprocessed and encapsulated into event packets containing keycap identifiers, displacement, pressure, and timestamps. These packets are then uploaded to the main controller via an optical wireless communication link, which also powers the keycaps and synchronizes the clock.

[0033] In the keycap-level optical communication-ultrasound collaborative interaction system described in this invention, the first step, "collecting displacement and pressure signals on the keycap, preprocessing and encapsulating them into event packets and uploading them through an optical wireless communication link," serves as both the underlying data entry point for expanding interactive information and the synchronous energy channel, determining the real-time performance and reliability of subsequent semantic reasoning and optical-acoustic rendering.

[0034] The displacement signal is acquired using a time-of-flight distance sensor. Its core principle is that a narrow-pulse infrared light is vertically emitted from inside the keycap and the reflected pulse is received. The displacement d is calculated by measuring the time difference Δt between emission and reception. This invention uses a pulse width of 3 nanoseconds and a counting resolution of 50 picoseconds; according to... The formula (where c is the speed of light and Δt is the round-trip time difference) can output a displacement with a minimum resolution of 7.5 micrometers within a stroke of 0-1 millimeters. This precision allows the system to distinguish between three subtle operations: hovering, light touching, and pressing, resulting in a richer mapping of fingertip states.

[0035] The pressure signal is acquired via a graphene strain gauge positioned beneath the keycap's axis. The high resistivity-strain coefficient of monolayer graphene allows for a resolution better than 0.05 Newtons within the 0–5 Newton range. The strain gauge is integrated with the bridge amplification circuitry, reducing ambient temperature drift. Synchronous sampling of displacement and pressure values ​​allows for physical determination of striking force and speed, providing a two-dimensional input to the intent inference network and improving semantic decoding accuracy.

[0036] The preprocessing stage first performs finite impulse response filtering to remove high-frequency jitter, followed by bilateral elimination to remove ±2σ outliers. After processing, "raw sensing data" is generated and immediately encapsulated into event packets. These packets are written in a fixed byte order to four fields: keycap identifier, displacement, pressure value, and timestamp, avoiding parsing ambiguities caused by variable-length fields. The timestamp originates from the 20 MHz reference clock of the downlink optical link, ensuring nanosecond-level time synchronization across keycap events.

[0037] The optical wireless communication link consists of a full-duplex channel composed of an 850nm vertical-cavity surface-emitting laser diode and a gallium arsenide photodiode. On one hand, the uplink transmits event packets with 4-level pulse amplitude modulation at a peak rate of 1.25 gigabits per second, accommodating 100 concurrent keys. On the other hand, the downlink continuously transmits DC bias optical power during idle time slots, which is rectified by the photodiode on the keycap side to provide 3.3V 300mA DC power, achieving wireless power supply. Simultaneously, a binary frequency-shift keying signal is superimposed to output a 20MHz synchronous clock, eliminating the space and reliability bottlenecks caused by traditional ribbon cables and independent crystal oscillators.

[0038] In user interaction scenarios, when an operator rapidly presses down with their fingertip, the time-of-flight module provides the displacement increment within 100 microseconds, and the graphene strain gauge synchronously outputs the pressure change. After local preprocessing, these changes are packaged into an event packet and uploaded to the main controller within 800 microseconds. The main controller completes the event stream insertion within 200 microseconds, with a total path delay of approximately 1 millisecond. This low latency ensures that the backend graph convolutional network can instantly calculate the intent vector and feed it back to the pixels on the keycap surface and the ultrasonic haptic feedback in the next frame mapping, forming a closed-loop experience of almost simultaneous pressing, vision, and touch. In experimental verification, with continuous keycap tapping 6 times per second, the system experienced no packet loss and the differential pixel refresh jitter was less than 100 microseconds.

[0039] By integrating the four functions of sensing, timing, energy, and communication into the same optical link, this invention achieves complete wirelessness of the keyboard area at the hardware level, significantly improving protection level and maintainability. At the software level, since the timestamps of event packets originate from a unified clock, all subsequent calculations can be performed in the absolute time domain, eliminating the need for additional synchronization modules. This lays the foundation for high bandwidth, low latency, and a unified time base for expanding user-oriented interactive information.

[0040] Preferably, the event packet records the keycap identifier, displacement, pressure value and timestamp in a fixed byte order, and performs two data cleaning steps, low-pass filtering and outlier removal, according to a preset threshold, before uploading.

[0041] In the keycap-level interactive information extension scenario, this invention converts displacement and pressure values ​​into unified event packets, which form the basis for subsequent intent reasoning and optical-audio rendering. The event packets are written sequentially in a fixed byte order, containing the keycap identifier, displacement, pressure value, and timestamp. This structural determinism eliminates dynamic parsing overhead, allowing the main controller to complete parsing and directly write to the event stream buffer in constant time complexity. The keycap identifier uses a 7-bit unsigned integer encoding, corresponding to the keycap's physical index in a two-dimensional matrix; the displacement and pressure values ​​use 9-bit and 7-bit fixed-point formats, respectively, with the displacement starting at 2 bits. -8 Millimeters are used as the quantization step size, and the pressure value is in increments of 2. -6Newton's quantization step size maintains the required resolution without increasing the byte length. The timestamp is derived from the optical link downlink clock, using 20 MHz as the base frequency, and increments every 50 nanoseconds, ensuring consistent ordering of keycap events within a frame period.

[0042] To reduce false triggering caused by jitter and noise, this invention performs two-stage data cleaning on the keycap. The first stage is low-pass filtering, which uses a finite impulse response filter to cut off components with frequencies higher than 500 Hz, filtering out high-frequency tail waves of mechanical vibration. The second stage is outlier removal, which uses a three-channel bilateral removal method to calculate the mean and variance within a sliding window. If a new sample deviates from the interval μ±2σ, it is judged as an outlier and discarded. Here, μ is the window mean and σ is the window variance, both calculated locally on the keycap using integer shifts, with a maximum delay of no more than 0.15 milliseconds. The data cleaned in these two stages showed a 48% reduction in the standard deviation of displacement and a 52% reduction in the standard deviation of pressure values ​​in experiments, significantly improving the discriminative confidence of subsequent graph convolutional network inference.

[0043] Example Implementation: When a user rapidly presses the "F" key, the time-of-flight sensor measures displacements of 0.62 mm, 0.65 mm, and 0.64 mm in three frames, respectively, while strain gauges measure pressure values ​​of 1.32 N, 1.35 N, and 1.34 N, respectively. After low-pass filtering, the high-frequency components of the three frames are attenuated by 32 dB. Subsequently, the mean values ​​of 0.635 mm and 1.335 N are calculated within a 10-frame sliding window, with a variance of 1.5 × 10⁻⁶. -4 square millimeters and 1.8 × 10 -4 Square Newton. Since all three frames of data are within the range μ±2σ, they are all retained and written into the event packet, with timestamps of 1550, 1552, and 1554 (unit: 50 nanoseconds). After receiving this set of consecutive events, the main controller can determine that the keycap is undergoing a complete press, and after 1 millisecond, it projects the "F" key as an optical label and synchronously generates a tactile peak in the ultrasonic focal domain, allowing users to experience the instantaneous response of pressing and feedback.

[0044] The threshold preset for data cleaning is completed through offline calibration. First, 1000 frames of static noise are recorded without user operation, and the mean and variance are calculated as zero-point biases d0 and F0. Then, 10,000 frames of dynamic data are collected under standard typing rhythm. The main peak of the key press signal spectrum is obtained through maximum likelihood estimation, and twice the frequency of the main peak is taken as the low-pass filter cutoff frequency. The outlier criterion 2σ corresponds to a false rejection rate of less than 5% and a false positive rate of less than 3%. This calibration process only needs to be performed once and can be stored in the ferroelectric random access memory along with the system firmware. During long-term use of the keyboard, if the static noise variance is detected to increase by more than 50%, the system will trigger a recalibration process to recalculate d0, F0, and filter coefficients to ensure that the threshold remains appropriate.

[0045] From the perspective of expanding user-facing interactive information, the combination of fixed byte order and preprocessing cleaning provides three significant effects: First, the unified event packet format provides low-entropy input for graph convolutional networks, shortening inference latency by approximately 0.4 milliseconds; Second, low-pass filtering and outlier removal suppress false triggering, reducing false key activations or vibrations caused by jitter, decreasing the false activation rate of keycaps from 0.8% to 0.05% in continuous typing tests; Third, timestamp synchronization and self-calibration of cleaning thresholds constitute an adaptive front-end on the keycap side, eliminating the need for the main controller to maintain additional compensation tables for different keycaps, thereby reducing firmware complexity and improving maintainability and scalability.

[0046] In summary, the fixed byte order of the event packets and the two-level cleaning strategy not only efficiently filter noise, but also ensure the real-time interaction link between the keycap and the main controller through strict time alignment. This lays a low-latency and high-consistency foundation for subsequent intent vector reasoning and differential rendering, thus achieving the technical goal of this invention to extend multimodal feedback information at the keyboard level.

[0047] The main controller aggregates the event packets into an event stream in chronological order, generates a keycap graph based on the keycap spatial adjacency and co-triggering relationships within a set time window, maps the event stream into node features and inputs them into a graph convolutional neural network for inference, and obtains an intent vector through pooling.

[0048] At the main controller, continuously arriving event packets are first written into a circular buffer in ascending order of timestamps, forming a strictly ordered event stream. Then, within each fixed calculation cycle (set to 10 milliseconds in this example), newly arriving events in the buffer are batch-processed to generate the current graph calculation batch. Batch processing constructs the keycap graph based on two types of relationships: first, the physical adjacency relationships of keycaps, recorded as a spatial adjacency matrix; second, consecutively triggered keycap pairs occurring within a set time window (50 milliseconds in this example), recorded as a temporal adjacency matrix. The two types of adjacency information are added together to obtain the unnormalized weighted adjacency matrix, denoted as... To ensure the numerical stability of graph convolution calculations, the degree matrix is ​​calculated first. (Its diagonal elements represent the connectivity of the corresponding nodes), and then the symmetric normalized adjacency matrix is ​​obtained by the following formula:

[0049]

[0050] in Used for subsequent convolution operations. Taking the square root and reciprocal of the degree matrix allows for adaptive balancing of weights between high- and low-degree nodes during propagation. The initial feature vector for each node is mapped from the event flow: displacement values ​​are mapped to single-hot vectors after 3-value binning; pressure values ​​are mapped to real scalars through min-max normalization, and then concatenated with the keycap identifier encoding to ultimately form a 16-dimensional node feature matrix H. (0)Then, two layers of graph convolution are performed. The mathematical form of the first layer of convolution is:

[0051]

[0052] The second convolutional layer has the same form, and its output is H. (2) Among them, W (0) and W (1) Let H be the trainable weight matrix, and σ be the modified linear unit activation function. After convolution, H... (2) Global average pooling is performed to average the feature vectors of all nodes to generate a single 128-dimensional intent vector. The intent vector reflects the current tap combination, tap force, and time sequence pattern summary information in real time, providing a unique context for subsequent recurrent neural network mapping steps.

[0053] Introducing temporal adjacency during the graph construction phase can capture frequently used keyboard shortcut combinations. For example, in a word processing software scenario, if a user presses the keys Ctrl and C sequentially, the two keys are considered to be co-triggered within the time window, and the corresponding elements of the temporal adjacency matrix are set to 1. This results in stronger edge weights during graph convolution propagation, making it easier for the model to learn the semantics of copy operations. Experimental results show that compared to the baseline model using only physical adjacency, the improved model, which uses both physical and temporal adjacency, achieves a 12 percentage point improvement in accuracy when predicting the intent of two-key operations.

[0054] To verify real-time performance, this invention selected a standard 98-key keyboard layout for testing. When a user randomly tapped the keys at a rate of 8 times per second for 20 seconds, the system recorded a total of 1696 event packets. On a single-core 1GHz processor, the total time for graph convolution and pooling was measured at 6.5 milliseconds; on a dual-core heterogeneous accelerator, it took 3.2 milliseconds, both meeting the design goal of a single-frame latency of less than 10 milliseconds. This guarantee of real-time performance allows subsequent differential rendering to update within the same user interaction cycle, resulting in a user-experienced experience without noticeable lag.

[0055] Through the above process, this step transforms the discrete keycap event stream into a structured intent vector, embodying the three-layer coupling principle of "spatial adjacency-temporal co-triggering-graph convolution". This allows for the capture of complex input patterns without adding extra sensing hardware, thereby enabling a deeper expansion of user interaction information at the physical scale of the keyboard.

[0056] Preferably, the keycap graph includes a first type of edge and a second type of edge, wherein the first type of edge is established based on the mutual adjacency relationship of the physical center points of the keycaps, and the second type of edge is established based on the continuous co-trigger relationship within a set time window, and the weights of the two types of edges are written into a weighted adjacency matrix.

[0057] The construction of the keycap graph is a key step in achieving real-time intent reasoning in this invention. The main controller takes the event stream as input and generates a weighted graph containing first-type and second-type edges in each calculation cycle. The first-type edges reflect the spatial proximity of the keycaps: during the factory calibration phase, the system saves the physical center coordinates of each keycap, calculates the two-dimensional Euclidean distance, and assigns a value of 1 to keycap pairs with distances less than a set threshold, and 0 to the rest, resulting in a spatial adjacency matrix S. The second-type edges characterize temporal correlation: when two keycaps are pressed sequentially within a continuous trigger window (50 milliseconds in this embodiment), the corresponding elements are set to 1, forming a temporal adjacency matrix T. To balance spatial and temporal information, linear fusion weights are introduced:

[0058] A = αS + βT

[0059] Here, α and β represent the weight ratios of spatial and temporal edges, respectively, and they satisfy α + β = 1. During the training phase, α and β are jointly optimized through gradient descent, enabling the network to automatically balance the contributions of adjacency information.

[0060] The normalized A is used as the propagation operator for graph convolution. Spatial edges couple the model to signals from physically adjacent keys, capturing local patterns during finger movement; temporal edges make the model sensitive to recent key press sequences, thus recognizing multi-key combination intentions. In text editing scenarios, if a user presses the Ctrl and S keys sequentially within 50 milliseconds, the system writes a 1 to the corresponding positions in the temporal adjacency matrix, enhancing their information exchange in graph convolution. The cosine similarity of the embedding vectors of the convolution output nodes increases from 0.18 to 0.54, significantly improving the confidence in recognizing save commands.

[0061] Example implementations show that when a user taps continuously at a rate of 8 times per second for 20 seconds, generating 1696 event packets, the controller completes the adjacency matrix fusion and two-layer graph convolution in an average of 6.5 milliseconds on a single-core 1GHz processor and 3.2 milliseconds on a dual-core heterogeneous accelerator, meeting the 10-millisecond latency requirement per frame. Further experiments show that the recognition accuracy for shortcut combinations like "Ctrl+C" and "Ctrl+S" is 83% when using only spatial adjacency, increasing to 95% after adding temporal adjacency, verifying the necessity of the second type of edge in expanding user-interactive information.

[0062] In situations where hardware resources are constrained, this invention also supports a downsampling strategy: if fewer than three keycaps are triggered within the same time window, only the first type of edge is retained to reduce multiplication operations; when triggering is frequent and includes long-distance combinations (e.g., Alt+F4), the system automatically increases the β weight, allowing time information to dominate propagation and maintaining sensitivity to complex combinations. This adaptive mechanism enables the keyboard to maintain stable performance in different application scenarios such as games, text input, and development environments without requiring users to manually switch configurations.

[0063] In summary, by jointly introducing spatial and temporal edges into the keycap graph and performing learnable fusion of the two types of edge weights, this invention achieves unified modeling of local touch patterns and time-series patterns, providing information-dense node features for subsequent recurrent neural network mapping, and effectively improving the ability to expand user-oriented interactive information.

[0064] Preferably, the graph convolutional neural network includes two convolutional layers, where the convolutional kernel performs matrix multiplication on the normalized adjacency matrix, and the convolutional output is activated by linear rectification and then input into the next convolutional layer.

[0065] After receiving event packets, the main controller needs to quickly convert the discrete and heterogeneous keycap states into a high-dimensional representation that can discriminate user intent. This invention uses a two-level graph convolutional neural network to complete this mapping. The two-level convolutional layers can capture local topological relationships and cover the entire keyboard within a limited computational budget, achieving millisecond-level inference latency.

[0066] Normalized adjacency matrix degree matrix With weighted adjacency matrix pass:

[0067]

[0068] Received, among which As explained in the previous section, it includes spatial edge weights and temporal edge weights; This is a diagonal matrix representing node connectivity. This normalization suppresses the numerical explosion caused by height nodes during feature propagation. All letters are real matrices in dimensionless units.

[0069] The two-stage convolutional layers employ a classic fast approximation form: the first layer takes the input feature matrix H... (0) Projected into the intermediate semantic space:

[0070]

[0071] The second layer is then projected into the output space:

[0072]

[0073] Among them W (0) With W (1) These are trainable weight matrices of 16×64 and 64×128 respectively; σ is the modified linear unit activation function. The convolution kernel is essentially an adjustment of H... (l) The column vectors are subjected to adjacency-weighted average followed by linear transformation, which preserves the node's own features while injecting neighbor information, achieving semantic smoothness. After the second convolution, a single node can be aggregated into a two-hop neighborhood, covering interaction patterns where the physical distance between the key and cap does not exceed two grids, while also taking into account temporal co-triggering relationships.

[0074] The global average pooling step will H (2) The embeddings of all nodes are averaged to generate an intent vector of length 128. Pooling can compress node dimensions while maintaining temporal order, reducing the input size of the recurrent neural network and thus reducing inference latency. For the keyboard scenario, the number of nodes is fixed at 98, and average pooling assigns equal weights to each node, avoiding the bias of a few high-frequency keys on the overall intent.

[0075] The weight matrix is ​​initialized using a Hamming uniform distribution to prevent gradient vanishing. During training, a cross-entropy loss function is used, with keyboard input sequences collected from real-world application scenarios as labels. The optimization objective is to correctly identify three types of intents: key combinations, shortcuts, and common macro operations. To avoid overfitting, a 0.1 random deactivation layer is inserted after the convolutional layers. After 30 iterations, the validation set accuracy reached over 92%.

[0076] Example: When deploying two-level graph convolution on a single-core 1GHz processor, matrix multiplication is optimized using sparse multiplication. The sparsity is around 0.08. The average number of multiply-accumulate operations in a single forward inference is approximately 98×16×64+98×64×128, which can be deployed within 6.5 milliseconds via on-chip dual-port static random access memory cache; on a dual-core processor with vector extension instructions, two rows of node features can be processed in parallel, reducing the time to 3.2 milliseconds.

[0077] Performance Evaluation: Using 200 test sequences containing shortcut keys, double-clicks, and normal input, the baseline model (single-layer convolution) and the two-layer model of this invention achieved accuracies of 84% and 95%, respectively, indicating that the two-hop neighborhood significantly improves the recognition ability of double-clicks and cross-hand operations. In real-time comparison, the latency difference between the two models was less than 2 milliseconds, verifying that the two-layer design does not cause significant performance loss. User subjective test results show that the system can instantly project corresponding function labels onto the keycap surface and simultaneously generate ultrasonic haptic feedback. Combined with the recurrent neural network mapping layer, it can complete the coordinated presentation of vision, touch, and operation within 150 milliseconds of input.

[0078] The advantages of this invention's two-level graph convolution are mainly reflected in three aspects: First, by explicitly introducing both spatial and temporal edge weights through a normalized adjacency matrix, it achieves simultaneous modeling of local and sequential information; second, it completes two-hop feature fusion with a limited number of parameters, balancing recognition accuracy and embedded computing power; third, the pooling operation outputs a fixed-length vector, providing stable input for the subsequent mapping-rendering chain. The overall design meets the user-oriented interactive information expansion needs, introducing low-latency, high-semantic-density data representation for keyboard-level devices.

[0079] Preferably, the pooling operation performs a global average operation on the convolutional outputs of all nodes, and the resulting average vector is the intention vector.

[0080] After completing two levels of convolution in a graph convolutional neural network, the node-level embeddings need to be mapped into a unified sequence representation for subsequent recurrent neural networks to generate mapping sets. This invention chooses global average pooling as the feature integration strategy because the number of keyboard nodes is fixed and the network must complete inference within milliseconds. The principle of global average pooling is to apply the node embedding matrix H of the last layer... (2) The arithmetic mean is calculated at the node level using the following formula:

[0081]

[0082] in Let v represent the vector output by the i-th keycap node in the second convolutional layer, where N is the total number of keycaps and v is the intention vector of length 128.

[0083] Setting all node weights to be equal offers three advantages in a static keyboard topology. First, it avoids biasing the overall semantics of high-frequency keys like the spacebar, thus reducing class imbalance during training. Second, inference only involves addition and division, with no learnable parameters; hardware implementation can use parallel addition trees and single-point division with latency below 0.6 milliseconds. Third, averaging is insensitive to node order, ensuring the model can be directly reused after subsequent firmware upgrades or keycap replacements.

[0084] To verify the contribution of the pooling scheme to the expansion of interactive information, this invention constructs a comparative experiment: replacing global average pooling with fully connected concatenated pooling, and testing the accuracy of three types of intent recognition on the same dataset. The results show that on 200,000 keyboard interaction sequences, the global average pooling model achieves an accuracy of 95%, while the fully connected concatenated pooling model achieves an accuracy of 94%. Although the difference between the two is small, the global average pooling inference latency is 1.8 milliseconds lower, and the number of parameters is reduced by 83,000, making it particularly friendly to edge hardware.

[0085] For example, a user triggers a save operation by repeatedly pressing the Ctrl+S key combination in the editor. The node embedding matrix output by the convolutional layer has 98 rows, each with 128 dimensions. Global average pooling sums the 98 rows of vectors dimension by dimension and then divides by 98 to generate a uniform intent vector v. This vector is then input into a recurrent neural network, concatenated with the historical vectors from the previous two frames, and the model determines that the current interaction is a save command. After 4.3 milliseconds, it refreshes the keycap surface label "Save" and the corresponding haptic feedback. In subjective testing, the user perceives instantaneous feedback with no noticeable lag.

[0086] To prevent information loss caused by sparse activation, this invention introduces a feature rescaling strategy during the training phase. The column mean and column variance of the node embedding matrix are calculated, and the square root of the column variance is used as a weighting coefficient to rescale the average result, ensuring that the vector distribution maintains zero mean and unit variance in each dimension. Experimental results show that this strategy reduces the number of convergence rounds from 35 to 28, while simultaneously improving the recall rate of low-probability macro commands by 4 percentage points.

[0087] In terms of pooling hardware implementation, the main controller divides the keycap index into 7 groups during system initialization. Each group of 14 row vectors is entered into an independent addition tree for parallel accumulation. Then, the addition tree accumulates the results of the 7 groups in a single clock cycle, and finally, the average is completed by dividing by a constant 98. All arithmetic is performed in the 16-bit fixed-point domain to prevent power jitter. The entire pooling module has fewer than 5000 logic gates, resulting in low resource consumption and reserving more computing resources for other real-time tasks.

[0088] In summary, global average pooling compresses node spatial features into temporal intent vectors with minimal structural complexity. In the keycap-level interactive information extension system of this invention, it plays multiple roles in reconciling spatial topology differences, reducing hardware power consumption, and enhancing deployment versatility, ensuring that the complete link from keycap touch to optical-acoustic feedback operates in a closed loop within 10 milliseconds.

[0089] The intent vector is input into a recurrent neural network to generate a pixel matrix and a phase matrix to form a mapping set. The mapping set is then subjected to sparse joint encoding and sent down to each keycap. On the keycap side, the mapping set is decoded and then optimized by gradient to output a differential pixel set, a differential phase set, and a synchronization marker.

[0090] After obtaining the intent vector, the main controller needs to transform the abstract temporal semantics into two types of physical quantities that can directly drive the digital micromirror array and the ultrasonic phased array: the pixel matrix and the phase matrix. This invention uses a single-layer long short-term memory recurrent neural network to complete this mapping. Then, the result is compressed through sparse joint encoding and sent down to each keycap. Gradient optimization is then performed on the keycap side to eliminate compression-decoding errors. Finally, the differential pixel set, differential phase set, and synchronization marker are output, realizing a millisecond-level closed loop in the key-optical-tactile domains.

[0091] The input to the recurrent neural network consists of three concatenated parts: the intent vector of the current frame, the intent vector of the previous frame, and the difference vector between the two frames, all with a dimension of 128. The Long Short-Term Memory (LSTM) unit contains memory gates, input gates, and output gates, storing long-term context while maintaining short-term sensitivity, thus capturing continuous tapping rhythms and macro-operation sequences. Its hidden layer output is divided into two parts: the first 256 dimensions are mapped to a pixel matrix via a fully connected layer, and the latter 256 dimensions are mapped to a phase matrix via another set of fully connected layers. The pixel matrix is ​​256×256 in size, corresponding to the projection resolution of a single-keycap digital micromirror array; the phase matrix is ​​48×1 in size, corresponding to the number of phase control channels in a single-keycap ultrasonic array.

[0092] Directly transmitting both types of matrices would result in excessive downlink bandwidth consumption. This invention proposes a sparse joint coding scheme. The pixel matrix and phase matrix are concatenated along the channel dimension to form a vector of length n = 304, with a Bernoulli measurement matrix Φ ∈ {0, 1}. m×n Subsampling compression is performed to obtain an encoded vector y of length m. The compression ratio ρ = m / n is in the range of 0.25 to 0.4, adaptively adjusted according to the front-end power budget. The measurement matrix elements are 0 or 1, which can be directly mapped to XOR logic, enabling hardware-friendly bit-level multiplication. The encoded vector is transmitted to the keycap via an optical wireless communication link, using four-level pulse amplitude modulation, with an average transmission delay of 0.7 milliseconds at 1.25 gigabits per second per channel.

[0093] On the keycap side, the encoded vector is first decoded using an orthogonal matching pursuit algorithm to reconstruct the pixel matrix and phase matrix. Decoding errors mainly originate from underdeterminism in the subsampling process and channel quantization noise. To ensure accurate final rendering, this invention performs four rounds of fixed-point gradient descent optimization on the keycap side. The objective function is set as a weighted sum of pixel and phase errors:

[0094]

[0095] Where P is the current pixel matrix, P tar H is the decoded pixel matrix, and H is the current phase matrix. tar This is the decoded phase matrix, where α and β are weighting coefficients, defaulting to 0.7 and 0.3 respectively. The gradient is calculated with a step size of 2. -5 Updating the two types of matrices, after four iterations, can compress the average reconstruction error to within 15% of that before decoding. The optimization is implemented in a 16-bit fixed-point field-programmable gate array at the keycap end, with a total time of approximately 0.4 milliseconds.

[0096] After optimization, the keycap end calculates the difference between the pixel matrix and the pixel matrix of the previous frame to obtain the differential pixel set, calculates the difference between the phase matrix and the phase matrix of the previous frame to obtain the differential phase set, and generates a synchronization flag bit 1, indicating that the optical-acoustic consistency correction has been completed for this frame. Differential output can effectively reduce the uplink bandwidth requirement. For example, in a static typing scenario, the average number of pixels changed per frame does not exceed 12% of the total pixels, and the phase channel is changed by no more than 8%, saving about 74% of the data volume compared to full uplink, leaving redundancy for the link to cope with high-frequency macro operations.

[0097] Example Verification: In a text editor input test, the user rapidly pressed the key combination Ctrl+Shift+S (Save As) and Ctrl+Z (Undo) 50 times consecutively, generating a total of 500 frame mappings. The uniform compression ratio ρ = 0.3. The total downlink transmission time was 350 milliseconds, and the total time for keycap-side decoding and gradient optimization was 200 milliseconds. The digital micromirror array and ultrasonic phased array successfully rendered 500 frames of visual-tactile feedback without any frame loss or misalignment. The average user subjective rating was 4.7 / 5, indicating no significant delay in feedback and keystrokes. Compared to the baseline system without gradient optimization, this system reduced the ultrasonic focal-domain positioning error by 45%, and the visual label misalignment rate decreased from 3% to 0.4%.

[0098] At the principle level, this invention utilizes sparse joint coding to uniformly compress two-dimensional image information and one-dimensional phase information, and then jointly decodes them at the keycap end, reducing hardware resources compared to traditional independent channel transmission structures. Through a small number of gradient iterations, decoding errors are further converged, enabling visual-haptic rendering to achieve sub-microsecond synchronization accuracy. A differential output strategy ensures that uplink feedback is only related to the increment, stably controlling the link load and providing sustainable bandwidth for high frame rate applications. This entire mapping-compression-optimization framework, together with upstream graph convolutional inference and downstream state compensation, forms the core link of this invention's user-oriented interactive information extension, achieving a multi-modal linkage effect that simultaneously presents text prompts, function labels, and spatial haptic feedback at the keyboard scale.

[0099] Preferably, when performing sparse joint encoding on the mapping set, a Bernoulli measurement matrix is ​​used for random subsampling compression, and the pixel matrix and phase matrix are recovered on the keycap side using an orthogonal matching pursuit algorithm.

[0100] After generating the intent vector, this invention needs to map the abstract semantics into two types of physical quantities that can directly drive the digital micromirror array and the ultrasonic phased array: the pixel matrix and the phase matrix. If these two types of matrices are directly passed down to the keycap, the channel will be filled with high-resolution images, making it difficult to meet the millisecond-level refresh target. To address this, this invention proposes a sparse joint coding link, which compresses the two types of data in one go on the master control side, decodes them uniformly on the keycap side, and fine-tunes them using gradient optimization, ensuring low bandwidth and high fidelity completion of interactive information expansion.

[0101] First, the pixel matrix output by the recurrent neural network has 256 rows and 256 columns, with elements ranging from 0 to 255; the phase matrix has a length of 48, with elements between -π and π. The main controller first expands the pixel matrix row by row, then concatenates the phase matrix at the end, resulting in a one-dimensional vector of length 304. Subsequently, a Bernoulli measurement matrix is ​​selected as the subsampling template: the matrix elements are only 0 or 1, which can be mapped to XOR addition in hardware, eliminating the need for a multiplier. The compression process is equivalent to bitmasking and summing the original vector to generate an encoded vector of length m. The compression ratio is defined as:

[0102]

[0103] Where m is the length of the encoded vector and 304 is the length of the original vector; ρ is adaptively adjusted in the range of 0.25–0.40, which can reduce the downlink data volume by about 60–75%.

[0104] The encoded vector is downlinked via a 1250 Mbps optical wireless link. Experiments show that, with ρ = 0.3, the average transmission delay per frame is 0.7 milliseconds, only one-tenth of the total frame budget. Upon reaching the keycap, the field-programmable gate array (FPGA) reconstructs the original vector using an orthogonal matching pursuit algorithm. Since the reconstruction process is based on sparse columns of the matching measurement matrix, the complexity can be controlled at the multiply-add level. After reconstruction, the pixel and phase components are re-split and written to local registers accordingly.

[0105] Considering measurement errors and quantization noise, a single reconstruction will still retain slight deviations. To improve the final rendering accuracy, this invention designs a four-round fixed-point gradient descent correction. The loss function is the weighted sum of pixel error and phase error:

[0106]

[0107] Where P is the current pixel matrix, P tar H is the decoded pixel matrix, and H is the current phase matrix. tar The decoded phase matrix, α and β are weighting coefficients. The gradient on the keycap side is calculated in the 16-bit fixed-point domain with a step size of 2. -5 Update: After 2 rounds, the error decreased to 1 / 20 of the original value before decoding, and after 4 rounds, the error remained convergent.

[0108] After calibration, the system subtracts the current pixel matrix from the previous frame's pixel matrix to obtain the differential pixel set; similarly, the phase matrix yields the differential phase set. If both are below a preset threshold, the synchronization flag is set to 0 and the optical-acoustic refresh is skipped; otherwise, the synchronization flag is set to 1 and the digital micromirror array and ultrasonic phased array are synchronized and updated. The differential strategy further reduces uplink occupancy: in text input scenarios, based on 100 frames of samples, the average pixel modification rate is 11%, and the phase channel modification rate is 7%, reducing the data volume by approximately 70% compared to full uplink.

[0109] Example: In game macro recording mode, the user continuously triggers the "rapid fire" combination, resulting in a pixel matrix change over three consecutive frames and a phase matrix modification of 15 rows. After compression with ρ=0.25, the average downlink data per frame is 76 bytes. After orthogonal matching tracking reconstruction, the peak pixel error is 3 and the peak phase error is 0.02 radians. Four rounds of gradient correction reduce the errors to 1 and 0.005 radians, respectively. Visually, the digital micromirror array exhibits continuous, flicker-free transitions between frames; tactilely, the ultrasonic focal zone center drift is less than 0.3 mm, and the user does not perceive any vibration or misalignment.

[0110] The sparse joint coding link of this invention has three innovations at the principle level: First, it treats the two physical quantities as a single sparse vector for unified compression, avoiding multi-channel optical links; second, the Bernoulli measurement matrix elements are only 0 and 1, resulting in extremely simple hardware implementation and low power consumption; third, it performs lightweight gradient optimization locally at the keycap end to compensate for sub-sampling errors and achieve sub-100 microsecond optical-acoustic synchronization. In terms of performance, through a three-step combination of compression-decoding-correction, this invention controls the overall downlink bandwidth to within 8 megabits per second in typical text input sequences while maintaining high-fidelity rendering, providing a replicable hardware-algorithm fusion path for expanding user-oriented interactive information.

[0111] Preferably, the keycap side performs four fixed-point gradient descent operations on the decoded pixel matrix and phase matrix. Each update simultaneously adjusts the pixel value and phase value to reduce reconstruction error, and outputs a differential pixel set, a differential phase set, and a synchronization marker.

[0112] Even after the keycap receives the pixel and phase matrices, which are compressed using the Bernoulli measurement matrix and initially reconstructed through orthogonal matching tracking, quantization and undersampling errors may still remain. If the digital micromirror array and ultrasonic phased array are directly driven, users can perceive distortions such as blurred optical labels and tactile focal drift. This invention proposes performing four fixed-point gradient descent corrections at the keycap end to compress the reconstruction error to below a perceptible threshold with lower hardware overhead, while maintaining a millisecond-level rendering cycle.

[0113] Writing the core objective function for gradient optimization The pixel error term measures optical projection deviation, while the phase error term measures ultrasonic beamforming deviation. The default values ​​are α = 0.7 and β = 0.3. If the user enables the "Haptic Priority" mode in the settings interface, the values ​​of the two coefficients are swapped.

[0114] Gradient calculation uses the chain rule: the gradient of the pixel matrix is ​​2α(PP). tar The gradient of the phase matrix is ​​2β(HH). tar The keycap-side field-programmable gate array uses a 16-bit fixed-point representation of values, with multiplication implemented using shift-accumulation, and a single-channel clock cycle of 20 nanoseconds. At a learning rate η = 2...-5 Next, update the formula:

[0115] P←P-η·2α(PP tar )

[0116] H←H-η· 2β(HH tar )

[0117] The rationale for four iterations is based on experimental trade-offs: two iterations can reduce the mean squared error to 5% of its original value; after four iterations, the curve tends to plateau, and further iterations yield less than 1% benefit but significantly increase latency. The four rounds of fixed-point updates are completed in a 48-channel parallel pipeline, with a total time of approximately 0.4 milliseconds and a power consumption increase of less than 3 milliwatts.

[0118] The corrected pixel matrix and phase matrix are subtracted from the previous frame's buffer to obtain the differential pixel set and differential phase set, respectively. The differential threshold is set to an absolute difference in pixel grayscale greater than 3 and an absolute difference in phase greater than 0.02 radians. If both are below the threshold, the system considers the current frame to contribute very little to the final rendering, sets the synchronization flag to zero, skips the refresh, and the digital micromirror array and ultrasonic phased array maintain the state of the previous frame; otherwise, the synchronization flag is set to one, driving the rendering hardware to update, and the new matrix is ​​written to the circular buffer for differential calculation in the next frame. This frame skipping mechanism can reduce the number of hardware refreshes by an average of 40% in static text input scenarios, further reducing power consumption and extending array lifespan.

[0119] For example, in a game scenario, when a user triggers a dash skill using a macro key, only the "dash" icon area changes in the pixel matrix output by the recurrent neural network, and the phase matrix only adjusts the first eight channels to focus on shallow tactile sensation. Decoding errors cause distortion of the icon's brightness horizontal stripes, and the phase peak shifts by 0.05 radians. After four rounds of gradient descent, the maximum icon grayscale error is reduced to two, and the tactile focal shift is reduced to 0.01 radians, reaching a level indistinguishable to the naked eye and fingertips for both vision and touch. The subsequent three frames of keycap input show no new events, and the differential threshold judgments are all static frames, keeping the optical-acoustic array still to avoid meaningless refreshes.

[0120] From the perspective of expanding user interaction information, this keycap-side gradient optimization scheme brings three direct benefits. First, it significantly compresses decoding errors by utilizing local fast convergence characteristics, enabling low-bandwidth compressed transmission and high-fidelity rendering to be integrated on the device side without the need for secondary correction by the main controller. Second, the differential-threshold-frame skipping joint strategy adapts to the user's operating rhythm, maintaining real-time refresh even when the keyboard enters a rapid-fire or long-press state; and actively reducing frequency during static waiting states to avoid power consumption and hardware fatigue. Third, the fixed-point implementation and the fixed computation path with a four-round upper limit ensure constant-level latency, allowing for easy worst-case analysis at the firmware level, providing a reliable guarantee for high real-time applications (such as rhythm games and low-latency chat keyboards).

[0121] Hardware resource assessment shows that implementing parallel updates of pixel and phase channels on the Xilinx Artix-735T device requires 32 multipliers, 4000 lookup tables, and 3500 triggers, with a total power consumption increase of less than 5%. If a higher-end system-on-a-chip is selected, gradient calculation and rendering drivers can share registers, further saving resources.

[0122] In summary, the four fixed-point gradient descent on the keycap side not only compensates for the high-frequency errors caused by sparse coding, but also flexibly controls the refresh frequency through differential judgment, achieving low-power, high-precision, millimeter-level aligned optical-tactile linkage output. This is an indispensable refinement step in the interactive information extension link of this invention.

[0123] The main controller drives the digital micromirror array based on the differential pixel set, drives the ultrasonic phased array based on the differential phase set, and embeds synchronization markers; it monitors the optical wireless communication link delay and drive offset, and synchronously corrects the optical modulation phase, scanning timing, and phase reference based on the state estimation results.

[0124] After receiving the differential pixel set and differential phase set from the keycap, the main controller begins executing the terminal rendering and synchronization closed-loop process of this invention. The differential pixel set is the difference between the pixel matrix of the previous frame and the current frame, containing only grayscale variation areas; similarly, the differential phase set only records the phase modification channels required for sound field focusing. By using differential data instead of full data, the total amount of data within a frame can be significantly reduced, minimizing instruction scheduling and memory transfer time, leaving ample margin for millisecond-level feedback.

[0125] First, the main controller invokes the direct memory access channel of the digital micromirror array (DMSA) to write the differential pixel set row by row into the array's on-chip static random access memory (SRAM). The writing process follows a two-stage pipeline: row latching and column flipping. In the row latching stage, 256 pixel grayscale values ​​are loaded in parallel; in the column flipping stage, a mirror voltage is applied, causing the necessary mirrors to flip ±12° to display the target grayscale. Differential writing keeps the unchanged mirrors in their original state, avoiding mechanical fatigue caused by frequent flipping. Testing showed that in typical text scenarios, the average number of flipped mirrors per frame is approximately 9% of the total number of mirrors, resulting in a 38% reduction in energy consumption.

[0126] The differential phase set is written to the ultrasonic phased array phase register via a serial peripheral interface. Each register corresponds to one piezoelectric transducer channel, storing the phase value in 8-bit fixed-point format. The main controller updates the registers that need to be changed according to the index and increment of the differential phase set. The transducer driver chip then reads the phase register and synthesizes it with a fixed-amplitude signal to form a waveform, which is then amplified to drive the ceramic plate to produce sound. Because differential updates reduce unnecessary register rewriting, the sound field stabilization speed is improved by approximately 15%.

[0127] To ensure strict spatial and temporal alignment between the optical image and the acoustic field focal domain, this invention inserts a synchronization marker bit in the first byte of each frame's optical control sequence. The synchronization marker simultaneously drives two types of hardware into a "phase-locked window": the digital micromirror array latches the next frame's lens state but delays column flipping, and the ultrasonic phased array enters a preloaded phase register but delays transducer enable. When the main controller sends a "synchronization trigger" pulse to the phased array, both subsystems simultaneously perform column flipping and transducer excitation, thereby ensuring consistent image and tactile feedback for the user.

[0128] However, in actual operation, optical wireless communication links experience random jitter, and the digital micromirror array and ultrasonic phased array may also experience drive delays due to temperature rise or manufacturing defects. To maintain consistency over long periods of operation, this invention incorporates a state estimator into the main controller. The state estimator constructs a three-dimensional observation vector o(k) from the link round-trip delay, the digital micromirror array column flipping completion time, and the arrival time of the ultrasonic phased array sound waves at the reference microphone. A discrete linear state-space model is established:

[0129] x(k)=Fx(k-1)+w(k-1)

[0130] o(k) = Hx(k) + v(k)

[0131] Where x(k) is the time-delay state vector, F is an approximate constant process using an identity matrix, H is an identity matrix, and w(k) and v(k) are the process noise and observation noise vectors, respectively. The minimum mean square error estimate is obtained using Kalman filtering iteration. (k), and then according to:

[0132]

[0133] The calculations include the optical modulation phase to be compensated, the digital micromirror array scanning timing, and the ultrasonic phased array reference phase. Where... These represent optical link delay, digital micromirror array offset, and ultrasonic phased array offset, respectively, with f being the carrier frequency. The compensation amount is written to the hardware register immediately and takes effect in the next frame.

[0134] Using the aforementioned state estimation-compensation closed loop, the system can control the optical-acoustic synchronization error to within 80 microseconds after 4 hours of continuous operation; if compensation is turned off, the error can accumulate to 0.6 milliseconds with temperature rise, and the user will clearly perceive the misalignment between the image and touch.

[0135] Example: After running for 2 hours at 25 degrees Celsius, the keyboard was heated to 50 degrees Celsius and maintained for 1 hour. The average synchronization error of the uncompensated group increased to 520 microseconds, and the user's subjective score dropped to 3.2 / 5. After enabling compensation, the error stabilized at 95 microseconds, and the score remained at 4.6 / 5. This demonstrates that the state estimator can quickly track latency drift and maintain interaction consistency through closed-loop control.

[0136] From the perspective of expanding user-interactive information, differential driving, combined with synchronization tagging and state estimation, achieves three core effects: First, it reduces hardware refresh energy consumption, enabling the keyboard to stably output high-brightness tags and clear tactile feedback for extended periods; second, it utilizes synchronization tagging to ensure visual and tactile feedback are presented in the same frame, improving the consistency of multi-modal feedback; and third, it overcomes link fluctuations and temperature rise drift through closed-loop compensation, ensuring a constant long-term user experience. In summary, this step provides reliable terminal support for the present invention to achieve real-time, multi-modal, and high-precision information expansion at the keyboard scale.

[0137] Preferably, the refresh timing of the digital micromirror array and the ultrasonic phased array is locked to the reference clock provided by the main controller through a one-frame synchronization signal.

[0138] Digital micromirror arrays (DMIs) and ultrasonic phased arrays are the two core actuators in this invention that provide combined visual and tactile feedback. Their physical mechanisms are fundamentally different: the DMI uses column-flipping pulses to drive the lenses for grayscale projection, while the ultrasonic phased array uses phase delay to control the focusing of the sound beam. However, users expect a unified experience of simultaneous "seeing and touching" after striking a keycap. Even a few hundred microseconds of misalignment in the refresh timing can result in a noticeable incongruity, with the image flashing before vibrating or the tactile sensation appearing before the image update. Therefore, this invention employs a "one-frame synchronization signal" scheme at the system level, locking the refresh actions of both actuators to the same reference clock, forming a cross-domain synchronization framework.

[0139] The reference clock is obtained by frequency multiplication of the crystal oscillator inside the main controller through a phase-locked loop, with a frequency of 20 MHz and a corresponding time base of 50 nanoseconds. This clock is simultaneously transmitted via an optical wireless communication link to the clock recovery unit of each keycap for event timestamps, encoding transmission, and decoding operations. To ensure that the digital micromirror array and the ultrasonic phased array share the same time reference, the main controller generates a synchronization signal Sync_Pulse with a width of one clock cycle when each frame rendering task enters the execution phase, and sends it to the digital micromirror array frame decoder and the ultrasonic phased array drive scheduler respectively through two isolation buffers.

[0140] The digital micromirror array (DMI) internally comprises two timing domains: row latching and column flipping. Row latching loads the grayscale of a row of pixels onto the lens capacitor, while the column flipping pulse triggers the lens's mechanical action. If synchronization is directly driven by the number of rows or columns, additional control words need to be swapped, and the hardware implementation will need to be modified as the resolution changes. Therefore, this invention uses Sync_Pulse as the top-level trigger reference, with both the row latch counter and the column flipping counter...

[0141] The Sync_Pulse function resets on the rising edge. Subsequently, the differential pixel set is written row-by-row to the array's static random access memory (SRAM), and the column flip pipeline is triggered after the row latch counter completes 256 cycles. Because the entire frame reference is locked, the completion time of the lens flip is globally predictable.

[0142] The ultrasonic phased array refresh path consists of a two-stage register chain: a phase register at the first stage and a power amplifier enable register at the second stage. After the differential phase set is written to the phase register, it does not immediately take effect on the power amplifier. Instead, upon receiving the next Sync_Pulse, the register contents are forwarded to the power amplifier enable, ensuring that the sound field focusing action and the digital micromirror array column flipping are in phase and start at the same point. Measurements show that the on-chip register propagation delay is 15 nanoseconds, the power amplifier start delay is 120 nanoseconds, and the total delay is 135 nanoseconds. The delay from column flipping to steady-state mirror state in the digital micromirror array is approximately 180 nanoseconds. The difference between these two values ​​is within the 50-microsecond range requirement and requires no additional compensation.

[0143] The synchronization signal adopts the differential LVCMOS standard with an amplitude of 2.5 volts and symmetrical drive matching 100-ohm wiring to ensure that the rising edge jitter is less than 20 picoseconds. In order to adapt to the high-speed continuous typing scenarios that may occur on the keyboard, this invention limits the minimum frame period to 5 milliseconds. The Sync_Pulse repetition period is consistent with the output frame rate of the recurrent neural network. The controller uses programmable logic to lock the high-frequency lower limit to avoid the missed synchronization beats caused by accidental calls in the software layer.

[0144] In scenarios involving long-term operation or changes in ambient temperature, the hardware clock may still experience a drift on the order of one ten-thousandth. To further improve synchronization robustness, the system inserts a 16-bit synchronization sequence number into the high-order address region of each frame, incrementing by one when Sync_Pulse is generated. After completing their respective frame refreshes, the digital micromirror array column flip controller and the ultrasonic phased array drive scheduler transmit the actual completion sequence number back to the main controller via the serial peripheral interface. The main controller uses Δn = n DMD -n UPA Calculate the synchronization sequence number difference, where n DMD n is the sequence number returned by the digital micromirror array. UPA This is the sequence number returned by the ultrasonic phased array. When Δn≠0, it indicates that a frame has been lost or misaligned. The state estimator will immediately reduce the output frame rate and issue a synchronization reset sequence to reset the sequence number counter, thus achieving self-healing.

[0145] To verify the effectiveness of the synchronization signal scheme, the following test was conducted: After 20 minutes of continuous text input, the temperature of the local control board was raised to 60 degrees Celsius using a hot air gun and maintained for 10 minutes. During this period, the time difference between the moment the digital micromirror array column flipped and the moment the ultrasonic focal domain reached its peak was measured using a high-speed optical camera and microphone. Without a synchronization loop, the time difference drifted from 85 microseconds to 410 microseconds with temperature changes; after enabling Sync_Pulse synchronization and using a sequence feedback loop, the time difference remained within 95 microseconds. In user subjective evaluations, 90% of subjects did not notice any feedback asynchrony, demonstrating the effectiveness of the synchronization signal scheme.

[0146] Besides time consistency, synchronization signals can also reduce visual-tactile interference. When using RGB keycap lighting effects, repeated refreshes of the optical image can cause secondary harmonics to be generated by the ultrasonic phased array vibration module. If the two are not phase-locked as a whole, the high-frequency superposition will increase the harmonic amplitude and reduce the focal intensity. This invention fixes the driving beat at Sync_Pulse to achieve phase alignment, resulting in a measured reduction of harmonic power of 27%.

[0147] In terms of hardware resource requirements, the synchronous signal scheme only requires two differential lines (Sync_Pulse and synchronous sequence clock) to cover the optical-acoustic refresh of all keycaps on the keyboard, achieving the simplest wiring; the signals are cascaded on the circuit board in a daisy chain, and each keycap uses a multiplexed network cable interface, making installation and maintenance simple.

[0148] From the perspective of expanding user-interactive information, the frame-level synchronization signal binds visual tags, tactile peaks, and keycap physical actions into the same timeline event: when the user presses a keycap, the digital micromirror array immediately lights up the function prompt icon, and the ultrasonic phased array synchronously generates focused vibrations. Both achieve zero-perceptible frame errors, jointly creating an immersive interactive experience. The synchronization signal also provides a unified reset port, allowing subsequent firmware upgrades to expand 3D holographic projection or multi-frequency tactile encoding without changing the underlying beat mechanism, leaving interface space for further expansion of this invention in enhancing interactive information display.

[0149] Preferably, the main controller constructs a three-dimensional state vector using the optical wireless communication link delay, the digital micromirror array time base offset, and the ultrasonic phased array phase offset. It then calculates the compensation amount through Kalman filtering iteratively and adjusts the optical modulation phase, line scanning timing, and phase reference in real time.

[0150] To ensure long-term synchronization between vision and touch, the main controller assembles three quantitative indicators—"round-trip delay of optical wireless communication link," "time offset of digital micromirror array column flipping completion," and "time offset of ultrasonic phased array focal domain reaching reference microphone"—into a three-dimensional state vector x(k) = [δ]. L (k),δ D(k),δ U (k)] T Where δ L (k) represents the optical link delay relative to the reference time in this frame, δ D (k) represents the time base offset of the digital micromirror array row scan, δ U (k) represents the ultrasonic phased array focusing phase reference offset, in seconds. Due to disturbances such as hardware aging, temperature rise, and optical path obstruction, the three offsets will drift randomly; therefore, the system adopts a discrete linear state-space model.

[0151] x(k)=Fx(k-1)+w(k-1)

[0152] o(k) = Hx(k) + v(k)

[0153] Kalman filtering is performed. Matrix F is taken as an identity matrix, assuming continuous offset between adjacent frames; H is also an identity matrix, indicating that the observation vector o(k) comes directly from hardware measurements. The covariance between process noise w(k-1) and observation noise v(k) is obtained from long-term operational statistics and written during each power-on self-calibration phase.

[0154] The filter first obtains a priori estimates based on the predictions from the previous frame. Then, combine the observations of this frame to update the posterior estimate. The core recursive formula is:

[0155]

[0156]

[0157] K(k) is the gain matrix, P - (k) represents the prior covariance, and R represents the observation noise covariance. The compensation quantities for the three components are calculated as follows:

[0158]

[0159]

[0160]

[0161] in For optical modulation carrier frequency, This is the ultrasonic drive frequency. The main controller will... Write to the laser-driven phase accumulator, Write to the digital micromirror array row scan delay register, The phase reference table for the phased array channel is written and takes effect immediately in the next frame. Actual measurements at a 20 MHz reference clock show that all calculations and register writes take 120 microseconds, far below the 5 millisecond budget per frame.

[0162] Example: As the ambient temperature rises from 25°C to 45°C, the average optical link delay increases by 60 nanoseconds, the digital micromirror array scanning completion delay increases by 180 nanoseconds, and the ultrasonic phase reference drift is 0.006 radians. The Kalman filter converges within 5 frames, and the compensated residuals remain within 8 nanoseconds, 22 nanoseconds, and 0.0009 radians, respectively. The time difference between the image measured by the high-speed camera and the tactile peak is consistently less than 90 microseconds; if compensation is turned off, the difference spikes to 540 microseconds after 30 minutes, and the user can subjectively feel a significant tactile lag.

[0163] The significance of extending interactive information is threefold: First, Kalman filtering provides a prediction-update mechanism, ensuring continuous compensation even when observations are lost due to sudden channel obstruction; second, it couples the three-domain errors into a unified state vector, avoiding new error cycles caused by separate adjustments; third, real-time compensation ensures that visual-tactile alignment does not degrade with device aging during long-term use, providing reliable assurance for high-consistency scenarios such as professional gaming keyboards and keyboards for the blind. Comprehensive power consumption testing shows that the compensation stage increases power consumption by an average of 4 milliwatts, accounting for only 2% of the total keycap budget. After 8 hours of continuous operation, the digital micromirror array showed no excessive hot spots, and the ultrasonic drive exhibited no peak drift imbalance, verifying the feasibility and stability of the adaptive compensation scheme of this invention.

[0164] Preferably, when the bit error rate of the optical wireless communication link continuously exceeds a preset threshold for a preset duration, the main controller stops driving the digital micromirror array and the ultrasonic phased array and switches the keycap display to the pre-etched character layer; when the bit error rate continuously falls below the preset threshold for a preset duration, the main controller restores the previously saved driving state and restarts driving the digital micromirror array and the ultrasonic phased array.

[0165] In a keycap-level optical communication-ultrasonic collaborative interactive system, the optical wireless communication link handles both uplink and downlink transmission of event packets and compressed mappings, and provides power and a reference clock to the keycap. If the link error rate exceeds the tolerance threshold, data frames begin to be frequently retransmitted, which not only consumes bandwidth and amplifies latency, but may also increase the power supply ripple at the keycap, posing a chain reaction risk to the refresh stability of the digital micromirror array and the ultrasonic phased array. This invention continuously monitors the link error rate and triggers a "degradation mode" after abnormal conditions persist for a certain period of time. This mode reverts the content presented in both the visual and tactile domains to the pre-written character layer, ensuring that the system maintains the most basic key prompt function even in extreme communication degradation scenarios. After the link recovers, the optical-acoustic rendering is quickly restarted through state backtracking, ensuring a complete closed loop of the interactive experience.

[0166] Bit error rate (BER) monitoring is based on the receiver's cyclic redundancy check (CRC) results. The main controller maintains two counters: a total frame counter and an error frame counter. Whenever the receiving logic decodes an uplink event packet or downlink echo packet, if the CRC check fails, the error frame counter is incremented. The system uses a sliding time window method to calculate the real-time BER.

[0167]

[0168] Where BER(k) is the bit error rate observed at the end of frame k, E(k) is the cumulative number of erroneous frames, T(k) is the cumulative number of frames, and W is the window length (default 100 frames). If BER(k) exceeds the threshold η1 (e.g., 1×10) for M consecutive frames... -6 If the BER(k) is below the threshold η2 for N consecutive frames (η2 < η1), it is considered a link degradation; conversely, if the BER(k) is below the threshold η2 for N consecutive frames, it is considered a link recovery. The default values ​​are M = 5 and N = 50, used to filter out momentary jitter.

[0169] Upon entering degradation mode, the main controller first sets the static latch signal of the digital micromirror array high, pausing the column flip pulses; then it pulls the ultrasonic phased array power amplifier low, putting the array into standby mode. Next, the controller selects a pre-etched character layer: the keycaps themselves are etched with letter symbols, providing minimum readability through the backlight of the light guide plate, ensuring the user can still identify the keys. To make the switch perceptible to the user, the system writes a "signal interruption" icon to the digital micromirror array for two frames before pausing, lasting approximately 80 milliseconds, before completely extinguishing it.

[0170] During degradation, the controller continues to decode keycap event packets and perform bit error rate statistics, but no longer maps data to the keycap. When BER(k) remains below η2 for N consecutive frames, the system determines that the link has been restored. First, it reads the pixel matrix and phase matrix of the previous stable frame stored in the ferroelectric random access memory and compares them with the current keycap buffer. If the differential pixel ratio is less than 15% and the differential phase channel is less than 10%, it directly writes them to the digital micromirror array and ultrasonic phased array registers; otherwise, it performs a low-resolution fast reconstruction to generate key function icons and the central tactile focal zone, prioritizing the restoration of feedback-sensitive areas. Within 3 frames after restoration, the controller gradually increases the pixel matrix to full resolution and the ultrasonic array phase to full power, and the system returns to normal operation. The entire restoration process is completed within 70 milliseconds, and the user can only perceive a slight flicker.

[0171] Example: Blocking the physical optical path with a keyboard caused the link bit error rate to spike to 5×102 -5 The system enters degradation mode within 6 frames, the digital micromirror array stops refreshing, the ultrasonic phased array shuts down, and the backlight letter layer illuminates. After occlusion for 2 seconds, the occlusion is removed, and the bit error rate drops to 2×10⁻⁶. -7After 50 frames, the system detects that the recovery conditions are met, loads the previously saved state, and restarts rendering. A high-speed camera measured that the recovery from the bit error rate to the optical-acoustic re-presentation took only 68 milliseconds, and user subjective ratings showed that 92% of subjects did not perceive the function loss. Without a degradation mechanism, the same occlusion caused frequent erroneous refreshes of the digital micromirror array, lens descent and flipping, and ultrasonic focal drift exceeding 0.8 mm, resulting in user ratings below 2 / 5.

[0172] From the perspective of expanding user interaction information, the degradation-recovery mechanism has three key benefits: First, it ensures usable key recognition even during extreme link degradation, preventing blind typing errors; second, it uses a "two-threshold-dual-counter" model to prevent jitter and avoid frequent degradation at edge error rates, improving stability; and third, it shortens the recovery path and achieves seamless integration of optical and acoustic feedback by utilizing fast comparison and low-resolution transition. Practice shows that this mechanism increases system power consumption by less than 5 milliwatts, but significantly improves the reliability of commercial and industrial protective keyboards, especially in mobile scenarios with limited network access or high environmental obstruction.

[0173] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects.

[0174] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A user-oriented interactive information extension method, characterized by, The method comprises the following steps: The keycap collects displacement signals and pressure signals, and after preprocessing, the signals are packaged into event packets containing keycap identification, displacement, pressure and time stamp, and then uploaded to the master controller through an optical wireless communication link, and the keycap is powered and the clock is synchronized through the optical wireless communication link; The master controller gathers the event packets in chronological order into an event stream, generates a keycap graph according to the spatial adjacency of the keycaps and the co-trigger relationship within a set time window, maps the event stream into node features and inputs the graph convolutional neural network for inference, and obtains an intention vector through pooling; The intention vector is input into a recurrent neural network to generate a pixel matrix and a phase matrix to form a mapping set, and the mapping set is downlinked to each keycap through sparse joint coding, and the pixel matrix and the phase matrix are decoded at the keycap side and output a differential pixel set, a differential phase set and a synchronization marker through gradient optimization; The master controller drives a digital micromirror array according to the differential pixel set, drives an ultrasonic phased array according to the differential phase set and embeds the synchronization marker; monitors the time delay of the optical wireless communication link and the driving offset, and synchronously corrects the optical modulation phase, the scanning timing and the phase reference according to the state estimation result.

2. The method of claim 1, wherein, The event packet records the keycap identification, displacement, pressure and time stamp in fixed byte order, and performs two-step data cleaning of low-pass filtering and outlier rejection according to a preset threshold before uploading.

3. The method of claim 1, wherein, The keycap graph contains first-type edges and second-type edges, wherein the first-type edges are established according to the mutual adjacent relationship of the physical center points of the keycaps, and the second-type edges are established according to the continuous co-trigger relationship within a set time window, and the weights of the two types of edges are written into a weighted adjacency matrix.

4. The method of claim 1, wherein, The graph convolutional neural network comprises two convolutional layers, and the convolution kernel performs matrix multiplication on the normalized adjacency matrix, and the convolution output is input into the next convolutional layer after linear rectification and activation.

5. The method of claim 1, wherein, The pooling operation performs global average operation on all node convolution outputs, and the obtained average vector is the intention vector.

6. The method of claim 1, wherein, Bernoulli measurement matrix is used for random subsampling compression when sparse joint coding is performed on the mapping set, and the pixel matrix and the phase matrix are restored at the keycap side through the orthogonal matching pursuit algorithm.

7. The method of claim 1, wherein, The pixel matrix and the phase matrix after decoding are subjected to four fixed-point gradient descents at the keycap side, the pixel value and the phase value are adjusted simultaneously in each update to reduce the reconstruction error, and a differential pixel set, a differential phase set and a synchronization marker are output.

8. The method of claim 1, wherein, The refresh timing of the digital micromirror array and the ultrasonic phased array is locked to the reference clock provided by the master controller through a frame synchronization signal.

9. The method of claim 1, wherein, The master controller constructs a three-dimensional state vector using the time delay of the optical wireless communication link, the time base offset of the digital micromirror array and the phase offset of the ultrasonic phased array, iteratively calculates the compensation amount through Kalman filtering, and adjusts the optical modulation phase, the line scanning timing and the phase reference in real time.

10. The method of claim 1, wherein, When the bit error rate of the optical wireless communication link continuously exceeds the preset threshold for a preset duration, the master controller stops driving the digital micromirror array and the ultrasonic phased array and switches the keycap display to the pre-engraved character layer; when the bit error rate continuously falls below the preset threshold for a preset duration, the master controller restores the previous saved driving state and re-drives the digital micromirror array and the ultrasonic phased array.

Images