A multi-modal dynamic perception-based smart watch gesture prediction system

Abstract

The application discloses a kind of based on multi-modal dynamic perception's smart watch gesture pre-judgment system, belong to smart watch technical field, including following module: adaptive sampling module, for monitoring inertia data and detecting gesture starting event with basic sampling rate;Predictive model, for the encoder based on time series self-attention, double-flow prediction head structure;Perception module, for extracting short-term dynamics, medium-term activity and long-term user context features;Reinforcement learning module, for the reward difference between actual execution result and generated counterfactual alternative trajectory of user;Adaptive module, for storing feedback log locally and performing online incremental learning under privacy protection;The application is by basic sampling rate and gesture starting triggered full-sampling rate dynamic switching strategy, so that the system is in no gesture interaction standby power consumption is only 0.9 mW;And in gesture execution process, part sequence prediction model can output result in advance in the stage of 30%-70% of gesture completion.

Description

Technical Field

[0001] This invention relates to the field of smartwatch technology, and more specifically, to a smartwatch gesture prediction system based on multimodal dynamic perception. Background Technology

[0002] As a personal interactive device, smartwatches have seen gesture control become one of the mainstream interaction paradigms due to its contactless and fast-response characteristics. Existing technologies mainly revolve around three types of solutions: The first type is based on inertial measurement units (IMUs), which collect wrist movement trajectories through accelerometers and gyroscopes, and use thresholding methods or traditional machine learning to recognize coarse-grained gestures such as raising the wrist, flipping, and tapping. A typical example is the Apple Watch's "pinch" and "flip" functions. The second type is based on electromyography (EMG), which collects forearm muscle electrical activity through bioelectrodes on the bottom of the watch, enabling the recognition of more refined finger movements, such as those described in Google Project. Soli's radar micro-motion solution and CTRL-labs' wristband-type EMG; the third type uses motion artifacts from optical heart rate sensors to assist in distinguishing gestures, but the accuracy is limited; in recent years, lightweight deep learning at the edge has begun to be applied to this field, such as temporal convolutional networks or MobileNetV2 to classify multi-sensor sequences, and the recognition capacity can be expanded to 10-20 gestures; however, existing technologies still have several fundamental defects: first, the contradiction between latency and power consumption is prominent. High-precision models require continuous high-frequency sampling, resulting in watch battery life of less than 1 day. After reducing the sampling rate, the gesture recognition latency exceeds 300ms, and the interaction feels sluggish; second, the false trigger rate is high. Everyday activities such as dressing, clapping, and clenching fists are frequently misjudged as control gestures, and users often need an extra "confirmation" step to correct them, which seriously weakens the convenience of gesture interaction; third, there is a lack of behavior prediction ability. All existing solutions can only recognize completed gestures and cannot predict the user's intention during the execution of the action, resulting in stiff and unnatural interaction.

[0003] Therefore, we have made improvements to this by proposing a smartwatch gesture prediction system based on multimodal dynamic perception. Summary of the Invention

[0004] In view of the above-mentioned problems in the existing technology, the purpose of this invention is to provide a smartwatch gesture prediction system based on multimodal dynamic perception.

[0005] To solve the above problems, the technical solution adopted by the present invention is as follows: it includes the following modules: The adaptive sampling module is used to monitor inertial data and detect gesture initiation events at the basic sampling rate. After the initiation is detected, it increases to the full sampling rate and simultaneously acquires inertial, electromyographic, photoplethysmography and vibration reflection signals. The prediction model is a temporal self-attention-based encoder and a two-stream prediction head structure, which maps the currently acquired partial gesture sequence to the complete gesture category probability distribution, the number of remaining frames, and the prediction confidence. The perception module is used to extract short-term dynamics, medium-term activities and long-term user context features, and dynamically adjust the decision threshold through a learnable threshold function. The reinforcement learning module is used to update the predictive model parameters unsupervised based on the reward difference between the user's actual performance and the generated counterfactual alternative trajectory. An adaptive module is used to store feedback logs locally and perform online incremental learning while protecting privacy.

[0006] Preferably, the adaptive sampling module specifically includes: an inertial measurement unit, electromyographic electrodes, a photoplethysmography sensor, and a vibration reflection sensing submodule composed of a vibration motor and an accelerometer; The base sampling rate is set to Only triaxial acceleration is collected. With triaxial angular velocity Gesture initiation detection network calculates motion energy ,in Represents the squared magnitude of the acceleration vector. Represents the squared magnitude of the angular velocity vector; when Three consecutive frames exceeding the preset energy threshold At that time, full sampling mode is triggered, and the sampling rate increases to Simultaneously, electromyography, photoplethysmography, and vibration reflex signal acquisition are activated. All sensor signals are aligned through a differentiable time offset calibration layer, which learns the delay parameters of each sensor channel to minimize the cross-correlation cost function and outputs a unified timestamp frame sequence.

[0007] Preferably, the prediction model includes: The spatial coding layer performs temporally and depthwise separable convolutions on each frame of input and outputs frame-level feature vectors. The temporal context encoder is composed of two stacked Transformer encoders, each containing a multi-head self-attention and feedforward network, and outputs a sequence of hidden states. Two-stream prediction head: The first stream is a multilayer perceptron, which outputs the probability distribution of gesture categories. ,in For the current frame index, The first stream represents the gesture category number; the second stream is a gated loop unit, which takes the current hidden state and position code as input and outputs the predicted number of remaining frames. During the training phase, a partial sequence generation strategy is employed, randomly extracting the first few segments from the complete gesture sequence. The ratio is used as input. The value ranges from 0.3 to 0.7; the loss function is: ; in Cross-entropy classification loss, One-hot encoding for complete gesture labels; The regression loss is for the remaining frames. This represents the actual number of remaining frames. To predict the number of remaining frames; Predict consistency loss for adjacent frames; , This is the balance coefficient.

[0008] Preferably, the sensing module includes: The short-term dynamic context extraction unit calculates the acceleration variance, peak angular velocity, and heart rate change slope over the past 2 seconds, and inputs them into a single-layer long short-term memory network to obtain a 32-dimensional vector. The mid-term activity context extraction unit maps the activity categories (stationary, walking, running, cycling, sleeping, driving) output by the upstream activity recognition module into a 16-dimensional vector through a learnable embedding matrix; The long-term user context extraction unit stores the historical gestures, scene co-occurrence matrix, and daily fatigue index calculated based on heart rate variability corresponding to anonymous user IDs, and obtains 16-dimensional features through sliding window statistics. The dynamic decision threshold generation unit linearly combines the three types of context vectors mentioned above using a learnable weight matrix, maps them to the 0-1 range using the sigmoid function, and then linearly transforms them to the 0.3-0.8 range to obtain the adaptive threshold. ; Arbitration rule unit, if the current maximum gesture probability Greater than or equal to If the gesture is not in the prohibited list of the mutual exclusion rule base, then the gesture is executed; otherwise, wait for more frames or refuse.

[0009] Preferably, it also includes a vibration-tactile reverse sensing module, which reuses a linear resonant motor and an accelerometer to actively transmit coded vibrations imperceptibly to the user and receive reflected signals to achieve muscle motion imaging. Specifically, it includes: The motor drive signal generation unit uses orthogonal frequency division multiplexing short-time pulses. The pulse waveform is a sine wave modulated by a Hamming window, and the amplitude is set to a level that is imperceptible to the user. The reflected signal receiving and processing unit uses an accelerometer to sample the reflected signal at 200Hz. It extracts the amplitude attenuation and phase delay of each frequency component through synchronous demodulation, and calculates the arrival time difference and multipath spectral entropy to form a vibration reflection feature vector. The fusion unit inputs the feature vector as an independent modality in parallel with inertial and electromyographic signals into the prediction model; when the electromyographic signal-to-noise ratio is less than 10dB, the model automatically increases the attention weight for vibration reflection features.

[0010] Preferably, it also includes an environmental radio frequency gesture projection module, which reuses a Bluetooth or Wi-Fi receiver to passively acquire channel state information of environmental radio frequency signals and extract electromagnetic disturbance characteristics caused by gestures, specifically including: The monitoring sampling unit activates the receiver's monitoring mode during communication idle time slots to perform sub-Nyquist sampling on broadcast signals in the 2.4GHz and 5GHz frequency bands at a sampling rate of 200Hz. The channel state information processing unit extracts the channel state information matrix from each data packet, calculates the difference sequence, and further extracts the angle of arrival change sequence and amplitude fluctuation spectrum to form a radio frequency feature vector. The gated fusion unit calculates the radio frequency feature by the edge layer only when the confidence level of the prediction model output is in the fuzzy range of 0.45 to 0.65, and determines the fusion weights through the expert hybrid gated network to output the corrected confidence level.

[0011] Preferably, the reinforcement learning module includes: The counterfact generator is a variational autoencoder structure. Its encoder compresses the real part of the sequence and the context into latent variables, and the decoder generates a counterfact sensor sequence. The reward calculation unit defines the actual reward. and counterfactual rewards : Calculated based on subsequent user behavior; The virtual execution result is obtained by inputting the counterfactual sequence into the prediction model, and then estimated by the user's historical preference model; The gradient loss function for the counterfactual policy is: ; in As a latent variable, For encoder distribution, For decoder distribution, For counterfactual sequence, This is a contextual feature; the loss is used to update the parameters of the prediction model.

[0012] Preferably, the adaptive module includes: The feedback log storage unit records the sequence, context, predicted label, user feedback type, and time stamp for each interaction event; user feedback is categorized into strong positive feedback, weak positive feedback, strong negative feedback, and weak negative feedback. The online incremental learning unit performs stochastic gradient descent once every 10 feedbacks to update the bias and context threshold parameters of the last two layers of the prediction model, with a learning rate of 0.01 and momentum of 0.9.

[0013] Preferably, it also includes a privacy-preserving federated aggregation unit, which calculates a Gaussian difference privacy perturbation version of the gradient every day at midnight in a Wi-Fi environment, with the perturbation strength determined by the privacy budget. and gradient clipping threshold The control mechanism uploads the perturbation gradient to the cloud server, and the cloud aggregates the global model every two weeks and then distributes it.

[0014] Preferably, it also includes an energy efficiency management unit for implementing dynamic sampling and inference scheduling, specifically including: The gesture initiation detection network uses a binary neural network with a computational cost of less than 0.5 MFLOPs and is triggered only once every 0.5 seconds at the base sampling rate. In full sampling mode, the prediction model uses the CMSIS-NN acceleration library for inference on the watch microcontroller, with a single inference time of 8ms and power consumption of 4.2mW. If there is no gesture interaction for more than 30 seconds, the system will automatically downgrade to the basic sampling rate and turn off electromyography, vibration reflex and radio frequency sampling.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. By adopting a dynamic switching strategy between the base sampling rate and the full sampling rate triggered by the gesture, the system's standby power consumption is only 0.9 mW when there is no gesture interaction. During the gesture execution process, some sequence prediction models can output results in advance when the gesture is 30%-70% complete, reducing the average recognition latency from 320 ms in the traditional solution to less than 48 ms, thus solving the contradiction between high accuracy and long battery life.

[0016] 2. By using short-term dynamic context, medium-term activity context, and long-term user context, an adaptive threshold is generated through a learnable threshold function and combined with a mutual exclusion rule base, reducing the false trigger rate in daily activities from 12.5% ​​to 1.9%. Users no longer need to frequently correct misjudged operations, and the interactive experience is significantly improved.

[0017] 3. By adopting an encoder and a dual-stream prediction head structure, and utilizing a partial sequence generation strategy and a triplet loss function during the training phase, the model can accurately predict the final intention midway through gesture execution and trigger the target application or command in advance. The smoothness of the interaction is close to that of physical buttons, fundamentally eliminating the lag of "waiting for the gesture to complete".

[0018] 4. Through the counterfactual intent reinforcement learning module, a counterfactual sequence of "potentially executed but not executed" is generated using a variational autoencoder. The difference between the actual reward and the counterfactual reward is used as the gradient signal to unsupervisedly mine users' implicit preferences from each interaction, shortening the personalized adaptation time for new users from 7 days to 2 days. At the same time, the adaptive module uses Gaussian difference privacy perturbation to upload gradients, realizing cloud-based global model and on-device online fine-tuning while protecting privacy. This effectively overcomes the accuracy decline caused by differences in wearing position, wrist size, and exercise habits.

[0019] 5. By utilizing existing vibration motors and accelerometers to achieve vibration-tactile reverse sensing, and Bluetooth / Wi-Fi receivers to achieve environmental radio frequency gesture projection, the system can automatically switch to vibration reflection features when the EMG signal is affected by sweat or looseness. Radio frequency features can be used for arbitration in the ambiguity range of the prediction model confidence, realizing the ability to predict gestures through fabric occlusion and airborne obstruction, significantly improving the robustness of gesture recognition in complex environments. Attached Figure Description

[0020] Figure 1 This application provides an architecture block diagram of a smartwatch gesture prediction system based on multimodal dynamic perception. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] A smartwatch gesture prediction system based on multimodal dynamic perception includes the following modules: The adaptive sampling module is used to monitor inertial data and detect gesture initiation events at the basic sampling rate. After the initiation is detected, it increases to the full sampling rate and simultaneously acquires inertial, electromyographic, photoplethysmography and vibration reflection signals. The prediction model is a temporal self-attention-based encoder and a two-stream prediction head structure, which maps the currently acquired partial gesture sequence to the complete gesture category probability distribution, the number of remaining frames, and the prediction confidence. The perception module is used to extract short-term dynamics, medium-term activities and long-term user context features, and dynamically adjust the decision threshold through a learnable threshold function. The reinforcement learning module is used to update the predictive model parameters unsupervised based on the reward difference between the user's actual performance and the generated counterfactual alternative trajectory. An adaptive module is used to store feedback logs locally and perform online incremental learning while protecting privacy.

[0023] Furthermore, the adaptive sampling module specifically includes: an inertial measurement unit, electromyographic electrodes, a photoplethysmography sensor, and a vibration reflection sensing submodule consisting of a vibration motor and an accelerometer; The base sampling rate is set to Only triaxial acceleration is collected. With triaxial angular velocity Gesture initiation detection network calculates motion energy ,in Represents the squared magnitude of the acceleration vector. Represents the squared magnitude of the angular velocity vector; when Three consecutive frames exceeding the preset energy threshold At that time, full sampling mode is triggered, and the sampling rate increases to Simultaneously, electromyography, photoplethysmography, and vibration reflex signal acquisition are activated. Energy threshold The value range is 0.3 to 0.8 m 2 / s 4 +0.1~0.3rad 2 / s 2 ; All sensor signals are aligned through a differentiable time offset calibration layer, which learns the delay parameters of each sensor channel to minimize the cross-correlation cost function and outputs a unified timestamp frame sequence.

[0024] Furthermore, the predictive models include: The spatial coding layer performs temporally and depthwise separable convolutions on each frame of input and outputs frame-level feature vectors. The temporal context encoder is composed of two stacked Transformer encoders, each containing a multi-head self-attention and feedforward network, and outputs a sequence of hidden states. Two-stream prediction head: The first stream is a multilayer perceptron, which outputs the probability distribution of gesture categories. ,in For the current frame index, The first stream represents the gesture category number; the second stream is a gated loop unit, which takes the current hidden state and position code as input and outputs the predicted number of remaining frames. During the training phase, a partial sequence generation strategy is employed, randomly extracting the first few segments from the complete gesture sequence. The ratio is used as input. The value ranges from 0.3 to 0.7; the loss function is: ; in Cross-entropy classification loss, One-hot encoding for complete gesture labels; The regression loss is for the remaining frames. This represents the actual number of remaining frames. To predict the number of remaining frames; Predict consistency loss for adjacent frames; , This is the balance coefficient; During the inference phase, when the maximum class probability exceeds 0.7 and the predicted class is consistent for three consecutive frames, the gesture is output in advance; The number of heads for multi-head self-attention is set to 4, and the feedforward network dimension is set to 256; the balance coefficient is... =0.5, =0.2.

[0025] Furthermore, the perception module includes: The short-term dynamic context extraction unit calculates the acceleration variance, peak angular velocity, and heart rate change slope over the past 2 seconds, and inputs them into a single-layer long short-term memory network to obtain a 32-dimensional vector. The mid-term activity context extraction unit maps the activity categories (stationary, walking, running, cycling, sleeping, driving) output by the upstream activity recognition module into a 16-dimensional vector through a learnable embedding matrix; The long-term user context extraction unit stores the historical gestures, scene co-occurrence matrix, and daily fatigue index calculated based on heart rate variability corresponding to anonymous user IDs, and obtains 16-dimensional features through sliding window statistics. The dynamic decision threshold generation unit linearly combines the three types of context vectors mentioned above using a learnable weight matrix, maps them to the 0-1 range using the sigmoid function, and then linearly transforms them to the 0.3-0.8 range to obtain the adaptive threshold. ; Arbitration rule unit, if the current maximum gesture probability Greater than or equal to If the gesture is not in the prohibited list of the mutual exclusion rule base, then the gesture is executed; otherwise, wait for more frames or refuse. The mutual exclusion rule base is stored locally on the watch in a configurable JSON format, such as "disable most gestures in sleep mode" and "only allow emergency or answer gestures in driving mode".

[0026] Furthermore, it also includes a vibration-tactile feedback sensing module. This module reuses a linear resonant motor and an accelerometer to actively emit coded vibrations that are imperceptible to the user and receive reflected signals to achieve muscle motion imaging. Specifically, it includes: The motor drive signal generation unit uses orthogonal frequency division multiplexing short pulses, each pulse lasting 5ms, with a pulse interval of 50ms, and the center frequency selected between 50Hz and 200Hz. The pulse waveform is a sine wave modulated by a Hamming window, and the amplitude is set to a level that is imperceptible to the user. The reflected signal receiving and processing unit uses an accelerometer to sample the reflected signal at 200Hz. It extracts the amplitude attenuation and phase delay of each frequency component through synchronous demodulation, and calculates the arrival time difference and multipath spectral entropy to form a vibration reflection feature vector. The fusion unit inputs the feature vector as an independent modality in parallel with inertial and electromyographic signals into the prediction model; when the electromyographic signal-to-noise ratio is below 10dB, the model automatically increases the attention weight for vibration reflex features. The amplitude is set to be less than 0.01g when the accelerometer receives the signal; the formula for calculating the multipath spectral entropy is... ,in This is the normalized value of the power delay profile.

[0027] Furthermore, it also includes an environmental radio frequency gesture projection module, which reuses a Bluetooth or Wi-Fi receiver to passively acquire channel state information of environmental radio frequency signals and extract electromagnetic disturbance characteristics caused by gestures, specifically including: The monitoring sampling unit activates the receiver's monitoring mode during communication idle time slots to perform sub-Nyquist sampling on broadcast signals in the 2.4GHz and 5GHz frequency bands at a sampling rate of 200Hz. The channel state information processing unit extracts the channel state information matrix from each data packet, calculates the difference sequence, and further extracts the angle of arrival change sequence and amplitude fluctuation spectrum to form a radio frequency feature vector. The gated fusion unit calculates the radio frequency feature by the edge layer only when the confidence level of the prediction model output is in the fuzzy range of 0.45 to 0.65, and determines the fusion weights through the expert hybrid gated network to output the corrected confidence level. The sub-Nyquist sampling rate can be reduced to 100 Hz to save power; the angle-of-arrival variation sequence is obtained by calculating the phase difference of multiple antennas.

[0028] Furthermore, the reinforcement learning module includes: The counterfact generator is a variational autoencoder structure. Its encoder compresses the real part of the sequence and the context into latent variables, and the decoder generates a counterfact sensor sequence that represents the user's "alternative gestures that could have been performed but were not". The reward calculation unit defines the actual reward. and counterfactual rewards : Calculated based on subsequent user actions (whether to withdraw, whether to continue using the result, whether to correct); The virtual execution result is obtained by inputting the counterfactual sequence into the prediction model, and then estimated by the user's historical preference model; The gradient loss function for the counterfactual policy is: ; in As a latent variable, For encoder distribution, For decoder distribution, For counterfactual sequence, The contextual features are used to update the parameters of the prediction model, maximizing the difference between real and counterfactual rewards, so that real gestures gradually approach the user's implicit preferences. This module runs at the edge layer (phone or gateway), triggers an offline update every 10 interactions, generates two counterfactual trajectories, and sends parameter increments to the watch via a security protocol.

[0029] Furthermore, the adaptive module includes: The feedback log storage unit records the sequence, context, predicted label, user feedback type, and time stamp for each interaction event; user feedback is categorized into strong positive feedback, weak positive feedback, strong negative feedback, and weak negative feedback. The online incremental learning unit performs stochastic gradient descent once every 10 feedbacks to update the bias and context threshold parameters of the last two layers of the prediction model. The learning rate is 0.01, the momentum is 0.9, and the loss function is weighted cross-entropy, with the weights determined by the feedback confidence. Each update takes less than 15ms and consumes less than 1mW.

[0030] Furthermore, it also includes a privacy-preserving federal aggregation unit that calculates a Gaussian difference privacy perturbation version of the gradient every morning in a Wi-Fi environment, with the perturbation strength determined by the privacy budget. and gradient clipping threshold The control mechanism uploads the perturbation gradient to the cloud server, and the cloud aggregates the global model every two weeks and distributes it. The gradient perturbation formula is ,in , This represents a Gaussian distribution.

[0031] Furthermore, it also includes an energy efficiency management unit for dynamic sampling and inference scheduling, specifically including: The gesture initiation detection network uses a binary neural network with a computational cost of less than 0.5 MFLOPs and is triggered only once every 0.5 seconds at the base sampling rate. In full sampling mode, the prediction model uses CMSIS and NN acceleration libraries for inference on the watch microcontroller, with a single inference time of 8ms and power consumption of 4.2mW. When there is no gesture interaction for more than 30 seconds, the system automatically degrades to the basic sampling rate and turns off electromyography, vibration reflex and radio frequency sampling, with a total standby power consumption of 0.9mW; The weights of a binary neural network can only be +1 or -1, and the multiplication-addition operation is simplified to addition-subtraction operation.

[0032] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of protection of this application is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of one or more embodiments of this application as described above, which are not provided in detail for the sake of brevity.

[0033] One or more embodiments in this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments in this application should be included within the protection scope of this application.

Claims

1. A smartwatch gesture prediction system based on multimodal dynamic perception, characterized in that, Includes the following modules: The adaptive sampling module is used to monitor inertial data and detect gesture initiation events at the basic sampling rate. After the initiation is detected, it increases to the full sampling rate and simultaneously acquires inertial, electromyographic, photoplethysmography and vibration reflection signals. The prediction model is a temporal self-attention-based encoder and a two-stream prediction head structure, which maps the currently acquired partial gesture sequence to the complete gesture category probability distribution, the number of remaining frames, and the prediction confidence. The perception module is used to extract short-term dynamics, medium-term activities and long-term user context features, and dynamically adjust the decision threshold through a learnable threshold function. The reinforcement learning module is used to update the predictive model parameters unsupervised based on the reward difference between the user's actual performance and the generated counterfactual alternative trajectory. An adaptive module is used to store feedback logs locally and perform online incremental learning with privacy protection; The prediction model includes: The spatial coding layer performs temporally and depthwise separable convolutions on each frame of input and outputs frame-level feature vectors. The temporal context encoder is composed of two stacked Transformer encoders, each containing a multi-head self-attention and feedforward network, and outputs a sequence of hidden states. Two-stream prediction head: The first stream is a multilayer perceptron, which outputs the probability distribution of gesture categories. ,in For the current frame index, The first stream represents the gesture category number; the second stream is a gated loop unit, which takes the current hidden state and position code as input and outputs the predicted number of remaining frames. During the training phase, a partial sequence generation strategy is employed, randomly extracting the first few segments from the complete gesture sequence. The ratio is used as input. The value ranges from 0.3 to 0.7; the loss function is: ； in Cross-entropy classification loss, One-hot encoding for complete gesture labels; The regression loss is for the remaining frames. This represents the actual number of remaining frames. To predict the number of remaining frames; Predict consistency loss for adjacent frames; , This is the balance coefficient.

2. The smartwatch gesture prediction system based on multimodal dynamic perception according to claim 1, characterized in that, The adaptive sampling module specifically includes: an inertial measurement unit, electromyographic electrodes, a photoplethysmography sensor, and a vibration reflection sensing submodule composed of a vibration motor and an accelerometer. The base sampling rate is set to Only triaxial acceleration is collected. With triaxial angular velocity Gesture initiation detection network calculates motion energy ,in Represents the squared magnitude of the acceleration vector. Represents the squared magnitude of the angular velocity vector; when Three consecutive frames exceeding the preset energy threshold At that time, full sampling mode is triggered, and the sampling rate increases to Simultaneously, electromyography, photoplethysmography, and vibration reflex signal acquisition are activated. All sensor signals are aligned through a differentiable time offset calibration layer, which learns the delay parameters of each sensor channel to minimize the cross-correlation cost function and outputs a unified timestamp frame sequence.

3. The smartwatch gesture prediction system based on multimodal dynamic perception according to claim 1, characterized in that, The sensing module includes: The short-term dynamic context extraction unit calculates the acceleration variance, peak angular velocity, and heart rate change slope over the past 2 seconds, and inputs them into a single-layer long short-term memory network to obtain a 32-dimensional vector. The mid-term activity context extraction unit maps the activity categories output by the upstream activity identification module into 16-dimensional vectors through a learnable embedding matrix. The long-term user context extraction unit stores the historical gestures, scene co-occurrence matrix, and daily fatigue index calculated based on heart rate variability corresponding to anonymous user IDs, and obtains 16-dimensional features through sliding window statistics. The dynamic decision threshold generation unit linearly combines the three types of context vectors mentioned above using a learnable weight matrix, maps them to the 0-1 range using the sigmoid function, and then linearly transforms them to the 0.3-0.8 range to obtain the adaptive threshold. ; Arbitration rule unit, if the current maximum gesture probability Greater than or equal to If the gesture is not in the prohibited list of the mutual exclusion rule base, then the gesture is executed; otherwise, wait for more frames or refuse.

4. The smartwatch gesture prediction system based on multimodal dynamic perception according to claim 1, characterized in that, It also includes a vibration-tactile feedback sensing module, which reuses a linear resonant motor and accelerometer to actively emit coded vibrations imperceptible to the user and receive reflected signals to achieve muscle motion imaging. Specifically, it includes: The motor drive signal generation unit uses orthogonal frequency division multiplexing short-time pulses. The pulse waveform is a sine wave modulated by a Hamming window, and the amplitude is set to a level that is imperceptible to the user. The reflected signal receiving and processing unit uses an accelerometer to sample the reflected signal at 200Hz. It extracts the amplitude attenuation and phase delay of each frequency component through synchronous demodulation, and calculates the arrival time difference and multipath spectral entropy to form a vibration reflection feature vector. The fusion unit inputs the feature vector as an independent modality along with inertial and electromyographic signals into the prediction model; when the electromyographic signal-to-noise ratio is below 10dB, the model automatically increases the attention weight for vibration reflection features.

5. The smartwatch gesture prediction system based on multimodal dynamic perception according to claim 1, characterized in that, It also includes an environmental radio frequency gesture projection module, which reuses a Bluetooth or Wi-Fi receiver to passively acquire channel state information of environmental radio frequency signals and extract electromagnetic disturbance characteristics caused by gestures, specifically including: The monitoring sampling unit activates the receiver's monitoring mode during communication idle time slots to perform sub-Nyquist sampling on broadcast signals in the 2.4GHz and 5GHz frequency bands at a sampling rate of 200Hz. The channel state information processing unit extracts the channel state information matrix from each data packet, calculates the difference sequence, and further extracts the angle of arrival change sequence and amplitude fluctuation spectrum to form a radio frequency feature vector. The gated fusion unit calculates the radio frequency feature by the edge layer only when the confidence level of the prediction model output is in the fuzzy range of 0.45 to 0.65, and determines the fusion weights through the expert hybrid gated network to output the corrected confidence level.

6. The smartwatch gesture prediction system based on multimodal dynamic perception according to claim 1, characterized in that, The reinforcement learning module includes: The counterfact generator is a variational autoencoder structure. Its encoder compresses the real part of the sequence and the context into latent variables, and the decoder generates a counterfact sensor sequence. The reward calculation unit defines the actual reward. and counterfactual rewards : Calculated based on subsequent user behavior; The virtual execution result is obtained by inputting the counterfactual sequence into the prediction model, and then estimated by the user's historical preference model; The gradient loss function for the counterfactual policy is: ； in As a latent variable, For encoder distribution, For decoder distribution, For counterfactual sequence, These are contextual features; this loss is used to update the parameters of the prediction model.

7. The smartwatch gesture prediction system based on multimodal dynamic perception according to claim 1, characterized in that, The adaptive module includes: The feedback log storage unit records the sequence, context, predicted label, user feedback type, and time stamp for each interaction event; user feedback is categorized into strong positive feedback, weak positive feedback, strong negative feedback, and weak negative feedback. The online incremental learning unit performs stochastic gradient descent once every 10 feedbacks to update the bias and context threshold parameters of the last two layers of the prediction model, with a learning rate of 0.01 and a momentum of 0.

9.

8. A smartwatch gesture prediction system based on multimodal dynamic perception according to claim 7, characterized in that, It also includes a privacy-preserving federated aggregation unit, which calculates a Gaussian difference privacy perturbation version of the gradient every day at midnight in a Wi-Fi environment, with the perturbation strength determined by the privacy budget. and gradient clipping threshold The control mechanism uploads the perturbation gradient to the cloud server, and the cloud aggregates the global model every two weeks and then distributes it.

9. A smartwatch gesture prediction system based on multimodal dynamic perception according to claim 1, characterized in that, It also includes an energy efficiency management unit for dynamic sampling and inference scheduling, specifically including: The gesture initiation detection network uses a binary neural network with a computational cost of less than 0.5 MFLOPs and is triggered only once every 0.5 seconds at the base sampling rate. In full sampling mode, the prediction model uses the CMSIS-NN acceleration library for inference on the watch microcontroller, with a single inference time of 8ms and power consumption of 4.2mW. If there is no gesture interaction for more than 30 seconds, the system will automatically downgrade to the basic sampling rate and turn off electromyography, vibration reflex and radio frequency sampling.

Images