Artificial intelligence-based multi-modal somatosensory interaction and context fusion method and system

Abstract

This invention relates to the field of artificial intelligence technology, and discloses a method and system for multimodal haptic interaction and context fusion based on artificial intelligence. The method includes acquiring multi-source sensory signals from the user and performing on-device preprocessing; extracting cross-modal features and generating a user interaction intent representation vector through a lightweight multimodal fusion model; simultaneously acquiring environmental context and physiological emotion parameters to construct a context-emotion joint context vector; coupling these two to generate context-enhanced interaction commands, driving AI glasses to execute adaptive responses; and triggering multi-level interaction mode switching when recognition is unstable. This application can improve the accuracy of interaction in complex dynamic environments, reduce the false trigger rate, and achieve a personalized and robust intelligent interaction experience.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a method and system for multimodal haptic interaction and context fusion based on artificial intelligence. Background Technology

[0002] With the deep integration of artificial intelligence and wearable device technology, motion-sensing interaction systems based on multimodal perception are becoming a core development direction for human-computer interaction terminals such as smart glasses and AR / VR devices. These systems aim to achieve more natural, accurate, and low-latency understanding of user intent and contextual responses by integrating multiple input modalities such as vision, voice, gestures, eye movements, and even physiological signals. However, existing technologies still have significant shortcomings in terms of multimodal fusion depth, real-time performance on the device side, robustness to complex environments, and context-aware collaboration, making it difficult to support a highly available intelligent interactive experience across all weather conditions and scenarios.

[0003] Currently, some attempts have been made to improve interaction performance by introducing multi-sensor fusion, but there are still several limitations: First, the perceptual modalities are relatively isolated, and visual, speech, eye-tracking, and haptic signals have not been deeply fused at the feature layer or decision layer, which limits the ability to understand context; second, most interaction algorithms are deployed in the cloud, and insufficient computing power on the edge leads to high response latency, making it difficult to meet real-time requirements; finally, existing systems rarely adapt to user emotional state and environmental context, and the interaction feedback tends to be mechanical, lacking naturalness and emotional resonance.

[0004] Therefore, how to build a multimodal haptic interaction method and system that supports efficient operation on the edge, has cross-modal dynamic fusion capabilities, and can adaptively respond to emotions and environmental contexts has become an urgent technical problem to be solved in order to improve the intelligence level and practicality of AI glasses. Summary of the Invention

[0005] This invention provides a multimodal somatosensory interaction and context fusion method based on artificial intelligence, which solves the technical problems in the prior art, such as insufficient fusion depth due to the isolation of multimodal perception signals, high latency of end-side response due to the reliance of interaction algorithms on the cloud, and mechanical and unnatural interactive feedback due to the lack of collaborative perception of emotional state and environmental context.

[0006] This invention provides a method and system for multimodal haptic interaction and context fusion based on artificial intelligence, including: The first aspect is an AI-based multimodal haptic interaction and context fusion method, including: Collect multi-source sensory signals from users while wearing AI glasses to form a raw sensory signal set; The original sensory signals include one or more of the following: visual image signals, speech audio signals, eye movement trajectory signals, head posture signals, and hand motion sensing signals; The original sensing signal set is preprocessed at the edge to obtain a structured sensing data stream. Based on the perception data stream, cross-modal feature extraction is performed on the edge device to generate visual semantic features, speech intent features, gaze focus features, posture context features, and gesture action features to obtain a feature set; The feature set is input into a multimodal fusion model deployed on the edge. Through a dynamic attention weight allocation mechanism, the features of each modality are weighted and fused to generate a unified user interaction intent representation vector. Simultaneously collect environmental context parameters and user physiological and emotional parameters; Based on the environmental context parameters and the user's physiological emotional parameters, a context-emotion joint context vector is constructed; The user interaction intent representation vector is coupled with the contextual emotion joint context vector to generate context-enhanced interaction instructions; Based on the context-enhanced interaction instructions, the AI ​​glasses are driven to perform interactive response operations, and the sensitivity threshold and feedback mode of the interaction strategy are dynamically adjusted. An interaction stability assessment index is set. When the confidence level of interaction intent recognition is lower than the preset confidence threshold for multiple consecutive time periods, or when the contextual emotion joint context vector undergoes a sudden change, an adaptive switching mechanism for the interaction mode is triggered, switching to a backup interaction channel.

[0007] Furthermore, the original sensing signal set includes a baseline sensing signal set collected during the initial wearing stage, and several time-series sensing signal subsets collected during different usage periods; It also includes clustering analysis of user interaction behavior based on the perceived data stream to identify multiple typical interaction pattern categories; The typical interaction mode categories are quantitatively evaluated, an interaction intent clarity scoring system is established, and the contribution score of each modal signal to intent clarity is calculated. Based on the contribution score, a modal importance threshold is set, and key modal signals with contribution scores higher than the threshold are selected as the dominant interaction modalities. Based on the dominant interaction mode, a modality reliability index is constructed; By combining the modal reliability index and the interaction intent clarity score, the current optimal fusion strategy is determined.

[0008] Furthermore, based on the aforementioned benchmark sensing signal set, a user-personalized interaction baseline model is established; Based on several time-series sensing signal subsets, the interaction feature sequences of different time periods are obtained and compared with the personalized interaction baseline model to form the interaction behavior drift trajectory. A lightweight graph neural network is used to model the association between cross-modal features, where nodes represent each modal feature and edge weights represent the degree of semantic consistency between modalities. The fusion weights of each modality feature are dynamically adjusted through an edge-trainable gating mechanism, and the control signal of the gating mechanism comes from the contextual emotion joint context vector. Based on a database of successful historical interaction cases, the parameters of the multimodal fusion model are updated through online incremental learning to adapt to the long-term evolution of user interaction habits.

[0009] Furthermore, anomaly detection is performed on the selected dominant interaction modes. When the signal quality of a certain mode is consistently lower than a preset quality threshold, it is marked as an unreliable mode and temporarily removed from the fusion process. The fusion weights are recalculated for the remaining reliable modalities, and the weight of the contextual emotion context in the decision-making is increased. The contextual emotion joint context vector is divided into two dimensions: environmental disturbance level and emotional arousal level. Based on the combined state of the two dimensions, multiple sets of interactive response templates are predefined, including response delay tolerance, voice feedback volume, visual cue intensity, and tactile feedback frequency. When generating an interactive response, the most suitable template is matched from the interactive response templates, and specific output parameters are configured accordingly.

[0010] Furthermore, the lightweight multimodal fusion model adopts a hierarchical distillation architecture, whose backbone network is derived from a large cloud model through knowledge distillation and compression, retaining key attention heads and high-level semantic paths; The model is deployed on a dedicated neural network acceleration unit for AI glasses, and inference latency is optimized through operator fusion and memory reuse techniques. The process of generating the contextual emotion joint context vector includes: normalizing the environmental context parameters and inputting them into the context encoder; extracting features from the user's physiological emotion parameters and inputting them into the emotion encoder; and then concatenating the output vectors of the two encoders and fusing them through a fully connected layer. The adaptive switching mechanism for the interaction mode includes a three-level response strategy: the first-level strategy maintains the current multimodal fusion mode, the second-level strategy downgrades to the dual-modal redundant verification mode, and the third-level strategy switches to the single-modal high-robust mode. The switching conditions are determined by both the interaction stability assessment index and the level of environmental interference.

[0011] Furthermore, the interactive response operation includes one or more combinations of information display, voice broadcast, tactile vibration, and optical cues; When a user is detected to be in a state of high stress and the ambient noise exceeds a preset noise threshold, non-auditory channels will be used to transmit information first. When it is detected that the user's gaze has lingered on a specific area for an extended period of time and the hand gesture indicates a confirmation intention, the area zoom-in or information expansion operation is automatically triggered. Establish a closed loop for interaction effect feedback. Determine whether the interaction has achieved the expected goal by the subsequent user behavior sequence. If not, record the failure context and use it for online model fine-tuning.

[0012] Furthermore, failure records are generated for the unreliable modes, including mode failure timestamps, failure cause categories, and signal quality degradation trends. Based on the failure records, a modal availability history map is constructed to predict the recovery timing of each modal signal quality. Based on the recovery timing predicted by the modal availability history map, when the quality of the temporarily removed modal signal exceeds the preset recovery threshold, a short-term confidence verification is performed on the recovered modality. Based on the results of the short-time confidence verification, the fusion weight of the recovered mode in the multimodal fusion model is gradually increased to a preset normal threshold, and the updated fusion weight is synchronized to the matching weight reference set of the interactive response template to prevent the fusion result from being unstable due to modal signal jitter. Based on the matching weight reference set, a fuzzy matching mechanism is introduced in the matching process of the interactive response template; When the combined state of environmental interference level and emotional arousal level is in the boundary region of multiple predefined templates, the response delay tolerance, voice feedback volume, visual cue intensity and tactile feedback frequency of adjacent templates are interpolated according to the fuzzy matching mechanism to generate a smooth transition response output parameter configuration. The actual execution status of the configured response output parameters is incorporated into the historical execution record of the context-enhanced interactive instructions. The actual effect of each interactive response template is statistically evaluated to obtain the evaluation results. Based on the evaluation results, the applicable scope boundary of the template and the interpolation range of the fuzzy matching mechanism are dynamically updated to achieve adaptive optimization of the interactive response template library.

[0013] Furthermore, the knowledge distillation process is divided into two stages: soft tag distillation and intermediate layer feature distillation. Among them, soft label distillation is used to transfer probability distribution knowledge of large cloud models, and intermediate layer feature distillation is used to align key semantic representation layers between teacher networks and student networks; Based on the key semantic representation layer aligned by the intermediate layer feature distillation, the context encoder and emotion encoder are constructed using a temporal convolutional network structure to dynamically encode the environmental context parameter sequence and the user physiological emotion parameter sequence within the sliding time window; The length of the time window is adaptively adjusted based on the complexity of the current interactive task reflected in the dynamic encoding process. When updating the parameters of the multimodal fusion model by using the perception data stream processed by the adaptive time window as input, an elastic weight consolidation strategy is adopted. The elastic weight consolidation strategy applies regularization constraints to the important parameters corresponding to historical high-value interaction tasks to prevent catastrophic forgetting in the long-term evolution of the model in adapting to user interaction habits, and outputs the recent parameter stability data obtained by the constraint process to the interaction stability evaluation module. The switching trigger threshold of the three-level response strategy is set as the superposition value of the static baseline threshold and the dynamic correction amount. The dynamic correction amount is calculated in real time based on the recent interaction stability statistics output by the interaction stability assessment module. When the level of environmental interference continues to rise, the switching triggering conditions corresponding to the switching triggering threshold are automatically relaxed; when the interactive environment tends to be stable, the switching triggering conditions are gradually tightened to avoid frequent switching jitter caused by a fixed threshold.

[0014] Furthermore, in the closed loop of the interaction effect feedback, a judgment is made on whether the expected goal has been achieved based on multi-dimensional evaluation indicators; The multidimensional evaluation indicators include the degree of consistency between the user's subsequent operation path and the preset expected path, the delay time of the user's response confirmation behavior, and the trend of changes in the user's physiological and emotional parameters before and after the interaction. Based on the data collected from each dimension of the multidimensional evaluation indicators, the three indicators are weighted and fused, and compared with a preset success threshold to obtain the comparison result. The comparison result indicates that when the fusion evaluation value is higher than the threshold, it is determined that the interaction has achieved the expected goal, and the corresponding multimodal feature snapshot and contextual emotion joint context vector state are stored in the historical successful case library, and the executed interaction response operation and output parameter configuration are positively reinforced and recorded; when the fusion evaluation value is lower than the threshold, it is determined that the interaction has not achieved the expected goal, the failure context recording process is triggered, and the difference between this failed interaction and the record in the historical successful case library is marked. Based on the failure flag triggered by the failure context recording process, a failure signal is fed back to the interaction strategy control module, driving it to reduce the usage priority of the current interaction response template in the next time period. Based on the interaction process corresponding to the failure signal, the failure context is recorded in a structured manner. The recorded content includes a multimodal feature snapshot at the time of failure, the corresponding situational emotion joint context vector state, the type of interaction response operation executed and its output parameter configuration, and is stored in the failure case library. Failed interaction samples are extracted from the failed case library, and semantically similar historical successful interaction samples are retrieved from the historical successful case library. A contrastive learning approach is used to compare features and guide the model to learn to distinguish the key feature differences between the two types of samples in the interaction intent representation vector. The online fine-tuned model's differentiation results for each interaction mode are fed back to the evaluation weight update module. Based on long-term user interaction history data, the weights of each dimension in the multi-dimensional evaluation indicators are personalized to adapt to the operating habits and perceptual preferences of different users.

[0015] Secondly, a multimodal somatosensory interaction and context fusion system based on artificial intelligence includes: Data acquisition module: used to collect multi-source sensory signals from the user while wearing AI glasses, forming a raw sensory signal set; Data preprocessing module: used to perform end-side preprocessing on the original sensing signal set to obtain a structured sensing data stream; Feature extraction module: Based on the perception data stream, it performs cross-modal feature extraction on the edge device to generate visual semantic features, speech intent features, gaze focus features, posture context features, and gesture action features to obtain a feature set; Vector generation module: used to input the feature set into the multimodal fusion model deployed on the edge, and to perform weighted fusion of the features of each modality through a dynamic attention weight allocation mechanism to generate a unified user interaction intent representation vector; Instruction generation module: used to synchronously collect environmental context parameters and user physiological and emotional parameters; construct a context-emotion joint context vector based on the environmental context parameters and user physiological and emotional parameters; couple the user interaction intent representation vector with the context-emotion joint context vector to generate context-enhanced interaction instructions; Interaction response module: used to drive the AI ​​glasses to perform interaction response operations according to the context-enhanced interaction instructions, and dynamically adjust the sensitivity threshold and feedback mode of the interaction strategy; and synchronously write the interaction intent recognition confidence and the current situation emotion joint context vector after each interaction response operation is executed into the interaction state buffer as real-time input for interaction stability assessment. Mode switching module: Used to set interaction stability evaluation indicators. When the confidence of interaction intent recognition is lower than the preset confidence threshold for multiple consecutive time periods, or when the context emotion joint context vector changes abruptly, the interaction mode adaptive switching mechanism is triggered to switch to the backup interaction channel.

[0016] The beneficial effects of this invention are as follows: By uniformly acquiring and preprocessing multi-source sensory signals such as visual images, voice audio, eye-tracking trajectories, head posture, and hand gestures on the edge device, this invention achieves full-link cross-modal deep fusion from the feature layer to the decision layer, overcoming the shortcomings of existing technologies where signals from different modalities are isolated and contextual understanding is limited. The localized deployment of the entire process of perception, extraction, and inference fundamentally eliminates the communication latency introduced by cloud transmission, meeting the stringent requirements of wearable devices for low-latency real-time interaction. By synchronously integrating environmental context parameters and user physiological emotion parameters to construct a contextual emotion joint context vector, and coupling it with the interaction intent representation vector to generate context-enhanced interaction commands, the system can adaptively perceive the user's emotional state and the environment, significantly improving the naturalness of the interaction response and the contextual adaptability. In addition, relying on the interaction stability evaluation index and adaptive switching mechanism, the system can automatically switch to the backup interaction channel under extreme conditions such as a decline in sensory signal quality or sudden environmental changes, ensuring a highly available intelligent interaction experience in all weather and all scenarios. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the process of a multimodal somatosensory interaction and context fusion method based on artificial intelligence provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of a multimodal somatosensory interaction and context fusion system module based on artificial intelligence provided in an embodiment of the present invention. Detailed Implementation

[0018] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0019] At least one embodiment of the present invention discloses a method and system for multimodal somatosensory interaction and context fusion based on artificial intelligence, including: like Figure 1 As shown, the AI-based multimodal haptic interaction and context fusion method includes the following steps: Step 1: Collect multi-source sensory signals from the user while wearing AI glasses to form a raw sensory signal set; Step 2: Perform edge-side preprocessing on the original sensing signal set to obtain a structured sensing data stream; Step 3: Based on the perception data stream, perform cross-modal feature extraction on the edge device to generate visual semantic features, speech intent features, gaze focus features, posture context features, and gesture action features to obtain a feature set; Step 4: Input the feature set into the multimodal fusion model deployed on the edge, and use the attention weight dynamic allocation mechanism to perform weighted fusion of the features of each modality to generate a unified user interaction intent representation vector; Step 5: Synchronously collect environmental context parameters and user physiological and emotional parameters; construct a contextual and emotional joint context vector based on the environmental context parameters and user physiological and emotional parameters; couple the user interaction intent representation vector with the contextual and emotional joint context vector to generate context-enhanced interaction instructions; Step 6: Drive the AI ​​glasses to perform interactive response operations according to the context-enhanced interactive instructions, and dynamically adjust the sensitivity threshold and feedback mode of the interactive strategy; synchronously write the confidence of the interactive intent recognition and the current context emotion joint context vector after each interactive response operation is executed into the interactive state buffer as real-time input for interactive stability assessment. Step 7: Set interaction stability evaluation indicators. When the confidence level of interaction intent recognition is lower than the preset confidence threshold for multiple consecutive time periods, or when the contextual emotion joint context vector undergoes a sudden change, the interaction mode adaptive switching mechanism is triggered to switch to the backup interaction channel.

[0020] Specifically, an interaction stability evaluation index is set, and the historical records in the interaction state buffer are continuously read. When the confidence of interaction intent recognition is lower than a preset confidence threshold for multiple consecutive time periods, it is determined that the intent recognition is continuously unstable. The mutation detection of the contextual emotion joint context vector is carried out by comparing the L2 norm of the vector difference between adjacent time moments with a preset mutation threshold, that is, calculating ‖ct-c{t-1}‖2, where ct and c{t-1} are the contextual emotion joint context vectors of the current time and the previous time moment, respectively. When ‖ct-c{t-1}‖2 exceeds the preset mutation threshold, it is determined that the contextual emotion joint context vector has mutated. When any of the above determination conditions are met, the interaction mode adaptive switching mechanism is triggered to switch to the backup interaction channel.

[0021] This invention uses an artificial intelligence-based multimodal haptic interaction and context fusion method and system deployed on AI glasses as its implementation vehicle. Based on the conventional understanding of the structure of intelligent wearable devices by those skilled in the art, AI glasses typically include hardware components such as an optical display module, camera array, microphone array, inertial measurement unit (IMU), eye-tracking sensor, hand haptic capture unit, physiological signal acquisition electrodes, neural network acceleration chip, main control processor, storage unit, and communication interface. These hardware components are electrically connected through board-level circuits or flexible circuit boards and are managed and scheduled by a unified operating system, forming a complete closed loop of edge-side perception, processing, and response.

[0022] In this embodiment, the data acquisition module consists of multiple physical sensors, including but not limited to: a front-facing RGB-D camera (for acquiring visual image signals), a stereo microphone array (for acquiring voice audio signals), an infrared eye-tracking sensor (for acquiring eye movement trajectory signals), a six-axis IMU (for acquiring head posture signals), and a millimeter-wave radar or ToF sensor integrated into the temples of glasses or a wristband (for acquiring hand motion signals). These sensors are all connected to the main control processor via I²C, SPI, or USB interfaces, and their sampling frequency and timing synchronization are uniformly managed by the driver layer in the operating system. All raw sensory signals are immediately timestamped with high precision after acquisition and cached in a shared memory area, forming a raw sensory signal set.

[0023] The data preprocessing module runs in the real-time task thread of the main control processor. Its functions include signal denoising, timestamp alignment, sampling rate unification, and invalid frame removal for the raw sensory signal set. Specifically, visual image signals are denoised using Gaussian filtering and motion blur compensation algorithms; speech audio signals are processed using spectral subtraction combined with an adaptive noise suppression model; eye movement signals are processed using Kalman filtering to eliminate jitter; head posture signals are processed using low-pass filtering to remove high-frequency vibration interference; and hand motion sensing signals are processed using sliding window mid-range filtering to improve stability. Timestamp alignment uses an interpolation resampling strategy with a 10ms time granularity to unify the signals of each modality onto the same time axis. Invalid frame removal is based on signal integrity check codes and amplitude thresholds, removing data frames missing critical channels or with a signal-to-noise ratio below 5dB, ultimately outputting a structured sensory data stream.

[0024] The feature extraction module is deployed on a dedicated Neural Processing Unit (NPU), which is interconnected with the main control processor at high speed via PCIe or AXI bus. The structured perceptual data stream is distributed to multiple lightweight feature extraction sub-networks: visual semantic features are extracted by the MobileNetV3 backbone network, outputting a 256-dimensional semantic vector; speech intent features are generated into a 128-dimensional intent embedding through a 1D convolutional network with a Squeeze-and-Excitation structure; gaze focus features are generated from eye movement trajectories encoded by LSTM, outputting 64-dimensional spatial attention coordinates; pose context features are generated by concatenating head pose quaternions and angular velocities, then compressed into a 64-dimensional vector by a fully connected layer; and gesture action features are generated from hand somatosensory temporal signals input to the Temporal Convolutional Network (TCN), generating a 128-dimensional action representation. All feature vectors are directly written to the shared feature buffer of the NPU via a DMA channel, forming a feature set.

[0025] The core of the vector generation module is a lightweight multimodal fusion model deployed on the NPU. This model adopts a hierarchical distillation architecture, and its backbone network is derived from a large Transformer model trained in the cloud through knowledge distillation and compression, retaining key attention heads and high-level semantic paths, with the number of parameters controlled within 1.2M. The model input is the aforementioned feature set. First, a learnable projection layer maps each modality feature to a unified dimension (256-dimensional). Then, a lightweight graph neural network (GNN) is constructed to model cross-modal associations: each modality feature serves as a graph node, and the edge weights between nodes are determined by the semantic consistency score calculated using cosine similarity, normalized by Softmax. For the embedding vectors ei and ej of the i-th and j-th modality nodes (the embedding representations of each modality feature mapped to the 256-dimensional common space by the projection layer), the cosine similarity sim(ei,ej) = (ei·ej) / (‖ei‖·‖ej‖), and the edge weight α{ij} = exp(sim(ei,ej)) / Σkexp(sim(ei,ek)), where k iterates through all modality nodes. The GNN aggregates neighborhood information through two layers of graph convolution, and the node update formula for each layer is hi. (l+1) =ReLU(W(l)·(hi (l) +Σ{j∈N(i)}α{ij}·hj (l) )), where hi (l) Let W(l) be the feature vector of node i in layer l, W(l) be the learnable weight matrix of layer l, N(i) be the set of neighboring nodes of node i, and output the enhanced modal features hi. 2Subsequently, a trainable gating mechanism is introduced on the edge. The control signal originates from the subsequently generated contextual emotion joint context vector c, and the gating weight vector w = Softmax(Wg·c + bg), where Wg is the gating projection matrix, bg is the bias vector, and wi is the fusion weight corresponding to the i-th modality. The fusion weights of each modality feature are dynamically adjusted. Finally, the weighted fusion feature vfused = Σiwi·hi 2 A 256-dimensional user interaction intent representation vector, vintent, is generated through global average pooling.

[0026] The instruction generation module comprises two parallel sub-processes. The first is the acquisition of environmental context parameters: illumination intensity is obtained from an ambient light sensor, background noise is estimated from the spectral energy of a microphone array, spatial position is output via a UWB module or SLAM positioning system, and motion state is determined by the integral of acceleration and angular velocity from an IMU. The second is the acquisition of user physiological and emotional parameters: heart rate variability (HRV) is extracted from a PPG photoelectric sensor through time-frequency analysis, skin conductance response (GSR) is obtained by measuring impedance changes using electrodes built into the temples of the glasses, and facial micro-expression features are obtained by cropping the eye and mouth areas from the front-facing camera and inputting them into a lightweight ResNet-18 micro-expression classifier, which outputs a 7-dimensional emotion probability distribution. These parameters are then input to the context encoder and emotion encoder—both are two-layer fully connected networks, outputting a 128-dimensional context vector and a 128-dimensional emotion vector. These two vectors are concatenated and fused through a third fully connected layer to generate a 256-dimensional context-emotion joint context vector.

[0027] The user interaction intent representation vector `vintent` and the contextual emotion joint context vector `c` are coupled in the NPU through element-wise multiplication and residual connections. The coupling formula is `z = (vintent⊙c) + vintent`, where `⊙` represents element-wise multiplication, the addition term is the residual connection, and `z` is a 256-dimensional context-enhanced interaction feature vector. This vector is then mapped to an interaction command probability distribution via a linear classification head `Wcls·z + bcls`, generating context-enhanced interaction commands. These commands include the interaction type (e.g., confirm / cancel zoom), the target object (e.g., UI element ID or spatial coordinates), and a confidence score. The mode switching module receives these commands and drives the AI ​​glasses to perform interactive response operations: if the command type is information display, the optical display module renders AR content on the waveguide lens; if it is voice broadcast, the TTS engine is activated and output through a bone conduction speaker; if it is tactile vibration, a linear motor in the temple is triggered; and if it is an optical cue, the LED indicator flashing mode is controlled. Meanwhile, the module dynamically adjusts the sensitivity threshold of the interaction strategy—for example, increasing the energy threshold of the voice wake-up word in a high-noise environment and reducing the minimum displacement requirement for gesture recognition in a high-stress emotional state.

[0028] The interaction stability evaluation index is continuously calculated by the main control processor: if the confidence score of interaction intent recognition is lower than 0.6 for three consecutive 100ms intervals, or the L2 norm change rate of the contextual emotion joint context vector exceeds 0.8 / s, it is judged as an unstable state, triggering the interaction mode adaptive switching mechanism. This mechanism includes a three-level response strategy: the first-level strategy maintains the current five-modal fusion; the second-level strategy degrades to visual + gesture dual-modal redundancy verification, only when both are valid; Figure 1 The response is executed only when the time is right; the three-level strategy switches to a single-modal high-robust mode, prioritizing the mode with the highest historical reliability (such as eye-tracking confirmation). The switching conditions are jointly determined by the interaction stability assessment index and the environmental interference level—the environmental interference level is determined by a combination of conditions such as light intensity < 50 lux, noise > 70 dB, and movement speed > 2 m / s, and is divided into three levels: low, medium, and high.

[0029] In terms of personalized adaptation, the system collects a 5-minute baseline perception signal set during the initial wearing phase. Through cluster analysis (such as K-means++), it identifies typical user interaction pattern categories (e.g., rapid squinting + nodding to confirm gaze + clenching fist), establishes a personalized interaction baseline model, and stores it in non-volatile memory. In subsequent use, a time-series perception signal subset is generated every 10 minutes. This subset is then compared with the baseline model to calculate the KL divergence: DKL(Pt‖P0)=ΣsPt(s)·log(Pt(s) / P0(s)), where Pt is the probability distribution of interaction behavior collected in the current time period t, P0 is the probability distribution of baseline interaction behavior established during the initial wearing phase, and s is the interaction pattern category index (distinct from the Kalman filter state vector xk), forming the interaction behavior drift trajectory. If the signal quality of a certain modality (such as speech) (weighted by signal-to-noise ratio and frame integrity rate) is below the threshold for 15 consecutive seconds (SNR < 8dB and integrity rate < 90%), it is marked as an unreliable modality, temporarily removed from the fusion process, and the weight of the context and emotion in the gating mechanism is increased.

[0030] The interactive response template library predefines 12 templates, categorized by environmental interference level (low / medium / high) and emotional arousal level (low / medium / high). Each template specifies response latency tolerance (50ms-500ms), voice feedback volume (30%-100%), visual cue intensity (brightness 20%-100%), and tactile feedback frequency (1Hz-5Hz). A fuzzy matching mechanism is used for matching: if the current context-emotion combination is within the template boundary (e.g., interference level 2.8 / 3.0, arousal level 1.9 / 2.0), linear interpolation is performed on adjacent template parameters to generate a smooth transition response output parameter configuration. After each interaction, the system records the user's subsequent 5-second behavior sequence (e.g., whether the operation was repeated, whether manual correction was performed). If the expected goal is not achieved (e.g., not clicking to expand content), the failure context (including all input features and parameter configurations) is stored in the online learning queue.

[0031] The online incremental learning module periodically draws samples from the queue and fine-tunes the parameters of the multimodal fusion model using the Elastic Weight Fixation (EWC) algorithm. Its regularized total loss function is L=Lnew(θ)+(λ / 2)·ΣiFi·(θi-θi*). 2 Where Lnew(θ) is the cross-entropy loss for the new task, θ is the current model parameter, θi* is the optimal parameter of the old task saved after completing the previous round of incremental learning, Fi is the diagonal element of the Fisher information matrix corresponding to parameter θi (measuring the importance of the parameter to the historical task), and λ is the regularization intensity hyperparameter to prevent catastrophic forgetting. Meanwhile, the recovery mechanism for failed modes continues to operate: based on the modality availability history map (recording each failure timestamp, cause category such as strong light interference, hand occlusion, and signal quality decline trend), the recovery timing is predicted. When the signal quality of the removed mode recovers to the recovery threshold (SNR > 12dB and integrity rate > 95%) and lasts for 2 seconds, short-term confidence verification is initiated—in the next 3 interactions, this mode is added to the fusion with a weight of 10%. If the consistency rate between the output intent and the dominant mode is > 80%, the weight is gradually increased to a normal level (e.g., 30%), and the updated weight is synchronized to the template matching reference set.

[0032] The entire system employs operator fusion (such as Conv-BN-ReLU merging) and memory reuse (feature buffer cyclic overlay) techniques to ensure edge-side inference latency is below 30ms, meeting real-time interaction requirements. All modules work collaboratively to form a complete closed loop from perception, fusion, decision-making to response, effectively solving problems such as inaccurate interaction, rigid feedback, and high false trigger rates in complex dynamic environments.

[0033] To enable those skilled in the art to fully understand and implement this invention, the specific implementation principles of this invention are further supplemented below with a specific application scenario.

[0034] In a scenario where a user wearing AI glasses walks on a noisy outdoor street and attempts to retrieve navigation information through gesture interaction, the system first uses a front-facing RGB-D camera to continuously capture images of the street scene ahead. Simultaneously, an infrared eye-tracking sensor records the user's gaze trajectory at a frequency of 120Hz, a six-axis IMU synchronously acquires head pitch and yaw angular velocities, a millimeter-wave radar integrated into the temples senses the hand gesture displacement in the chest area in real time, and a stereo microphone array picks up the ambient sound field and potential voice commands. All raw signals are transmitted to the main control processor via an I²C or USB interface, and the driver layer uniformly adds a microsecond-level timestamp based on a hardware clock to form the raw perception signal set.

[0035] Subsequently, the data preprocessing module performs multimodal signal alignment in the real-time thread of the main control processor: the visual image is spatially filtered using a Gaussian kernel σ=1.2, and optical flow is used to compensate for motion blur caused by head movement; the speech signal is extracted for spectrum through short-time Fourier transform, and background traffic noise above 70dB is suppressed using spectral subtraction based on minimum mean square error; the eye-tracking trajectory is input into a Kalman filter, and the state vector is defined as xk=[px,py,vx,vy] (px and py are the coordinates of the current gaze point, and vx and vy are the gaze velocity components), and the state transition matrix Φ is set to a uniform motion model Φ=[[1,0,Δt,0],[0,1,0,Δt],[0,0,1,0],[0,0,0,1] (Δt is the sampling interval, Φ is different from the GNN adjacency matrix A mentioned later), the observation matrix Hobs=[I2,o2] only observes the position component (Hobs is different from the Butterworth transfer function HBW mentioned later), the observation noise covariance matrix Rk=rk·I2 is dynamically adjusted according to the pupil visibility ρ∈[0,1] (ρ is different from the velocity components vx, vy in the state vector), the adjustment formula is rk=rmax / (1+β·ρ), where rmax is the maximum observation noise variance, β is the sensitivity coefficient, the higher ρ is, the smaller the observation noise and the more the filter trusts the measurement value; the acceleration and angular velocity output by the IMU are processed by a second-order Butterworth low-pass filter with a cutoff frequency fc=8Hz, and its amplitude frequency response is |HBW(f)|= Where f is the signal frequency scalar (distinct from the GNN embedding vector ei), and the attenuation slope is -40dB / dec when n=2 to eliminate high-frequency jitter caused by gait. The hand millimeter-wave signal is filtered using a sliding median filter with a window length of 5 frames to remove jump points caused by instantaneous occlusion. Each modal signal is linearly interpolated and resampled with a granularity of 10ms to ensure strict alignment of the time axis. Invalid frames are removed based on the signal-to-noise ratio threshold (e.g., speech SNR < 5dB) or the absence of data integrity check codes, and a structured perceptual data stream is output.

[0036] The data stream is distributed to multiple lightweight sub-networks within the Neural Processing Unit (NPU): the vision branch uses MobileNetV3-small as its backbone, taking 160×120 resolution RGB-D image patches as input, extracting high-level semantics through depthwise separable convolutions, and outputting a 256-dimensional vector representing road signs or UI elements in the current field of view; the speech branch employs a one-dimensional convolutional network with a Squeeze-and-Excitation module, taking a 40-dimensional MFCC feature sequence as input and outputting a 128-dimensional intent embedding vector; eye tracking is encoded using two layers of LSTM with 32 hidden units, outputting 64-dimensional gaze focus coordinates; head pose quaternions and angular velocities are concatenated into a 7-dimensional input, compressed into a 64-dimensional context vector through two fully connected layers; hand temporal signals are input into a 5-layer TemporalConvolutionalNetwork with convolutional kernel dilation rates of 1, 2, 4, 8, and 16, generating a 128-dimensional gesture representation. All feature vectors are directly written to the NPU shared buffer via DMA, forming a feature set.

[0037] The vector generation module loads a distilled, lightweight Transformer-GNN fusion model. The backbone of this model originates from a 12-layer Transformer trained in the cloud, and after knowledge distillation, retains three key attention heads, compressing the parameter count to 1.18M. The model first maps each modal feature to a 256-dimensional common space using a learnable linear projection. Then, a graph structure is constructed: five modal features serve as nodes, and the edge weights between nodes are calculated using cosine similarity α{ij}=exp((ei·ej) / (‖ei‖·‖ej‖)) / Σkexp((ei·ek) / (‖ei‖·‖ek‖)). This is then normalized using Softmax to obtain the adjacency matrix A (A is distinct from the Kalman state transition matrix Φ), where ei and ej are the embedding vectors of the i-th and j-th modalities after the projection layer (e is distinct from the Butterworth filter signal frequency f), and k iterates through all modal nodes. The graph neural network performs two layers of message passing, with each layer using mean pooling as the aggregation function. The node update formula is hi. (l+1) =ReLU(W(l)·(hi (l) +(1 / |N(i)|)·Σ{j∈N(i)}α{ij}·hj (l) )), where W l Let |N(i)| be the learnable weight matrix of the l-th layer, and |N(i)| be the number of neighboring nodes of node i. Let hi be the enhanced feature of the output of the two layers of the GNN. 2 Weighted fusion based on gate weight wi: vfused=Σiwi·hi 2 Finally, a 256-dimensional user interaction intent representation vector, `vintent`, is generated through global average pooling.

[0038] Meanwhile, the instruction generation module collects contextual and emotional parameters in parallel: the ambient light sensor reads a light intensity of 45 lux, indicating a low-light environment; the microphone array spectral energy integral shows a background noise level of 75 dB; the SLAM system outputs a user movement speed of 2.3 m / s, indicating a brisk walking state; the PPG sensor measures a HRV time-domain index of SDNN=35 ms and a frequency-domain LF / HF ratio of 2.8. Combined with the frowning probability of 0.72 output by the ResNet-18 micro-expression classifier, the emotion encoder determines that the user is in a state of medium to high stress. The context encoder and emotion encoder each output a 128-dimensional vector, which are concatenated and fused through a third fully connected layer to form a 256-dimensional contextual-emotional joint context vector.

[0039] The context vector *c* and the user interaction intent representation vector *vintent* are coupled in the NPU using the operation *z = (vintent⊙c) + *vintent* (⊙ represents element-wise multiplication, and the addition term represents residual connection). *z* is then mapped to the probability distribution of each interaction category via the linear classification head *Wcls·z + bcls*, generating context-enhanced interaction commands. The system identifies the user intent as enlarging the navigation arrow with a confidence level of 0.82. Based on this, the mode switching module activates the optical display module, increasing the brightness of the AR navigation icon on the diffractive waveguide lens to 85%. Simultaneously, due to the detection of a high-noise environment, the system automatically sets the voice feedback volume to 90% and outputs path prompts through a bone conduction speaker. Furthermore, due to a high level of emotional stress, the system dynamically reduces the minimum effective displacement threshold for gesture recognition from 5cm to 3cm, allowing users to complete confirmation operations with a small wave of the hand.

[0040] During the interaction, the main control processor continuously monitors stability indicators: if the intent confidence level is below 0.6 for 300ms consecutively (e.g., due to visual feature distortion caused by strong backlighting), and the L2 norm change rate of the situational emotion vector exceeds 0.8 / s (reflecting drastic emotional fluctuations), a level 3 adaptive switch is triggered. At this time, the system degrades to a level 2 strategy, only executing a response when the gaze target area recognized by visual recognition and the pointing coordinate of gesture recognition are both true, avoiding misjudgment of a single modality. If environmental interference is further aggravated (e.g., entering a tunnel resulting in illumination <30 lux and movement speed >2.5 m / s), the system switches to a level 3 strategy, activating the eye-tracking confirmation modality with the highest historical reliability—the user must gaze at the target for more than 2 seconds to trigger the operation.

[0041] During the initial wearing phase, the system established a personalized baseline through 5 minutes of baseline data collection: cluster analysis identified the user's typical interaction pattern as gaze + slight fist clenching, with a corresponding gesture amplitude of 4.2cm. In subsequent operation, every 10 minutes, the KL divergence between the current gesture distribution Pt and the baseline distribution P0 is calculated as DKL(Pt‖P0)=ΣsPt(s)·log(Pt(s) / P0(s)) (where s is the gesture category index, distinct from the Kalman state vector xk). If the speech signal-to-noise ratio is below 8dB and the frame integrity rate is below 90% for 15 consecutive seconds, the speech modality is temporarily removed, and the weight of the contextual emotion in the gating mechanism is increased from the default 30% to 50%. The interaction response template library matches template number 10 based on the current interference level (high) and emotional arousal level (medium-high), setting the response latency tolerance to 300ms and the haptic feedback frequency to 3Hz. If the user does not perform a subsequent action on the AR prompt within 5 seconds, the interaction context is stored in the online learning queue.

[0042] The online incremental learning module is scheduled weekly, and the Elastic Weight Consolidation (EWC) algorithm is used to fine-tune the multimodal fusion model: the regularized total loss function is L=Lnew(θ)+(λ / 2)·ΣiFi·(θi-θi*)², where Fi, the diagonal element of the Fisher information matrix of the old task parameters, is used as the importance weight, θi* is the optimal parameter of the previous round, and λ is the regularization strength to prevent new samples from overwriting key historical knowledge. Simultaneously, the failed modality recovery mechanism detects that the speech signal-to-noise ratio has risen to 13dB and lasts for 2 seconds, and then initiates the verification process: in the next 3 interactions, speech features are introduced with a weight of 10%. If the consistency rate between the output intent and the dominant modality (eye movement + gesture) exceeds 80%, the weight is gradually restored to 30%, and the modality reliability score in the template matching reference set is updated.

[0043] The entire process relies on operator fusion technology (such as merging convolution, batch normalization, and ReLU activation into a single inference operator) and memory reuse strategies (using a circular queue to overwrite old data in the feature buffer) to ensure that the end-to-end latency from signal acquisition to response output is controlled within 28ms, meeting the requirements of real-time interaction. The synergistic effect of these steps enables the system to accurately interpret user intent even in complex and dynamic environments, generating personalized feedback that adapts to the current context and emotional state, effectively suppressing false triggers and maintaining interaction continuity.

[0044] In terms of power management, the system employs a heterogeneous computing architecture with always-on coprocessors and on-demand main processors to support the aforementioned continuous sensing capabilities. Low-power coprocessors (such as the Cortex-M series) continuously run basic sensor drivers and lightweight wake-up detection algorithms with extremely low power consumption, monitoring IMU acceleration characteristics and low-resolution image frames in real time. When a user interaction intent trigger signal is detected (such as specific head movements, keyword triggers, or sudden changes in the environment), the main processor and neural network acceleration unit are woken up to execute high-computational inference tasks. After the task is completed, the main processor quickly returns to sleep mode. The power management unit (PMU) works in conjunction with the context-aware module for scheduling: in static reading scenarios, only the computing power required for OCR and text translation is activated; in walking navigation scenarios, SLAM localization and obstacle detection-related computing resources are activated; in multimodal interaction scenarios, the NPU is fully activated for feature extraction and fusion inference. Through this dynamic computing power allocation mechanism, the system significantly reduces overall power consumption while ensuring real-time interaction capabilities, effectively extending single-charge battery life and meeting the power supply requirements for all-day wearable use scenarios. In the aforementioned outdoor street scenarios, the system operates in a high-computing-power mode for walking navigation when the user is walking. When the user stops to read, the system automatically switches to a low-computing-power static reading mode, reducing power consumption by approximately 45% compared to full load, thus extending battery life without affecting the interactive experience.

[0045] In terms of privacy and security protection, the system adopts a multi-layered protection mechanism that combines hardware and software. At the hardware level, the camera and microphone circuits are linked to force the LED privacy indicator light to illuminate: whenever the sensor is powered on, the indicator light will illuminate synchronously and cannot be turned off by software, ensuring that people in the surrounding area are aware of the device's acquisition status in real time. In strong light environments (light intensity exceeding 3000 lux), the indicator light brightness automatically increases by 3 times to ensure visibility. At the software level, the system prioritizes processing the raw perception data containing biometric information such as facial features and voiceprints locally on the device's NPU. Only anonymized results (such as detecting a person's movement area rather than a specific facial feature vector) or necessary desensitized data are uploaded to the cloud as needed after end-to-end encryption, preventing the leakage of raw biometric data from the source. Furthermore, users can preset privacy rules according to different scenarios: in meeting mode, the system automatically disables external recording and audio capture functions, retaining only inward services such as local subtitle rendering; in public place mode, it automatically enables real-time face mosaic processing of the video stream, blurring the faces of unauthorized personnel in the image locally before storage or transmission, achieving a balance between functionality and privacy protection. In the aforementioned outdoor street scenario, the indicator light is always on when the front-facing camera captures street scene images. The facial features of pedestrians in front are anonymized on the device side, and the original image frames are not stored. Only the depth map and semantic annotation results used for spatial perception are retained.

[0046] In terms of cross-device ecosystem collaboration, the system supports deep collaboration with mobile phones, car infotainment systems, and smart home terminals through a unified cross-device collaboration protocol, enabling file synchronization, command linkage, and data sharing. The context awareness module automatically triggers multi-device collaborative responses after recognizing the user's location: when the system detects that the user's gaze is focused on a specific point of interest for an extended period, it automatically retrieves navigation map data from the mobile phone and overlays path guidance onto the AR interface; when the system recognizes that the user has entered a meeting scene, it automatically synchronizes the mobile phone's calendar events and projects a directional arrow for the meeting room into the user's field of vision; when the user returns to their home scene, they can directly control smart home devices paired with the AI ​​glasses via gestures or eye-tracking commands. The intelligent service orchestrator is responsible for automatically discovering and orchestrating the scene and intent recognition results to cross-device service atoms: first, the image recognition atom identifies the target object; then, the information overlay atom renders AR content in the field of vision; simultaneously, the voice broadcast atom outputs a brief introduction; and finally, the collection atom stores the current context in the user's personal knowledge base. The entire process requires no manual switching or launching of multiple independent applications by the user. In the aforementioned outdoor navigation scenario, the user's interactive command to enlarge the navigation arrow is synchronized to the mobile navigation application through a cross-device collaboration protocol. The mobile phone returns high-precision navigation data of the current road segment in real time, which is then rendered as an AR overlay path by the AI ​​glasses. This enables real-time linkage of navigation information between the glasses and the mobile phone, leveraging the core value of AI glasses as an intelligent interaction portal.

[0047] like Figure 2 As shown, the AI-based multimodal haptic interaction and context fusion system includes: Data acquisition module: used to collect multi-source sensory signals from the user while wearing AI glasses, forming a raw sensory signal set; Data preprocessing module: used to perform end-side preprocessing on the original sensing signal set to obtain a structured sensing data stream; Feature extraction module: Based on the perception data stream, it performs cross-modal feature extraction on the edge device to generate visual semantic features, speech intent features, gaze focus features, posture context features, and gesture action features to obtain a feature set; Vector generation module: used to input the feature set into the multimodal fusion model deployed on the edge, and to perform weighted fusion of the features of each modality through a dynamic attention weight allocation mechanism to generate a unified user interaction intent representation vector; Instruction generation module: used to synchronously collect environmental context parameters and user physiological and emotional parameters; construct a context-emotion joint context vector based on the environmental context parameters and user physiological and emotional parameters; couple the user interaction intent representation vector with the context-emotion joint context vector to generate context-enhanced interaction instructions; Interaction response module: used to drive the AI ​​glasses to perform interaction response operations according to the context-enhanced interaction instructions, and dynamically adjust the sensitivity threshold and feedback mode of the interaction strategy; and synchronously write the interaction intent recognition confidence and the current situation emotion joint context vector after each interaction response operation is executed into the interaction state buffer as real-time input for interaction stability assessment. Mode switching module: Used to set interaction stability evaluation indicators. When the confidence of interaction intent recognition is lower than the preset confidence threshold for multiple consecutive time periods, or when the context emotion joint context vector changes abruptly, the interaction mode adaptive switching mechanism is triggered to switch to the backup interaction channel.

[0048] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A multimodal haptic interaction and context fusion method based on artificial intelligence, characterized in that, include: Collect multi-source sensory signals from users while wearing AI glasses to form a raw sensory signal set; The original sensory signals include one or more of the following: visual image signals, speech audio signals, eye movement trajectory signals, head posture signals, and hand motion sensing signals; The original sensing signal set is preprocessed at the edge to obtain a structured sensing data stream. Based on the perception data stream, cross-modal feature extraction is performed on the edge device to generate visual semantic features, speech intent features, gaze focus features, posture context features, and gesture action features to obtain a feature set; The feature set is input into a multimodal fusion model deployed on the edge. Through a dynamic attention weight allocation mechanism, the features of each modality are weighted and fused to generate a unified user interaction intent representation vector. Simultaneously collect environmental context parameters and user physiological and emotional parameters; construct a context-emotion joint context vector based on the environmental context parameters and user physiological and emotional parameters; The user interaction intent representation vector is coupled with the contextual emotion joint context vector to generate context-enhanced interaction instructions; Based on the context-enhanced interaction instructions, the AI ​​glasses are driven to perform interactive response operations, and the sensitivity threshold and feedback mode of the interaction strategy are dynamically adjusted. The confidence score of the interaction intent recognition and the current situation emotion joint context vector are synchronously written into the interaction state buffer after each interaction response operation is executed, as a real-time input for interaction stability evaluation; An interaction stability assessment index is set. When the confidence level of interaction intent recognition is lower than the preset confidence threshold for multiple consecutive time periods, or when the contextual emotion joint context vector undergoes a sudden change, an adaptive switching mechanism for the interaction mode is triggered, switching to a backup interaction channel.

2. The multimodal haptic interaction and context fusion method based on artificial intelligence according to claim 1, characterized in that, The original sensing signal set includes a baseline sensing signal set collected during the initial wearing phase, and several time-series sensing signal subsets collected during different usage periods. It also includes clustering analysis of user interaction behavior based on the perceived data stream to identify multiple typical interaction pattern categories; The typical interaction mode categories are quantitatively evaluated, an interaction intent clarity scoring system is established, and the contribution score of each modal signal to intent clarity is calculated. Based on the contribution score, a modal importance threshold is set, and key modal signals with contribution scores higher than the threshold are selected as the dominant interaction modalities. Based on the dominant interaction mode, a modality reliability index is constructed; By combining the modal reliability index and the interaction intent clarity score, the current optimal fusion strategy is determined.

3. The multimodal haptic interaction and context fusion method based on artificial intelligence according to claim 1, characterized in that, Based on the aforementioned benchmark sensing signal set, a user personalized interaction baseline model is established; Based on several time-series sensing signal subsets, the interaction feature sequences of different time periods are obtained and compared with the personalized interaction baseline model to form the interaction behavior drift trajectory. A lightweight graph neural network is used to model the association between cross-modal features, where nodes represent each modal feature and edge weights represent the degree of semantic consistency between modalities. The fusion weights of each modality feature are dynamically adjusted through an edge-trainable gating mechanism, and the control signal of the gating mechanism comes from the contextual emotion joint context vector. Based on a database of successful historical interaction cases, the parameters of the multimodal fusion model are updated through online incremental learning to adapt to the long-term evolution of user interaction habits.

4. The multimodal haptic interaction and context fusion method based on artificial intelligence according to claim 1, characterized in that, Anomaly detection is performed on the selected dominant interaction modes. When the signal quality of a certain mode is consistently lower than a preset quality threshold, it is marked as an unreliable mode and temporarily removed from the fusion process. The fusion weights are recalculated for the remaining reliable modalities, and the weight of the contextual emotion context in the decision-making is increased. The contextual emotion joint context vector is divided into two dimensions: environmental disturbance level and emotional arousal level. Based on the combined state of the two dimensions, multiple sets of interactive response templates are predefined, including response delay tolerance, voice feedback volume, visual cue intensity, and tactile feedback frequency. When generating an interactive response, the most suitable template is matched from the interactive response templates, and specific output parameters are configured accordingly.

5. The multimodal haptic interaction and context fusion method based on artificial intelligence according to claim 1, characterized in that, The lightweight multimodal fusion model adopts a hierarchical distillation architecture, and its backbone network is derived from a large cloud model through knowledge distillation and compression, retaining key attention heads and high-level semantic paths. The model is deployed on a dedicated neural network acceleration unit for AI glasses, and inference latency is optimized through operator fusion and memory reuse techniques. The process of generating the contextual emotion joint context vector includes: normalizing the environmental context parameters and inputting them into the context encoder; extracting features from the user's physiological emotion parameters and inputting them into the emotion encoder; and then concatenating the output vectors of the two encoders and fusing them through a fully connected layer. The adaptive switching mechanism for the interaction mode includes a three-level response strategy: the first-level strategy maintains the current multimodal fusion mode, the second-level strategy downgrades to the dual-modal redundant verification mode, and the third-level strategy switches to the single-modal high-robust mode. The switching conditions are determined by both the interaction stability assessment index and the level of environmental interference.

6. The multimodal haptic interaction and context fusion method based on artificial intelligence according to claim 1, characterized in that, The interactive response operations include one or more combinations of information display, voice broadcast, tactile vibration, and optical prompts; When a user is detected to be in a state of high stress and the ambient noise exceeds a preset noise threshold, non-auditory channels will be used to transmit information first. When it is detected that the user's gaze has lingered on a specific area for an extended period of time and the hand gesture indicates a confirmation intention, the area zoom-in or information expansion operation is automatically triggered. Establish a closed loop for interaction effect feedback. Determine whether the interaction has achieved the expected goal by the subsequent user behavior sequence. If not, record the failure context and use it for online model fine-tuning.

7. The multimodal haptic interaction and context fusion method based on artificial intelligence according to claim 4, characterized in that, The unreliable modes are recorded for failure, and the failure record includes the mode failure timestamp, failure cause category, and signal quality degradation trend; Based on the failure records, a modal availability history map is constructed to predict the recovery timing of each modal signal quality. Based on the recovery timing predicted by the modal availability history map, when the quality of the temporarily removed modal signal exceeds the preset recovery threshold, a short-term confidence verification is performed on the recovered modality. Based on the results of the short-time confidence verification, the fusion weight of the recovered mode in the multimodal fusion model is gradually increased to a preset normal threshold, and the updated fusion weight is synchronized to the matching weight reference set of the interactive response template to prevent the fusion result from being unstable due to modal signal jitter. Based on the matching weight reference set, a fuzzy matching mechanism is introduced in the matching process of the interactive response template; When the combined state of environmental interference level and emotional arousal level is in the boundary region of multiple predefined templates, the response delay tolerance, voice feedback volume, visual cue intensity and tactile feedback frequency of adjacent templates are interpolated according to the fuzzy matching mechanism to generate a smooth transition response output parameter configuration. The actual execution status of the configured response output parameters is incorporated into the historical execution record of the context-enhanced interactive instructions. The actual effect of each interactive response template is statistically evaluated to obtain the evaluation results. Based on the evaluation results, the applicable scope boundary of the template and the interpolation range of the fuzzy matching mechanism are dynamically updated to achieve adaptive optimization of the interactive response template library.

8. The multimodal haptic interaction and context fusion method based on artificial intelligence according to claim 5, characterized in that, The knowledge distillation process is divided into two stages: soft tag distillation and intermediate layer feature distillation. Among them, soft label distillation is used to transfer probability distribution knowledge of large cloud models, and intermediate layer feature distillation is used to align key semantic representation layers between teacher networks and student networks; Based on the key semantic representation layer aligned by the intermediate layer feature distillation, the context encoder and emotion encoder are constructed using a temporal convolutional network structure to dynamically encode the environmental context parameter sequence and the user physiological emotion parameter sequence within the sliding time window; The length of the time window is adaptively adjusted based on the complexity of the current interactive task reflected in the dynamic encoding process. When updating the parameters of the multimodal fusion model by using the perception data stream processed by the adaptive time window as input, an elastic weight consolidation strategy is adopted. The elastic weight consolidation strategy applies regularization constraints to the important parameters corresponding to historical high-value interaction tasks to prevent catastrophic forgetting in the long-term evolution of the model in adapting to user interaction habits, and outputs the recent parameter stability data obtained by the constraint process to the interaction stability evaluation module. The switching trigger threshold of the three-level response strategy is set as the superposition value of the static baseline threshold and the dynamic correction amount. The dynamic correction amount is calculated in real time based on the recent interaction stability statistics output by the interaction stability assessment module. When the level of environmental interference continues to rise, the switching triggering conditions corresponding to the switching triggering threshold are automatically relaxed; when the interactive environment tends to be stable, the switching triggering conditions are gradually tightened to avoid frequent switching jitter caused by a fixed threshold.

9. The multimodal haptic interaction and context fusion method based on artificial intelligence according to claim 6, characterized in that, In the aforementioned interactive effect feedback loop, the achievement of the expected goal is judged based on multi-dimensional evaluation indicators; The multidimensional evaluation indicators include the degree of consistency between the user's subsequent operation path and the preset expected path, the delay time of the user's response confirmation behavior, and the trend of changes in the user's physiological and emotional parameters before and after the interaction. Based on the data collected from each dimension of the multidimensional evaluation indicators, the three indicators are weighted and fused, and compared with a preset success threshold to obtain the comparison result. The comparison result indicates that when the fusion evaluation value is higher than the threshold, it is determined that the interaction has achieved the expected goal, and the corresponding multimodal feature snapshot and contextual emotion joint context vector state are stored in the historical successful case library, and the executed interaction response operation and output parameter configuration are positively reinforced and recorded; when the fusion evaluation value is lower than the threshold, it is determined that the interaction has not achieved the expected goal, the failure context recording process is triggered, and the difference between this failed interaction and the record in the historical successful case library is marked. Based on the failure flag triggered by the failure context recording process, a failure signal is fed back to the interaction strategy control module, driving it to reduce the usage priority of the current interaction response template in the next time period. Based on the interaction process corresponding to the failure signal, the failure context is recorded in a structured manner. The recorded content includes a multimodal feature snapshot at the time of failure, the corresponding situational emotion joint context vector state, the type of interaction response operation executed and its output parameter configuration, and is stored in the failure case library. Failed interaction samples are extracted from the failed case library, and semantically similar historical successful interaction samples are retrieved from the historical successful case library. A contrastive learning approach is used to compare features and guide the model to learn to distinguish the key feature differences between the two types of samples in the interaction intent representation vector. The online fine-tuned model's differentiation results for each interaction mode are fed back to the evaluation weight update module. Based on long-term user interaction history data, the weights of each dimension in the multi-dimensional evaluation indicators are personalized to adapt to the operating habits and perceptual preferences of different users.

10. A multimodal somatosensory interaction and context fusion system based on artificial intelligence, used to execute the multimodal somatosensory interaction and context fusion method based on artificial intelligence as described in any one of claims 1-9, characterized in that, include: Data acquisition module: used to collect multi-source sensory signals from the user while wearing AI glasses, forming a raw sensory signal set; Data preprocessing module: used to perform end-side preprocessing on the original sensing signal set to obtain a structured sensing data stream; Feature extraction module: Based on the perception data stream, it performs cross-modal feature extraction on the edge device to generate visual semantic features, speech intent features, gaze focus features, posture context features, and gesture action features to obtain a feature set; Vector generation module: used to input the feature set into the multimodal fusion model deployed on the edge, and to perform weighted fusion of the features of each modality through a dynamic attention weight allocation mechanism to generate a unified user interaction intent representation vector; Instruction generation module: used to synchronously collect environmental context parameters and user physiological and emotional parameters; construct a context-emotion joint context vector based on the environmental context parameters and user physiological and emotional parameters; couple the user interaction intent representation vector with the context-emotion joint context vector to generate context-enhanced interaction instructions; Interaction response module: used to drive the AI ​​glasses to perform interaction response operations according to the context-enhanced interaction instructions, and dynamically adjust the sensitivity threshold and feedback mode of the interaction strategy; and synchronously write the interaction intent recognition confidence and the current situation emotion joint context vector after each interaction response operation is executed into the interaction state buffer as real-time input for interaction stability assessment. Mode switching module: Used to set interaction stability evaluation indicators. When the confidence of interaction intent recognition is lower than the preset confidence threshold for multiple consecutive time periods, or when the context emotion joint context vector changes abruptly, the interaction mode adaptive switching mechanism is triggered to switch to the backup interaction channel.

Images