User state intervention method and system based on multi-modal perception and interpretable decision

Abstract

The invention discloses a user state intervention method and system based on multi-modal perception and interpretable decision, and aims to solve the problems of insufficient depth of multi-modal fusion, rough state recognition granularity, passive intervention and lack of common information in the prior art. The method comprises the steps of obtaining a multi-modal physiological data sequence through timestamp alignment, and extracting a modal feature vector sequence; dynamically fusing the multi-modal features by using a selective scanning mechanism to generate a deep fusion feature vector; the method comprises the following steps of: realizing interpretable user state mapping on the basis of a Kolmogorov-Arnold network; the future state is predicted in combination with the RWKV network, and a personalized intervention strategy is generated through reinforcement learning. The system comprises a multi-mode perception input module, a fusion and state understanding engine, a future state prediction module and an intervention execution module. According to the method, the fusion precision is improved through long-time-sequence modeling and dynamic weight adjustment, the decision interpretability is enhanced by using the KAN, active and predictive co-estrus intervention is realized, the method is suitable for the fields of education, medical treatment, human-computer interaction and the like, and the user experience and task efficiency are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a user state intervention method and system based on multimodal perception and interpretable decision-making. Background Technology

[0002] Currently, multimodal AI-assisted systems are widely used in fields such as emotion recognition, cognitive load assessment, adaptive learning, and driver monitoring. These systems integrate data from facial expressions, voice, physiological signals (such as heart rate and EEG), eye tracking, and operational behavior to make preliminary judgments about the user's state and adjust interaction strategies accordingly. For example, educational technology platforms dynamically optimize learning content, while fatigue monitoring systems analyze blinking frequency and steering wheel operations to warn of potential risks.

[0003] However, existing methods have significant limitations: First, multimodal fusion relies heavily on simple feature splicing or rule-based decision-making, making it difficult to model complex dynamic relationships across modalities, and it faces a contradiction between computational efficiency and modeling depth in long-term time-series analysis; second, state recognition is coarse-grained, mostly remaining at basic emotion classification or broad cognitive levels, lacking the ability to recognize complex composite states such as "anxiety-induced confusion caused by conceptual misunderstandings," and the black-box model decision-making process lacks interpretability, hindering system optimization and user trust; third, the system generally adopts a passive response mechanism, which cannot predict potential difficulties based on user behavior trajectories, nor can it provide preventative assistance, and the interaction mode is fixed, failing to dynamically adjust in combination with user personality characteristics and real-time context; finally, existing empathic interactions are superficial, with mechanical and rigid feedback content, failing to accurately adapt to the rhythm of user emotional changes, and lacking in-depth communication strategies that evoke emotional resonance, severely restricting the naturalness and affinity of human-computer interaction.

[0004] While existing technologies have made progress in certain scenarios, their shortcomings in deep integration, fine-grained understanding, and proactive empathy limit the accuracy and human-like nature of the systems. Therefore, breakthroughs are needed in key technologies such as long-term modeling, interpretable reasoning, and personalized prediction to achieve truly intelligent, natural, and emotionally supportive human-computer interaction. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a user state intervention method and system based on multimodal perception and interpretable decision-making.

[0006] This invention is achieved through the following technical solution: A user state intervention method based on multimodal perception and interpretable decision-making includes the following steps: Step 1: Obtain multimodal physiological data sequences. Based on the quantization characteristics of each modal physiological data sequence, obtain the modal feature vector sequence of each modal physiological data sequence. Step 2: After obtaining the deep temporal feature vectors of each modality feature vector sequence and aligning them in the time dimension, all deep temporal feature vectors are concatenated at each time step to obtain a multimodal feature sequence. A selective scanning mechanism is used to fuse the modal features in the multimodal feature sequence to obtain a deep fused feature vector. The selective scanning mechanism achieves spatiotemporal fusion of cross-modal features by dynamically adjusting the weights of each modal feature. Step 3: Based on the deep fusion feature vector and combined with the KAN method, determine the historical time period and the current user state vector sequence; Step 4: Embed the historical time period user state vector sequence into the current state vector sequence to obtain the input sequence. Based on the input sequence and combined with the RWKV network, predict the state vector sequence for future time steps. Determine the user state based on the state vector sequence. Obtain the corresponding state profile information and intervention strategy based on the user state. Determine the optimal state profile information based on the user state information. Determine the optimal intervention strategy in the intervention strategy based on the optimal state profile information.

[0007] Preferably, step 1, which involves acquiring multimodal physiological data sequences and obtaining modal feature vector sequences for each modal physiological data sequence based on the quantization characteristics of each sequence, includes: Multimodal physiological signals are acquired and aligned with timestamps to obtain the quantization features of each modal physiological data sequence. These quantization features are used to characterize the user's state. The quantization features of each sequence are then normalized to obtain the modal feature vector sequence corresponding to each sequence.

[0008] Preferably, after obtaining the depth temporal feature vectors of each modality feature vector sequence and aligning them in the time dimension in step 2, all depth temporal feature vectors are concatenated at each time step to obtain a multimodal feature sequence, including: Feature normalization is performed on the feature vector sequences of each modality to eliminate dimensional differences; At least one feature vector is generated in each time window based on the modal feature vector sequence, forming a temporally aligned sequence of modal feature vectors; Linear projection is performed on the feature vector sequences of each mode to obtain the dimension-reduced feature vector sequences of each mode; A single-modal Mamba encoder is used to obtain the depth-time feature vectors of each modality feature vector sequence after dimensionality reduction; At each time step, the deep temporal feature vectors of all modalities are concatenated to obtain a multimodal feature sequence.

[0009] Preferably, step 2 further includes: User profile information is obtained from user data. The user profile information is then concatenated with a multimodal feature sequence using a Mamba encoder. A deep fusion feature vector is generated based on the concatenated multimodal feature sequence. Preferably, step 3, which involves determining the historical time period and the current user state vector sequence based on the deep fusion feature vector and the KAN method, includes: The deep fusion feature vector is input into the KAN model. The deep fusion feature vector passes through multiple KAN layers. The feature components input to each KAN layer are transformed by a learnable spline function connecting all edges. The transformed feature components are weighted and summed at the nodes of the next KAN layer. The final KAN layer outputs a sequence of historical time periods and the current user state vector through an activation function.

[0010] Preferably, step 4, which involves embedding the historical time period user state vector sequence into the current state vector sequence to obtain the input sequence, predicting the state vector sequence for future time steps based on the input sequence and the RWKV network, and determining the user state based on the state vector sequence, includes: The timestamp corresponding to each state vector in the historical time period user state vector sequence is embedded and concatenated with the current state vector sequence to obtain the input sequence; The RWKV blocks of each layer of the RWKV network obtain the temporal dependencies of the input sequence through Time-mixing, then obtain the output sequence through feature interaction through Channel-mixing, and use MLP decoding to decode the output sequence to obtain the state vector sequence, and determine the user state based on the state vector sequence.

[0011] Preferably, after determining the optimal intervention strategy based on the optimal state profile information in step 4, the following steps are also included: Structured intervention information is constructed based on intervention strategies and user preferences, and then transformed into multi-source output information.

[0012] A user state intervention system based on multimodal perception and interpretable decision-making includes: A module is used to acquire multimodal physiological data sequences. Based on the quantization characteristics of each modal physiological data sequence, the modal feature vector sequence of each modal physiological data sequence is obtained. The fusion module is used to obtain the deep temporal feature vectors of each modality feature vector sequence and align them in the time dimension. At each time step, all deep temporal feature vectors are concatenated to obtain a multimodal feature sequence. A selective scanning mechanism is used to fuse the modal features in the multimodal feature sequence to obtain a deep fused feature vector. The selective scanning mechanism achieves spatiotemporal fusion of cross-modal features by dynamically adjusting the weights of each modality feature. The state module is used to determine the historical time period and the current user state vector sequence based on the deep fusion feature vector and the KAN method. The intervention module is used to embed the user state vector sequence of historical time periods into the current state vector sequence to obtain the input sequence, predict the state vector sequence of future time steps based on the input sequence and combined with the RWKV network, determine the user state based on the state vector sequence, obtain the corresponding state profile information and intervention strategy based on the user state, determine the optimal state profile information based on the user information, and determine the optimal intervention strategy in the intervention strategy based on the optimal state profile information.

[0013] An electronic device, comprising: Memory, used to store computer programs; A processor is used to implement the steps of the user state intervention method based on multimodal perception and interpretable decision-making when executing the computer program.

[0014] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the user state intervention method based on multimodal perception and interpretable decision-making.

[0015] Compared with the prior art, the present invention has the following beneficial technical effects: This application provides a user state intervention method based on multimodal perception and interpretable decision-making. The technical solution comprehensively utilizes advanced technologies such as multimodal physiological data acquisition, deep temporal feature extraction, cross-modal feature fusion, interpretable state recognition, and future state prediction to achieve accurate perception and intelligent intervention of user states. The method first acquires multimodal physiological data sequences and extracts their quantitative features to ensure the comprehensiveness and accuracy of user state representation. Subsequently, using deep temporal feature extraction and cross-modal feature fusion techniques, it aligns and splices various modal features in the time dimension, achieving deep fusion of cross-modal information and spatiotemporal correlation modeling. This effectively overcomes the problems of simple feature splicing and limited modeling capabilities in traditional methods. In particular, by introducing a selective scanning mechanism to dynamically adjust the weights of each modal feature, it further enhances the flexibility and effectiveness of feature fusion. In the state recognition stage, the method uses a KAN network for state mapping, which not only improves the fine-grainedness and accuracy of state recognition but also endows the decision-making process with good interpretability through its unique network structure, making the system's decisions more transparent and credible. Furthermore, by combining the RWKV network for future state prediction, the system becomes more predictive, enabling it to identify potential user predicaments in advance and provide preventative assistance, significantly improving the initiative and intelligence level of human-computer interaction.

[0016] This application also proposes a user state intervention system based on multimodal perception and interpretable decision-making, an electronic device, and a computer storage medium, which possess all the advantages of the aforementioned user state intervention method based on multimodal perception and interpretable decision-making. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 The flowchart below shows a user state intervention method based on multimodal perception and interpretable decision-making according to the present invention. Figure 2 This is a structural block diagram of a user state intervention system based on multimodal perception and interpretable decision-making according to the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0020] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0021] A user state intervention method based on multimodal perception and interpretable decision-making includes the following steps: Step 1: Acquire multimodal physiological signals and align them with timestamps to obtain physiological data sequence groups; Step 2: Obtain the quantization features of each sequence in the physiological data sequence group. These quantization features are used to characterize the user's state. The quantization features of each sequence are normalized to obtain the modal feature vector sequence corresponding to each sequence. Step 3: Obtain the depth temporal feature vectors of each modality feature vector sequence, align them in the time dimension, and then concatenate all depth temporal feature vectors at each time step to obtain the multimodal feature sequence. ; Step 4: Employ a selective scanning mechanism to fuse the features of each modality in the multimodal feature sequence to obtain a deep fused feature vector. Among them, the selective scanning mechanism achieves spatiotemporal fusion of cross-modal features by dynamically adjusting the weights of each modal feature.

[0022] Step 5: Deeply fuse the feature vectors Input the KAN network to determine the historical time period and the current user state vector sequence.

[0023] Step 6: Embed the historical time period user state vector sequence into the current state vector sequence to obtain the input sequence. Based on the input sequence and combined with the RWKV network, predict the state vector sequence for future time steps.

[0024] Step 7: Determine the user state based on the state vector sequence, obtain the corresponding state profile information and intervention strategy based on the user state, determine the optimal state profile information based on the user information, and determine the optimal intervention strategy in the intervention strategy based on the optimal state profile information. Step 8: Construct structured intervention information based on intervention strategies and user preferences, and convert the structured intervention information into natural language.

[0025] This method acquires and timestamps-aligns multimodal physiological signals, extracts quantified features, and normalizes them into a sequence of modal feature vectors. It then obtains deep temporal feature vectors and concatenates them into a multimodal feature sequence. A selective scanning mechanism is used to achieve spatiotemporal fusion of cross-modal features, resulting in a deeply fused feature vector. A KAN network is combined to determine historical and current user state vector sequences, and an RWKV network is used to predict future states. This allows for accurate determination of user states and acquisition of corresponding state profiles and intervention strategies. Finally, structured intervention information is constructed based on user preferences and converted into natural language. This achieves a complete closed loop from multimodal data perception to accurate user state identification and personalized intervention strategy generation. It boasts significant advantages such as deep fusion of multi-source information, fine-grained state identification, cross-modal spatiotemporal correlation modeling, predictive future state prediction, interpretable decision-making processes, and natural human-computer interaction, effectively improving the accuracy, proactivity, and humanization of user state intervention.

[0026] Example 1 A user state intervention method based on multimodal perception and interpretable decision-making includes the following steps: Step 1: Use a multimodal input sensing module to acquire multimodal physiological signals and align them with timestamps to obtain a physiological data sequence group; The multimodal input sensing layer includes a sensor interface array, a data acquisition unit, a central synchronization clock module, a data buffer and queue module, and a configuration and management interface.

[0027] Sensor Interface Array (SIA): This array contains standardized physical interfaces and wireless communication modules for connecting to and receiving raw data from various existing types of sensors. Physical interfaces may include USB, serial ports, Ethernet ports, analog input channels, etc.; the wireless communication modules support protocols such as Bluetooth (BLE), Wi-Fi, Zigbee, and ANT+.

[0028] Data Acquisition Units: For each type or category of sensor, there is a dedicated data acquisition unit. These units can be hardware (such as channels on a dedicated data acquisition card), firmware (such as sampling programs on a microcontroller), or software drivers (such as operating system-level device drivers or user-space SDK library functions).

[0029] Central Synchronization Clock Module: This module has a built-in or connected high-precision clock source (e.g., a system clock synchronized via Network Time Protocol (NTP) or a dedicated hardware clock). It provides a uniform, high-resolution timestamp for all acquired data points or data segments.

[0030] Data Buffering and Queueing System: Each data acquisition unit is equipped with a data buffer to temporarily store the incoming raw data stream and prevent data loss. One or more data queues are also provided to orderly pass timestamped data to subsequent processing stages.

[0031] Configuration and Management Interface (Optional, but recommended): Provides a software interface that allows the system to configure which sensors are active, set the sampling rate, resolution, and other necessary acquisition parameters for each sensor.

[0032] The sensor interface array is directly connected to external physical sensors or sensor networks. Each data acquisition unit is logically or physically tightly coupled to its corresponding sensor interface and is responsible for reading data from that interface. When acquiring data, all data acquisition units request the central synchronization clock module to timestamp the data samples or data blocks. The timestamped data is sent by each data acquisition unit to its own data buffer and then pushed into a shared or dedicated data queue. The configuration and management interface allows upper-layer applications or users to control the parameters of each acquisition unit and interface.

[0033] This multimodal input sensing module collects user-related data from multiple modalities in real time and synchronously, and transforms this data into raw physiological data sequences with precise timestamps. Its working principle and steps are as follows: Initialization and Configuration: Upon system startup, all selected sensor interfaces and corresponding data acquisition units are initialized according to preset configurations or instructions via the configuration management interface. This includes establishing communication connections and setting sampling parameters (such as sampling frequency, resolution, data format, etc.).

[0034] Parallel data acquisition: Each data acquisition unit independently acquires raw data from its connected sensors in real time in parallel. The acquisition method can be continuous (such as EEG, ECG signals) or event-driven (such as keyboard key presses, mouse clicks).

[0035] Real-time timestamp assignment: For each data point acquired (e.g., an EEG sample, the start of a video frame, a mouse click event) or a predefined data block (e.g., a 1-second audio clip), the data acquisition unit immediately obtains the current high-precision timestamp from the central synchronization clock module and closely associates this timestamp with the data. This is a crucial step in ensuring that subsequent multimodal data can be effectively fused.

[0036] Data buffering: The raw data with timestamps is first sent to the internal buffer of the corresponding acquisition unit. This can smooth out data congestion that may be caused by instantaneous load fluctuations in data transmission or subsequent processing modules.

[0037] Serialization and queuing: Buffered data is organized into physiological data sequences in chronological order. For example, a series of consecutive timestamped physiological signal samples constitutes a physiological data sequence; a series of consecutive timestamped video frames constitutes a visual data sequence. These raw data sequences are then placed into one or more output queues, awaiting consumption by the next module (feature extraction module). The "serialization" of data here refers to the temporal order and continuity between data points.

[0038] Examples of specific acquisition steps for different modal data: a) Physiological Signal Acquisition: Sensor examples: wearable electrocardiogram (ECG) sensors, electrical activity of the skin (EDA) sensors, photoplethysmography (PPG) sensors, electroencephalogram (EEG) caps / headbands, body temperature sensors, etc.

[0039] The steps for acquiring physiological signals are as follows: The data acquisition unit establishes a connection with the corresponding physiological sensor (e.g., wired connection or wireless pairing). It continuously reads the raw voltage, resistance, or light intensity values ​​output by the sensor at a preset sampling rate (e.g., 250Hz for ECG, 10Hz for EDA). Each sample or a data block consisting of a fixed number of samples is precisely timestamped. The timestamps and the raw physiological signal sample values ​​(possibly after very basic onboard filtering) are used as sequence elements to form the raw physiological data sequence. For example, the ECG sequence is [(ts1, val1), (ts2, val2), ...]. b) Behavioral Visual Signal Acquisition: Examples of sensors: webcams, depth cameras (such as Kinect, RealSense), and dedicated eye trackers.

[0040] The steps for acquiring behavioral visual signals are as follows: The data acquisition unit initializes the camera or eye tracker, setting parameters such as resolution and frame rate (e.g., 30fps). It continuously captures video image frames or data such as gaze coordinates and pupil diameter output by the eye tracker at the set frame rate. A precise timestamp is added to the capture time of each frame or each eye-tracking data point. The timestamps and raw image frame data (such as RGB pixel matrices and depth maps) or eye-tracking data points are organized into a visual behavior data sequence. For example, the video sequence is [(ts1,frame1_data), (ts2, frame2_data), ...]; the eye-tracking sequence is [(ts1, gaze_x1, gaze_y1,pupil_d1), ...]. c) Auditory Signal Acquisition: Sensor examples: microphone arrays, single microphones (such as headphone microphones, desktop microphones).

[0041] The steps for acquiring auditory signals are as follows: The data acquisition unit initializes the audio input device, setting the sampling rate (e.g., 16kHz or 44.1kHz), bit depth (e.g., 16-bit), and number of channels (mono / stereo). It continuously acquires raw audio waveform data. The audio stream is segmented into short frames (e.g., one frame every 20ms), and a precise timestamp is added to the start time of each frame. The timestamps and corresponding raw audio frame sample values ​​are organized into an auditory data sequence. For example, the audio sequence is [(ts1, audio_frame1_samples), (ts2, audio_frame2_samples), ...]. d) Operational Behavior and Context Acquisition: Examples of data sources: Operating system-level APIs (for keystroke logging, mouse events, and window focus changes), application-specific plugins or logging interfaces, and browser extensions (for logging web page interactions).

[0042] The steps for collecting operational behavior and context are as follows: (3) Output of multimodal input sensing module The final output of the multimodal input sensing module is a set of parallel (time-synchronized) raw multimodal physiological data sequences. Each physiological sequence consists of data points or data frames with precise timestamps.

[0043] For example, at a certain time t, module 1 can output: Step 2: Use a multimodal fusion and state understanding engine to obtain the quantitative features of each sequence in the physiological data sequence group. These quantitative features are used to characterize the user's state. The quantitative features of each sequence are normalized to obtain the modal feature vector sequence corresponding to each sequence. After obtaining the depth temporal feature vectors of each modality feature vector sequence and aligning them along the time dimension, all depth temporal feature vectors are concatenated at each time step to obtain the multimodal feature sequence. ; A selective scanning mechanism is used to fuse the features of each modality in the multimodal feature sequence to obtain a deep fused feature vector. .

[0044] Deeply fuse feature vectors Input the KAN network and predict the user state vector sequence.

[0045] This multimodal fusion and state understanding engine processes the input multimodal physiological data sequence through timestamp synchronization, feature extraction, context-aware cross-modal fusion, and interpretable state output to obtain the user's current cognitive-emotional state. The multimodal fusion and state understanding engine includes a multimodal feature extraction module, a Mamba-based context-aware cross-modal fusion module, and a KAN-based interpretable state output module.

[0046] A. Multimodal Feature Extraction Module This multimodal feature extraction module contains a set of dedicated feature extractors for different data modalities. It extracts quantifiable features that effectively characterize user states from the timestamped raw multimodal data sequences output by module 1. Each extractor consists of optimized signal processing algorithms, statistical analysis tools, or pre-trained deep learning models (e.g., facial recognition models for vision, acoustic models for speech). Specifically, it includes, but is not limited to: 1. Physiological signal feature extractor: Processes signals such as electrocardiogram (ECG), electrical activity of the skin (EDA), electroencephalogram (EEG), photoplethysmography (PPG), and body temperature.

[0047] 2. Behavioral visual signal feature extractor: Processes visual data such as facial expressions, eye tracking, and head posture.

[0048] 3. Auditory signal feature extractor: processes speech (content, emotion, rhythm) and ambient sounds.

[0049] 4. Operation Behavior and Context Feature Extractor: Processes keyboard input, mouse operations, application interaction logs, and task context information.

[0050] The working principle of this multimodal feature extraction module is as follows: 1. Data input: Multimodal physiological data sequences.

[0051] Windowing: This involves dividing continuous signal data (such as physiological signals, audio, and some visual data) into windows. Windows can be of fixed length (e.g., 5 seconds of EEG data) or event-dependent (e.g., 2 seconds of data before and after a mouse click). Each window inherits the timestamp of its starting point.

[0052] 2. Extract features from each physiological data sequence 1) Physiological signals: ECG / PPG: Calculates heart rate (HR) and heart rate variability (HRV) related parameters, such as time-domain SDNN, RMSSD, pNN50, frequency-domain low-frequency power (LF), high-frequency power (HF), LF / HF ratio, and nonlinear parameters such as Poincaré plot descriptors and the alpha exponent of detrended fluctuation analysis (DFA).

[0053] 2) EDA: Extract skin conductance level (SCL) as an indicator of the tetanic phase, and the amplitude, number, rise time, and recovery time of skin conductance response (SCRs) as indicators of the phase phase.

[0054] 3) EEG: Calculate the power spectral density or relative power for each frequency band (e.g., Delta (1-4Hz), Theta (4-8Hz), Alpha (8-13Hz), Beta (13-30Hz), Gamma (30-100Hz)). If task-related, event-related potentials (ERPs) can be extracted.

[0055] 4) Body temperature: Calculate the average body temperature and the trend of body temperature changes.

[0056] 5) Behavioral visual signals: Facial expressions: Extract the intensity and activation state of action units (AUs) using a Facial Action Coding System (FACS) (via OpenFace, FACET). Alternatively, use a pre-trained Facial Expression Recognition (FER) model to output the probability distribution of basic emotions (such as happiness, sadness, anger, surprise, fear, disgust, and neutrality). Extract facial keypoint coordinates.

[0057] 6) Eye tracking: Calculate the number of fixations, average fixation duration, saccade parameters (amplitude, velocity, direction), saccade path length and entropy, and pupil diameter (mean and variability).

[0058] 7) Head posture: Extract the three-dimensional rotation angles of the head (pitch, yaw, roll) and their angular velocity and angular acceleration.

[0059] 8) Auditory signals, including speech acoustic features, speech prosody features, and speech content and emotional features.

[0060] Speech acoustic features: Extracting Mel frequency cepstral coefficients (MFCCs), fundamental frequency (Pitch / F0), formants, short-time energy, zero-crossing rate, harmonic noise ratio (HNR), jitter, shimmer, etc.

[0061] Speech prosodic features: extract speech rate, pause frequency and duration, pitch profile, energy profile, etc.

[0062] Speech content and sentiment features (if ASR and NLU are integrated): Automatic speech recognition (ASR) extracts text content, and natural language understanding (NLU) extracts the sentiment tendency (positive, negative, neutral), semantic topics, etc.

[0063] 9) Operational behavior and contextual characteristics: Keyboard behavior: Input speed, error rate, key combination frequency (e.g., N-grams), key interval duration.

[0064] Mouse behavior: click frequency, movement distance / speed, curvature and entropy of cursor trajectory, idle time.

[0065] 10) Application interaction and context: Application switching frequency, dwell time on a specific application / document, task step completion time, API call pattern, current window title, file type being processed, etc.

[0066] 3. Feature Normalization / Standardization: The feature values ​​of each extracted physiological data sequence were normalized (Z-score normalization: ,in It is the mean. (It is the standard deviation) to eliminate dimensional differences and ensure that different features have comparable contributions in subsequent fusion models.

[0067] 4. Data Output: For each input data window (corresponding to a timestamp), one or more feature vectors are generated, forming a sequence of modal feature vectors aligned with the time of the original data stream. For example, for a 5-second window, an ECG feature vector, a facial expression feature vector, etc., might be output.

[0068] B. Mamba-based context-aware cross-modal fusion module This Mamba-based context-aware cross-modal fusion module leverages the powerful sequence modeling capabilities of the Selective State Space Model (Mamba) to deeply fuse feature vector sequences from various modalities, resulting in multimodal feature sequences.

[0069] This Mamba-based context-aware cross-modal fusion module includes an input embedding layer, a unimodal Mamba encoder, a feature sequence splicing / alignment module, a cross-modal Mamba fusion network, a context injection module, and an output pooling layer.

[0070] 1. Input Embedding Layer: Performs independent linear projection or embedding of each modality feature vector sequence using a small MLP, mapping it to a dimensional space more suitable for Mamba model processing, and selectively adds positional encoding.

[0071] Unimodal Mamba Encoders: Before cross-modal fusion, a separate Mamba encoder is set up for each modality's feature vector sequence. The Mamba encoder is used to capture the complex temporal dynamics and contextual dependencies within each modality, and outputs the deep temporal feature vector of that modality.

[0072]

[0073] in, It is modal The characteristic sequence, It is its encoded deep temporal feature vector.

[0074] 2. Feature Sequence Concatenation / Alignment Module: The deep temporal feature vector processed by the single-modal Mamba encoder. Alignment is performed along the time dimension, and the deep temporal feature vectors of all modalities are concatenated at each time step to form a unified multimodal feature sequence.

[0075]

[0076] 3. Core Cross-modal Mamba Fusion Network: This is the key to the fusion process. It consists of multiple stacked Mamba encoders. It receives the concatenated multimodal feature sequences. As input.

[0077] Mamba encoder structure: Each Mamba encoder mainly consists of a Selective State-Space Model (SSM) core and a gating mechanism. The key lies in the Selective Scan Mechanism, which allows the model to dynamically adjust the parameters of the SSM based on the input (especially...). (Parameters), thus effectively focusing on important information in the sequence and filtering out irrelevant information, while maintaining linear computational complexity. The continuous form of SSM is:

[0078] Discretization (e.g., zero-order ZOH preservation) yields the discrete form:

[0079] In the Mamba encoder, and discretization step size It is based on the input It is dynamically generated through a small neural network, which gives the model powerful selection capabilities.

[0080] The Mamba encoder processes sequences using an efficient parallel scanning algorithm, enabling it to efficiently handle extremely long sequences.

[0081] 4. Context Injection Mechanism (C): The task context information (such as task type and difficulty) and user profile features (such as the user's baseline emotion and skill level) of the knowledge base and user profile module can be spliced ​​in as additional channels. In, or used to conditionalize parameters in Mamba blocks (via the FiLM layer), the fusion process can adapt to the current specific context and user characteristics.

[0082] 5. Output Pooling Layer: The output of a Mamba fusion network is typically a sequence of fused feature vectors of the same length as the input sequence. .

[0083] Depending on the requirements of the downstream task, a pooling layer (such as taking the representation of the last time step of the sequence, performing average pooling or max pooling on the entire sequence, or using an attention mechanism for weighted pooling) is used to obtain a fixed-dimensional fused feature vector. This represents the overall user status of the current window.

[0084] The following explains the working principle of the Mamba-based context-aware cross-modal fusion module, including the following steps: Input preparation: Receive the feature vector sequences of each modality, ensuring that these sequences are strictly aligned in time.

[0085] Single-modal encoding: The temporal dependencies within each modality feature vector sequence are extracted by its single-modal Mamba encoder.

[0086] Feature concatenation and context integration: The (encoded or raw) feature vectors of each modality are concatenated at each time step. If context information is used, it is integrated as well.

[0087] Cross-modal Mamba fusion: The concatenated multimodal sequences are input into the cross-modal Mamba fusion module. The Mamba network, through its selective scanning mechanism, captures complex, non-linear, long-range dependencies and interactions between features of different modalities and across time. It learns how to dynamically "select" which features of which modalities to focus on, and their temporal evolution patterns, based on the input.

[0088] Cross-modal Mamba fusion output: The Mamba network outputs a deeply fused sequence of feature representations. If a fixed-length vector is needed, it can be obtained through a pooling layer. This representation contains a comprehensive encoding of various aspects of the current user's state.

[0089] Loss Function: The Mamba-based context-aware cross-modal fusion module does not have a separate loss function. Instead, it functions as a feature extractor, trained end-to-end with the KAN-based interpretable state output module. Its parameters are optimized by backpropagating the loss calculated from the KAN-based interpretable state output module.

[0090] C. KAN-based interpretable state output module This KAN-based interpretable state output module uses a Kolmogorov-Arnold network to map deeply fused feature vectors to specific user state dimensions, while ensuring the high interpretability of this mapping relationship. An interpretable state output module based on KAN, comprising an input layer, a KAN network layer, and an output layer.

[0091] 1. The input layer is used to receive deep fusion feature vectors.

[0092] 2. KAN Network Layers: Unlike traditional multilayer perceptrons (MLPs), KAN's nonlinear activation functions reside on the network's "edges" rather than "nodes." Nodes simply perform weighted summations.

[0093] Each edge connecting node i in the previous layer and node j in the current layer is associated with a learnable one-dimensional function. This function is parameterized as a linear combination of B-splines:

[0094] in, It is the k-th degree d (e.g., d=3 for cubic splines) B-spline basis function, defined on a set of nodes. Above (G is the number of grid intervals). These are learnable spline coefficients. The positions of the grid points can also be learnable to achieve adaptive splines.

[0095] A KAN layer (e.g., from Dimension input The j-th output of the dimension output) It can be represented as: in It is the input vector The i-th component.

[0096] This module stacks multiple such KAN layers to increase the model's complexity and expressive power.

[0097] 3. Output layer Classification tasks (e.g., emotion recognition): The output of the last layer of KAN can be viewed as the logit values ​​corresponding to each category. These logit values ​​are then transformed into a probability distribution using the Softmax function.

[0098]

[0099] Where C is the number of state categories.

[0100] For regression tasks (e.g., cognitive load rating, engagement level): the output of the last layer of KAN can be directly used as a continuous value for prediction. A linear activation or bounded activation (e.g., Sigmoid, if the value is between 0 and 1) may be required.

[0101] Multi-task output: KANs can be designed with multiple output heads, each corresponding to a specific user state dimension (e.g., one head outputs the emotion category, and another outputs the cognitive load score). Each head can have its own specific output activation function.

[0102] The working principle of the KAN-based interpretable state output module is as follows: Input reception: Receive deep fusion feature vectors .

[0103] Forward propagation: The process involves multiple KAN layers. In each layer, each input feature component is transformed by a learnable spline function connected to all its edges, and these transformed values ​​are then weighted and summed at the nodes of the next layer.

[0104] State Output: The output of the final KAN layer is processed by an appropriate activation function (such as Softmax or linear) to obtain the final user state result. For example, a state vector containing emotion probability distribution, cognitive load score, and engagement level.

[0105] Interpretability Analysis: This is a key advantage of KAN.

[0106] Visualizing spline functions: due to the activation function on each edge They are one-dimensional, and can be directly plotted. This allows analysts to visually see the first... i Input features (from) How and in what functional relationship (linear, nonlinear, threshold, periodic, etc.) does ) affect the first j One intermediate neuron or the final output state.

[0107] Feature importance analysis: If a certain input feature All spline functions connected to the next layer If the values ​​are close to constants (i.e., do not change much), it indicates that the feature has little impact on subsequent calculations. Conversely, if the spline function has a complex shape and changes significantly, it indicates that the feature is important.

[0108] Interaction identification (for multi-layer KAN): By analyzing layer by layer, it is possible to understand how features combine and interact to form the final judgment.

[0109] Symbolic Formula Discovery: KAN points out that by performing symbolic regression on learned spline functions, they can sometimes be simplified to known mathematical expressions (such as...). (etc.), thus providing deeper insights.

[0110] Loss Function Design: The training of this module is usually supervised and requires labeled data (i.e., the user's real cognitive-emotional state labeled within a specific time window).

[0111] Classification Loss: Categorical Cross-Entropy Loss is commonly used.

[0112]

[0113] Where N is the number of samples and K is the number of categories. It is the true label (one-hot encoded) of sample n. It is the probability that the model predicts that sample n belongs to class k.

[0114] Regression Loss: Use Mean Squared Error (MSE) or Root Mean Squared Error (RMSE).

[0115] in That's the actual score. It is the predicted score.

[0116] Multi-task Loss: If multiple state dimensions are predicted simultaneously (e.g., sentiment is classification, load is regression), then the total loss is a weighted sum of the losses from each task.

[0117] The weight w can be set manually, or used as a learnable parameter, or automatically adjusted using methods such as uncertainty weighting.

[0118] Regularization for KAN: To improve the smoothness or sparsity of spline functions, regularization terms can be added. For example, for spline coefficients... L1 or L2 regularization.

[0119] The interpretable state output module of this KAN leverages the powerful selective state-space modeling capabilities of the Mamba model to effectively capture complex dynamic relationships and long-range dependencies between multimodal data, making it particularly suitable for handling continuous data streams like user states that evolve over time. By introducing KAN as the output mapping network for the final state, the "decision-making process" from highly abstract fused features to specific user states becomes transparent and analyzable. This overcomes the "black box" problem of traditional deep learning models in human-computer interaction understanding, enhancing the system's credibility and debuggability.

[0120] Step 3: Using the RWKV-based user future state prediction module, the historical state sequence is embedded into the state vector sequence to obtain the input sequence, and the state vector sequence of the future time step is predicted based on the input sequence.

[0121] Determine the user state based on the state vector sequence, obtain the corresponding state profile information and intervention strategy based on the user state, determine the optimal state profile information based on the user information, and determine the optimal intervention strategy in the intervention strategy based on the optimal state profile information. The RWKV-based user future state prediction module outputs the optimal intervention strategy through a reinforcement learning-based decision-making process, based on the predicted user state vector sequence, the prediction of the user's future state, user profile information, and a configurable intervention strategy knowledge base.

[0122] The RWKV-based user future state prediction module includes an RWKV-based user future state prediction module, an intervention strategy knowledge base, and a KAN-enhanced reinforcement learning decision-making module.

[0123] A. User Future State Prediction Module Based on RWKV The RWKV-based user future state prediction module includes an input sequence construction and embedding module, an RWKV core network, and a prediction head.

[0124] 1. The input sequence construction and embedding module is used to embed timestamps and state vectors into the input user state vector sequence.

[0125] Timestamp Embedding (optional): Embeds the timestamp (or relative time difference) corresponding to each state vector in the user state vector sequence, and concatenates or adds it to the state vector to provide more explicit time information.

[0126] State Vector Embedding: Using a small MLP to embed the original state vector Working dimensions projected onto the RWKV model .

[0127] 2. RWKV Core Network Depend on It consists of stacked RWKV blocks. Each layer contains Time-mixing and Channel-mixing sub-blocks.

[0128] Understanding Time-mixing: Its core is the recursive update of the state, which can be written in form as: (This is a highly simplified analogy; the actual recursive state updates in RWKV are much more complex and incorporate...) (Calculation).

[0129] (Output weights) and time decay parameters (such as) Or learnable parameters in the latest version and The _first) parameters are key learnable weights. These parameters allow the model to dynamically determine how and with what strength historical information influences the processing of the current step.

[0130] Understanding Channel-mixing:

[0131] Usually The form, which incorporates the channel state from the previous time step. This gives it the ability to dynamically adjust the output based on the input, similar to the ability in FFN.

[0132] 3. Predicting the head The last layer of RWKV operates at each time step i (corresponding to the input sequence) The output of ) .

[0133] Typically, the output of the last time step of the input sequence is taken. (Or pooling the output of the most recent few time steps) as an encoded representation of the entire historical sequence.

[0134] This encoding indicates that the future is generated through one or more MLP layers (prediction heads). Prediction of the state vector sequence at each time step:

[0135] Prediction can be autoregressive (predicting the future step by step) or direct multi-step prediction. For state vectors, direct multi-step prediction is usually used, where each... .

[0136] The working principle of this RWKV-based user future state prediction module is as follows: 1) Historical state data preparation: Maintain a sliding window to store the user state vectors of the most recent N time steps. The choice of N requires a trade-off between information sufficiency and computational efficiency.

[0137] 2) Embedding and Sequence Input: The historical state sequence is embedded to form the input sequence of RWKV. .

[0138] 3) RWKV Forward Propagation: The input sequence passes through RWKV blocks at each layer. Each layer captures temporal dependencies through time-mixing and performs feature interactions through channel-mixing. Its linear computational complexity and constant memory consumption enable it to efficiently process long sequences.

[0139] 4) Feature aggregation and prediction: Using the sequence representation output by RWKV (usually the representation of the last time step), the prediction head MLP decodes the sequence of predicted state vectors for the next H time steps.

[0140]

[0141] The predicted state vector sequence F can be a flattened vector or it can maintain its sequence structure.

[0142] 5) Loss Function Design: Consider the multidimensionality of the state vector: If the user state vector s contains variables of different properties (e.g., emotion category probabilities and continuous cognitive load values), the loss function should be a weighted sum of the losses of each variable. in This is the loss for the j-th dimension of the state vector. If the j-th dimension is a category (such as emotion), then... It is the cross-entropy loss; if it is a continuous value (such as load), it is the MSE or Huber loss. It's the weight.

[0143] Weighting of multi-step forecasts: For multi-step forecasts Different weights can be assigned to the prediction error at different future time steps. For example, predictions of the near future may be more important or easier to predict accurately, and can be assigned a higher weight. ,in .

[0144] Training data: requires a large amount Pairing data. This data can be generated through actual system operation or a simulated environment.

[0145] B-profile knowledge base and intervention strategy knowledge base The profile knowledge base stores various profile information corresponding to user status. For example, if a user's status is running and their heart rate exceeds the standard, the profile information will be the running status information of different groups of people.

[0146] The intervention strategy knowledge base stores intervention information corresponding to user status. For example, if a user is running and their heart rate exceeds the standard, the intervention information represents various strategies to intervene in the running heart rate.

[0147] The RL agent loads the knowledge base at startup and can be configured to periodically check for updates or receive update notifications via API.

[0148] The RL agent uses the intervention_id in the knowledge base (possibly plus a discretized combination of key parameters) as the basis for its discrete action space.

[0149] For interventions with parameters, the RL agent may need to learn to select intervention IDs and learn to output these parameter values ​​(forming a hybrid action space), or the knowledge base may provide a set of discrete options for each parameter.

[0150] RL agents can use trigger_conditions_template or other metadata in the knowledge base to perform preliminary action pruning, reducing the complexity of decision-making, although the final decision depends on the learned policy.

[0151] The user profile knowledge base and intervention strategy knowledge base serve as the system's central information storage and management center, responsible for maintaining personalized information related to users (user profiles) as well as general or domain-specific knowledge required for system operation. This information ensures the system's personalization, adaptability, and context awareness. While the user profile knowledge base and intervention strategy knowledge base do not directly participate in the core processing of real-time data streams, they provide other modules with crucial background information and dynamically updated data support.

[0152] The core functions of the user profile knowledge base and intervention strategy knowledge base are: to store, manage, and provide user profile information, user preferences, interaction history, and domain-specific knowledge.

[0153] User initial configuration data for the profiling knowledge base and intervention strategy knowledge base (such as demographic information, explicitly defined preferences, and initial baseline physiological data).

[0154] Long-term behavioral and environmental data summaries from the multimodal perception input module (e.g., common behavioral patterns of users in specific environments).

[0155] Long-term, aggregated user state sequences and identified patterns from the multimodal fusion inference module.

[0156] Intervention history from the proactive intervention decision-making module and its corresponding (indirectly observed) user responses / effects.

[0157] Explicit feedback (such as ratings) or implicit feedback (such as skipping or accepting) from users at the multimodal output and execution layer regarding the way the intervention is presented.

[0158] User profile parameters or models that have been learned and optimized from the feedback optimization module.

[0159] Outputs of the profiling knowledge base and intervention strategy knowledge base: User profile features (such as personality traits, cognitive ability tendencies, emotional baseline, stress response patterns, and learned skill levels).

[0160] User preferences (such as notification method preferences, intervention type preferences, and acceptable level of disturbance).

[0161] Domain-specific knowledge (such as the context of the current task and relevant background information).

[0162] User Profile Sub-module: Function: Build and maintain a dynamic profile for each user.

[0163] Static / semi-static data: User ID, age, gender (if relevant), occupation, personality traits (such as the Big Five personality traits) obtained through questionnaires or initial setup, cognitive style, long-term goals, etc.

[0164] Dynamic data: Physiological and behavioral baselines include: average heart rate, normal HRV range, common facial expression patterns, average speech rate, and typical operating habits. These baselines are updated slowly over time.

[0165] State history and patterns: Records the historical changes in users' emotions, cognitive load, and engagement in different contexts (such as different tasks and different times).

[0166] Learning and performance history: The user's learning curve, error rate, efficiency, etc. on a specific task.

[0167] Intervention Preferences and Effectiveness: Record users' acceptance of different interventions, historical feedback, and the actual impact of these interventions on the user's state.

[0168] Intervention strategy knowledge base module: Function: Store knowledge related to specific application domains to help the system better understand the context and make reasonable decisions.

[0169] Task knowledge: For learning support systems, this may include course structure, knowledge point connections, and common difficulties; for work support systems, this may include project processes and tool usage methods.

[0170] Environmental knowledge: Define the typical characteristics of different environments (such as office, home, and travel) and their potential impact on user status.

[0171] General knowledge: Psychological common sense related to emotions and cognition, used to initially interpret the user's state or suggest interventions.

[0172] C. KAN-enhanced reinforcement learning decision-making module The structure of the KAN-enhanced reinforcement learning decision module is as follows: Construction and representation of state space (S) (Dimensionality: ): Input concatenation: .in (Current state) yes The future state prediction sequence can be flattened into 1D vectors, or compressed into fixed-dimensional representations via attention mechanisms / RNNs / mini-Mamba. . (User profile) (If flattened) or (If compressed).

[0173] Normalization: All input features should be normalized before concatenation (e.g., Z-score or Min-Max to the [0,1] or [-1,1] interval) to ensure that the spline functions in the KAN network work within similar input ranges.

[0174] Feature engineering: A small amount of manual feature engineering can be performed before stitching, such as calculating the difference between the current load and the predicted load, or the difference between the current sentiment and the portrait baseline sentiment.

[0175] Construction and representation of action space (A): Discrete actions: If the number of interventions If the action space is finite and without complex parameters, then the action space is of size . The output of KAN (if it's a Q-network) is a discrete set of Q-values ​​for each action: $Q \in \mathbb{R}^{K_{actions}+1}$. If it's a policy network, it's the probability distribution of action selection.

[0176] Parameterized Actions: If an intervention has continuous or a large number of discrete parameters (such as duration_seconds), a more complex action representation is needed. One approach is to use a hybrid action space: first, select a basic intervention ID (discrete), and then predict its parameters (continuous or discrete) for the selected intervention ID. KANs can be used in different parts of the policy network, with one part outputting the probability of intervention ID selection and other parts outputting the corresponding parameters.

[0177] reward function ( In-depth design considerations: Goal alignment: Rewards must precisely reflect the desired system behavior and user well-being. For example, the goal is to reduce user stress and improve task efficiency.

[0178] Reward Shaping: To accelerate learning, intermediate rewards can be introduced. For example, if the predicted future state indicates an increase in load, and the agent takes preventative measures to slow the predicted increase (even if the actual load has not yet decreased significantly), a small positive reward can be given. Care must be taken to avoid introducing unexpected biases through reward shaping (potential-based reward shaping is a theoretically safe method).

[0179] Long-term vs. short-term trade-offs (Discount Factor) ): Control the degree of emphasis placed on future rewards. A score close to 1 indicates a greater emphasis on long-term cumulative rewards.

[0180] Sparse reward handling: If the primary reward (such as task success) is very sparse, consider the following: Auxiliary tasks / intrinsic incentives: For example, rewarding the agent for exploring new state-action pairs, or rewarding it for improving the accuracy of its predictions of user states (if the prediction model is also tuned by RL).

[0181] Hierarchical Reinforcement Learning: High-level policies set sub-goals, and low-level policies learn to achieve these sub-goals, thereby decomposing complex problems.

[0182] Preference-based learning (RL): When it is difficult to explicitly define the reward function, human users can compare different behavioral trajectories produced by the agent and learn an implicit reward function or a direct learning strategy from them.

[0183] KAN policy network ( ) or KAN value network ( - Structural details: Number of layers and width: KANs typically use fewer layers (e.g., 2-3 layers) and a narrower width because their expressive power primarily comes from the spline functions on the edges. For example, a structure could be... .

[0184] Spline parameters: Number of grid points (G): Controls the complexity and fitting ability of the spline. More grid points can fit more complex functions, but may also lead to overfitting. Typically, G is between 3 and 10.

[0185] Spline order (d): Cubic B-splines with d=3 are typically used to provide sufficient smoothness and flexibility.

[0186] Grid Update: KAN allows the grid point positions to be updated adaptively based on the distribution of the input data during training, which can significantly improve its efficiency and accuracy.

[0187] Why KAN is superior to MLP in this scenario: Local interpretability and global composition: Features of each input state With the next level node Relationship It is a directly visualized one-dimensional function. This allows us to precisely understand how a single state dimension (such as "current cognitive load") contributes to the value assessment of an intervention in a non-linear way (e.g., the impact increases sharply after reaching a certain threshold). Furthermore, multiple such functions, when summed and combined, can form complex global decision boundaries. While the ReLU activation function in MLP is simple, its combined effects are difficult to decompose intuitively.

[0188] Fewer parameters, higher accuracy potential: The KAN paper shows that KAN can achieve higher accuracy with far fewer parameters than MLP when fitting certain complex functions, which may be beneficial for sample efficiency and generalization ability in RL.

[0189] Directly linking input features with decision logic: For a Q-network If action 'a' is also part of the input (e.g., concatenated with 's' after one-hot encoding), then we can directly see the state features. and specific actions How the interaction between them affects the final Q value through spline functions.

[0190] Smoothness and robustness: The properties of B-splines make the learned functions generally smoother, which may make the RL policy more robust to small perturbations of the input state.

[0191] The working principle of KAN-enhanced reinforcement learning decision-making module is as follows: Initialization: Initialization Network (main network) and Network (target network, parameters) Initialize the experience playback buffer D.

[0192] Looping interaction and learning: For each time step a. State awareness: Combining the current state And standardize it. b. Action selection: Use Strategy: Based on probability Randomly select an action (Explore), otherwise select (use). It typically decays with training. c. Execution and Observation: Execution Observe the instant rewards and new user status and changes in portrait Module 3.1 Predicting New Future States The combination yields the next state. d. Storage experience: [This will be used in conjunction with other methods] Store in D. e. Sampling and training: Randomly sample a batch of experience from D. f. Calculate the target Q value: (if It is not in a terminated state, otherwise g. Calculate the loss: (Huber loss is more robust to outliers). h. Gradient update: For parameters (mainly spline coefficients) Perform gradient descent updates (and possible grid points): i. Update the target network: Every C steps, let (Hard update) or (Soft update).

[0193] Applications of interpretability analysis in practice: Decision path visualization: For a specific decision It can track which input state components Through spline functions of which shapes It makes a major positive or negative contribution to the Q value of action a.

[0194] "What-if" analysis: Change a dimension of the input state s (e.g., manually increase the "cognitive load" value), observe how the Q value (or policy probability) of the KAN output changes, and understand the reasons by examining the spline function.

[0195] Rule extraction (approximation): For certain input dimensions, if the corresponding spline function exhibits a clear pattern (such as step, saturation, linear segment), heuristic rules such as "IF cognitive load > 0.7 THEN tend to choose intervention X" can be approximately extracted.

[0196] Debugging and Verification: If the RL agent makes counterintuitive decisions, the problem can be located by examining the internal functions of KAN to determine whether it lies in the state representation, reward design, or learning process.

[0197] Computational Costs and Optimization: Training: Training KAN may be slightly slower than training an MLP with the same number of parameters because the computation of spline functions and grid updates can be more complex. However, its better sample efficiency may compensate for this.

[0198] Reasoning: KAN's reasoning speed is usually very fast because it mainly involves table lookup (B-spline basis function values) and linear combination.

[0199] Pruning and Sparsification: The KAN paper mentions that unimportant spline functions or nodes can be removed through regularization and pruning techniques, which can further simplify the model and improve efficiency and interpretability.

[0200] User Profile By directly incorporating state inputs into the KAN, the policies (or value assessments) learned by the RL agent are inherently user-specific. The interpretability of KANs can reveal which features in the user profile (such as prior preferences for a certain type of intervention, specific stages of the learning curve) influence decision-making.

[0201] Dynamic adaptability: As user interaction data accumulates, the RL agent (KAN network parameters) is continuously updated, and the intervention strategy also evolves to better adapt to long-term changes and needs of users.

[0202] The Practice of Trustworthy AI: In human-computer interaction, especially in scenarios involving proactive intervention, the understanding and trust of users and designers in the system's decisions are crucial. KAN-enhanced RL provides a powerful tool for this purpose, making "why the system chooses this intervention" no longer a black box question.

[0203] Step 4: The multimodal output and execution module constructs structured intervention information based on the intervention strategy and user preferences, and converts the structured intervention information into multi-source output information.

[0204] The multimodal output and execution module transforms the optimal intervention strategy into a user-perceptible and interactive multimodal output and presents it to the user. As an execution layer, it ensures that intervention information is delivered in the most appropriate, natural, and minimally intrusive way, and adaptively adjusts according to the user's current device environment and preferences.

[0205] The multimodal output and execution module includes an intervention strategy parsing and distribution unit, a UI / UX adaptation module, a natural language generation (NLG) module, a text-to-speech (TTS) module, and a visual / auditory cue generation module. (1) Intervention Strategy Parsing and Dispatch Unit Function: As a data input port, it is responsible for receiving and parsing the output intervention strategy. Based on the strategy type and parameters, as well as the acquired user preferences, it determines which subsequent sub-modules to call and how to configure them.

[0206] (2) UI / UX Adaptive Module Function: Dynamically adjust the application's user interface (UI) and user experience (UX) based on intervention strategies and user preferences.

[0207] Information density adjustment: If the intervention is to "reduce cognitive load", this module can reduce the amount of information displayed on the interface at the same time, and hide secondary function buttons or information panels.

[0208] Content highlighting / guidance: Based on the intervention strategy, highlight key information related to the current task or suggest buttons for the next step.

[0209] Font and color adjustments: For example, use softer tones and more legible fonts when relaxation is needed; use striking colors when a warning is required.

[0210] Interaction mode adjustment: Simplify the operation process: For tasks that need to be completed quickly or when users are under high load, provide a more direct operation path and reduce the number of clicks.

[0211] Provide auxiliary input: For example, provide voice input or easier-to-use virtual buttons when the user may be fatigued.

[0212] Notifications and feedback presentation: Non-intrusive notifications: Messages are delivered using Toast notifications, status bar icon changes, micro-animations, etc., to avoid forcibly interrupting the user.

[0213] Modal dialog boxes: Use modal dialog boxes for interventions that require explicit user confirmation or input (such as "Start a 5-minute relaxation exercise?").

[0214] (3) Natural Language Generation (NLG) Module Function: Transforms the output of structured intervention information or intent into natural, fluent, personalized, and empathetic human language text.

[0215] By leveraging pre-trained language models (such as lightweight versions of the GPT series, T5, or BART, or models fine-tuned for specific domains), more context-aware and emotionally resonant text can be generated based on the input intervention objective, user status, and user profile. For example, if the intervention strategy is "encourage the user," the model could generate: "I've noticed you've put a lot of effort into this task and made great progress! Keep it up!" (4) Text-to-Speech (TTS) Module Function: Synthesizes text information into natural, clear, and expressive human speech.

[0216] Voice selection: Select an appropriate synthesized voice based on user preferences (such as gender, language, and timbre preference) or the emotional goals of the intervention strategy (such as the need for a calm or lively voice).

[0217] Rhythm control: Adjust speech rate, pitch, volume, and pauses to match the emotion and emphasis of the text. For example, encouraging messages can be delivered with a slightly faster and rising intonation, while relaxation guidance can be delivered with a slow and gentle tone.

[0218] (5) Visual / Auditory Cue Generation Module Function: Generate non-textual visual or auditory cues to attract user attention, convey simple status information, or supplement other modal outputs.

[0219] Visual cues: Indicator / icon blinking: For example, when the system detects user distraction, the taskbar icon blinks slightly.

[0220] Color Changes: The application's borders or background subtly change based on the user's state (e.g., green when highly focused, orange when fatigued).

[0221] Micro-animation / graphic cues: Short animated cues (such as a deep breathing animation to guide the user to relax).

[0222] Auditory cues: Earcons / Auditory Icons: Use short, distinctive sounds to indicate specific events or states (such as task completion sounds, error sounds, and warning sounds that require attention).

[0223] Background sound effects / ambient music: Play soft background music to help relax or focus, or play brief alert sounds when needed, depending on the intervention strategy.

[0224] Step 5: Data update.

[0225] Update system data to achieve long-term adaptation and performance improvement. It is responsible for collecting user feedback and system operation data, and using this data to regularly or event-triggered update and optimize the learnable models in the core modules (such as the fusion model in the multimodal fusion inference module, the state prediction and RL decision model in the proactive intervention decision module).

[0226] enter: Multimodal outputs and explicit user feedback on the intervention at the execution layer (e.g., ratings, text comments, selection of "helpful" / "not helpful").

[0227] The empirical tuples $(s, a, R, s')$ of the RL decision module are used for updating the reinforcement learning model.

[0228] Metrics for the accuracy of state understanding in the multimodal fusion inference module (e.g., if there is some form of "true label" or user correction, or through verification with other information).

[0229] Indirect metrics related to user experience and task performance in the system operation log (such as changes in task completion time, changes in error rate, and the frequency with which users abandon a certain function).

[0230] The updated model parameters or complete model file are pushed to the multimodal perception input module, the multimodal fusion inference module, the RWKV-based user future state prediction module, the RL decision module, and the corresponding learnable components in the knowledge base and user profile.

[0231] Feedback Collection and Data Annotation / Preprocessing Unit: Aggregates feedback data from all sources. Cleans, formats, and annotates the raw data as needed (e.g., converts text comments into structured tags using sentiment analysis tools).

[0232] Model Training and Evaluation Engine: Based on the collected data, the machine learning models in the relevant modules are retrained or fine-tuned periodically or when specific conditions are met (such as collecting enough new data or when the model performance drops to a certain threshold).

[0233] Specific tasks: Multimodal Fusion Inference Module Update: Fine-tunes the parameters of the Mamba fusion network and KAN output network using new multimodal data and (potential) state labels. RWKV-Based User Future State Prediction Module Update: Optimizes the parameters of the RWKV prediction model using updated user state sequences to improve the accuracy of future state predictions. RL Agent Update: The core is to continuously optimize the policy network and / or value network of the KAN-enhanced RL agent using new $(s, a, R, s')$ empirical data. This can be offline batch updates or more frequent online fine-tuning (stability must be handled carefully). User Profile Model Update: If the user profile contains learnable dynamic parameters or a small model (such as a model predicting user preferences), it is also updated here.

[0234] Evaluation: Before deploying a new model, evaluate it on the retained validation set or in a simulated environment to ensure that the new model outperforms the old model (e.g., in RL, evaluate cumulative reward; in state recognition, evaluate accuracy / F1 score).

[0235] The Model Version Control and Deployment Unit (MPU) manages different versions of models, ensuring traceability and rollback capabilities. After successful verification, the updated model is securely deployed to the corresponding modules in the production environment.

[0236] The user state intervention method based on multimodal perception and interpretable decision-making in this embodiment has the following advantages: 1. It can proactively identify potential issues such as excessive cognitive load, negative emotions, or task bottlenecks in users, and provide timely, empathetic intervention before these issues escalate. This helps improve users' emotional state (e.g., relieving stress and anxiety, and enhancing positive emotions), reduces cognitive load, and thus improves users' subjective experience and overall well-being in work, study, or specific tasks.

[0237] Through intelligent task assistance, cognitive support, and attention guidance, the system can help users complete tasks more efficiently, reduce errors, improve learning outcomes or work output, and thus enhance overall task performance.

[0238] 2. By continuously collecting and analyzing user data, a dynamic and multi-dimensional user profile is constructed, enabling the system to deeply understand the user's personality traits, behavioral patterns, preferences, and current baseline status.

[0239] The KAN reinforcement learning decision engine selects the most suitable intervention for each individual user from a configurable knowledge base of intervention strategies, based on accurate user profiles, current state, and future state predictions. This ensures the relevance and effectiveness of the intervention, avoiding a "one-size-fits-all" approach.

[0240] The continuous learning mechanism enables the system to learn from user feedback and interaction history and continuously optimize its internal models (such as state understanding models and intervention decision models), thereby adapting to user changes and growth in the long term and providing increasingly accurate and effective support.

[0241] 3. The Kolmogorov-Arnold Networks (KANs) structure employed enhances the interpretability of complex fusion and decision-making processes. Users or system administrators can understand how specific input features (such as changes in a physiological signal or a dimension of a user profile) affect the system's judgment of the user's state and the selection of intervention strategies.

[0242] 4. By fusing multimodal inputs through advanced sequence models such as Mamba and combining them with KAN for feature weighting and state mapping, a fast, accurate and comprehensive understanding of the user's cognitive-emotional state is achieved.

[0243] The RWKV-based future state prediction module endows the system with "foresight," enabling it not only to respond to the current state but also to predict the future trajectory of the user's state, thereby achieving more proactive and forward-looking intervention and preventing problems before they occur.

[0244] It should be noted that, in the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another device, or some features may be ignored or not executed. The modules described as separate components may or may not be physically separated. The components shown as modules may be one or more physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs.

[0245] Furthermore, in the various embodiments of the present invention, the modules can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The integrated unit described above can be implemented in hardware or as a software functional unit.

[0246] An electronic device provided in this application includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of a user state intervention method based on multimodal perception and interpretable decision-making as described in any of the above embodiments.

[0247] Another electronic device provided in this application embodiment may further include: an input port connected to a processor for transmitting multimodal data collected by an external acquisition device to the processor; a display unit connected to the processor for displaying the processor's processing results to the outside world; and a communication module connected to the processor for enabling communication between the electronic device and the outside world. The display unit may be a display panel, a laser scanning display, etc.; the communication method adopted by the communication module includes, but is not limited to, Mobile High Definition Link (HML), Universal Serial Bus (USB), High Definition Multimedia Interface (HDMI), and wireless connection (including Wi-Fi, Bluetooth, Bluetooth Low Energy, and IEEE 802.11s-based communication technology).

[0248] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the steps of a user state intervention method based on multimodal perception and interpretable decision-making as described in any of the above embodiments.

[0249] For descriptions of relevant parts of the user state intervention system, electronic device, and computer-readable storage medium based on multimodal perception and interpretable decision-making provided in this application, please refer to the detailed description of the corresponding parts in the user state intervention method based on multimodal perception and interpretable decision-making provided in this application, which will not be repeated here. Furthermore, parts of the technical solutions provided in this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.

[0250] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A user state intervention method based on multimodal perception and interpretable decision-making, characterized in that, Includes the following steps: Step 1: Obtain multimodal physiological data sequences. Based on the quantization characteristics of each modal physiological data sequence, obtain the modal feature vector sequence of each modal physiological data sequence. Step 2: After obtaining the deep temporal feature vectors of each modality feature vector sequence and aligning them in the time dimension, all deep temporal feature vectors are concatenated at each time step to obtain a multimodal feature sequence. A selective scanning mechanism is used to fuse the modal features in the multimodal feature sequence to obtain a deep fused feature vector. The selective scanning mechanism achieves spatiotemporal fusion of cross-modal features by dynamically adjusting the weights of each modal feature. Step 3: Based on the deep fusion feature vector and combined with the KAN method, determine the historical time period and the current user state vector sequence; Step 4: Embed the historical time period user state vector sequence into the current state vector sequence to obtain the input sequence. Based on the input sequence and combined with the RWKV network, predict the state vector sequence for future time steps. Determine the user state based on the state vector sequence. Obtain the corresponding state profile information and intervention strategy based on the user state. Determine the optimal state profile information based on the user state information. Determine the optimal intervention strategy in the intervention strategy based on the optimal state profile information.

2. The user state intervention method based on multimodal perception and interpretable decision-making according to claim 1, characterized in that, Step 1 involves acquiring multimodal physiological data sequences. Based on the quantization characteristics of each modal physiological data sequence, the modal feature vector sequence for each modal physiological data sequence is obtained, including: Multimodal physiological signals are acquired and aligned with timestamps to obtain the quantization features of each modal physiological data sequence. These quantization features are used to characterize the user's state. The quantization features of each sequence are then normalized to obtain the modal feature vector sequence corresponding to each sequence.

3. The user state intervention method based on multimodal perception and interpretable decision-making according to claim 1, characterized in that, Step 2 involves obtaining the depth temporal feature vectors of each modality feature vector sequence and aligning them along the time dimension. Then, at each time step, all depth temporal feature vectors are concatenated to obtain a multimodal feature sequence, including: Feature normalization is performed on the feature vector sequences of each mode to eliminate dimensional differences; At least one feature vector is generated in each time window based on the modal feature vector sequence, forming a temporally aligned sequence of modal feature vectors; Linear projection is performed on the feature vector sequences of each mode to obtain the dimension-reduced feature vector sequences of each mode; A single-modal Mamba encoder is used to obtain the depth-series temporal feature vectors of each modality feature vector sequence after dimensionality reduction; At each time step, the depth temporal feature vectors of all modalities are concatenated to obtain a multimodal feature sequence.

4. The user state intervention method based on multimodal perception and interpretable decision-making according to claim 3, characterized in that, Step 2 also includes: User profile information is obtained from user data. The user profile information is concatenated with multimodal feature sequences using a Mamba encoder. A deep fusion feature vector is generated based on the concatenated multimodal feature sequences.

5. A user state intervention method based on multimodal perception and interpretable decision-making according to claim 1, characterized in that, Step 3, which involves determining the historical time period and the current user state vector sequence based on the deep fusion feature vector and the KAN method, includes: The deep fusion feature vector is input into the KAN model. The deep fusion feature vector passes through multiple KAN layers. The feature components input to each KAN layer are transformed by a learnable spline function connecting all edges. The transformed feature components are weighted and summed at the nodes of the next KAN layer. The final KAN layer outputs a sequence of historical time periods and the current user state vector through an activation function.

6. The user state intervention method based on multimodal perception and interpretable decision-making according to claim 1, characterized in that, Step 4 involves embedding the historical time period user state vector sequence into the current state vector sequence to obtain the input sequence. Based on the input sequence and the RWKV network, the state vector sequence for future time steps is predicted. The user state is then determined based on the state vector sequence, including: The timestamp corresponding to each state vector in the historical time period user state vector sequence is embedded and concatenated with the current state vector sequence to obtain the input sequence; The RWKV blocks of each layer of the RWKV network obtain the temporal dependencies of the input sequence through Time-mixing, then obtain the output sequence through feature interaction through Channel-mixing, and use MLP decoding to decode the output sequence to obtain the state vector sequence, and determine the user state based on the state vector sequence.

7. The user state intervention method based on multimodal perception and interpretable decision-making according to claim 1, characterized in that, After determining the optimal intervention strategy based on the optimal state profile information in step 4, the following steps are also included: Structured intervention information is constructed based on intervention strategies and user preferences, and then transformed into multi-source output information.

8. A user state intervention system based on multimodal perception and interpretable decision-making, characterized in that, include: A module is used to acquire multimodal physiological data sequences. Based on the quantization characteristics of each modal physiological data sequence, the modal feature vector sequence of each modal physiological data sequence is obtained. The fusion module is used to obtain the deep temporal feature vectors of each modality feature vector sequence and align them in the time dimension. At each time step, all deep temporal feature vectors are concatenated to obtain a multimodal feature sequence. A selective scanning mechanism is used to fuse the modal features in the multimodal feature sequence to obtain a deep fused feature vector. The selective scanning mechanism achieves spatiotemporal fusion of cross-modal features by dynamically adjusting the weights of each modality feature. The state module is used to determine the historical time period and the current user state vector sequence based on the deep fusion feature vector and the KAN method. The intervention module is used to embed the user state vector sequence of historical time periods into the current state vector sequence to obtain the input sequence, predict the state vector sequence of future time steps based on the input sequence and combined with the RWKV network, determine the user state based on the state vector sequence, obtain the corresponding state profile information and intervention strategy based on the user state, determine the optimal state profile information based on the user information, and determine the optimal intervention strategy in the intervention strategy based on the optimal state profile information.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the user state intervention method based on multimodal perception and interpretable decision-making as described in any one of claims 1-8 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the user state intervention method based on multimodal perception and interpretable decision-making as described in any one of claims 1-8.

Images