A multimodal perceptual intelligent interaction feedback system

Abstract

This invention relates to the field of multimodal interaction technology and discloses a multimodal perception-based intelligent interactive feedback system. The system includes generating a multimodal data slice sequence with precisely aligned timestamps by synchronously recording and time-stamping speech, facial images, and vital signs; extracting text, facial expressions, gaze patterns, and physiological feature sequences through parsing, and jointly analyzing these sequences using an inference engine to obtain state results; and querying a knowledge graph to generate a composite feedback strategy containing content identifiers, rhythm parameters, and channel instructions. This method improves the reliability of state inference through precise temporal alignment of multimodal data; and through the collaborative generation of the composite strategy, it achieves adaptive and unified control of feedback content, rhythm, and channels, thereby enhancing the naturalness of the interaction and the overall system response performance.

Description

Technical Field

[0001] This invention relates to the field of multimodal interaction technology, specifically to a multimodal sensing intelligent interactive feedback system. Background Technology

[0002] Current intelligent interaction systems are evolving towards multimodal perception, aiming to understand user states by integrating signals from multiple sources such as speech, vision, and physiological data. Existing technologies typically acquire data from each modality in a parallel but independent manner, followed by later fusion and simple joint judgment in feature extraction or decision-making stages. The system's feedback generation mainly relies on text-based dialogue management or triggering mechanisms based on single-modal recognition results.

[0003] Existing solutions have limitations. Because the clocks and sampling start points of each sensor differ, the raw data stream lacks a unified time stamp, making it impossible to accurately correlate the time relationships between signals of different modes. This timing inconsistency makes it difficult to establish reliable cross-modal causal relationships in subsequent analysis, directly reducing the reliability of user state inference. Furthermore, system feedback behavior often exhibits isolated and singular characteristics. Feedback content, interaction rhythm, and multi-channel output are typically controlled by independent modules, lacking a unified top-level mechanism based on a comprehensive understanding of the user's state, resulting in rigid and insufficiently adaptable feedback.

[0004] It is necessary to address the time synchronization issue of multi-source heterogeneous data at the acquisition and processing front-end to support reliable joint analysis. Simultaneously, it is necessary to solve the problem of mapping the state from comprehensive inference to a coordinated, consistent, multi-dimensional, adaptive interaction strategy to improve the overall efficiency and naturalness of system feedback. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal sensing intelligent interactive feedback system to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides a multimodal sensing intelligent interactive feedback system, the system comprising: The data synchronization and recording module receives and synchronously records the real-time voice waveforms, user facial image streams, and user vital sign sensor signals collected during the interaction period, forming the original synchronous interactive data stream. The data alignment and slicing module performs time stamp unification and data slicing processing on the original synchronous interactive data stream to generate a multimodal data slice sequence with aligned timestamps; The hierarchical feature parsing module performs hierarchical parsing processing on the multimodal data slice sequence to extract pure speech segments and their text symbol sequences, facial expression change coding sequences, gaze placement sequences, and physiological state indicator sequences. The interaction state inference module inputs the text symbol sequence, facial expression change encoding sequence, gaze point sequence, and physiological state index sequence into the interaction state inference engine for joint analysis, and the interaction state inference engine outputs the state inference result for the current interaction period. The feedback strategy generation module, based on the state inference result, starts the strategy generator. The strategy generator queries a preset interaction rule knowledge graph based on the state inference result to generate a composite feedback strategy that includes feedback content template identifiers, interaction rhythm adjustment parameters, and perception channel enhancement instructions.

[0007] Preferably, the hierarchical parsing process for the multimodal data slice sequence includes: Each multimodal data slice contains a slice of speech waveform, a slice of facial image, and a slice of vital signs within the same time segment; Layered parsing processing includes: noise filtering and endpoint detection of speech waveform slices, extraction of clean speech segments and conversion into text symbol sequences; Micro-expression unit segmentation and eye-tracking are performed on facial image slices to generate expression change coding sequences and gaze placement sequences; Baseline calibration and feature point extraction are performed on the vital sign signal slices to generate a physiological state index sequence.

[0008] Preferably, the process of performing noise filtering and endpoint detection on the speech waveform slices, extracting clean speech segments, and converting them into text symbol sequences includes: An adaptive spectral subtraction algorithm is applied to the speech waveform slice to suppress environmental steady-state noise, resulting in a preliminary denoised speech waveform; The initial denoised speech waveform is subjected to dual endpoint detection based on energy and zero-crossing rate to mark the start and end time points of the effective speech segment, and the clean speech segment is extracted. The clean speech segment is input into a streaming speech recognition model, which outputs the corresponding text symbol sequence in an incremental manner. Each symbol in the text symbol sequence is associated with a timestamp and a confidence level. The text symbol sequence is subjected to context correction and domain terminology normalization processing, which includes: correcting incorrectly identified symbols based on the co-occurrence probability of preceding and following symbols; mapping synonymous domain terms to standard terms to form a standardized text symbol sequence.

[0009] Preferably, the process of performing micro-expression unit segmentation and eye-tracking on facial image slices to generate expression change coding sequences and gaze point sequences includes: A facial key point grid is located on the facial image slice, and the facial motion unit intensity value is calculated based on the motion displacement of the facial key point grid to generate an initial expression code; The initial facial expression code is subjected to temporal smoothing filtering and discrete event detection to identify micro-expression events with intensity exceeding a threshold and lasting for a certain duration. Each micro-expression event is encoded as an entry in the facial expression change encoding sequence containing type, intensity, and start and end time. Simultaneously, the pupil center position is tracked on the facial image slice and mapped to a predefined screen area coordinate system to generate the original gaze coordinate sequence; Cluster analysis and fixation point detection are performed on the original gaze coordinate sequence to aggregate the continuous gaze coordinates into the gaze focus sequence, each gaze focus containing the center coordinates and gaze duration.

[0010] Preferably, the process of baseline calibration and feature point extraction of the vital sign signal slices to generate a physiological state index sequence includes: The original physiological signals in the vital sign signal slices are subjected to power frequency interference filtering and baseline drift correction to obtain preprocessed physiological signal waveforms. From the preprocessed physiological signal waveforms, the heart rate interval, skin conductance level, and pulse wave conduction time are periodically detected to form the original physiological indicator time series. The original physiological index time series is compared with the baseline reference value of the user at rest, and the normalized offset is calculated. The normalized offset is statistically feature-extracted based on a sliding time window. The extracted features include mean, variance, and trend slope, forming the physiological state index sequence.

[0011] Preferably, the step of inputting the text symbol sequence, facial expression change encoding sequence, gaze focus sequence, and physiological state indicator sequence into the interaction state inference engine for joint analysis includes: A multimodal temporal fusion network is constructed, in which the text symbol sequence, facial expression change encoding sequence, gaze point sequence, and physiological state index sequence are embedded into a unified feature space to generate corresponding modal feature temporal sequences. Within the multimodal temporal fusion network, a cross-modal attention mechanism is designed to calculate the correlation weight between the feature vectors of any two different modal feature time series at the same time. Using the aforementioned correlation weights, feature enhancement guided by features from other modalities is performed on the temporal features of each modality to generate enhanced multimodal fusion temporal features; The enhanced multimodal fusion temporal features are input into a bidirectional recurrent neural network for temporal context encoding to obtain a hidden state sequence containing complete interaction time context information; The last state of the hidden state sequence is input into a fully connected classification layer, which outputs the state inference result. The state inference result is a multi-dimensional vector, with different dimensions corresponding to different cognitive load levels and intent clarity levels. The state inference result is used to characterize the user's cognitive load level and intent clarity level.

[0012] Preferably, within the multimodal temporal fusion network, a cross-modal attention mechanism is designed to calculate the correlation weights between feature vectors of any two different modal feature time series at the same time, including: For a given feature vector of a target modal feature time series at time t, calculate the dot product similarity between it and the feature vectors of all other modal feature time series at the same time t; The dot product similarity is input into a soft maximization function for normalization to obtain a set of attention weights, which represent the importance distribution of other modal features to the target modal feature at time t; The attention weights are used to perform a weighted summation of the feature vectors of other modal features at time t to obtain a context vector; The context vector is concatenated with the feature vector of the target modality at time t, and then an enhanced feature vector of the target modality at time t is generated by passing it through a learnable linear transformation layer. Repeat the process for each modality and each time step to generate the enhanced multimodal fusion temporal features for all modalities at all times.

[0013] Preferably, based on the state inference result, a strategy generator is activated. The strategy generator queries a preset interaction rule knowledge graph based on the state inference result to generate a composite feedback strategy that includes feedback content template identifiers, interaction rhythm adjustment parameters, and perception channel enhancement instructions, including: The cognitive load level dimension and intent clarity level dimension in the state inference results are analyzed and mapped to knowledge graph query conditions. Using the knowledge graph query conditions as input, the nodes and edges in the preset interaction rule knowledge graph are traversed. The nodes in the interaction rule knowledge graph represent the interaction state, and the edges represent the state transition conditions and the feedback actions that can be taken. In the interaction rule knowledge graph, find one or more target state nodes whose matching degree with the query conditions of the knowledge graph exceeds a preset threshold; Obtain all outgoing edges associated with the one or more target state nodes, with each outgoing edge accompanied by a description of the feedback action and the confidence level of the expected effect; Based on the current interaction context and system resource status, select a set of compatible and executable feedback actions from all outgoing edge-attached feedback actions to form a basic action set; The basic action set is instantiated with parameters, and each feedback action is assigned specific execution parameters, including content template identifier, rhythm adjustment parameters and channel enhancement instructions, thereby instantiating and generating the composite feedback strategy.

[0014] Preferably, the step of instantiating the basic action set into parameters and assigning specific execution parameters to each feedback action includes: For each feedback action in the basic action set, the historical interaction log library is searched to find historical records in which the feedback action was successfully executed under similar state inference results; Extract the specific parameter values ​​used in the feedback action from the historical records, including the specific content template identifiers used, the specific numerical range of the rhythm adjustment parameters, and the specific code of the channel enhancement command; Analyze the correlation between the extracted specific parameter values ​​and the corresponding historical interaction performance scores, and establish a parameter-performance mapping model; The contextual features of the current interaction are input into the parameter-effect mapping model to predict the expected effects that different parameter values ​​may bring. Select the parameter value that yields the highest expected results as the execution parameter currently assigned to the feedback action.

[0015] Preferably, the step of selecting a set of compatible and executable feedback actions from all outgoing edge-related feedback actions based on the current interaction context and system resource status to form a basic action set includes: Obtain the real-time resource status of the current interactive system, including available computing resources, available output channel status, and the length of the queue of pending interactive tasks. Assess the real-time resource requirements of the feedback actions associated with each outgoing edge, including computational overhead, output channel occupancy time, and execution latency; Check the logical compatibility and timing conflict possibility between different feedback actions within the basic action set; Establish a multi-constraint optimization model with the goal of maximizing the overall expected utility of the set of feedback actions, and with system resource constraints and compatibility between actions as constraints. Solve the multi-constraint optimization model to obtain the feedback action combination with the highest expected utility under all constraints, and determine the feedback action combination as the basic action set.

[0016] Compared with the prior art, the beneficial effects of the present invention are: By uniformly time-stamping the original synchronized data stream and assigning precise alignment timestamps to the segmented data packets, a multimodal data slice sequence is constructed. This technique eliminates inherent signal time differences caused by hardware variations, ensuring strict temporal coherence among audio, video frames, and sensor data within the same time slice. The extracted feature sequences thus possess inherent millisecond-level temporal alignment relationships. The state inference engine can perform reasoning based on precise temporal patterns such as "specific audio segments, corresponding facial micro-expressions, and synchronized physiological signal peaks," fusing discrete multimodal evidence into a high-confidence user state description, thus improving the reliability and inference accuracy of joint analysis.

[0017] The interaction state inference engine performs joint analysis of text, facial expressions, gaze patterns, and physiological indicator sequences, outputting a comprehensive state inference result. The strategy generator then queries a pre-defined interaction rule knowledge graph, which encodes a structured mapping from complex states to multi-dimensional feedback actions. This process generates a composite strategy that simultaneously includes feedback content identifiers, interaction rhythm adjustment parameters, and perceptual channel enhancement instructions. The system can then execute actions that are consistent in content, timing, and presentation. All dimensions of feedback are generated based on unified state decisions, forming an organic whole that coordinates in time and space, enhancing the system's adaptability, naturalness, and guidance efficiency. Attached Figure Description

[0018] Figure 1 This is a timing diagram of the multimodal sensing intelligent interactive feedback system described in this invention. Figure 2 A flowchart for speech processing and text symbol sequence generation; Figure 3 A flowchart for facial image processing and expression / gaze sequence generation; Figure 4 A bar chart showing the similarity comparison before and after multimodal feature enhancement; Figure 5 A heatmap for predicting the expected results of the combination of action parameters. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see Figure 1This invention provides a multimodal perception intelligent interactive feedback system. The system includes: a data synchronization recording module that receives real-time speech waveforms, user facial image streams, and user vital sign sensor signals collected during the interaction period, and synchronously records these three types of data to form a time-stricken original synchronous interactive data stream. This data stream is then sent to a data alignment and slicing module, which performs time-stamping and data slicing processing on the original synchronous interactive data stream. Based on a unified time reference, the continuous stream data is cut into time segments of equal or unequal length, generating a multimodal data slice sequence with aligned timestamps. A hierarchical feature parsing module receives this sequence and performs hierarchical parsing processing on each slice. The processing objective is to extract from each slice a clean speech segment and its corresponding text symbol sequence, an expression change encoding sequence describing subtle changes in the user's face, a gaze placement sequence reflecting the user's attention, and a physiological state indicator sequence representing the user's physiological state. These extracted sequences are then input into the interaction state inference engine within the interaction state inference module for joint analysis. This engine comprehensively evaluates the multimodal information through internal algorithms and outputs a state inference result representing the user's state during the current interaction period. Based on this state inference result, the feedback strategy generation module starts its internal strategy generator. Based on the obtained state inference result, the strategy generator queries a preset interaction rule knowledge graph that contains the relationship between various interaction states and feedback rules, and generates a structured composite feedback strategy. This strategy explicitly includes the feedback content template identifier to be used, the interaction rhythm parameters that need to be adjusted, and the specific perception channel instructions that need to be enhanced.

[0021] Example 1: See Figure 2 The data alignment and slicing module processes the original synchronous interactive data stream to ensure that each generated multimodal data slice contains speech waveform slices, facial image slices, and vital sign signal slices within the same time segment. The hierarchical feature parsing module performs hierarchical parsing processing on each multimodal data slice. The processing is performed on the speech waveform slices. First, an adaptive spectral subtraction algorithm is applied to the speech waveform slices to suppress environmental steady-state noise, resulting in a preliminary denoised speech waveform. Next, dual endpoint detection based on energy and zero-crossing rate is performed on the preliminary denoised speech waveform to mark the start and end times of effective speech segments, and clean speech segments are extracted from the original slices based on these times. The clean speech segments are then input into a streaming speech recognition model, which outputs a corresponding text symbol sequence incrementally. Each symbol in the text symbol sequence is associated with a timestamp and confidence information. Finally, context correction and domain terminology normalization are performed on the text symbol sequence. This processing includes correcting incorrectly recognized symbols based on the co-occurrence probability of preceding and following symbols, and mapping synonymous domain terms to standard terms, ultimately forming a standardized text symbol sequence.

[0022] In practical implementation, the hierarchical feature parsing module's processing of multimodal data slice sequences, particularly the noise filtering, endpoint detection, speech recognition, and text post-processing of speech waveform slices, can be described in detail through a scenario with specific data examples. The scenario is set as the system processing an online educational question-and-answer interaction. The data synchronization recording module simultaneously collects a segment of the student's speech, facial image, and heart rate sensor signal, forming the original synchronous interactive data stream. The data alignment slicing module slices this original synchronous interactive data stream at fixed intervals of 100 milliseconds. Each multimodal data slice contains a speech waveform slice, facial image slice, and vital sign signal slice within a 100-millisecond time window, and has a unified alignment timestamp. The hierarchical feature parsing module processes one of the speech waveform slices, corresponding to the time segment when the student utters the sentence "The area formula of a triangle is 1 / 2 base multiplied by the height." The speech waveform slice contains a mixture of the student's speech signal and background noise in the classroom.

[0023] In specific implementations, an adaptive spectral subtraction algorithm is applied to the speech waveform slice to suppress environmental steady-state noise. The adaptive spectral subtraction algorithm estimates the noise power spectrum from the initial portion of the speech waveform slice and then subtracts the estimated noise power spectrum from the overall power spectrum of the speech waveform slice. After processing by the adaptive spectral subtraction algorithm, the spectral components of the continuous air conditioner hum in the background environment are significantly suppressed, resulting in a preliminary denoised speech waveform with cleaner waveform amplitude and more prominent speech components. In some embodiments, the noise estimation of the adaptive spectral subtraction algorithm is based on a 50-millisecond non-speech segment at the beginning of the speech waveform slice. It is assumed that this segment of the signal is mainly environmental steady-state noise, and the noise power spectrum is estimated by calculating the mean value of the spectrum of this segment of the signal.

[0024] A dual endpoint detection method based on energy and zero-crossing rate is used to analyze the initial denoised speech waveform. The endpoint detection process calculates the short-time energy and short-time average zero-crossing rate of the initial denoised speech waveform within each short frame. When the short-time energy value of multiple consecutive short frames exceeds a preset energy threshold, and the short-time average zero-crossing rate is lower than a preset zero-crossing rate threshold, it is determined as the start point of a valid speech segment. When the short-time energy value of multiple consecutive short frames falls below the energy threshold, it is determined as the end point of a valid speech segment. Using this dual criterion, the start and end times of the valid speech segment "1 / 2 base multiplied by height" can be accurately marked within the slice. Based on these start and end times, a clean speech segment containing only pure speech signal is extracted from the original speech waveform slice, eliminating the silent section at the beginning and the invalid noise at the end.

[0025] The extracted clean speech segment is input into a streaming speech recognition model, which incrementally recognizes the input audio stream. In practice, the streaming speech recognition model outputs an intermediate recognition result every 50 milliseconds of clean speech segment audio data it receives and processes. For the speech segment "1 / 2 base multiplied by height", the streaming speech recognition model may output incomplete intermediate text symbol sequences sequentially, and finally output the complete text symbol sequence "1 / 2 base multiplied by height" after processing the entire clean speech segment. Each text symbol in the text symbol sequence is associated with its recognition timestamp and recognition confidence level. It can be understood that the dual endpoint detection of energy and zero-crossing rate sets two thresholds: the energy threshold is used to distinguish between speech and silence, and the zero-crossing rate threshold is used to distinguish between unvoiced and voiced sounds. Only when the signal simultaneously satisfies the condition that the energy exceeds the energy threshold and the zero-crossing rate is lower than the zero-crossing rate threshold is it determined to be a valid speech segment.

[0026] Contextual error correction and domain terminology normalization are performed on the text symbol sequence. Contextual error correction corrects incorrectly identified symbols based on the co-occurrence probability of preceding and following symbols. For example, a streaming speech recognition model might misidentify "base" as "low" due to unclear pronunciation. However, in the context of the sequence "1 / 2 low multiplied by high," the co-occurrence probability of "low multiplied by high" is much lower than that of "base multiplied by high," so the error correction process will correct "low" to "base." Domain terminology normalization maps synonymous domain terms to standard terms. Therefore, the normalization process will replace "multiply" in the text symbol sequence with "multiply by," forming the final standardized text symbol sequence "1 / 2 base multiplied by high." In some embodiments, when processing clean speech segments, the streaming speech recognition model performs a recognition every 20 milliseconds of accumulated speech data, outputs intermediate results, and updates previous outputs until the entire clean speech segment is processed, forming the final text symbol sequence. It is understandable that context correction uses a pre-trained language model, which is trained on a large-scale text corpus and can calculate the probability of any sequence of symbols appearing in the language. When a low-probability combination of symbols appears in the text sequence, the language model will suggest replacing it with a higher-probability combination to achieve error correction.

[0027] In practical implementation, the core operation of the adaptive spectral subtraction algorithm can be described by a formula. For the processing of the f-th frequency component in a speech waveform slice, the amplitude of the enhanced speech spectrum is... Calculated using the following formula: Where: symbol The spectral amplitude of a noisy speech waveform slice at the f-th frequency component is represented by the symbol. This represents the value of the noise power spectrum estimated from the initial silent segment of the speech waveform slice at the f-th frequency component, denoted by the symbol. The over-subtraction factor is a real number greater than 1 used to control the intensity of noise subtraction. Its sign is... The lower limit parameter is a decimal between 0 and 1, used to prevent the processed spectrum amplitude from being too small, thus introducing musical noise. Optional, an over-subtraction factor. The value is dynamically adjusted based on the preliminary estimate of the signal-to-noise ratio (SNR) in the speech waveform slice; a value of 3 is used when the SNR is low, and a value of 1.5 is used when the SNR is high. Optional, the lower spectral limit parameter... The value is fixed at 0.01, which means the enhanced speech spectrum amplitude At least retain the original noisy speech spectrum amplitude. 1%.

[0028] Example 2: See Figure 3 The hierarchical feature parsing module performs hierarchical parsing processing on each multimodal data slice. In the facial image processing workflow, facial key point grids are located on the facial image slices, and the intensity values ​​of facial action units are calculated based on the motion displacement of the facial key point grids to generate initial expression codes. Temporal smoothing filtering and discrete event detection are performed on the initial expression codes to identify micro-expression events with intensity exceeding a threshold and lasting for a certain duration. Each micro-expression event is encoded as an entry in an expression change coding sequence containing type, intensity, and start and end times. Simultaneously, the pupil center position is tracked on the facial image slices and mapped to a predefined screen area coordinate system to generate the original gaze coordinate sequence. Cluster analysis and fixation point detection are performed on the original gaze coordinate sequence to aggregate continuous gaze coordinates into a gaze focus sequence representing the focus of attention. Each gaze focus includes the center coordinates and gaze duration. In the vital sign signal processing workflow, power frequency interference filtering and baseline drift correction are performed on the original physiological signals in the vital sign signal slices to obtain preprocessed physiological signal waveforms. From the preprocessed physiological signal waveforms, heart rate intervals, skin conductance levels, and pulse wave transit time are periodically detected to construct the original physiological indicator time series. The original physiological indicator time series is compared with the user's baseline reference values ​​at rest to calculate the normalized offset. Statistical features are extracted from the normalized offset based on a sliding time window, including the mean, variance, and trend slope, forming a physiological state indicator sequence.

[0029] In practical implementation, the hierarchical feature parsing module processes multimodal data slice sequences, involving micro-expression and eye-tracking analysis of facial image slices and physiological indicator extraction from vital sign signal slices. This can be described in detail through a scenario with specific data examples. The scenario continues the setup of online education question-and-answer interaction. A multimodal data slice generated by the data alignment and slicing module contains facial image slices and vital sign signal slices within the same 100-millisecond time interval. The facial image slice is a continuous sequence of video frames, and the vital sign signal slice is a segment of photoplethysmography (PPG) wave signal and skin conductance signal from a wrist sensor.

[0030] In practice, the process of segmenting facial image slices into micro-expression units and tracking eye movements begins by locating a facial keypoint grid on each frame of the facial image slice. The facial keypoint grid contains the coordinates of dozens of keypoints for features such as eyebrows, eyes, nose, and lips. The intensity value of facial action units is calculated based on the motion displacement of the facial keypoint grid. For example, by calculating the vertical distance change between the inner and outer keypoints of the left eyebrow, the intensity of the "inner eyebrow raising" action unit can be quantified. Based on the displacement changes of all relevant keypoints, intensity values ​​are calculated for multiple facial action units, generating an initial expression encoding vector containing the intensity values ​​of multiple facial action units. In practice, the initial expression encoding vector undergoes temporal smoothing filtering and discrete event detection. Temporal smoothing filtering uses a moving average method to eliminate interference from minor muscle tremors, while discrete event detection sets an intensity threshold and a duration threshold. When the intensity value of a facial action unit continuously exceeds the intensity threshold and reaches the duration threshold, a micro-expression event is identified. Each micro-expression event is encoded as an entry containing type, intensity, and start and end times, and added to the expression change encoding sequence.

[0031] The pupil center position is tracked on facial image slices, and the pixel coordinates of the pupil center in the image coordinate system are determined by identifying the boundary between the iris and sclera in each frame. The pupil center pixel coordinates are then mapped to a predefined screen area coordinate system. The mapping process uses a pre-calibrated mapping function to convert the pupil position in the camera image into corresponding two-dimensional coordinates on the computer screen, generating a raw gaze coordinate sequence arranged in chronological order. In some embodiments, the mapping function is a polynomial transformation model, calibrated by having the user gaze at several known points on the screen. In a specific implementation, cluster analysis and fixation point detection are performed on the raw gaze coordinate sequence. The cluster analysis uses a density-based spatial clustering method to aggregate gaze coordinate points that are spatially and temporally close on the screen coordinate plane into a cluster. Fixation point detection sets a minimum gaze duration threshold; when the time span corresponding to all points within a coordinate cluster exceeds the minimum gaze duration threshold, the cluster is considered a valid gaze fixation point. Each gaze fixation point is described by the center coordinates and gaze duration of all coordinate points within its cluster, forming a gaze landing point sequence. It is understandable that density-based spatial clustering methods can effectively distinguish between rapidly moving coordinate points during saccades and stable clustered coordinate points during fixation.

[0032] In some embodiments, when performing micro-expression unit segmentation on facial image slices, the formula for calculating the facial motion unit intensity value is as follows: Where: symbol Indicates the intensity value of a specific facial action unit, symbol Indicates the number of keypoint pairs involved in calculating this facial action unit, symbol [symbol missing]. This represents the weight coefficient of the i-th keypoint pair, with the sign... This represents the normalized distance change of the i-th keypoint pair between the current frame and the neutral expression reference frame. Optional, weighting coefficients. Trained on a dataset labeled with facial motion unit intensities using machine learning methods. Optional, distance variation. The result is obtained by calculating the difference between the Euclidean distance of the keypoint pair in the current frame and the Euclidean distance of the corresponding keypoint pair in the reference frame, and then normalizing it by dividing by the pupil distance of the reference frame.

[0033] In practice, baseline calibration and feature point extraction are performed on the vital sign signal slices. Each slice contains a 100-millisecond raw photoplethysmography (PPG) pulse wave signal and raw skin conductance signal. Power frequency interference filtering and baseline drift correction are performed on the raw physiological signals in the slices. A 50Hz notch filter is used for PPG interference filtering, and a high-pass filter is used for baseline drift correction to remove slowly changing components with frequencies below 0.5Hz, resulting in a preprocessed physiological signal waveform. From the preprocessed physiological signal waveform, feature points are periodically detected. For the PPG pulse wave signal, the peak position of each pulse wave is detected, and the time interval between consecutive peaks is calculated as the heart rate interval. For the skin conductance signal, the average value of the waveform within a 100-millisecond time window is calculated as the skin conductance level. For two consecutive pulse waves, the time difference between specific feature points on their waveforms is calculated as an estimate of the pulse wave conductance time. Through these detections, a raw physiological index time series containing the heart rate interval, skin conductance level, and pulse wave conductance time estimate is constructed.

[0034] The raw physiological indicator time series is compared with the baseline reference value of the user at rest, and the normalized offset is calculated. For example, if the average heart rate interval at rest is 800 milliseconds, and the heart rate interval detected in the current slice is 720 milliseconds, then the normalized offset of the heart rate interval is calculated as (800-720) / 800=0.1. Statistical features of the normalized offset are extracted based on a sliding time window, with a sliding time window length of 1 second and a sliding interval of 100 milliseconds. Within each window, the mean, variance, and trend slope of the normalized offset of the heart rate interval are calculated. The trend slope is obtained by linearly fitting the change of the offset over time within the window. The extracted features include the mean, variance, and trend slope, forming a physiological state indicator sequence. It can be understood that the baseline reference value at rest is obtained by collecting a physiological signal and calculating its average value before the interaction begins, when the user is in a relaxed state. Optionally, the statistical features of the sliding time window also include kurtosis and skewness, used to describe the distribution pattern of physiological indicator changes.

[0035] Example 3: Joint Analysis Process of the Intermodal State Inference Engine. A multimodal temporal fusion network is constructed, embedding text symbol sequences, facial expression change encoding sequences, gaze placement sequences, and physiological state indicator sequences into a unified feature space to generate corresponding modal feature temporal sequences. Within the multimodal temporal fusion network, a cross-modal attention mechanism is designed to calculate the correlation weight between the feature vectors of any two different modal feature temporal sequences at the same time. Using the correlation weight, feature enhancement guided by other modal features is performed on the feature temporal sequences of each modality, generating enhanced multimodal fusion temporal features. The enhanced multimodal fusion temporal features are input into a bidirectional recurrent neural network for temporal context encoding, obtaining a hidden state sequence containing complete interaction time context information. The final state of the hidden state sequence is input into a fully connected classification layer, which outputs a state inference result. The state inference result is a multidimensional vector, with different dimensions corresponding to different cognitive load levels and intent clarity levels. The state inference result is used to characterize the user's cognitive load level and intent clarity level. The cross-modal attention mechanism involves calculating the dot product similarity between a given target modality feature vector at time t and the feature vectors of all other modality features at the same time t. This dot product similarity is normalized using a soft maximization function to obtain a set of attention weights, representing the importance distribution of other modality features to the target modality at time t. The attention weights are then used to weight and sum the feature vectors of other modality features at time t to obtain a context vector. This context vector is concatenated with the target modality feature vector at time t and passed through a learnable linear transformation layer to generate an enhanced feature vector for the target modality at time t. This process is repeated for each modality and at each time point to generate enhanced multimodal fusion temporal features for all modalities at all times.

[0036] In practical implementation, the interaction state inference module executes the joint analysis process, involving the construction and computation of a multimodal temporal fusion network and a cross-modal attention mechanism. This can be described in detail through a scenario with specific data examples. The scenario continues the online education question-and-answer interaction. After a student is asked a relatively complex geometric proof problem, within a 5-second interaction period, the hierarchical feature parsing module outputs the corresponding text symbol sequence, facial expression change encoding sequence, gaze focus sequence, and physiological state indicator sequence. In practical implementation, a multimodal temporal fusion network is constructed, embedding the text symbol sequence, facial expression change encoding sequence, gaze focus sequence, and physiological state indicator sequence into a unified feature space. The embedding process is as follows: each symbol in the text symbol sequence is converted into a dense vector through a word embedding layer; each entry in the facial expression change encoding sequence is encoded into a vector through a fully connected layer; each entry in the gaze focus sequence is encoded into a vector through a fully connected layer; and each statistical feature vector of the physiological state indicator sequence is directly used as input. All vectors are mapped to the same dimension, generating the corresponding text modality feature temporal sequence, facial expression modality feature temporal sequence, gaze modality feature temporal sequence, and physiological modality feature temporal sequence. In some embodiments, the word embedding layer uses a pre-trained word vector model, the fully connected layer encoding facial expressions and gaze has 64 neurons, and the uniform feature space dimension is set to 128.

[0037] Within the multimodal temporal fusion network, a cross-modal attention mechanism is designed to calculate the correlation weights between feature vectors of any two different modal feature time series at the same time. For a given feature vector of the target modal feature time series at time t, the dot product similarity between it and the feature vectors of all other modal feature time series at the same time t is calculated. The dot product similarity is normalized using a soft maximization function to obtain a set of attention weights, representing the importance distribution of other modal features to the target modal feature at time t. The attention weights are then used to perform a weighted summation of the feature vectors of other modal feature time series at time t, resulting in a context vector. This context vector is concatenated with the feature vector of the target modal feature time series at time t, and passed through a learnable linear transformation layer to generate an enhanced feature vector for the target modality at time t. This process is repeated for each modality and each time series to generate enhanced multimodal fused temporal features for all modalities at all times.

[0038] In practice, the feature vector of the text modality at time t is used as the query to calculate the attention weights between it and the feature vectors of the facial expression, gaze, and physiological modalities at time t, thus generating an enhanced feature vector for the text modality. Similarly, the feature vector of the facial expression modality is also used as the query to calculate the attention weights between it and the feature vectors of the text, gaze, and physiological modalities. This cross-modal attention mechanism allows each modality's features to dynamically focus on and absorb complementary information provided by other modalities at the same time during fusion. Optionally, after calculating the dot product similarity, the cross-modal attention mechanism is divided by a scaling factor to stabilize training.

[0039] In some embodiments, the calculation of attention weights from mode A to mode B in a cross-modal attention mechanism can be described by the following formula: Where: symbol This indicates that at time t, the information from mode B is the context vector generated by mode A. (Symbol) This represents the query vector obtained by linearly transforming the eigenvector of mode A at time t. (Symbol) This represents the key vector obtained by linearly transforming the eigenvectors of mode B at time t. (Symbol) This represents the value vector obtained by linearly transforming the eigenvector of mode B at time t. (Symbol) This represents the scaling factor, typically taken as the dimension of the key vector. The square root of. The function operates along the feature vector dimension of all modalities B. Optionally, the linear transformation layer projects the original feature vector of each modality into a query vector, key vector, and value vector using independent weight matrices.

[0040] By leveraging relevance weights, feature enhancement guided by features from other modalities is applied to the temporal sequence of features for each modality, generating enhanced multimodal fusion temporal features. After this process, the text modality feature temporal sequence not only contains linguistic information but also incorporates the confusion signal that a frowning expression might represent at the same time; the facial expression modality feature temporal sequence is associated with the linguistic context when the user says "Let me think." The enhanced multimodal fusion temporal features are then input into a bidirectional recurrent neural network for temporal context encoding. The bidirectional recurrent neural network consists of two long short-term memory networks, forward and backward, which process the fusion feature sequence over the entire 5-second time period from left to right and from right to left, respectively, ultimately obtaining a hidden state sequence containing complete contextual information of the interaction time period. In the specific implementation, the number of hidden layer units in the bidirectional long short-term memory network is set to 256.

[0041] The final state of the hidden state sequence is input into a fully connected classification layer. This layer contains two parallel output heads: one for regressing or classifying cognitive load level, and the other for regressing or classifying intent clarity level. The fully connected classification layer outputs a state inference result, which is a multi-dimensional vector. Different dimensions correspond to different cognitive load levels and intent clarity levels, representing the user's cognitive load level and intent clarity level. For example, in the scenario described, the state inference result might be a two-dimensional vector, where a first dimension of 0.8 represents a high cognitive load level, and a second dimension of 0.3 represents a low intent clarity level. In some embodiments, the fully connected classification layer consists of two independent fully connected layers with 64 neurons each and a final output layer. The activation function is the sigmoid function, mapping the output values ​​to between 0 and 1.

[0042] See Figure 4 This is a bar chart comparing the similarity of features before and after multimodal feature enhancement. The feature similarity of all modalities significantly improved after enhancement, indicating that cross-modal fusion effectively enhanced feature consistency and information complementarity. The physiological modality showed the largest enhancement (from approximately 0.7 to 0.9+), indicating the most significant fusion gain between physiological signals and other modalities. The text modality, already having a high original similarity (0.65), further improved to 0.85+ after enhancement, reflecting the fundamental value of textual information and the optimization potential after fusion. This type of chart is a feature fusion effect evaluation tool for multimodal perception systems, used to verify the effectiveness of cross-modal attention mechanisms and ensure that information from different modalities can effectively complement each other. The improvement in feature similarity is directly related to the accuracy of interaction state inference (such as cognitive load and intent clarity judgment), and is a core reference indicator for system performance optimization.

[0043] Example 4: The workflow of the strategy generator is as follows: First, it parses the cognitive load level dimension and intent clarity level dimension from the state inference results and maps them to knowledge graph query conditions. Using the knowledge graph query conditions as input, it traverses the nodes and edges in a preset interaction rule knowledge graph. Nodes in this knowledge graph represent interaction states, and edges represent state transition conditions and available feedback actions. In the interaction rule knowledge graph, it searches for one or more target state nodes whose matching degree with the knowledge graph query conditions exceeds a preset threshold. It obtains all outgoing edges associated with one or more target state nodes, each edge accompanied by a description of the feedback action and a confidence level of the expected effect. Based on the current interaction context and system resource status, it selects a set of compatible and executable feedback actions from all the feedback actions attached to the outgoing edges to form a basic action set. It then performs parameter instantiation on the basic action set, assigning specific execution parameters to each feedback action. These execution parameters include content template identifiers, rhythm adjustment parameters, and channel enhancement instructions, thereby instantiating and generating a composite feedback strategy.

[0044] In practical implementation, the feedback strategy generation module generates a complete process of composite feedback strategies based on the state inference results. Its core is that the strategy generator queries a preset interaction rule knowledge graph based on the state inference results and generates the strategy's actions. Taking an online education scenario where the interaction state inference engine outputs a multi-dimensional vector [0.8, 0.3] as an example, the first dimension 0.8 corresponds to a high cognitive load level, and the second dimension 0.3 corresponds to a low intent clarity level. The strategy generator begins its work, parsing the cognitive load level dimension and intent clarity level dimension in the state inference results and mapping them to knowledge graph query conditions. Using the knowledge graph query condition "(high load, low clarity)" as input, the strategy generator begins traversing the nodes and edges in the preset interaction rule knowledge graph. The preset interaction rule knowledge graph is a directed graph structure, where nodes represent different interaction states, and edges represent the conditions for transitioning from one state to another and the feedback actions that can be taken under those conditions. In the interaction rule knowledge graph, it searches for one or more target state nodes whose matching degree with the knowledge graph query conditions exceeds a preset threshold. Matching degree calculation can be based on the similarity of state attributes. For example, if there is a node named "high cognitive load and ambiguous intent", its defined load level is "high" and its clarity level is "low", which completely matches the query conditions, then this node is identified as the target state node.

[0045] Retrieve all outgoing edges associated with one or more target state nodes. Each outgoing edge includes a description of the feedback action and a confidence level for the expected effect. For example, starting from the "high cognitive load and ambiguous intent" node, there may be multiple outgoing edges, each pointing to a possible new state and indicating the feedback action required to trigger the state transition. Refer to Table 1 for these feedback actions and their confidence levels.

[0046] Table 1: Feedback Action Table Associated with Target State Nodes Action ID Feedback Action Description Target state / transition conditions Confidence level of expected results A1 Provide step-by-step prompts Shift to "medium load, medium resolution" 0.85 A2 Slow down the pace of voice broadcast Shift to "medium load, low definition" 0.75 A3 Highlight key information about the problem. Shift to "medium load, medium resolution" 0.80 A4 Repeat the current problem Shift to "high load, medium definition" 0.65 Based on the current interaction context and system resource status, a set of compatible and executable feedback actions is selected from all feedback actions attached to outgoing edges to form a basic action set. The current interaction context includes the task type (geometric proof problem), the user (student), and the number of historical interactions. System resource status includes available computing resources, available output channel status, and the length of the queue of pending interaction tasks. The real-time resource requirements of each feedback action attached to an outgoing edge are evaluated, including computational overhead, output channel occupancy time, and execution latency. A multi-constraint optimization model is established to maximize the overall expected utility of the feedback action set, with system resource limitations and inter-action compatibility as constraints. The overall expected utility can be calculated by summing the confidence scores of the expected effects of each action in the selected action set and subtracting the potential conflict losses due to parallel execution. Resource constraints include that the total computational overhead cannot exceed 80% of the available computing resources, and the total voice output channel occupancy time cannot exceed the duration of the next interaction cycle. The multi-constraint optimization model is solved to obtain the feedback action combination with the highest expected utility under all constraints.

[0047] In some embodiments, the objective function of a multi-constraint optimization model can be expressed as: Where: symbol This represents the overall expected utility to be maximized. (Symbol) This represents a subset of actions selected from all candidate actions, i.e., the base set of actions to be determined. (Symbol) Indicates action The utility weighting coefficient, which can be adjusted based on the priority of actions. (Symbol) Indicates action The confidence level of the expected effect. (Symbol) Indicates when the action and actions Penalty items incurred due to logical or resource conflicts when both are selected simultaneously. Summation item. Calculate the total weighted confidence of the selected action, summation term Calculate the total penalty for pairwise conflicts between selected actions. Under resource and compatibility constraints, the model seeks to minimize such conflicts. The subset of actions with the largest value Optional, utility weighting coefficient Based on the urgency of the feedback action, the weight of basic guidance actions is higher than that of enhancement prompt actions.

[0048] Example 5: The process of parameter instantiation for the basic action set includes: for each feedback action in the basic action set, searching the historical interaction log library to find historical records of successfully executing the feedback action under similar state inference results. Extracting the specific parameter values ​​used for the feedback action from the historical records, including the specific content template identifiers used, the specific numerical range of the rhythm adjustment parameters, and the specific code of the channel enhancement instruction. Analyzing the correlation between the extracted specific parameter values ​​and the corresponding historical interaction performance scores to establish a parameter-performance mapping model. Inputting the contextual features of the current interaction into the parameter-performance mapping model to predict the expected performance that different parameter values ​​may bring. Selecting the parameter value with the highest expected performance as the execution parameter assigned to the current feedback action.

[0049] In constructing the basic action set, based on the current interaction context and system resource status, a set of compatible and executable feedback actions is selected from all feedback actions attached to outgoing edges. This process includes obtaining the real-time resource status of the current interaction system, including available computing resources, available output channel status, and the length of the queue of pending interaction tasks. The real-time resource requirements of each feedback action attached to an outgoing edge are evaluated, including computational overhead, output channel occupancy time, and execution latency. The logical compatibility and timing conflict possibilities between different feedback actions within the basic action set are checked. A multi-constraint optimization model is established with the objective of maximizing the overall expected utility of the feedback action set, constrained by system resource limitations and inter-action compatibility. Solving the multi-constraint optimization model yields the feedback action combination with the highest expected utility under all constraints, and this combination is determined as the basic action set.

[0050] In practical implementation, the complete process of parameter instantiation of the basic action set, as well as the specific method for filtering the basic action set based on the current interaction context and system resource status, can be described in detail through a scenario combined with specific data examples. The scenario continues the online education question-and-answer interaction, and the state inference result is a multi-dimensional vector [0.8, 0.3], representing a high cognitive load level and a low intent clarity level. Based on this result, the policy generator selects two feedback actions from the preset interaction rule knowledge graph to form a basic action set. The basic action set includes action A1 "provide step-by-step prompts" and action A3 "highlight key information of the question".

[0051] In practical implementation, the parameter-performance mapping model determines the expected performance for a given combination of parameters. The prediction can be expressed in the following functional form: Where: symbol This represents the expected outcome predicted by the model, and is a real number between 0 and 1. (Symbol) This represents the set of all adjustable parameters for the feedback action, for example, for action A1. ,in Represents the content template identifier, Represents the duration of pauses between steps. (Symbol) This represents the contextual feature vector of the current interaction, including task type, user attributes, and environmental information. (Symbol) This represents all trainable weight parameters of the parameter-performance mapping model. (Symbol) This represents the mapping function defined by the model, whose structure can be a neural network or a gradient boosting tree. The model is trained on historical data, learning the mapping relationship from parameters and context to performance. Optional, performance scoring. These are standardized values ​​derived from a comprehensive evaluation of user satisfaction surveys and task completion efficiency after interaction.

[0052] A multi-constraint optimization model is established to maximize the overall expected utility of the set of feedback actions, with system resource limitations and inter-action compatibility as constraints. The overall expected utility is the sum of the confidence levels of the expected effects of the selected action sets, minus penalties incurred due to resource contention or logical conflicts. System resource constraints include: total CPU usage not exceeding 80% of available CPU computing power, the voice output channel being occupied by only one action at a time, and the graphics display channel occupancy not exceeding 90%. Inter-action compatibility constraints are based on predefined mutual exclusion rules; for example, "providing step-by-step hints" and "repeating the current problem" are logically mutually exclusive and cannot be selected simultaneously. The multi-constraint optimization model is solved to obtain the combination of feedback actions with the highest expected utility under all constraints, and this combination is determined as the basic action set. In some embodiments, an integer programming solver is used to solve the multi-constraint optimization model. Optionally, the resource requirements of the actions are empirical values ​​obtained based on statistical analysis of historical execution data. It is understood that available computing resources are typically measured by CPU utilization and memory availability percentage.

[0053] See Figure 5 This is a heatmap predicting the expected effectiveness of combinations of feedback action parameters. The effectiveness scores follow a pattern of "high in the middle and low at both ends," with the template 3 + 1.5s combination showing the highest effectiveness (approximately 0.8), while the template 1 / 5 + 0.5s / 2.5s combination shows the lowest effectiveness (approximately 0.4). The combination of content template 3 and a 1.5s pause time is the optimal choice in the current interaction scenario, corresponding to the peak of the expected effectiveness. This type of chart serves as a decision-making tool for instantiating feedback action parameters, used to quickly locate the optimal parameter combination to ensure maximum effectiveness of feedback actions; it also intuitively illustrates the interactive influence between parameters, providing a basis for dynamic parameter adjustments in intelligent interaction systems.

[0054] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0055] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal sensing intelligent interactive feedback system, characterized in that, The system includes the following processing procedures: The data synchronization and recording module receives and synchronously records the real-time voice waveforms, user facial image streams, and user vital sign sensor signals collected during the interaction period, forming the original synchronous interactive data stream. The data alignment and slicing module performs time stamp unification and data slicing processing on the original synchronous interactive data stream to generate a multimodal data slice sequence with aligned timestamps; The hierarchical feature parsing module performs hierarchical parsing processing on the multimodal data slice sequence to extract pure speech segments and their text symbol sequences, facial expression change coding sequences, gaze placement sequences, and physiological state indicator sequences. The interaction state inference module inputs the text symbol sequence, facial expression change encoding sequence, gaze point sequence, and physiological state index sequence into the interaction state inference engine for joint analysis, and the interaction state inference engine outputs the state inference result for the current interaction period. The feedback strategy generation module, based on the state inference result, starts the strategy generator. The strategy generator queries a preset interaction rule knowledge graph based on the state inference result to generate a composite feedback strategy that includes feedback content template identifiers, interaction rhythm adjustment parameters, and perception channel enhancement instructions.

2. The multimodal sensing intelligent interactive feedback system according to claim 1, characterized in that, The hierarchical parsing process for the multimodal data slice sequence includes: Each multimodal data slice contains a slice of speech waveform, a slice of facial image, and a slice of vital signs within the same time segment; Layered parsing processing includes: noise filtering and endpoint detection of speech waveform slices, extraction of clean speech segments and conversion into text symbol sequences; Micro-expression unit segmentation and eye-tracking are performed on facial image slices to generate expression change coding sequences and gaze placement sequences; Baseline calibration and feature point extraction are performed on the vital sign signal slices to generate a physiological state index sequence.

3. The multimodal sensing intelligent interactive feedback system according to claim 2, characterized in that, The process of performing noise filtering and endpoint detection on speech waveform slices, extracting clean speech segments, and converting them into text symbol sequences includes: An adaptive spectral subtraction algorithm is applied to the speech waveform slice to suppress environmental steady-state noise, resulting in a preliminary denoised speech waveform; The initial denoised speech waveform is subjected to dual endpoint detection based on energy and zero-crossing rate to mark the start and end time points of the effective speech segment and extract the clean speech segment. The clean speech segment is input into a streaming speech recognition model, which outputs the corresponding text symbol sequence in an incremental manner. Each symbol in the text symbol sequence is associated with a timestamp and a confidence level. The text symbol sequence is subjected to context correction and domain terminology normalization processing, which includes: correcting incorrectly identified symbols based on the co-occurrence probability of preceding and following symbols; mapping synonymous domain terms to standard terms to form a standardized text symbol sequence.

4. The multimodal sensing intelligent interactive feedback system according to claim 2, characterized in that, The process of segmenting facial image slices into micro-expression units and tracking eye movements to generate expression change coding sequences and gaze placement sequences includes: A facial key point grid is located on the facial image slice, and the facial motion unit intensity value is calculated based on the motion displacement of the facial key point grid to generate an initial expression code; The initial facial expression code is subjected to temporal smoothing filtering and discrete event detection to identify micro-expression events with intensity exceeding a threshold and lasting for a certain duration. Each micro-expression event is encoded as an entry in the facial expression change encoding sequence containing type, intensity, and start and end time. Simultaneously, the pupil center position is tracked on the facial image slice and mapped to a predefined screen area coordinate system to generate the original gaze coordinate sequence; Cluster analysis and fixation point detection are performed on the original gaze coordinate sequence to aggregate the continuous gaze coordinates into the gaze focus sequence, each gaze focus containing the center coordinates and gaze duration.

5. The multimodal sensing intelligent interactive feedback system according to claim 2, characterized in that, The process of baseline calibration and feature point extraction of the vital sign signal slices to generate a physiological state index sequence includes: The original physiological signals in the vital sign signal slices are subjected to power frequency interference filtering and baseline drift correction to obtain preprocessed physiological signal waveforms. From the preprocessed physiological signal waveforms, the heart rate interval, skin conductance level, and pulse wave conduction time are periodically detected to form the original physiological indicator time series. The original physiological index time series is compared with the baseline reference value of the user at rest, and the normalized offset is calculated. The normalized offset is statistically feature-extracted based on a sliding time window. The extracted features include mean, variance, and trend slope, forming the physiological state index sequence.

6. The multimodal sensing intelligent interactive feedback system according to claim 1, characterized in that, The step of inputting the text symbol sequence, facial expression change encoding sequence, gaze focus sequence, and physiological state indicator sequence into the interaction state inference engine for joint analysis includes: A multimodal temporal fusion network is constructed, in which the text symbol sequence, facial expression change encoding sequence, gaze point sequence, and physiological state index sequence are embedded into a unified feature space to generate corresponding modal feature temporal sequences. Within the multimodal temporal fusion network, a cross-modal attention mechanism is designed to calculate the correlation weight between the feature vectors of any two different modal feature time series at the same time. Using the aforementioned correlation weights, feature enhancement guided by features from other modalities is performed on the temporal features of each modality to generate enhanced multimodal fusion temporal features; The enhanced multimodal fusion temporal features are input into a bidirectional recurrent neural network for temporal context encoding to obtain a hidden state sequence containing complete interaction time context information; The last state of the hidden state sequence is input into a fully connected classification layer, which outputs the state inference result. The state inference result is a multi-dimensional vector, with different dimensions corresponding to different cognitive load levels and intent clarity levels. The state inference result is used to characterize the user's cognitive load level and intent clarity level.

7. The multimodal sensing intelligent interactive feedback system according to claim 6, characterized in that, Within the multimodal temporal fusion network, a cross-modal attention mechanism is designed to calculate the correlation weights between feature vectors of any two different modal feature time series at the same time, including: For a given feature vector of a target modal feature time series at time t, calculate the dot product similarity between it and the feature vectors of all other modal feature time series at the same time t; The dot product similarity is input into a soft maximization function for normalization to obtain a set of attention weights, which represent the importance distribution of other modal features to the target modal feature at time t; The attention weights are used to perform a weighted summation of the feature vectors of other modal features at time t to obtain a context vector; The context vector is concatenated with the feature vector of the target modality at time t, and then an enhanced feature vector of the target modality at time t is generated by passing it through a learnable linear transformation layer. Repeat the process for each modality and each time step to generate the enhanced multimodal fusion temporal features for all modalities at all times.

8. The multimodal sensing intelligent interactive feedback system according to claim 1, characterized in that, Based on the state inference result, a strategy generator is activated. The strategy generator queries a preset interaction rule knowledge graph based on the state inference result to generate a composite feedback strategy that includes feedback content template identifiers, interaction rhythm adjustment parameters, and perception channel enhancement instructions, including: The cognitive load level dimension and intent clarity level dimension in the state inference results are analyzed and mapped to knowledge graph query conditions. Using the knowledge graph query conditions as input, the nodes and edges in the preset interaction rule knowledge graph are traversed. The nodes in the interaction rule knowledge graph represent the interaction state, and the edges represent the state transition conditions and the feedback actions that can be taken. In the interaction rule knowledge graph, find one or more target state nodes whose matching degree with the query conditions of the knowledge graph exceeds a preset threshold; Obtain all outgoing edges associated with the one or more target state nodes, with each outgoing edge accompanied by a description of the feedback action and the confidence level of the expected effect; Based on the current interaction context and system resource status, select a set of compatible and executable feedback actions from all outgoing edge-attached feedback actions to form a basic action set; The basic action set is instantiated with parameters, and each feedback action is assigned specific execution parameters, including content template identifier, rhythm adjustment parameters and channel enhancement instructions, thereby instantiating and generating the composite feedback strategy.

9. The multimodal sensing intelligent interactive feedback system according to claim 8, characterized in that, The parameter instantiation process for the basic action set, assigning specific execution parameters to each feedback action, includes: For each feedback action in the basic action set, the historical interaction log library is searched to find historical records in which the feedback action was successfully executed under similar state inference results; Extract the specific parameter values ​​used in the feedback action from the historical records, including the specific content template identifiers used, the specific numerical range of the rhythm adjustment parameters, and the specific code of the channel enhancement command; Analyze the correlation between the extracted specific parameter values ​​and the corresponding historical interaction performance scores, and establish a parameter-performance mapping model; The contextual features of the current interaction are input into the parameter-effect mapping model to predict the expected effects that different parameter values ​​may bring. Select the parameter value that yields the highest expected results as the execution parameter currently assigned to the feedback action.

10. The multimodal sensing intelligent interactive feedback system according to claim 8, characterized in that, The basic action set is formed by selecting a set of compatible and executable feedback actions from all outgoing edge-attached feedback actions based on the current interaction context and system resource status, including: Obtain the real-time resource status of the current interactive system, including available computing resources, available output channel status, and the length of the queue of pending interactive tasks. Assess the real-time resource requirements of the feedback actions associated with each outgoing edge, including computational overhead, output channel occupancy time, and execution latency; Check the logical compatibility and timing conflict possibility between different feedback actions within the basic action set; Establish a multi-constraint optimization model with the goal of maximizing the overall expected utility of the set of feedback actions, and with system resource constraints and compatibility between actions as constraints. Solve the multi-constraint optimization model to obtain the feedback action combination with the highest expected utility under all constraints, and determine the feedback action combination as the basic action set.

Images