A multimodal affective interaction system combining dynamic time warping and factorization machines

Abstract

The application relates to the technical field of human-computer interaction and emotional computing, and specifically discloses a multimodal emotional interaction system combining dynamic time warping and a factorization machine. The system acquires multimodal time sequence data in user interaction and extracts features; an optimal warping path between different modal feature sequences is calculated by using a dynamic time warping algorithm; a structured interaction constraint template is constructed based on the path, the template encodes the time alignment relationship between cross-modal features; the template is integrated into the second-order feature interaction calculation process of a factorization machine model, selective activation and weighting of feature interaction are carried out by using a guide model; the model is trained by using labeled data and is used for real-time emotional recognition feedback. The feature interaction process of the factorization machine is explicitly structured and constrained by the dynamic time warping path, the time asynchrony interference of the multimodal data and the blindness of feature interaction modeling are overcome, and the accuracy of emotional recognition and the interaction intelligence of the system are improved.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction and affective computing technology, and more specifically, to a multimodal affective interaction system that combines dynamic time warping and factorization machine. Background Technology

[0002] With the development of human-computer interaction technology, the accurate understanding of users' emotional states has become crucial for improving the naturalness and intelligence of interactions. Multimodal emotion recognition, by integrating information from multiple channels such as voice, vision, and text, can perceive user emotions more comprehensively and robustly, and has wide application needs in fields such as intelligent customer service, in-vehicle systems, and online education.

[0003] Currently, machine learning-based multimodal emotion recognition methods typically face two core challenges. First, signals from different modalities (such as audio and video) often exhibit inherent asynchronicity and different rates of change over time; for example, changes in facial expressions may lag slightly behind changes in tone of voice. Existing methods often employ simple time window alignment or direct feature concatenation, failing to finely model this cross-modal temporal dynamic correspondence. This results in the fused features containing significant temporally misaligned noise, affecting the model's understanding of emotional co-expression patterns. Second, in the feature fusion stage, widely used deep neural networks or traditional factorization machines tend to perform indiscriminate interaction modeling of all possible feature combinations. This "blind" interaction modeling introduces a large number of irrelevant or even interfering cross-modal associations, making it difficult for the model to focus on those truly co-occurring and semantically co-located key emotional cues. This not only increases model complexity and the risk of overfitting but also reduces the interpretability and accuracy of emotion judgment.

[0004] Therefore, this paper proposes a multimodal emotion interaction system that combines dynamic time warping and factorization machine to address the problems in the prior art: how to overcome the interference caused by the time asynchrony of multimodal data on feature fusion, and how to avoid the blindness of cross-modal feature interaction modeling, thereby improving the accuracy and robustness of emotion recognition. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a multimodal emotional interaction system that combines dynamic time warping and factorization machine to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multimodal emotional interaction system combining dynamic time warping and factorization machine, wherein the system achieves emotional interaction by executing the following steps:

[0007] Acquire raw time-series data of at least two different modalities generated during user interaction;

[0008] Extract the corresponding initial feature sequence from the original time series data of each modality;

[0009] For at least one pair of initial feature sequences of different modes that need to be fused, a dynamic time warping algorithm is applied to calculate and generate the optimal warping path that characterizes the temporal correspondence between the mode pairs.

[0010] Based on the optimal regularization path, a structured interaction constraint template is constructed. The template is used to encode the feature interaction relationships that should be of interest between different modal features as determined by the optimal regularization path.

[0011] A factorization machine model is constructed, and the structured interaction constraint template is integrated into the second-order feature interaction calculation process of the model, so that when the model performs second-order interaction calculation between any two features, it can regulate the corresponding interaction calculation according to the template.

[0012] The factorization machine model integrating the interaction constraint template was trained using a training dataset with sentiment labels to optimize the model parameters.

[0013] The multimodal interaction data of the user to be identified is input into the trained factorization machine model, which outputs the emotion recognition result, and the interactive feedback content of the system is generated or adjusted based on the result.

[0014] Furthermore, to improve robustness to noise, after generating the optimal regular path, a path confidence score can be calculated for each alignment point on the path; and when constructing the structured interaction constraint template, the element values ​​in the template are weighted according to the path confidence score, so that the model pays more attention to alignment relationships with high confidence.

[0015] Preferably, the construction of the structured interaction constraint template specifically includes:

[0016] Based on the optimal regularization path, the alignment and pairing relationship between the features in the initial feature sequence of the first mode and the features in the initial feature sequence of the second mode is determined;

[0017] Based on the alignment pairing relationship, a cross-modal interaction constraint matrix is ​​generated. The row dimension of this matrix corresponds to the total number of features in the first modality, and the column dimension corresponds to the total number of features in the second modality. Each element in the matrix represents a feature pair consisting of a feature from the first modality and a feature from the second modality.

[0018] For cross-modal feature pairs that have an alignment relationship on the optimal regularized path, the element at the corresponding position in the interaction constraint matrix is ​​assigned a first type of value; for cross-modal feature pairs that do not have an alignment relationship, the element at the corresponding position is assigned a second type of value.

[0019] The integration into the second-order feature interaction calculation process refers to using the value of the corresponding element in the interaction constraint matrix as a multiplier in the calculation result of the interaction term when the factorization machine model calculates the second-order interaction term of the corresponding cross-modal feature pair.

[0020] Preferably, the first type of value is 1 and the second type of value is 0, so that the second-order interaction terms generated by feature pairs that do not have alignment relationships are directly masked by the multiplier effect.

[0021] Preferably, the first type of value is a continuous weight calculated based on the distance metric between corresponding alignment points on the optimal regularization path, and the second type of value is 0;

[0022] The calculation of the continuous weights satisfies the following condition: the smaller the distance metric value, the larger the calculated continuous weight value.

[0023] Preferably, after generating the optimal regularized path, a path confidence score is calculated for each alignment point on the path;

[0024] The regulation of interactive computing based on the template includes:

[0025] The path confidence score is integrated into the structured interaction constraint template, so that for the second-order interaction computation involving cross-modal feature pairs aligned on the path, the output result is further multiplied by the corresponding path confidence score.

[0026] Preferably, the path confidence score is calculated based on the local normalization distance of the current alignment point and the smoothness of the local path segment containing that point.

[0027] Preferably, the loss function used in the training step includes sentiment classification loss and path consistency loss;

[0028] The path consistency loss is used to constrain the implicit feature interaction intensity pattern learned by the factorization machine model during training, so that it is close to the expected interaction intensity pattern mapped from the structured interaction constraint template.

[0029] Preferably, the expected interaction intensity pattern mapped from the interaction constraint template is specifically represented as an interaction intensity matrix consistent with the template logic;

[0030] The implicit feature interaction intensity pattern learned by the model is characterized by calculating the absolute value of the inner product between the latent vectors corresponding to each feature pair in the factorization machine model.

[0031] The path consistency loss is achieved by calculating the difference norm between the expected interaction strength matrix and the absolute value matrix of the inner product of the latent vectors.

[0032] Preferably, during online interaction, the system dynamically generates the corresponding interaction constraint template for each time window of multimodal data received in real time, and combines this template with the parameters of a pre-trained general factorization machine model to perform real-time sentiment inference.

[0033] The parameters of the pre-trained general factorization machine model are obtained by training the model using training data and simulating reasonable variations of the interaction constraint template, thereby enabling the model to generalize and generate different templates to adapt to different input data.

[0034] Preferably, the at least two different modalities include an audio modality and a video modality;

[0035] The initial feature sequence of the audio modality includes a Mel frequency cepstral coefficient sequence, and the initial feature sequence of the video modality includes a facial motion unit intensity sequence.

[0036] The application of the dynamic time warping algorithm is specifically used to perform cross-modal time alignment between the Mel frequency cepstral coefficient sequence and the facial motion unit intensity sequence.

[0037] The structured interaction constraint template is specifically used to guide the factorization machine model to focus on modeling the cross-modal interaction relationship between temporally aligned audio features and facial expression features.

[0038] The technical effects and advantages of this invention are as follows:

[0039] Compared to existing technologies, this invention transforms the optimal warping path generated by the dynamic temporal warping algorithm into a structured interaction constraint template, which is then deeply integrated into the second-order feature interaction calculation process of the factorization machine. This template explicitly specifies which cross-modal feature pairs should have their interactions activated or enhanced. During model computation, different feature interaction terms are selectively weighted or masked based on this template. This approach ensures that feature interaction calculation is no longer indiscriminate but strictly constrained by the prior structure derived from the temporal dynamics of the data itself. This effectively filters out noisy interactions caused by temporally misaligned features, forcing the model to focus on learning temporally synchronized cross-modal association patterns that are more likely to have emotional discriminative power, thus improving the accuracy of feature fusion and the interpretability of the model.

[0040] Compared to existing technologies, this invention introduces a path consistency loss function during model training, constructing an additional supervisory signal. This loss function directly compares the difference between the latent feature interaction strength actually learned by the model and the prior interaction strength derived from the dynamic time-warped path, driving the two to converge. Its working mechanism involves continuously injecting the time-aligned strength of the underlying signal as a form of structured knowledge into the modeling process of the upper-level feature interaction relationships. This enables the factorization machine model not only to learn and fit sentiment labels, but also for the feature interaction patterns represented by its internal parameters to be calibrated with the temporal co-occurrence patterns of cross-modal signals, thereby enhancing the model's ability to model sentiment consistency and improving the robustness of sentiment recognition in complex asynchronous scenarios.

[0041] Compared to existing technologies, this invention employs an inference mechanism that combines offline training of a general model with online dynamic templates. In the offline phase, the model learns and adapts to the general rules of cross-modal time alignment by interacting with diverse path perturbation samples, obtaining a universal parameter representation. In online applications, the system only needs to quickly calculate the specific regularized path for real-time input data and generate the corresponding interaction constraint template, which is then combined with fixed general model parameters for inference. This mechanism separates time-consuming model retraining from lightweight path computation, ensuring that the system can perform personalized and precise feature interaction control based on the temporal characteristics of specific input data, while also meeting the high real-time response requirements of practical interactive systems, achieving a balance between accuracy and efficiency. Attached Figure Description

[0042] Figure 1 This is a flowchart illustrating the overall system workflow of the present invention.

[0043] Figure 2 This is a diagram of the feature interaction constraint mechanism of the present invention.

[0044] Figure 3 This is a flowchart illustrating the training and reasoning process of this invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] Example 1

[0047] As attached Figures 1 to 3The illustrated multimodal emotional interaction system combines dynamic time warping and factorization machines. The system's implementation primarily includes the following steps: data acquisition and feature extraction (S1, S2), cross-modal temporal alignment (S3), structured interaction constraint template construction (S4), factorization machine model construction and integration (S5), model training (S6), and online inference and interaction (S7). It should be understood that the step numbers are used only to clearly illustrate the technical solution and do not constitute any limitation on the claims. Through the collaborative work of the above steps, the system can dynamically configure the fusion model based on the real-time characteristics of the input data, prioritizing the modeling of highly temporally synchronized cross-modal emotional cues, ultimately providing a reliable basis for generating personalized and intelligent interactive feedback.

[0048] Furthermore, the structured interaction constraint template construction step (S4) is implemented as follows. The core function of this step is to convert the optimal regularization path output in step S3 into a regularized path. This is mapped to a matrix structure that matches the dimension of the input feature space of the factorization machine model, i.e., the interaction constraint matrix. .path It consists of a series of coordinate pairs, denoted as ,in and These represent the frame indices in the first modality (e.g., audio) feature sequence and the second modality (e.g., video) feature sequence, respectively. Let be the path length. Assume the total dimension of the cross-modal fusion feature vector extracted by the system is . ,in Let be the feature dimension of the first mode. Let be the characteristic dimension of the second mode. Then the matrix... It was constructed.

[0049] The construction process first initializes It is a zero matrix. For the optimal regularized path... Each alignment point in It indicates that, in the time dimension, the first mode is the first Frame features and the second modality's first There is a correspondence between frame features.

[0050] Given that each frame's features are multi-dimensional vectors, this alignment point implies a potential semantic relationship between all feature dimensions contained in the two frames. Therefore, in the matrix... In the middle, the row index corresponds to all of the first mode. Each feature and column index corresponds to all of the second modalities. The rectangular region formed by the features (i.e.) Top left The submatrix or the corresponding cross-modal region is marked as a "candidate interaction region" based on the alignment point. More specifically, all elements within this region can be... (in , The value of ) is set according to certain rules. (Path traversal) After aligning all points in the matrix, Positions covered by one or more alignment points will be assigned non-zero values, indicating that the interaction between the cross-modal feature pairs corresponding to these positions should be considered or enhanced; while the remaining positions (including most of the same-modal feature interaction areas and the uncovered cross-modal areas) will remain zero, indicating that the interaction between these feature pairs should be suppressed or ignored.

[0051] The interaction constraint matrix The physical significance lies in the fact that it transforms the temporal co-occurrence relationships discovered by dynamic time warping from the temporal index space to the feature interaction space, forming a data-dependent, sparse "interactive attention blueprint." This blueprint will serve as the static or dynamic configuration parameters for subsequent factorization machine model calculations.

[0052] Furthermore, the interaction constraint matrix The assignment strategy for non-zero elements can adopt a binary approach. In this embodiment, for all matrix elements covered by at least one optimal normalized path alignment point... directly set its value Set as For all uncovered elements, their values ​​remain unchanged. Therefore, the matrix It becomes a binary mask matrix.

[0053] This matrix is ​​integrated into the second-order interactive computation part of the factorization machine model in step S5. The standard factorization machine model uses the input feature vector... The prediction includes the summation of second-order interaction terms. ,in It is a feature and corresponding Hidden vectors This indicates the inner product operation.

[0054] After integrating the binary mask matrix, the calculation is corrected to .because The mechanism of this correction is element-wise multiplication gating. When At that time, regardless of the latent vector and What are the parameter values, features? and The interaction contribution is forced to zero during summation, and the gradient cannot be transferred through this term during backpropagation during model training. and An update is generated. When In this way, the interactive contributions are fully preserved and can participate in subsequent calculations.

[0055] This binary strategy implements a hard constraint, the direct technical effect of which is that the model's feature interaction capabilities are strictly limited to a sparse subgraph predefined by the dynamic time-warped path. It completely eliminates the possibility of the model establishing spurious statistical associations between feature combinations that have never been observed in time, thus significantly improving the model's interpretability. With limited training data, this constraint effectively reduces the risk of overfitting and guides the model to focus on learning the most physically meaningful (i.e., time-synchronized) cross-modal sentiment expression patterns.

[0056] Furthermore, the interaction constraint matrix The assignment strategy for non-zero elements can also employ a continuous weighting approach to achieve more refined guidance. In this embodiment, the calculation of weights is closely dependent on the local alignment quality metric generated when generating the optimal regular path.

[0057] For path Each alignment point on It can obtain its local alignment distance. This distance The instantaneous difference between the two modal feature vectors at this point is quantified; a smaller value indicates tighter alignment and higher quality. This is achieved through a pre-defined monotonically decreasing function. Map the distance to a value between and The weight values ​​between them, i.e. A typical function is the exponentially decaying function. ,in This is the scale parameter.

[0058] In constructing the matrix At that time, for those aligned with the point Each element within the identified candidate interaction area Its value is assigned If the same element If a feature is covered by multiple alignment points (e.g., the same pair of feature dimensions are aligned across multiple frames), then aggregation strategies such as taking the maximum value or averaging can be used to determine its final weight. When integrated into the factorization machine calculation, the corrected sum of second-order interaction terms is: ,at this time These are continuous values. This weight acts as a multiplier, scaling the interaction contribution of each pair of features.

[0059] The continuous weighting strategy implements a soft guidance. Its technical advantage lies in incorporating alignment confidence into the interaction modeling process. High-quality synchronization (low distance) amplifies the interaction contribution in that region, resulting in stronger gradient signals during model training; conversely, low-quality synchronization (high distance) suppresses this contribution. This allows the model to adaptively focus on the most reliable cross-modal cues without completely discarding alignments that may contain weak information, thus enhancing the model's robustness while maintaining discriminative power, especially suitable for real-world scenarios where alignment noise may exist.

[0060] Furthermore, to improve the robustness of the interaction constraint template, an evaluation of the structural quality of the optimal regular path itself can be introduced when calculating the weights, that is, calculating the comprehensive path confidence score for each alignment point. This score is determined by the local alignment quality. (As mentioned above, based on) (Calculation) and local path smoothness The decision is made jointly. Path smoothness assesses whether the local path segment formed by the alignment point and its neighboring points conforms to natural temporal continuity, in order to filter out anomalous jumps that may be caused by transient noise or regularization algorithms.

[0061] For alignment points Its smoothness weight One method for calculation is to examine the directional consistency of its neighboring points. Define its forward vector. and backward vector Calculate the cosine of the angle between two vectors. . The closer This indicates that the path's direction changes gently at that point, exhibiting high smoothness. (This can be...) Normalized to the [0, 1] interval as the smoothness weight, for example The final overall confidence level can be obtained through a linear combination: ,in This is the balance coefficient. This will be used as the final weight value when constructing the weighted interaction constraint matrix.

[0062] The design offers two technical benefits: firstly, it generates more reliable alignment confidence estimates by combining local and contextual information; secondly, it effectively suppresses points that, although locally small, cause unreasonable path transitions, making the generated interaction constraint templates more globally consistent and physically plausible, thus providing higher-quality prior guidance for upper-level models.

[0063] Furthermore, the path smoothness The computation can be extended to a wider local window to enhance robustness. For points Considering its preceding and following aspects A window composed of points .

[0064] One method for calculating the overall smoothness of a window is to use the direction vectors between all consecutive pairs of points within the window. ( The analysis proceeds by calculating the average cosine similarity between each pair of these direction vectors. . A higher value indicates a more consistent path direction within the window, resulting in better smoothness. After normalization, it can be used as a point. Smoothness weight Windowed computation reduces sensitivity to abrupt changes in a single direction, better reflects the macroscopic trend of the path, filters out high-frequency jitter, and thus generates confidence scores that are more representative of stable, long-range time-series correlation patterns.

[0065] Furthermore, the loss function used in the model training step (S6) is a joint loss function. Loss due to sentiment classification and path consistency loss It consists of two parts. Sentiment classification loss. To supervise the matching of model output with real sentiment labels, methods such as cross-entropy loss can be used. Path consistency loss. This ensures that the implicit feature interaction patterns learned internally by the factorization machine model are consistent with the explicit temporal priors (i.e., interaction constraint templates) provided externally by dynamic time warping, thereby injecting the temporal structure knowledge of the underlying signals into the learning of high-level semantic features.

[0066] Path consistency loss The specific construction method is as follows: First, the interaction constraint matrix generated in step S4 is... (Regardless of whether it is binary or weighted) Normalization is performed to obtain a prior interaction strength matrix. ,in .matrix It can be viewed as a probability distribution that describes the relative interactive importance that different feature pairs "should" have.

[0067] Simultaneously, during the model's forward propagation, an actual interaction strength matrix is ​​calculated. , of which elements Features and The absolute value of the inner product of the corresponding latent vectors reflects the correlation strength between the two features encoded by the current parameters of the model. Path consistency loss is defined as... and The difference between the distributions of two matrices, for example, using mean squared error: The total loss during model training is , This is a hyperparameter.

[0068] Through optimization While minimizing the classification error, the model is also forced to adapt the structure of its latent vector space (reflected in...). Approximating the prior structure derived from the temporal path ( The core technical effect it brings is the realization of cross-level knowledge alignment. This provides additional supervisory signals for model training, derived from the structure of the data itself, which helps improve generalization ability when label data is limited. It ensures that the feature interaction patterns finally learned by the model have a clear physical interpretation, that is, they are strongly correlated with the observed cross-modal temporal synchronization, thereby enhancing the interpretability and reliability of the entire system.

[0069] Furthermore, in actual training, to improve computational efficiency, an importance-based sampling method can be used to approximate the path consistency loss. Due to the prior matrix It is usually sparse (with many elements having zero), and we are more concerned with The consistency of the medium-to-high weight regions can therefore be determined based on the distribution. Sample a small batch of interaction pairs Sampling probability and Proportional. Then approximate calculation. .

[0070] At the same time, in order to encourage model suppression The interaction strength of zero-weight regions (i.e., feature pairs that should not interact) can be further regularized by introducing a small additional term: uniformly sampling another small batch of samples from the zero-weight regions. ,calculate ,in It is a small positive number.

[0071] Ultimately, the combined loss was This sampling strategy significantly reduces computational overhead while maintaining the core role of path consistency constraints, enabling the design to be efficiently applied to model training with large-scale feature dimensions.

[0072] Furthermore, the system employs an inference mechanism that trains a general model offline and dynamically applies templates online to achieve a balance between accuracy and efficiency. During the offline training phase, the goal is to obtain a general parameter set for the factorization machine model. The model has the ability to adapt to various reasonable interaction constraint templates that may be generated by different input data.

[0073] Therefore, path perturbation enhancement is introduced during training data preparation. For each training sample, a standard optimal regularized path is used. By applying slight Gaussian noise to its coordinates And perform reasonable rounding and adjustment (to ensure path constraints) to generate Disturbance path Based on the original path and all perturbation paths, generate Interaction constraint template .

[0074] During training iterations, for the same set of low-level features, this is used sequentially. Each template is combined with the model for forward computation to obtain... Each prediction and its corresponding joint loss are used, and the average of these losses is used for backpropagation and parameter updates. This process forces the model to learn a generalization ability: even when faced with inputs containing the same content but with slight differences in temporal alignment details, its internal parameters can maintain stable sentiment judgments, thus learning to understand the paradigm of "interaction guided by dynamic temporal warping paths" itself, rather than memorizing a specific template.

[0075] During the online inference phase, for new input data, the system only needs to execute steps S1 to S4 to quickly calculate its unique optimal regularization path. And generate the corresponding single interaction constraint template. Then, load the fixed, general model parameters. ,Will As part of the model's computation, a single efficient forward pass can yield the sentiment prediction result.

[0076] This mechanism decouples computationally intensive model training from lightweight real-time path computation, enabling the system to meet the stringent low-latency requirements of real-time interactive applications while maintaining data-specific temporal guidance for each inference.

[0077] Furthermore, in a specific audio-visual emotion recognition embodiment, the data acquisition and feature extraction steps (S1, S2) are implemented as follows.

[0078] Audio modality: After framing the original audio waveform (25ms frame length, 10ms frame shift), pre-emphasizing, and applying a Hamming window, the Mel-frequency cepstral coefficients (MFCCs) of each frame are calculated. In this embodiment, the first 13 static MFCC coefficients are extracted, and their first and second order differences are calculated to obtain a 39-dimensional feature vector for each frame. The entire audio segment is represented as a sequence. ,in This represents the total number of audio frames.

[0079] Video modality: Face detection and tracking are performed on the video stream. For each frame of face image, a set of facial action unit (AU) intensity values ​​are calculated using a pre-trained facial behavior encoder (such as a ResNet-based network).

[0080] In this embodiment, 17 AUs related to emotional expression are selected to obtain a 17-dimensional feature vector for each frame. The entire video segment is represented as a sequence. ,in This represents the total number of video frames. Due to differences in acquisition frequency, Usually much larger The cross-modal timing alignment step (S3) is specifically designed for... and These two sequences are performed.

[0081] Calculate the cosine distance between each pair of frames in the sequence. Then, the optimal regular path is obtained through dynamic programming. This path Essentially, this establishes a fine-grained temporal mapping between audio events (such as pitch rises and energy bursts) and video events (such as specific AU activations). The subsequent construction of the structured interaction constraint template... Its rows correspond to 39-dimensional audio features, and its columns correspond to 17-dimensional video features (in the complete...). (Located in a specific cross-modal region within the matrix).

[0082] The position and weight of non-zero elements in the template directly encode the temporal co-occurrence and degree of co-occurrence of "what acoustic features" and "what facial muscle movements," thus providing precise guidance for factorization machine models to model cross-modal emotional resonance (such as the synchronization of laughter and smiles).

[0083] To more fully demonstrate the workflow and technical effectiveness of the method, a specific example from an intelligent online education scenario is provided below. This example demonstrates how the system analyzes students' emotional states in the classroom in real time.

[0084] Scenario Setting: The system is deployed in the cloud, processing audio and video streams uploaded by students in real time and analyzing their emotional states (such as "focused," "confused," and "bored"). A sliding window analysis method is used, with a window length of... s, step size s.

[0085] Detailed workflow and calculation examples:

[0086] Data Acquisition (S1): The system acquires the current time window (e.g., the 10th time window). Audio sampling data within 14 seconds and video frame sequence .

[0087] Feature extraction (S2):

[0088] Audio: Yes Perform MFCC extraction. Assuming a sampling rate of 16kHz, approximately... Each frame contains 39-dimensional features, forming a matrix. .

[0089] Video: To (30fps, 120 frames total) Perform face recognition (AU) to obtain a matrix. .

[0090] Dynamic Time Warping (S3):

[0091] To simplify the calculation, assume that the 39-dimensional audio features and 17-dimensional video features of each frame are mapped through a small coding network (such as a fully connected layer) to... A time-series representation vector of dimension is used to obtain the sequence. and .

[0092] Calculate the Euclidean distance matrix between A' and V', and run the DTW algorithm. Assume that the optimal normalized path is obtained. ,common This indicates that, for example, the first two frames of audio are aligned with the first frame of the video, and the third frame of audio is aligned with the second frame of the video, depicting the temporal warp relationship between the audio and video.

[0093] Constructing an interaction constraint template (S4):

[0094] Using the original feature dimensions ( , Constructing cross-modal interaction submatrices. It is initialized to zero.

[0095] For path Each alignment point Calculate cosine distance using original features Suppose for point... , .

[0096] Using exponential weighting .

[0097] Will In the middle, the row corresponds to all 39 features of the 10th frame of the audio, and the column corresponds to the 10th frame of the video. The region of all 17 features of the frame ( The value of (the number of positions) is set to (If the maximum value strategy is adopted, and the current value in this region is less than 0.741).

[0098] Traverse all After one point, Some positions in the middle have non-zero weights, while the rest have zero weights. Assuming location (corresponding to the 15th MFCC feature and the 1st MFCC feature) The final weights of the AU features (Formed by the aggregation of multiple high-quality alignment points).

[0099] Model inference (S5, S7):

[0100] Loading pre-trained general factorization machine model parameters: global bias First-order weight vector Latent vector matrix (Latent vector dimension) ).

[0101] Generate window-level input feature vectors Calculate separately and The mean over the time dimension is obtained. and spliced ​​together Assuming , .

[0102] Forward computation:

[0103] a. First-order part: .

[0104] b. Guided second-order part:

[0105] i. Extract the hidden vector: (correspond )and (correspond (due to index offset), all are A dimensional vector. Assume... , .

[0106] ii. Calculate the inner product: (Example value).

[0107] iii. Query constraint weights: .

[0108] iv. Calculate the weighted interaction contribution of this feature pair:

[0109] .

[0110] v. Similarly, calculate all other variables. Covered feature pairs (i.e.) of The contributions of each group are summed to obtain the total constrained second-order interaction value. .

[0111] c. Final prediction score: The score is then input into the softmax layer (assuming...). (emotional categories), to obtain probability distributions, for example These correspond to "focus", "confusion", and "boredom" respectively.

[0112] Interactive Feedback (S7 Follow-up): The system determines that the student's current state is primarily "focused," but with some "confusion." This result is pushed to the teacher's console for visual feedback. Additionally, if the probability of "confusion" is higher than 0.3 in two consecutive windows, the system may automatically send the student an encouraging message or mark the time point for the teacher to review later.

[0113] This example fully demonstrates the entire process of the described method, from raw signals to emotional decisions. Through... By visualizing the DTW path as feature interaction weights and applying these weights in real time during factorization machine computation, direct and interpretable micro-control of the feature fusion model by cross-modal temporal knowledge is achieved.

[0114] Finally, the following points should be noted: First, in the description of this application, it should be noted that, unless otherwise specified and limited, the terms "installation", "connection", and "linkage" should be interpreted broadly, and can be mechanical or electrical connections, or internal connections between two components, or direct connections. "Up", "down", "left", "right", etc. are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may change.

[0115] Secondly: The accompanying drawings of the embodiments disclosed in this invention only involve the structures involved in the embodiments disclosed in this invention. Other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of this invention can be combined with each other.

[0116] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal emotional interaction system combining dynamic time warping and factorization machine, characterized in that, The system achieves emotional interaction by performing the following steps: Acquire raw time-series data of at least two different modalities generated during user interaction; Extract the corresponding initial feature sequence from the original time series data of each modality; For at least one pair of initial feature sequences of different modes that need to be fused, a dynamic time warping algorithm is applied to calculate and generate the optimal warping path that characterizes the temporal correspondence between the mode pairs. Based on the optimal regularization path, a structured interaction constraint template is constructed. The template is used to encode the feature interaction relationships that should be of interest between different modal features as determined by the optimal regularization path. A factorization machine model is constructed, and the structured interaction constraint template is integrated into the second-order feature interaction calculation process of the model, so that when the model performs second-order interaction calculation between any two features, it can regulate the corresponding interaction calculation according to the template. The factorization machine model integrating the interaction constraint template was trained using a training dataset with sentiment labels to optimize the model parameters. The multimodal interaction data of the user to be identified is input into the trained factorization machine model, which outputs the emotion recognition result, and the interactive feedback content of the system is generated or adjusted based on the result.

2. The multimodal emotional interaction system combining dynamic time warping and factorization machine as described in claim 1, characterized in that, The construction of the structured interaction constraint template specifically involves: Based on the optimal regularization path, the alignment and pairing relationship between the features in the initial feature sequence of the first mode and the features in the initial feature sequence of the second mode is determined; Based on the alignment pairing relationship, a cross-modal interaction constraint matrix is ​​generated. The row dimension of this matrix corresponds to the total number of features in the first modality, and the column dimension corresponds to the total number of features in the second modality. Each element in the matrix represents a feature pair consisting of a feature from the first modality and a feature from the second modality. For cross-modal feature pairs that have an alignment relationship on the optimal regularized path, the element at the corresponding position in the interaction constraint matrix is ​​assigned a first type of value; for cross-modal feature pairs that do not have an alignment relationship, the element at the corresponding position is assigned a second type of value. The integration into the second-order feature interaction calculation process refers to using the value of the corresponding element in the interaction constraint matrix as a multiplier in the calculation result of the interaction term when the factorization machine model calculates the second-order interaction term of the corresponding cross-modal feature pair.

3. A multimodal emotional interaction system combining dynamic time warping and factorization machine as described in claim 2, characterized in that, The first type of value is 1, and the second type of value is 0. The multiplier effect directly masks the second-order interaction terms generated by cross-modal feature pairs that do not have alignment relationships.

4. A multimodal emotional interaction system combining dynamic time warping and factorization machine as described in claim 2, characterized in that, The first type of value is a continuous weight calculated based on the distance metric between corresponding alignment points on the optimal regularization path, and the second type of value is 0. The calculation of the continuous weights satisfies the following condition: the smaller the distance metric value, the larger the calculated continuous weight value.

5. A multimodal emotional interaction system combining dynamic time warping and factorization machine as described in claim 1, characterized in that, After generating the optimal regularized path, a path confidence score is calculated for each alignment point on the path; The regulation of interactive computing based on the template includes: The path confidence score is integrated into the structured interaction constraint template, so that for the second-order interaction computation involving cross-modal feature pairs aligned on the path, the output result is further multiplied by the corresponding path confidence score.

6. A multimodal emotional interaction system combining dynamic time warping and factorization machine as described in claim 5, characterized in that, The path confidence score is calculated based on the local normalization distance of the current alignment point and the smoothness of the local path segment containing that point.

7. A multimodal emotional interaction system combining dynamic time warping and factorization machine as described in claim 1, characterized in that, The loss function used in the training step includes sentiment classification loss and path consistency loss; The path consistency loss is used to constrain the implicit feature interaction intensity pattern learned by the factorization machine model during training, so that it is close to the expected interaction intensity pattern mapped from the structured interaction constraint template.

8. A multimodal emotional interaction system combining dynamic time warping and factorization machine as described in claim 7, characterized in that, The expected interaction intensity pattern mapped from the interaction constraint template is specifically represented as an interaction intensity matrix consistent with the template logic. The implicit feature interaction intensity pattern learned by the model is characterized by calculating the absolute value of the inner product between the latent vectors corresponding to each feature pair in the factorization machine model. The path consistency loss is achieved by calculating the difference norm between the expected interaction strength matrix and the absolute value matrix of the inner product of the latent vectors.

9. A multimodal emotional interaction system combining dynamic time warping and factorization machine as described in claim 1, characterized in that, During online interaction, the system dynamically generates corresponding interaction constraint templates for multimodal data received in real time within each time window, and combines these templates with the parameters of a pre-trained general factorization machine model to perform real-time sentiment inference. The parameters of the pre-trained general factorization machine model are obtained by training the model using training data and simulating reasonable variations of the interaction constraint template, thereby enabling the model to generalize and generate different templates to adapt to different input data.

10. A multimodal emotional interaction system combining dynamic time warping and factorization machine according to claim 1, characterized in that, The at least two different modalities include an audio modality and a video modality; The initial feature sequence of the audio modality includes a Mel frequency cepstral coefficient sequence, and the initial feature sequence of the video modality includes a facial motion unit intensity sequence. The application of the dynamic time warping algorithm is specifically used to perform cross-modal time alignment between the Mel frequency cepstral coefficient sequence and the facial motion unit intensity sequence. The structured interaction constraint template is specifically used to guide the factorization machine model to focus on modeling the cross-modal interaction relationship between temporally aligned audio features and facial expression features.

Images