An aspect-level sentiment analysis method and system based on multi-modal information fusion

Abstract

The application discloses a kind of multi-modal information fusion's aspect level sentiment analysis method and system, specifically including the following steps: extracting the original text features of student experimental reflection process, original visual features and original physiological features are aligned in time and space;Through the visual prototype and physiological prototype guided debiasing cross-attention mechanism, respectively from the aligned original visual features and original physiological features, obtain purified visual features and purified physiological features, then respectively with original text features are cross-modal perception and alignment, obtain fusion visual information text features and fusion physiological information text features, with original text features are hierarchically fused to obtain multi-modal information fusion after fine-grained sentiment feature;Aspect level sentiment prediction is output by inputting fine-grained sentiment feature into BART model.The application effectively blocks the error accumulation between cross-modal, systematically improves the accuracy of student reflection sentiment tuple analysis in experimental teaching process.

Description

Technical Field

[0001] This invention discloses a method and system for aspect-level sentiment analysis based on multimodal information fusion, which belongs to the application of multimodal sentiment analysis technology in experimental teaching scenarios. Background Technology

[0002] Current scientific inquiry experiment evaluations focus more on the experimental procedure and final results, lacking effective automated analysis methods for the post-experiment reflection process. This makes it difficult to identify higher-order thinking activities such as process monitoring, strategy adjustment, and emotional experience during the reflection process, which can reflect students' metacognitive abilities and emotional states.

[0003] In the context of evaluating the reflection process during scientific inquiry experiments in experimental teaching, traditional reflection evaluation mainly relies on teachers' subjective observation or single text analysis, which suffers from problems such as coarse evaluation granularity, strong subjectivity, and difficulty in integrating multimodal information. With the advancement of model algorithms, multimodal sentiment analysis methods based on text information, video image information, and students' physiological signals are being used to achieve fine-grained and multimodal sentiment analysis of students' post-experiment reflection process. However, during the student reflection process, there are problems such as modal heterogeneity, temporal asynchrony, and difficulties in cross-modal semantic association among the collected text signals, video image signals, and physiological signals, making it difficult for existing multimodal sentiment analysis methods to accurately depict students' true cognitive and emotional states.

[0004] Existing research on multimodal aspect-level sentiment analysis mainly revolves around cross-modal feature fusion and alignment mechanisms. To achieve effective interaction between modalities, researchers have proposed various fusion strategies, including cross-modal attention, collaborative attention, gating fusion mechanisms based on attention mechanisms, and relational modeling methods based on graph neural networks. Examples include a multimodal sentiment analysis method and device based on teacher-guided distillation and selective alignment of text queries, disclosed in patent application CN122065290A, and a noise-perceived graph neural network method and system for multimodal sentiment analysis, disclosed in patent application CN121434876A. These methods can align image regions and text aspect words at the aspect level, enabling more refined sentiment reasoning. However, applying multimodal aspect-level sentiment analysis technology to reasoning analysis in the context of scientific inquiry experiment reflection and evaluation in experimental teaching presents the following challenges:

[0005] First, the model tends to over-rely on explicit text, while implicit visual and physiological features are susceptible to strong noise interference, leading to severe cross-modal semantic bias. During reflection, the model easily relies excessively on high-frequency emotional expressions in the text while ignoring implicit emotional information in facial expressions and physiological states. Simultaneously, student facial expression videos contain background noise, lighting variations, and irrelevant actions, and radar physiological signals are characterized by strong continuity and significant noise fluctuations. While traditional cross-modal attention mechanisms can establish connections between text and visual / physiological aspects, they struggle to effectively focus on the truly emotion-discriminating facial expressions and radar regions.

[0006] Secondly, multimodal features exhibit significant hierarchical and heterogeneous differences, and emotional reasoning in educational scenarios has its own inherent hierarchy. The existence of hierarchical differences between reflective text cognitive semantics, explicit facial behavior, and implicit physiological states makes the expression of fusion features after shallow weighting limited and prone to error accumulation, making it difficult to support deep emotional reasoning in complex educational scenarios.

[0007] Therefore, the core technical problem that this application needs to solve is how to integrate multimodal data such as text, video and physiological signals, and design an analysis method that can achieve cross-modal feature alignment and fine-grained reflection state recognition, so as to improve the accuracy of the post-experiment reflection evaluation of students in experimental teaching. Summary of the Invention

[0008] The technical problem solved by this invention is to provide a method and system for aspect-level sentiment analysis that addresses the serious cross-modal semantic bias, difficulty in effectively modeling implicit emotional information, and coarse granularity of multimodal fusion in the context of student reflection in scientific inquiry experiments.

[0009] This invention is achieved using the following technical solution:

[0010] This invention first discloses an aspect-level sentiment analysis method based on multimodal information fusion, comprising the following steps:

[0011] S1. Collect audio text, video facial expressions, and radar physiological data of the experimental reflection process, align them temporally according to the timeline of the audio text, and extract the corresponding original text features, original visual features, and original physiological features for spatial alignment given the reflection segments of the audio text.

[0012] S2. Through the debiased cross-attention mechanism guided by visual and physiological prototypes, purified visual features and purified physiological features are obtained from the aligned original visual features and original physiological features, respectively.

[0013] S3. Perform cross-modal perception and alignment between the original text features and the purification visual features and purification physiological features respectively to obtain text features fused with visual information and text features fused with physiological information.

[0014] S4. The original text features, the fused visual information text features, and the fused physiological information text features are hierarchically fused to obtain fine-grained emotional features after multimodal information fusion.

[0015] S5. Input the fine-grained emotional features into the BART model and output the aspect word boundaries and their emotional polarity predictions in the experimental reflection process.

[0016] In a multimodal information fusion aspect-level sentiment analysis method of the present invention, further, in step S1, the audio text data is segmented into several semantically complete reflective sentence segments through speech recognition transcription;

[0017] For video facial expression data, extract video frame sequences according to the start and end times of the reflective sentence segments;

[0018] For physiological data, radar physiological signals are extracted from the corresponding time intervals based on the start and end times of the reflection segments. The radar physiological signals are then resampled onto the time axis of the video frame sequence to extract a physiological feature sequence that is aligned with the video facial expression data at the frame level.

[0019] In a multimodal information fusion aspect-level sentiment analysis method of the present invention, the given audio text reflection segment is further mapped by a text encoder to obtain a context-related vector representation as the original text feature, the spatiotemporal features of the corresponding video frame sequence are extracted as the original visual features, and the temporal feature matrices of various types of physiological indicators in the corresponding physiological feature sequence are extracted as the original physiological features; the original text features, original visual features and original physiological features are projected onto a unified shared dimension through a linear projection layer to achieve spatial dimension alignment of the three-modal feature sequences.

[0020] In a multimodal information fusion aspect-level sentiment analysis method of the present invention, further, in step S2, learnable visual prototypes and physiological prototypes are defined.

[0021] The visual prototype-guided debiasing cross-attention mechanism includes Y stacked debiasing layers with identical structures. In the y-th debiasing layer, the output of the previous debiasing layer is mapped to the visual prototype to obtain a visual query vector. Combined with the key vector and value vector mapped from the original visual features after spatial alignment, attention is focused on local salient visual features. The local salient visual features are mapped to the same dimension as the original text features through MLP, and then concatenated with the original visual features to obtain the output of the debiasing layer. The output of the last debiasing layer is used to purify the visual features.

[0022] The physiological prototype-guided debiasing cross-attention mechanism includes Y stacked debiasing layers with identical structures. In the y-th debiasing layer, the output of the previous debiasing layer is mapped to the physiological prototype to obtain a physiological query vector. Combined with the key vector and value vector mapped from the original physiological features after spatial alignment, attention is focused on local salient physiological features. The local salient physiological features are mapped to the same dimension as the original text features through MLP, and then concatenated with the original physiological features to obtain the output of the debiasing layer. The physiological features are purified by the output of the last debiasing layer.

[0023] In a multimodal information fusion aspect-level sentiment analysis method of the present invention, further, in step S3, the original text features are used as semantic anchors, and cross-modal perception and alignment of purification visual features and purification physiological features are performed respectively through a dual-path independent text-guided multi-head cross-attention mechanism.

[0024] In text-guided visual cross-attention, the original text features are used as queries, and the purified visual features are used as keys and values. Through multi-head cross-attention, the fused visual information text features are obtained.

[0025] In text-guided physiological cross-attention, the original text features are used as queries, and the purified physiological features are used as keys and values. Through multi-head cross-attention, the fused physiological information text features are obtained.

[0026] In a multimodal information fusion aspect-level sentiment analysis method of the present invention, further, in step S4, when the cosine similarity expression between the fused visual information text features and the fused physiological information text features is consistent, element-wise multiplication is used to model the collaborative information between the fused visual information text features and the fused physiological information text features to capture the consistent region of multimodal sentiment.

[0027] When the cosine similarity between the fused visual information text features and the fused physiological information text features is inconsistent, the absolute difference is used to model the conflict information between the fused visual information text features and the fused physiological information text features.

[0028] Combining original text features with collaborative and conflict information to construct interactively enhanced text representations. :

[0029] ;

[0030] in, B represents the original text features, and C represents collaborative and conflicting information, respectively. These are learnable weights;

[0031] Dynamically fuse interactive enhanced text representations, original text features, text features fused with visual information, and text features fused with physiological information:

[0032] ;

[0033] in, , These are text features that fuse visual information and text features that fuse physiological information, respectively. , , Weights for different modalities, This refers to fine-grained sentiment features derived from the fusion of multimodal information.

[0034] In a multimodal information fusion aspect-level sentiment analysis method of the present invention, further, in step S5, the encoder of the BART model processes the fine-grained sentiment features of the input. Encode the hidden state sequence to output a context-aware fusion. :

[0035] ;

[0036] The decoder of the BART model fuses context-aware hidden state sequences. Decode the code and output the context information of the current time step. :

[0037] ;

[0038] in, The decoder output from the previous time step is used to calculate the predicted experimental aspect word boundaries and sentiment polarity target sequences using softmax(). :

[0039] ;

[0040] in, This is the weight matrix. This is the bias vector.

[0041] In a multimodal information fusion aspect-level sentiment analysis method of the present invention, further, cross-entropy loss is calculated between the predicted aspect word boundaries and sentiment polarity and the real sentiment labels of the student's experimental reflection process. All losses are used for backpropagation to update the learnable parameters of the debiased cross-attention mechanism and the BART model.

[0042] This invention also discloses an aspect-level sentiment analysis system based on multimodal information fusion, comprising:

[0043] The data acquisition and alignment module collects audio text, video facial expressions, and physiological data from the experimental reflection process, aligns them temporally according to the timeline of the audio text, and extracts the corresponding original text features, original visual features, and original physiological features for spatial alignment given the reflection segments of the audio text.

[0044] The data debiasing and purification module uses a debiasing cross-attention mechanism guided by visual and physiological prototypes to obtain purified visual features and purified physiological features from the aligned original visual features and original physiological features, respectively.

[0045] The cross-modal perception and alignment module performs cross-modal perception and alignment of the original text features with the purified visual features and purified physiological features respectively, to obtain text features that fuse visual information and text features that fuse physiological information.

[0046] The hierarchical fusion module hierarchically fuses the original text features with fused visual information text features and fused physiological information text features to obtain fine-grained emotional features after multimodal information fusion.

[0047] The prediction module inputs the fine-grained sentiment features into the BART model and outputs the aspect word boundaries and their sentiment polarity predictions during the experimental reflection process.

[0048] In a multimodal information fusion aspect-level sentiment analysis system of the present invention, a parameter update module is further included, which calculates the cross-entropy loss between the aspect word boundaries and sentiment polarity predicted by the prediction module and the real sentiment labels of the student's experimental reflection process, and uses all the losses for backpropagation to update the learnable parameters of the debiased cross-attention mechanism and the BART model.

[0049] The present invention, by adopting the above technical solution, has the following beneficial effects:

[0050] (1) This invention employs a dual-prototype-guided debiasing cross-attention mechanism. Debiasing cross-attention guided by visual and physiological prototypes derives purified visual and physiological features from aligned original visual and physiological features, respectively. This mechanism constructs learnable visual and physiological prototypes in parallel, serving as semantic anchors for explicit micro-expression patterns in visual features and implicit physiological rhythms in physiological features, respectively. Through multi-head cross-attention, it achieves fine-grained intramodal focusing and dynamic debiasing, automatically suppressing noise interference such as irrelevant backgrounds and random physiological fluctuations, outputting pure visual and radar-like emotional representations. This model effectively breaks the modal imbalance dominated by a single text and improves the ability to recognize students' deep emotional states in experimental reflection scenarios.

[0051] (2) This invention proposes a hierarchical trimodal fusion framework. This framework takes the semantics of students' reflection text as the core. First, through cross-modal cross attention, the text features are interacted with the bimodal representations of the debiased visual features and physiological features. The precise alignment of text semantics and fine-grained emotional cues is established to obtain text features that integrate visual information and text features that integrate physiological information. Then, the original text semantics, the explicit emotions in the visual features and the implicit physiological states in the physiological features are hierarchically fused to achieve fine-grained structured fusion of multimodal information, so as to systematically improve the accuracy and robustness of aspect-level emotional tuple analysis.

[0052] In summary, the aspect-level sentiment analysis method and system provided by this invention abandons unguided global attention calculation, and outputs high-purity visual and physiological representations through a dual-prototype parallel guided debiased cross-attention mechanism. Then, through hierarchical alignment, it achieves fine-grained structured fusion of multimodal information, effectively blocking the accumulation of cross-modal errors and systematically improving the accuracy of student reflection sentiment tuple analysis in experimental teaching.

[0053] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0054] Figure 1 This is a flowchart illustrating the steps of an aspect-level sentiment analysis method based on multimodal information fusion according to the present invention.

[0055] Figure 2 This is a structural block diagram of an aspect-level sentiment analysis system based on multimodal information fusion, as shown in the embodiment. Detailed Implementation

[0056] Example

[0057] See Figure 1 The aspect-level sentiment analysis method based on multimodal information fusion of the present invention specifically includes the following steps:

[0058] S1. Collect audio text, video facial expressions, and physiological data from the experimental reflection process. Align the audio text according to the timeline. Given a reflection segment of the audio text, extract the corresponding original text features, original visual features, and original physiological features for spatial alignment.

[0059] S2. Through the debiased cross-attention mechanism guided by visual and physiological prototypes, purified visual features and purified physiological features are obtained from the aligned original visual features and original physiological features, respectively.

[0060] S3. Perform cross-modal perception and alignment between the original text features and the purified visual features and purified physiological features respectively to obtain text features fused with visual information and text features fused with physiological information.

[0061] S4. The original text features are hierarchically fused with the fused visual information text features and the fused physiological information text features to obtain fine-grained emotional features after multimodal information fusion.

[0062] S5. Input the fine-grained sentiment features into the BART model to output the experimental aspect word boundaries and sentiment polarity predictions during the experimental reflection process.

[0063] For details, please refer to [link / reference]. Figure 2 The following is a specific emotional analysis and evaluation process for collecting students' textual reflection reports, video reflection reports, and radar physiological information in a real scientific inquiry experiment reflection and evaluation scenario.

[0064] As described in step S1, firstly, based on the timeline of the collected text reflection report, the data of the three modalities of text, vision, and radar are time-series aligned.

[0065] The text-based reflection report is generated from the audio recordings of students' reflection process. The audio-text data is transcribed into several semantically complete reflection segments through speech recognition. The video reflection report is generated from the facial expression data of students collected during the reflection process. The video expression data is extracted into a video frame sequence according to the start and end times of the reflection segments in the text. The radar physiological data is generated from the students' physiological indicators such as heart rate, respiratory rate, and heart rate variability collected in a non-contact manner during the reflection process. The physiological data is extracted into radar physiological signals corresponding to the start and end times of the reflection segments. The radar physiological signals are resampled onto the time axis of the video frame sequence to obtain a physiological feature sequence that is aligned with the video expression data at the frame level.

[0066] Specifically, such as Figure 2 As shown, the text reflection report T, after being transcribed by an automatic speech recognition system (such as Paraformer), is segmented into several semantically complete reflection segments N. .

[0067] Next, for each reflective sentence / paragraph... Based on the start and end times of the reflective paragraph text The video facial expression data is sampled at a fixed frame rate, and the resulting video frame sequence is obtained after extraction. , ,in For the first The image of the frame, The padding time added to each sentence segment is used to preserve some contextual information at the sentence segment boundaries, preventing the emotional expression near the boundaries from being truncated.

[0068] Since the sampling frequency of the radar physiological signal is inconsistent with the video frame rate, the radar physiological signal is first extracted from the corresponding time interval according to the time window of the reflection sentence text, and then the radar physiological signal is resampled into the video frame sequence. For time points that could not be matched, temporal linear interpolation was used to fill in the missing radar physiological signals, achieving frame-level temporal alignment between radar physiological features and video frame features. Finally, the extracted physiological feature sequence was obtained. .

[0069] Then, the given audio text reflection segments are mapped by the text encoder to obtain context-related vector representations as original text features, the spatiotemporal features of the corresponding video frame sequence are extracted as original visual features, and the temporal feature matrices of various types of physiological indicators in the corresponding physiological feature sequence are extracted as original physiological features. The original text features, original visual features and original physiological features are projected onto a unified shared dimension through a linear projection layer to achieve spatial dimension alignment of the three-modal feature sequences.

[0070] Specifically, such as Figure 2 As shown, for text features, given a reflective sentence segment The pre-trained language model BERT-base-Chinese is used as the text encoder to process given reflective sentence segments. Mapping to context-dependent vector representation Each reflection segment contains L tokens. For text feature dimensions.

[0071] Corresponding reflection paragraphs Video frame sequence within timestamp A pre-trained I3D convolutional neural network is used as a video encoder to extract video frame sequences. spatiotemporal feature set , , For the i-th reflection sentence segment The total number of video frames sampled within the time window, To extract the spatiotemporal features of video frames, This represents the video feature dimension.

[0072] Physiological characteristic sequence Physiological features extracted by a Long Short-Term Memory (LSTM) network are organized into a temporal feature matrix. , , The physiological characteristic dimension.

[0073] Building upon the timestamp alignment at the sentence / segment level, a linear projection layer is used to project all three onto a unified shared dimension d, achieving dimensional alignment of the three-modal feature sequences.

[0074] ;

[0075] ;

[0076] .

[0077] in, These are the learnable projection matrices of the linear projection layer. As a bias term, all parameters of the linear projection layer are updated through backpropagation during training to obtain the aligned original text features. Original visual features and primitive physiological characteristics .

[0078] Step S2 outputs purified visual and radar emotion representations from the original visual features and original physiological features through a dual-prototype guided debiased cross-attention mechanism.

[0079] Specifically, such as Figure 2 As described above, a learnable prototype is defined to capture typical expression patterns within a modality. Specifically, this invention includes a visual prototype. and physiological prototype , where there is a set of E learnable tokens, which are randomly initialized.

[0080] The visual prototype-guided debiasing cross-attention mechanism comprises Y stacked debiasing layers with identical structures. Each debiasing layer's input is the output of the previous layer, and this process is repeated multiple times to refine the visual features. Specifically, in the y-th debiasing layer, the output of the previous debiasing layer is used as the input. With the visual prototype Mapping to obtain visual query vector Combined with the original visual features after spatial alignment Mapped key vector Sum value vector By focusing attention on local salient visual features :

[0081] ;

[0082] .

[0083] in, This is a learnable weight matrix for a visual prototype-guided debiased cross-attention mechanism.

[0084] Next, local salient visual features are extracted using MLP. Mapped to the same dimension d as the original text features, and then compared with the original visual features. The output of the bias removal layer is obtained by splicing the components. This is then used as a visual prototype input to the next debiasing layer, and finally the last debiasing layer outputs the purified visual features. :

[0085] ;

[0086] .

[0087] The physiological prototype-guided debiasing cross-attention mechanism comprises Y stacked debiasing layers with identical structures. Similarly, the input to each debiasing layer is the output of the previous debiasing layer, and this process is repeated multiple times to purify physiological features. In the y-th debiasing layer, the output of the previous debiasing layer is... With the physiological prototype Mapping to obtain physiological query vector Combined with the original physiological characteristics after spatial alignment Mapped key vector Sum value vector By focusing attention on localized, significant physiological features :

[0088] ;

[0089] .

[0090] in, This is a learnable weight matrix for a physiological prototype-guided debiased cross-attention mechanism.

[0091] Then, MLP is used to identify local salient physiological features. Mapped to the same dimension d, and then compared with the original physiological characteristics The output of the debiasing layer is obtained by splicing the layers together, and the purified physiological characteristics are then output from the last debiasing layer. :

[0092] ;

[0093] .

[0094] Step S3 uses the given original text features as semantic anchors to dynamically perceive segments in the purified visual and physiological representations that are strongly related to the text semantics. Through a dual-path independent text-guided multi-head cross-attention mechanism, cross-modal perception and alignment are performed on the purified visual features and purified physiological features respectively to obtain text features that integrate visual information and text features that integrate physiological information.

[0095] In text-guided visual cross-attention, the original text features are used as queries, and the purified visual features are used as keys and values. Through multi-head cross-attention, the fused visual information text features are obtained.

[0096] In text-guided physiological cross-attention, the original text features are used as queries, and the purified physiological features are used as keys and values. Through multi-head cross-attention, the fused physiological information text features are obtained.

[0097] Specifically, such as Figure 2 As shown, in text-guided visual cross-attention, the original text features For the query, refine the visual features. Using key-value pairs, text features that fuse visual information are obtained through multi-head cross-attention. :

[0098] ;

[0099] ;

[0100] .

[0101] in, , and These are the query vector, key vector, and value vector for text-guided visual cross-attention, respectively. For text-guided visual cross-attention, a learnable weight matrix is ​​used. For the output of the j-th attention head in multi-head cross-attention, concat() concatenates the outputs of h attention heads along the feature dimension.

[0102] Similarly, in text-guided physiological cross-attention, the original text features... For Query, purify physiological characteristics Using key-value pairs, text features incorporating physiological information are obtained through multi-head cross-attention. :

[0103] ;

[0104] ;

[0105] .

[0106] in, , and These are the query vector, key vector, and value vector for text-guided physiological cross-attention, respectively. The text-guided physiological cross-attention learnable weight matrix. For the output of the j-th attention head in multi-head cross-attention, concat() concatenates the outputs of h attention heads along the feature dimension.

[0107] Step S4 involves hierarchically fusing the text semantics of the original text features, the explicit emotions of the text features fused with visual information, and the implicit physiological states of the text features fused with physiological information to model the consistency and differences of multimodal emotions and obtain fine-grained emotional features after multimodal information fusion.

[0108] The specific conditions for hierarchical integration are as follows:

[0109] The cosine similarity between the fused visual information text features and the fused physiological information text features is calculated. When the cosine similarity between the fused visual information text features and the fused physiological information text features is within a set threshold range, they are considered to express the same meaning, and the corresponding emotional signal is strengthened. Element-wise multiplication is used to model the fused visual information text features. Text features with fusion of physiological information The collaborative information B between them captures the consistent regions of multimodal emotions:

[0110] .

[0111] When merging visual information and text features Text features with fusion of physiological information When there is inconsistency in expression, absolute difference modeling is used to model the conflicting information C between visual information text features and physiological information text features:

[0112] .

[0113] Combining original text features with collaborative and conflict information to construct interactively enhanced text representations. :

[0114] .

[0115] in, B represents the original text features, and C represents collaborative and conflicting information, respectively. These are learnable weights.

[0116] Dynamically fuse interactive enhanced text representations, original text features, text features fused with visual information, and text features fused with physiological information:

[0117] .

[0118] in, , These are text features that fuse visual information and text features that fuse physiological information, respectively. , , To assign weights to different modalities, a dynamic weight allocation mechanism is introduced, and the weights for different modalities are calculated using softmax(). , This refers to fine-grained sentiment features derived from the fusion of multimodal information.

[0119] Finally, in step S5, the fused fine-grained sentiment features are input into the BART model to predict the aspect word boundaries and their sentiment polarities in the student's experimental reflection process in an end-to-end manner. For example, for "I am confused about the experimental procedure", the model should identify the aspect word "experimental procedure" and output the sentiment polarity "negative".

[0120] Specifically, the BART model includes an encoder and a decoder. The encoder of the BART model processes fine-grained sentiment features of the input. Encode the hidden state sequence to output a context-aware fusion. :

[0121] .

[0122] The decoder of the BART model fuses the context-aware hidden state sequence of the encoder output. Decode the code and output the context information of the current time step. :

[0123] .

[0124] in, This is the decoder output from the previous time step.

[0125] Finally, the predicted experimental aspect words and reflection sentiment polarity sequences were calculated using softmax(). :

[0126] ;

[0127] in, This is the weight matrix. This is the bias vector.

[0128] As a widely adopted baseline model, the BART model generates experimental aspect words and their corresponding emotional polarity target sequences in an autoregressive manner by inputting fine-grained sentiment features fused from multiple sources into the encoder and decoder.

[0129] The total loss is calculated by combining the predicted aspect word boundaries and sentiment polarity with the true sentiment labels of the student's experimental reflection process. All losses are then used for backpropagation to update all learnable parameters in the models, attention mechanisms, and BART encoder-decoder involved in the above steps. Iterative training continues by repeating step S2 until the set number of iterations is reached, at which point training stops.

[0130] Specifically, this invention uses cross-entropy loss to optimize the tuple extraction task. First, the loss is generated. The supervised decoder autoregressively generates complete sequences containing aspect word boundaries and corresponding sentiment polarities to ensure the overall correctness of the model output, where l is the length of the decoder output sequence Y:

[0131] .

[0132] To enhance the fine-grained sentiment features after multimodal information fusion To enhance sentiment discrimination capabilities and avoid ineffective fusion features, an auxiliary classification head is used to predict sentiment output based solely on the fusion features. .

[0133] .

[0134] The auxiliary classification head uses fine-grained sentiment features fused from multimodal information. Given input, output the predicted sentiment polarity. And calculate the cross-entropy loss with the real sentiment labels s. The prediction process relies solely on fused features, and its prediction accuracy is used to evaluate the gains made by visual and physiological modal information in emotion discrimination.

[0135] .

[0136] The final model's total loss ,in The weighting coefficients are used to assist in the classification loss.

[0137] like Figure 2As shown, this embodiment also discloses a multimodal information fusion aspect-level sentiment analysis system to implement the above-mentioned steps and processes of this embodiment, specifically including a data acquisition and alignment module 100, a data debiasing and purification module 200, a cross-modal perception and alignment module 300, a hierarchical fusion module 400, and a prediction module 500. The data acquisition and alignment module 100 collects audio text, video facial expressions, and physiological data from the experimental reflection process, aligns them temporally according to the audio text timeline, and extracts corresponding original text features, original visual features, and original physiological features for spatial alignment for given reflection segments of the audio text. The data debiasing and purification module 200 obtains purified visual features and purified physiological features from the aligned original visual features and original physiological features through a debiasing cross-attention mechanism guided by visual and physiological prototypes. The cross-modal perception and alignment module 300 performs cross-modal perception and alignment of the original text features with the purified visual features and purified physiological features to obtain text features that fuse visual information and text features that fuse physiological information. The hierarchical fusion module 400 performs hierarchical fusion of the original text features with the text features that fuse visual information and text features that fuse physiological information to obtain fine-grained emotional features after multimodal information fusion. The prediction module 500 inputs the fine-grained emotional features into the BART model and outputs the aspect word boundaries and their emotional polarity predictions during the experimental reflection process.

[0138] It also includes a parameter update module, which calculates the cross-entropy loss of the aspect word boundaries and sentiment polarity predicted by the prediction module 500 and the real sentiment labels of the student's experimental reflection process, and uses all the losses for backpropagation to update the parameters of the debiased cross-attention mechanism and the learnable parameters of the BART model.

[0139] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0140] In this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a step or method that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such a step or method.

[0141] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A multimodal information fusion aspect-level sentiment analysis method, characterized in that... Includes the following steps: S1. Collect audio text, video facial expressions, and radar physiological data of the experimental reflection process, align them temporally according to the timeline of the audio text, and extract the corresponding original text features, original visual features, and original physiological features for spatial alignment given the reflection segments of the audio text. S2. Through the debiased cross-attention mechanism guided by visual and physiological prototypes, purified visual features and purified physiological features are obtained from the aligned original visual features and original physiological features, respectively. Define learnable visual and physiological prototypes; The visual prototype-guided debiasing cross-attention mechanism includes Y stacked debiasing layers with identical structures. In the y-th debiasing layer, the output of the previous debiasing layer is mapped to the visual prototype to obtain a visual query vector. Combined with the key vector and value vector mapped from the original visual features after spatial alignment, attention is focused on local salient visual features. The local salient visual features are mapped to the same dimension as the original text features through MLP, and then concatenated with the original visual features to obtain the output of the debiasing layer. The output of the last debiasing layer is used to purify the visual features. The physiological prototype-guided debiasing cross-attention mechanism includes Y stacked debiasing layers with identical structures. In the y-th debiasing layer, the output of the previous debiasing layer is mapped to the physiological prototype to obtain a physiological query vector. Combined with the key vector and value vector mapped from the original physiological features after spatial alignment, attention is focused on local salient physiological features. The local salient physiological features are mapped to the same dimension as the original text features through MLP, and then concatenated with the original physiological features to obtain the output of the debiasing layer. The physiological features are purified by the output of the last debiasing layer. S3. Perform cross-modal perception and alignment between the original text features and the purification visual features and purification physiological features respectively to obtain text features fused with visual information and text features fused with physiological information. S4. The original text features, the fused visual information text features, and the fused physiological information text features are hierarchically fused to obtain fine-grained emotional features after multimodal information fusion. When the cosine similarity between the fused visual information text features and the fused physiological information text features is consistent, element-wise multiplication is used to model the collaborative information between the fused visual information text features and the fused physiological information text features to capture the consistent regions of multimodal emotions. When the cosine similarity between the fused visual information text features and the fused physiological information text features is inconsistent, the absolute difference is used to model the conflict information between the fused visual information text features and the fused physiological information text features. Combining original text features with collaborative and conflict information to construct interactively enhanced text representations. : ； in, B represents the original text features, and C represents collaborative and conflicting information, respectively. These are learnable weights; Dynamically fuse interactive enhanced text representations, original text features, text features fused with visual information, and text features fused with physiological information: ； in, , These are text features that fuse visual information and text features that fuse physiological information, respectively. , , Weights for different modalities, Fine-grained sentiment features derived from the fusion of multimodal information; S5. Input the fine-grained emotional features into the BART model and output the aspect word boundaries and their emotional polarity predictions in the experimental reflection process.

2. The aspect-level sentiment analysis method based on multimodal information fusion according to claim 1, characterized in that: In step S1, the audio-text data is segmented into several semantically complete reflective sentence segments through speech recognition transcription. For video facial expression data, extract video frame sequences according to the start and end times of the reflective sentence segments; For physiological data, radar physiological signals are extracted from the corresponding time intervals based on the start and end times of the reflection segments. The radar physiological signals are then resampled onto the time axis of the video frame sequence to extract a physiological feature sequence that is aligned with the video facial expression data at the frame level.

3. The aspect-level sentiment analysis method based on multimodal information fusion according to claim 2, characterized in that: The given audio text reflection segment is mapped by a text encoder to obtain a context-related vector representation as the original text feature. The spatiotemporal features of the corresponding video frame sequence are extracted as the original visual features. The temporal feature matrix of each type of physiological indicator in the corresponding physiological feature sequence is extracted as the original physiological feature. By using a linear projection layer, the original text features, original visual features, and original physiological features are projected onto a unified shared dimension, achieving spatial dimension alignment of the three-modal feature sequences.

4. The aspect-level sentiment analysis method based on multimodal information fusion according to claim 1, characterized in that: In step S3, using the original text features as semantic anchors, a dual-path independent text-guided multi-head cross-attention mechanism is used to perform cross-modal perception and alignment of the purification visual features and purification physiological features, respectively. In text-guided visual cross-attention, the original text features are used as queries, and the purified visual features are used as keys and values. Through multi-head cross-attention, the fused visual information text features are obtained. In text-guided physiological cross-attention, the original text features are used as queries, and the purified physiological features are used as keys and values. Through multi-head cross-attention, the fused physiological information text features are obtained.

5. The aspect-level sentiment analysis method based on multimodal information fusion according to claim 1, characterized in that: In step S5, the encoder of the BART model processes the fine-grained sentiment features of the input. Encode the hidden state sequence to output a context-aware fusion. : ； The decoder of the BART model fuses context-aware hidden state sequences. Decode the code and output the context information of the current time step. : ； in, The decoder output from the previous time step is used to calculate the predicted aspect word boundaries and sentiment polarity target sequence using softmax(). : ； in, This is the weight matrix. This is the bias vector.

6. The aspect-level sentiment analysis method based on multimodal information fusion according to claim 1, characterized in that: Cross-entropy loss is calculated for the predicted aspect word boundaries and sentiment polarity and the true sentiment labels of the student's experimental reflection process. All losses are used for backpropagation to update the learnable parameters of the debiased cross-attention mechanism and the BART model.

7. A multimodal information fusion aspect-level sentiment analysis system, characterized in that, include: The data acquisition and alignment module collects audio text, video facial expressions, and physiological data from the experimental reflection process, aligns them temporally according to the timeline of the audio text, and extracts the corresponding original text features, original visual features, and original physiological features for spatial alignment given the reflection segments of the audio text. The data debiasing and purification module uses a debiasing cross-attention mechanism guided by visual and physiological prototypes to obtain purified visual features and purified physiological features from the aligned original visual features and original physiological features, respectively. Define learnable visual and physiological prototypes; The visual prototype-guided debiasing cross-attention mechanism includes Y stacked debiasing layers with identical structures. In the y-th debiasing layer, the output of the previous debiasing layer is mapped to the visual prototype to obtain a visual query vector. Combined with the key vector and value vector mapped from the original visual features after spatial alignment, attention is focused on local salient visual features. The local salient visual features are mapped to the same dimension as the original text features through MLP, and then concatenated with the original visual features to obtain the output of the debiasing layer. The output of the last debiasing layer is used to purify the visual features. The physiological prototype-guided debiasing cross-attention mechanism includes Y stacked debiasing layers with identical structures. In the y-th debiasing layer, the output of the previous debiasing layer is mapped to the physiological prototype to obtain a physiological query vector. Combined with the key vector and value vector mapped from the original physiological features after spatial alignment, attention is focused on local salient physiological features. The local salient physiological features are mapped to the same dimension as the original text features through MLP, and then concatenated with the original physiological features to obtain the output of the debiasing layer. The physiological features are purified by the output of the last debiasing layer. The cross-modal perception and alignment module performs cross-modal perception and alignment of the original text features with the purified visual features and purified physiological features respectively, to obtain text features fused with visual information and text features fused with physiological information. The hierarchical fusion module hierarchically fuses the original text features with fused visual information text features and fused physiological information text features to obtain fine-grained emotional features after multimodal information fusion. When the cosine similarity between the fused visual information text features and the fused physiological information text features is consistent, element-wise multiplication is used to model the collaborative information between the fused visual information text features and the fused physiological information text features to capture the consistent regions of multimodal emotions. When the cosine similarity between the fused visual information text features and the fused physiological information text features is inconsistent, the absolute difference is used to model the conflict information between the fused visual information text features and the fused physiological information text features. Combining original text features with collaborative and conflict information to construct interactively enhanced text representations. : ； in, B represents the original text features, and C represents collaborative and conflicting information, respectively. These are learnable weights; Dynamically fuse interactive enhanced text representations, original text features, text features fused with visual information, and text features fused with physiological information: ； in, , These are text features that fuse visual information and text features that fuse physiological information, respectively. , , Weights for different modalities, Fine-grained sentiment features derived from the fusion of multimodal information; The prediction module inputs the fine-grained sentiment features into the BART model and outputs the aspect word boundaries and their sentiment polarity predictions during the experimental reflection process.

8. The aspect-level sentiment analysis system based on multimodal information fusion according to claim 7, characterized in that: It also includes a parameter update module, which calculates the cross-entropy loss between the aspect word boundaries and sentiment polarity predicted by the prediction module and the real sentiment labels of the students' experimental reflection process, and uses all the losses for backpropagation to update the parameters of the debiased cross-attention mechanism and the learnable parameters of the BART model.

Images