A multimodal sentiment analysis method for online education

Abstract

The present invention relates to a multimodal sentiment analysis method for online education. The method comprises obtaining a text comprehensive modal representation, a speech comprehensive modal representation, and a visual comprehensive modal representation in data to be analyzed, and performing multimodal fusion. The method performs analysis in a multimodal manner and utilizes the complementarity between different modalities to improve the comprehensiveness and accuracy of sentiment recognition. The method further comprises determining an optimal wavelet basis function and an optimal number of decomposition layers based on the fusion result, performing a discrete wavelet transform on the fusion result to obtain high-frequency features and low-frequency features, and finely separating local frequency characteristics. The method further comprises determining a multi-frequency graph attention weight based on the high-frequency features and the low-frequency features, performing feature fusion on the high-frequency features and the low-frequency features based on the multi-frequency graph attention weight to obtain a multimodal sentiment analysis result, capturing the synergy between modalities at different frequency levels, effectively fusing heterogeneous data, more comprehensively understanding complex sentiment expressions, and improving the accuracy of multimodal sentiment recognition.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal sentiment analysis method for online education. Background Art

[0002] With the prevalence of online education, students' learning processes are gradually shifting from traditional face-to-face instruction to virtual, digital environments. However, a key challenge facing online education is the difficulty for teachers to perceive students' emotional states in real time, such as their focus, interest, confusion, or anxiety. These emotional states directly impact students' learning outcomes and experience. Emotion recognition, a key research area in artificial intelligence, aims to identify human emotions by analyzing multimodal data such as speech, text, and images. Although various methods have attempted to address the problem of sentiment analysis in online education, these methods still have significant shortcomings in separating high- and low-frequency signals and interacting with multimodal features.

[0003] Existing methods have the following problems in processing high- and low-frequency signals: Insufficient separation of high- and low-frequency signals: Traditional methods (such as Fourier transforms) have difficulty finely separating high-frequency signals based on the local frequency characteristics of the signal, resulting in loss or confusion of high- and low-frequency information. Insufficient interaction of multimodal features: Emotional data in online education scenarios (such as text, speech, and vision) have different frequency characteristics and time scales, and existing methods have difficulty effectively fusing these heterogeneous data. Existing methods typically directly fuse raw features and lack the collaborative modeling of multimodal features at the frequency level, resulting in limited understanding of complex emotional expressions.

[0004] Therefore, there is an urgent need to propose a multimodal sentiment analysis method for online education to solve the technical problems that the high- and low-frequency signal processing methods in the existing technology cannot finely separate local frequency characteristics and cannot effectively fuse heterogeneous data, resulting in low accuracy of multimodal sentiment recognition. Summary of the Invention

[0005] In view of this, it is necessary to provide a multimodal sentiment analysis method for online education to solve the technical problem of low accuracy of multimodal sentiment recognition in the existing technology.

[0006] In order to solve the above problems, in a first aspect, the present invention provides a multimodal sentiment analysis method for online education, comprising:

[0007] Obtain the data to be analyzed during the conversation process, extract features from the data, and encode them to obtain textual, speech, and visual comprehensive modal representations.

[0008] Performing multimodal fusion on the text comprehensive modal representation, the speech comprehensive modal representation, and the visual comprehensive modal representation to obtain a fusion result;

[0009] Determining an optimal wavelet basis function and an optimal number of decomposition layers according to the fusion result, and performing discrete wavelet transform on the fusion result according to the optimal wavelet basis function and the optimal number of decomposition layers to obtain high-frequency features and low-frequency features;

[0010] According to the high-frequency features and the low-frequency features, a multi-frequency graph attention weight is determined, and the high-frequency features and the low-frequency features are feature fused according to the multi-frequency graph attention weight to obtain a multimodal sentiment analysis result.

[0011] In a possible implementation, the performing multimodal fusion on the text comprehensive modal representation, the speech comprehensive modal representation, and the visual comprehensive modal representation to obtain a fusion result includes:

[0012] Determining a plurality of nodes according to the text comprehensive modality representation, the voice comprehensive modality representation, and the visual comprehensive modality corresponding to each utterance in the conversation process;

[0013] Connect each node to other nodes of the same modality in the same conversation to obtain the intra-modal connection structure;

[0014] Connecting each of the nodes to other nodes of other modalities in the same conversation to obtain an inter-modal connection structure;

[0015] An undirected graph is obtained according to the multiple nodes, the intra-modality connection structure, and the inter-modality connection structure; the undirected graph is a fusion result.

[0016] In a possible implementation, before determining the optimal wavelet basis function and the optimal number of decomposition layers according to the fusion result, and performing discrete wavelet transform on the fusion result according to the optimal wavelet basis function and the optimal number of decomposition layers to obtain high-frequency features and low-frequency features, the method further includes:

[0017] Normalizing the features of each node in the fusion result to obtain a unified feature;

[0018] The high-frequency signal in the unified feature is processed in sections to obtain the input signal of each node.

[0019] In a possible implementation, determining an optimal wavelet basis function and an optimal number of decomposition layers according to the fusion result, and performing discrete wavelet transform on the fusion result according to the optimal wavelet basis function and the optimal number of decomposition layers to obtain high-frequency features and low-frequency features includes:

[0020] Calculating the input signal according to a preset wavelet basis function candidate set to determine an optimal wavelet basis function;

[0021] Determining the optimal number of decomposition layers according to the frequency characteristics of the input signal;

[0022] Performing discrete wavelet transform on the input signal according to the optimal wavelet basis function and the optimal number of decomposition layers to obtain low-frequency coefficients and high-frequency coefficients;

[0023] The input signal is reconstructed according to the low-frequency coefficients and the high-frequency coefficients to obtain a segmented signal; the segmented signal includes high-frequency features and low-frequency features.

[0024] In a possible implementation, calculating the input signal according to a preset wavelet basis function candidate set to determine an optimal wavelet basis function includes:

[0025] Calculating each wavelet basis function in the preset wavelet basis function candidate set with the input signal to obtain a corresponding matching degree;

[0026] According to all the matching degrees, the optimal wavelet basis function with the highest matching degree in the wavelet basis function candidate set is determined.

[0027] In a possible implementation, determining the optimal number of decomposition layers according to the frequency characteristics of the input signal includes:

[0028] Performing Fourier transform on the input signal to obtain a frequency spectrum;

[0029] Obtaining energy distribution of the input signal according to the frequency spectrum;

[0030] determining a main frequency range according to the energy distribution;

[0031] The main frequency range and the preset sampling frequency are calculated to obtain the optimal number of decomposition layers.

[0032] In a possible implementation, determining the multi-frequency graph attention weight according to the high-frequency features and the low-frequency features includes:

[0033] Performing linear transformation on the high-frequency features and the low-frequency features respectively to obtain high-frequency linear features and low-frequency linear features;

[0034] Calculating the high-frequency linear features of each node and the high-frequency linear features of the corresponding neighboring nodes to obtain the high-frequency attention weight of the corresponding node;

[0035] Calculating the low-frequency linear features of each node and the low-frequency linear features of the corresponding neighboring nodes to obtain the low-frequency attention weight of the corresponding node;

[0036] A multi-frequency graph attention weight is obtained according to the high-frequency attention weight and the low-frequency attention weight.

[0037] In a possible implementation, the performing feature fusion on the high-frequency features and the low-frequency features according to the multi-frequency graph attention weights to obtain a multimodal sentiment analysis result includes:

[0038] Weighting the high-frequency features of the neighboring nodes of each node according to the multi-frequency graph attention weight to obtain a high-frequency aggregation result;

[0039] Dynamically fusing the low-frequency features with the high-frequency aggregation results to obtain a multimodal fusion result;

[0040] The multimodal fusion result is input into the classifier to obtain the multimodal sentiment analysis result.

[0041] In a possible implementation, determining a main frequency range according to the energy distribution includes:

[0042] According to the energy distribution, a cumulative energy distribution curve is obtained;

[0043] The main frequency range is determined according to the position of the preset total energy in the cumulative energy distribution curve.

[0044] In one possible implementation, extracting and encoding features of the data to be analyzed to obtain a textual comprehensive modal representation, a speech comprehensive modal representation, and a visual comprehensive modal representation includes:

[0045] Extracting features from the data to be analyzed based on a multilingual model to obtain text features;

[0046] Extracting features from the data to be analyzed based on a preset toolkit to obtain speech features;

[0047] Extracting features from the data to be analyzed based on a three-dimensional deep learning model to obtain visual features;

[0048] Encoding the text features according to the bidirectional GRU to obtain a comprehensive modal representation of the text;

[0049] The speech features and the visual features are respectively encoded according to a single-layer MLP to obtain a speech comprehensive modality representation and a visual comprehensive modality representation.

[0050] The beneficial effects of the present invention are as follows: feature extraction and encoding are performed on the data to be analyzed to obtain text comprehensive modal representation, speech comprehensive modal representation and visual comprehensive modal representation, so that the text comprehensive modal representation, speech comprehensive modal representation and visual comprehensive modal representation can be multimodally fused to obtain a fusion result, so that analysis can be performed in a multimodal manner (text, speech and vision), making full use of the complementarity between different modalities, and improving the comprehensiveness and accuracy of emotion recognition; the optimal wavelet basis function and the optimal number of decomposition layers can be determined according to the fusion result, and the fusion result is subjected to discrete wavelet transform according to the optimal wavelet basis function and the optimal number of decomposition layers to obtain high-frequency features and low-frequency features, so as to more accurately extract emotion-related high- and low-frequency information, so as to finely separate local frequency characteristics; the multi-frequency graph attention weight can also be determined according to the high-frequency features and low-frequency features, and the high-frequency features and low-frequency features are feature fused according to the multi-frequency graph attention weight to obtain a multimodal emotion analysis result, so that the synergy between modalities can be captured at different frequency levels, so that heterogeneous data can be effectively integrated, complex emotional expressions can be more comprehensively understood, and the accuracy of multimodal emotion recognition can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of an embodiment of the multimodal sentiment analysis method for online education provided by the present invention;

[0052] Figure 2 For the present invention Figure 1 A schematic flow chart of an embodiment of step S101;

[0053] Figure 3 For the present invention Figure 1 A schematic flow chart of an embodiment of step S102;

[0054] Figure 4 For the present invention Figure 1 A schematic flow chart of an embodiment of step S103;

[0055] Figure 5 For the present invention Figure 4 A schematic flow chart of an embodiment of step S402;

[0056] Figure 6 For the present invention Figure 1 A schematic flow chart of an embodiment of step S104;

[0057] Figure 7 A schematic structural diagram of an embodiment of the multimodal sentiment analysis device for online education provided by the present invention. DETAILED DESCRIPTION

[0058] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0059] like Figure 1 As shown, a specific embodiment of the present invention discloses a multimodal sentiment analysis method for online education, including:

[0060] S101. Obtain data to be analyzed during a conversation, perform feature extraction and encoding on the data to be analyzed, and obtain a text comprehensive modal representation, a speech comprehensive modal representation, and a visual comprehensive modal representation.

[0061] The multimodal sentiment analysis method for online education provided in the embodiment of the present application can be applied to a multimodal sentiment analysis system, wherein the multimodal sentiment analysis can be based on a software system running on a terminal device, and the terminal device can be a server, a tablet computer, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a mobile phone and other terminal devices. The embodiment of the present application does not impose any restrictions on the specific type of the terminal device.

[0062] Among them, students and teachers communicate and learn through an online education platform, but online education teachers cannot know the students' emotions and cannot teach students according to their actual situation. Therefore, an embodiment of the present invention proposes a multimodal sentiment analysis method for online education, which can analyze the data in the communication process between students and teachers, so that teachers can know the students' situation more clearly. The communication between students and teachers includes but is not limited to text, voice, video and other methods. The communication process between students and teachers can include multiple dialogues, thereby forming a dialogue mode. When teachers and students have a dialogue on the online education platform, the data to be analyzed in the dialogue process can be obtained, so that the data to be analyzed can be feature extracted and encoded to obtain text comprehensive modal representation, voice comprehensive modal representation and visual comprehensive modal representation.

[0063] S102: Perform multimodal fusion on the text comprehensive modal representation, the speech comprehensive modal representation, and the visual comprehensive modal representation to obtain a fusion result.

[0064] Among them, multimodal fusion can be used for construction. Specifically, a graph can be constructed based on the text comprehensive modal representation, speech comprehensive modal representation and visual comprehensive modal representation to obtain an undirected graph, thereby using multimodal data (such as text, speech and vision) for sentiment analysis, making full use of the complementarity between different modalities, and improving the comprehensiveness and accuracy of sentiment recognition.

[0065] S103 , determining an optimal wavelet basis function and an optimal number of decomposition layers according to the fusion result, and performing discrete wavelet transform on the fusion result according to the optimal wavelet basis function and the optimal number of decomposition layers to obtain high-frequency features and low-frequency features.

[0066] Among them, to address the problem of insufficient separation of high and low frequency signals, a high and low frequency separation method based on adaptive wavelet transform is proposed. Unlike the traditional fixed frequency separation strategy, the adaptive wavelet transform can dynamically adjust the frequency separation parameters according to the local frequency characteristics of the signal, thereby more accurately extracting emotion-related high and low frequency information. After completing the multimodal fusion to obtain an undirected graph, the feature representation of each node in the graph can be used as the input signal to perform an adaptive wavelet transform operation. Specifically, the optimal wavelet basis function and the optimal number of decomposition layers are determined based on the fusion results. The input signal is then discretely wavelet transformed based on the optimal wavelet basis function and the optimal number of decomposition layers to obtain high-frequency features and low-frequency features. This method is more flexible in capturing the non-stationarity and local characteristics of the signal, and can better adapt to the complexity of the spectral distribution in multimodal emotional data.

[0067] S104: Determine the multi-frequency graph attention weights based on the high-frequency features and the low-frequency features, and perform feature fusion on the high-frequency features and the low-frequency features based on the multi-frequency graph attention weights to obtain a multimodal sentiment analysis result.

[0068] To address the problem of insufficient interaction between multimodal features, an embodiment of the present invention proposes a multi-frequency graph attention mechanism. This mechanism constructs a multimodal graph structure based on the high- and low-frequency signals extracted by adaptive wavelet transform, calculates high-frequency and low-frequency features, and determines the multi-frequency graph attention weights. This allows the interaction between multimodal features to be modeled using the graph attention mechanism. In this way, the model can capture the synergy between modalities at different frequency levels, thereby providing a more comprehensive understanding of complex emotional expressions.

[0069] Compared with the prior art, this embodiment provides feature extraction and encoding of the data to be analyzed to obtain a text comprehensive modal representation, a speech comprehensive modal representation, and a visual comprehensive modal representation, thereby performing multimodal fusion on the text comprehensive modal representation, the speech comprehensive modal representation, and the visual comprehensive modal representation to obtain a fusion result. This allows for analysis in a multimodal manner (text, speech, and vision), fully utilizing the complementarity between different modalities to improve the comprehensiveness and accuracy of emotion recognition. The optimal wavelet basis function and the optimal number of decomposition layers can also be determined based on the fusion result, and the fusion result can be discretely wavelet transformed based on the optimal wavelet basis function and the optimal number of decomposition layers to obtain high-frequency features and low-frequency features, thereby more accurately extracting emotion-related high- and low-frequency information, thereby enabling fine separation of local frequency characteristics. The multi-frequency graph attention weights can also be determined based on the high-frequency and low-frequency features, and the high-frequency and low-frequency features can be feature fused based on the multi-frequency graph attention weights to obtain a multimodal emotion analysis result, thereby capturing the synergy between modalities at different frequency levels, effectively fusing heterogeneous data, more comprehensively understanding complex emotional expressions, and improving the accuracy of multimodal emotion recognition.

[0070] In some embodiments of the present invention, Figure 2 As shown, step S101 includes:

[0071] S201 : Extract features from the data to be analyzed based on a multilingual model to obtain text features.

[0072] Among them, a multilingual model can be constructed, and the multilingual model can be a fine-tuned RoBERTa Large model. The fine-tuned RoBERTa Large model can be used to perform feature extraction on the data to be analyzed, and text features can be extracted.

[0073] S202: Extract features from the data to be analyzed based on a preset toolkit to obtain speech features.

[0074] A preset toolkit may be set, and the preset toolkit may be an openSMILE toolkit. The openSMILE toolkit may be used to perform feature extraction on the data to be analyzed, and speech features may be extracted.

[0075] S203: Extract features from the data to be analyzed based on the three-dimensional deep learning model to obtain visual features.

[0076] Among them, a three-dimensional deep learning model can be constructed. The three-dimensional deep learning model can be a pre-trained DenseNet and 3D-CNN model. By using the pre-trained DenseNet and 3D-CNN models to extract features of the data to be analyzed, visual features can be obtained.

[0077] S204, encoding the text features according to the bidirectional GRU to obtain a comprehensive modal representation of the text;

[0078] S205 , encoding the speech features and the visual features according to the single-layer MLP to obtain a speech comprehensive modal representation and a visual comprehensive modal representation.

[0079] Among them, text features can be encoded by bidirectional GRU, and speech and visual features can be encoded by a single-layer MLP. In addition, one-hot encoding and lookup table are used to generate speaker embedding representation, which is added to the encoded modal features to generate a comprehensive modal representation containing contextual information. That is, the text features generate a text comprehensive modal representation, and the speech features and visual features generate a speech comprehensive modal representation and a visual comprehensive modal representation, respectively.

[0080] In some embodiments of the present invention, Figure 3 As shown, step S102 includes:

[0081] S301. Determine multiple nodes according to the text comprehensive modal representation, voice comprehensive modal representation, and visual comprehensive modal representation corresponding to each utterance in the conversation process.

[0082] Among them, for the first The textual comprehensive modal representation, speech comprehensive modal representation and visual comprehensive modal representation of an utterance are respectively expressed as 、 、 In the initial state, the feature embedding of each node is consistent with its sequential encoding representation, which is recorded as .

[0083] S302: Connect each node with other nodes of the same modality in the same conversation to obtain an intra-modal connection structure.

[0084] Among them, each node Connected to other nodes of the same modality in the same conversation, i.e. The definition of the intra-modality connection structure is shown in formula (1), and this edge models the contextual relationship between the same modality.

[0085] (1)

[0086] Where, Represents the intra-modal connection structure.

[0087] S303: Connect each node with other nodes of other modalities in the same conversation to obtain an inter-modal connection structure.

[0088] Among them, each node is connected to nodes of other modalities in the same discourse, i.e. The definition of inter-modal connections is shown in formula (2), which models the complementary information between different modalities.

[0089] (2)

[0090] Where, Represents the inter-modal connection structure.

[0091] S304: Obtain an undirected graph based on the multiple nodes, the intra-modal connection structure, and the inter-modal connection structure. The undirected graph is a fusion result.

[0092] After completing the construction of nodes and edges, the embodiment of the present invention constructs an undirected graph , represents a node set consisting of a specific mode of discourse in the conversation, i.e., multiple nodes in step S301, and Represents an edge set used to model the relationship between nodes by step S302 and S303 The edges of .

[0093] In some embodiments of the present invention, before step S103, the method further includes:

[0094] Normalize the features of each node in the fusion result to obtain unified features;

[0095] The high-frequency signal in the unified feature is processed segmentally to obtain the input signal of each node.

[0096] Among them, after the undirected graph structure is constructed, the feature representation of each node in the graph can be used as the input signal to perform adaptive wavelet transform operation. However, due to the different modal characteristics and complex edge connection structure of the nodes in the graph, directly processing these node signals may be affected by scale differences or modal noise. Therefore, before decomposing the node signal, the input node signal needs to be preprocessed to ensure that it is suitable for wavelet decomposition. First, the node signal in the undirected graph is converted into The features of are normalized to a unified scale to obtain unified features and eliminate dimensional differences. The method is shown in formula (3):

[0097] (3)

[0098] Where, is the mean of the signal, is the standard deviation of the signal.

[0099] Then the high-frequency signal in the unified feature is segmented, and the length of each segment is , get the input signal of each node, the method is shown in formula (4):

[0100] (4)

[0101] Where, T is the total signal length, t is the time variable, is the segment length.

[0102] In some embodiments of the present invention, Figure 4 As shown, step S103 includes:

[0103] S401 : Calculate the input signal according to a preset wavelet basis function candidate set to determine the optimal wavelet basis function.

[0104] Among them, a preset wavelet basis function candidate set can be set, and the preset wavelet basis function candidate set can be determined according to actual conditions. The preset wavelet basis function candidate set can be ,in, It is k wavelet basis functions, so that the input signal can be calculated according to the preset wavelet basis function candidate set to determine the optimal wavelet basis function.

[0105] In some embodiments of the present invention, step S401 includes:

[0106] Calculate each wavelet basis function in the preset wavelet basis function candidate set with the input signal to obtain the corresponding matching degree.

[0107] Among them, each wavelet basis function in the preset wavelet basis function candidate set can be calculated one by one with the input signal, and the matching degree corresponding to each wavelet basis function can be obtained. , calculated as shown in formula (5):

[0108] (5)

[0109] Where, is the time shift parameter.

[0110] According to all matching degrees, the optimal wavelet basis function with the highest matching degree in the wavelet basis function candidate set is determined.

[0111] Among them, after obtaining all the matching degrees of all wavelet basis functions in the preset wavelet basis function candidate set, the wavelet basis function with the highest matching degree can be determined as the optimal wavelet basis function, as shown in formula (6):

[0112] (6)

[0113] S402: Determine the optimal number of decomposition layers according to the frequency characteristics of the input signal.

[0114] Among them, the optimal number of decomposition layers can be dynamically determined according to the frequency characteristics of the input signal.

[0115] In some embodiments of the present invention, Figure 5 As shown, step S402 includes:

[0116] S501: Perform Fourier transform on the input signal to obtain a frequency spectrum.

[0117] Among them, the input signal can be Perform Fourier transform to obtain the spectrum , as shown in formula (7):

[0118] (7)

[0119] Where, f is the frequency, j Is an imaginary unit.

[0120] S502: Obtain energy distribution of the input signal according to the frequency spectrum.

[0121] The energy distribution of the input signal can be obtained based on the spectrum, as shown in formula (8):

[0122] (8)

[0123] S503: Determine the main frequency range according to the energy distribution.

[0124] Among them, the main frequency range of the signal [f min ,f max ].

[0125] In some embodiments of the present invention, determining the main frequency range according to the energy distribution includes:

[0126] According to the energy distribution, a cumulative energy distribution curve is obtained.

[0127] Among them, according to the energy distribution , calculate the cumulative energy distribution curve , calculated as shown in formula (9):

[0128] (9)

[0129] in, is the frequency of the integration variable, The sampling frequency of the signal can be set according to the actual situation. In the embodiment of the present invention, 4 Hz can be used for text, 16 kHz can be used for voice signal, and 25 Hz can be used for video signal.

[0130] The main frequency range is determined according to the position of the preset total energy in the cumulative energy distribution curve.

[0131] Among them, you can set the preset total energy, for example, 5% and 95%, you can observe The growth of the energy is used to determine the frequencies at which the cumulative energy first reaches 5% and 95% of the total energy, which are defined as the lower and upper bounds of the main frequency range, respectively, as shown in formulas (10) and (11):

[0132] (10)

[0133] (11)

[0134] S504: Calculate the main frequency range and the preset sampling frequency to obtain the optimal number of decomposition layers.

[0135] Among them, the optimal number of decomposition layers can then be calculated based on the main frequency range and the preset sampling frequency , calculated as shown in formula (12):

[0136] (12)

[0137] S403 , performing discrete wavelet transform on the input signal according to the optimal wavelet basis function and the optimal number of decomposition layers to obtain low-frequency coefficients and high-frequency coefficients.

[0138] Among them, the selected optimal wavelet basis function can be used and decomposition levels , perform discrete wavelet transform (DWT) on the input signal. Specifically, in In layer decomposition, the input signal is decomposed into low-frequency coefficients and high frequency coefficients , as shown in formulas (13) and (14):

[0139] (13)

[0140] (14)

[0141] Where, and are the coefficients of the low-pass filter and the high-pass filter, respectively. is the low-frequency coefficient of the previous layer. Then repeat the above process until the number of decomposition layers is reached .

[0142] S404 , reconstructing the input signal according to the low-frequency coefficients and the high-frequency coefficients to obtain a segmented signal; the segmented signal includes high-frequency features and low-frequency features.

[0143] Among them, the low-frequency and high-frequency coefficients obtained by decomposition are used to reconstruct the original signal. Specifically, in the In layer reconstruction, the low-frequency and high-frequency coefficients are merged into the low-frequency coefficients of the previous layer. The calculation method is shown in formula (15):

[0144] (15)

[0145] Where, and are the coefficients of the reconstructed low-pass filter and high-pass filter respectively. Repeat the above process until all layers are reconstructed and the segmented signal is restored. , segmented signal Including high-frequency features and low-frequency features obtained by segmentation.

[0146] In some embodiments of the present invention, Figure 6 As shown, step S104 includes:

[0147] S601 , performing linear transformation on the high-frequency features and the low-frequency features respectively to obtain high-frequency linear features and low-frequency linear features.

[0148] Among them, firstly, linear transformation is applied to the divided high-frequency features and low-frequency features to map them to the same output dimension. After the high-frequency features and low-frequency features are linearly transformed, a new representation is obtained, namely, the high-frequency linear feature and low-frequency linear characteristics The linear transformation methods of high-frequency features and low-frequency features are shown in formulas (16) and (17):

[0149] (16)

[0150] (17)

[0151] Where, and is the weight matrix of the linear transformation, and is the bias term.

[0152] S602: Calculate the high-frequency linear features of each node and the high-frequency linear features of the corresponding neighboring nodes to obtain the high-frequency attention weight of the corresponding node.

[0153] Among them, for each node’s high-frequency linear features and low-frequency linear features, the corresponding attention weights are calculated respectively. Assume that node The high-frequency linear characteristics are , the high-frequency linear characteristics of neighbor nodes are , high-frequency attention coefficient The definition of is shown in formula (18):

[0154] (18)

[0155] S603: Calculate the low-frequency linear features of each node and the low-frequency linear features of the corresponding neighboring nodes to obtain the low-frequency attention weight of the corresponding node.

[0156] Among them, for low-frequency linear features, it is assumed that the node The low-frequency linear characteristics are , the low-frequency linear characteristics of the neighbor nodes are , low-frequency attention coefficient The calculation method of is shown in formula (19):

[0157] (19)

[0158] Where, and It is a learnable parameter matrix that controls the information fusion between high-frequency linear features and low-frequency linear features.

[0159] S604: Obtain a multi-frequency graph attention weight according to the high-frequency attention weight and the low-frequency attention weight.

[0160] Among them, a learning-based weighted strategy is introduced to assign different weights to high and low frequencies, dynamically adjust the contribution of the two, and obtain the multi-frequency graph attention weight , calculated as shown in formula (20):

[0161] (20)

[0162] Where, and is a learnable weight that represents the importance of high-frequency attention and low-frequency attention.

[0163] In some embodiments of the present invention, step S104 includes:

[0164] The high-frequency features of each node’s neighbor nodes are weighted according to the multi-frequency graph attention weight to obtain the high-frequency aggregation result;

[0165] Dynamically fuse low-frequency features with high-frequency aggregation results to obtain multimodal fusion results;

[0166] The multimodal fusion results are input into the classifier to obtain the multimodal sentiment analysis results.

[0167] In a specific embodiment of the present invention, during the information transmission process in the graph neural network, since high-frequency features are usually more sensitive in emotion recognition, but their stability is poor and they are easily disturbed by noise, instead of directly fusing high-frequency and low-frequency features, a strategy of focusing on high-frequency features and supplementing low-frequency features is adopted. First, a multi-frequency graph attention weight is used. The high frequencies of neighboring nodes are weighted. At the same time, in order to further improve the expressive power of the model and enhance the flow of information, residual connections are introduced on the above basis to dynamically fuse the low-frequency information with the high-frequency aggregation results to obtain multimodal fusion results. , the method is shown in formula (21):

[0168] (twenty one)

[0169] Through this weighted fusion strategy, the model can better capture the sensitivity of local emotional fluctuations while retaining global semantic stability, thereby improving the ability to express and recognize complex emotional states.

[0170] Furthermore, a classifier is set up, which can identify emotions, so that the modal fusion results can be By performing emotion recognition, we can obtain multimodal sentiment analysis results.

[0171] The embodiments of the present invention have the following advantages:

[0172] 1. Adaptability: Compared to traditional fixed-frequency separation methods (such as Fourier transforms or Laplace-based filters), adaptive wavelet transforms can flexibly adjust the frequency separation strategy based on the local frequency characteristics of the signal, thereby more accurately capturing emotion-related high- and low-frequency information. This paper uses an adaptive wavelet transform method to dynamically select the optimal frequency separation parameters, significantly improving the accuracy of high- and low-frequency information extraction.

[0173] 2. Modal Feature Interactivity: Unlike existing simple weighted fusion or single attention mechanisms, the multi-frequency graph attention mechanism achieves synergistic effects of multimodal features at the frequency level by modeling high- and low-frequency signals separately and introducing an attention mechanism into the graph structure. By proposing a multi-frequency graph attention mechanism, this paper models the interactive relationship between multimodal features at the frequency level, significantly improving the model's ability to understand complex emotional expressions.

[0174] 3. Robustness: Traditional methods often suffer from performance degradation when processing multimodal data due to fixed frequency separation strategies or insufficient feature interaction, addressing the complex spectral distribution of multimodal data in online education scenarios. However, the present invention combines adaptive wavelet transforms with a multi-frequency graph attention mechanism to dynamically adjust the spectral distribution of multimodal sentiment data, demonstrating enhanced robustness when handling complex conversational scenarios and significantly improving the model's generalization capabilities in practical applications.

[0175] In order to better implement the multimodal sentiment analysis method for online education in the embodiment of the present invention, based on the multimodal sentiment analysis method for online education, the embodiment of the present invention also provides a multimodal sentiment analysis device for online education, such as Figure 7 As shown, the multimodal sentiment analysis device 700 for online education includes:

[0176] The data acquisition module 701 is used to acquire the data to be analyzed during the conversation process, and perform feature extraction and encoding on the data to be analyzed to obtain a text comprehensive modal representation, a speech comprehensive modal representation, and a visual comprehensive modal representation;

[0177] A graph construction module 702 is used to perform multimodal fusion on the text comprehensive modal representation, the speech comprehensive modal representation, and the visual comprehensive modal representation to obtain a fusion result;

[0178] The feature transformation module 703 is used to determine the optimal wavelet basis function and the optimal number of decomposition layers according to the fusion result, and perform discrete wavelet transformation on the fusion result according to the optimal wavelet basis function and the optimal number of decomposition layers to obtain high-frequency features and low-frequency features;

[0179] The feature fusion module 704 is used to determine the multi-frequency graph attention weight based on the high-frequency features and the low-frequency features, and perform feature fusion on the high-frequency features and the low-frequency features based on the multi-frequency graph attention weight to obtain a multimodal sentiment analysis result.

[0180] The multimodal sentiment analysis device 700 for online education provided in the above embodiment can implement the technical solution described in the above embodiment of the multimodal sentiment analysis method for online education. The specific implementation principles of the above modules or units can be found in the corresponding contents in the above embodiment of the multimodal sentiment analysis method for online education, which will not be repeated here.

[0181] The above is a detailed introduction to the multimodal sentiment analysis method and device for online education provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A multimodal sentiment analysis method for online education, characterized by: include: Obtain the data to be analyzed during the conversation process, extract features from the data, and encode them to obtain textual, speech, and visual comprehensive modal representations. Determining a plurality of nodes according to the text comprehensive modality representation, the voice comprehensive modality representation, and the visual comprehensive modality corresponding to each utterance in the conversation process; Connect each node to other nodes of the same modality in the same conversation to obtain the intra-modal connection structure; Connecting each of the nodes to other nodes of other modalities in the same conversation to obtain an inter-modal connection structure; Obtaining an undirected graph according to the plurality of nodes, the intra-modality connection structure, and the inter-modality connection structure; The undirected graph is a fusion result; Determining an optimal wavelet basis function and an optimal number of decomposition layers according to the fusion result, and performing discrete wavelet transform on the fusion result according to the optimal wavelet basis function and the optimal number of decomposition layers to obtain high-frequency features and low-frequency features; Performing linear transformation on the high-frequency features and the low-frequency features respectively to obtain high-frequency linear features and low-frequency linear features; Calculating the high-frequency linear features of each node and the high-frequency linear features of the corresponding neighboring nodes to obtain the high-frequency attention weight of the corresponding node; Calculating the low-frequency linear features of each node and the low-frequency linear features of the corresponding neighboring nodes to obtain the low-frequency attention weight of the corresponding node; According to the high-frequency attention weight and the low-frequency attention weight, a multi-frequency graph attention weight is obtained, and the high-frequency features and the low-frequency features are feature fused according to the multi-frequency graph attention weight to obtain a multimodal sentiment analysis result.

2. The multimodal sentiment analysis method for online education according to claim 1, characterized in that: Before determining the optimal wavelet basis function and the optimal number of decomposition layers according to the fusion result, and performing discrete wavelet transform on the fusion result according to the optimal wavelet basis function and the optimal number of decomposition layers to obtain high-frequency features and low-frequency features, the method further includes: Normalizing the features of each node in the fusion result to obtain a unified feature; The high-frequency signal in the unified feature is processed in sections to obtain the input signal of each node.

3. The multimodal sentiment analysis method for online education according to claim 2, characterized in that: The determining of an optimal wavelet basis function and an optimal number of decomposition layers according to the fusion result, and performing discrete wavelet transform on the fusion result according to the optimal wavelet basis function and the optimal number of decomposition layers to obtain high-frequency features and low-frequency features includes: Calculating the input signal according to a preset wavelet basis function candidate set to determine an optimal wavelet basis function; Determining the optimal number of decomposition layers according to the frequency characteristics of the input signal; Performing discrete wavelet transform on the input signal according to the optimal wavelet basis function and the optimal number of decomposition layers to obtain low-frequency coefficients and high-frequency coefficients; The input signal is reconstructed according to the low-frequency coefficients and the high-frequency coefficients to obtain a segmented signal; the segmented signal includes high-frequency features and low-frequency features.

4. The multimodal sentiment analysis method for online education according to claim 3 is characterized in that: The step of calculating the input signal according to a preset wavelet basis function candidate set to determine an optimal wavelet basis function includes: Calculating each wavelet basis function in the preset wavelet basis function candidate set with the input signal to obtain a corresponding matching degree; According to all the matching degrees, the optimal wavelet basis function with the highest matching degree in the wavelet basis function candidate set is determined.

5. The multimodal sentiment analysis method for online education according to claim 3, characterized in that: The determining of the optimal number of decomposition layers according to the frequency characteristics of the input signal includes: Performing Fourier transform on the input signal to obtain a frequency spectrum; Obtaining energy distribution of the input signal according to the frequency spectrum; determining a main frequency range according to the energy distribution; The main frequency range and the preset sampling frequency are calculated to obtain the optimal number of decomposition layers.

6. The multimodal sentiment analysis method for online education according to claim 1, characterized in that: The high-frequency features and the low-frequency features are subjected to feature fusion according to the multi-frequency graph attention weights. Get multimodal sentiment analysis results, including: Weighting the high-frequency features of the neighboring nodes of each node according to the multi-frequency graph attention weight to obtain a high-frequency aggregation result; Dynamically fusing the low-frequency features with the high-frequency aggregation results to obtain a multimodal fusion result; The multimodal fusion result is input into the classifier to obtain the multimodal sentiment analysis result.

7. The multimodal sentiment analysis method for online education according to claim 5, characterized in that: Determining a main frequency range according to the energy distribution includes: According to the energy distribution, a cumulative energy distribution curve is obtained; The main frequency range is determined according to the position of the preset total energy in the cumulative energy distribution curve.

8. The multimodal sentiment analysis method for online education according to claim 1, characterized in that: The feature extraction and encoding of the data to be analyzed to obtain a text comprehensive modal representation, a voice comprehensive modal representation, and a visual comprehensive modal representation includes: Extracting features from the data to be analyzed based on a multilingual model to obtain text features; Extracting features from the data to be analyzed based on a preset toolkit to obtain speech features; Extracting features from the data to be analyzed based on a three-dimensional deep learning model to obtain visual features; Encoding the text features according to the bidirectional GRU to obtain a comprehensive modal representation of the text; The speech features and the visual features are respectively encoded according to a single-layer MLP to obtain a speech comprehensive modality representation and a visual comprehensive modality representation.

Images