Teacher and student sentiment analysis method fusing cross-modal data

Through the multimodal feature interactive fusion network, the problem of insufficient accuracy of single modal data in the existing technology is solved, and the information complementarity of cross-modal data and the accuracy of emotional classification results is improved, which is suitable for emotional analysis of teachers and students.

CN119066485AActive Publication Date: 2024-12-03SHAANXI NORMAL UNIV
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411001290.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-12-03
Estimated Expiration
2044-07-25

AI Technical Summary

Technical Problem

The prior art relies on single modal data in teacher-student sentiment analysis, with limited accuracy, and the information complementarity and redundancy suppression of cross-modal data are difficult to achieve, resulting in insufficient accuracy of sentiment analysis.

Method used

Multimodal feature interactive fusion network is adopted to extract and transform visual, auditory, and text modal features through tools such as Facet, COVAREP, SentiwordNet, and Bert models, and build an emotional classification network, and use clustering and embedding enhancement gates to perform feature fusion. and classification.

Benefits of technology

It improves the accuracy and information complementarity of emotional classification results, and is suitable for emotional analysis of teachers and students in real educational and teaching scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066485A_ABST
    Figure CN119066485A_ABST
Patent Text Reader

Abstract

The invention discloses a teacher and student sentiment analysis method fusing cross-modal data. The method comprises the steps of dividing a data set, extracting multi-modal features, converting cross-modal features, constructing a sentiment classification network, training the sentiment classification network, verifying the sentiment classification network and testing the sentiment classification network. A first interaction fusion module of the multi-modal feature interaction fusion network performs interaction fusion on speech features and visual and listening features to form speech text fusion features and multi-modal features after visual and listening information is supplemented; the second interaction fusion module is used for mining the internal incidence relation of the multi-modal features, the speech text features and the audio-visual embedded features to obtain advanced potential multi-modal features and multi-modal emotion representations, and the multi-modal emotion representations are input into an emotion classifier for classification. The method has the advantages of cross-modal data information complementation, accurate sentiment classification result and the like, and is suitable for teacher and student sentiment classification in education and teaching scenes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to computer vision technology, and specifically relates to a teacher-student sentiment analysis method integrating cross-modal data. Background Art

[0002] Multimodal data teacher-student emotion analysis technology uses teachers' and students' facial expressions, body postures, voice features, and spoken words to identify teachers' and students' emotions in the process of education and teaching. Positive emotional communication helps to create a good teacher-student relationship, improve teaching quality, and promote the all-round development of students; while negative emotional communication will hinder students' growth and affect the effectiveness of education and teaching. Therefore, building a positive channel for emotional communication between teachers and students is of great significance to promoting the development of education and teaching.

[0003] Traditional sentiment analysis technology mainly relies on information from a single modality, such as text, speech, image or video, and has certain limitations in accuracy. Multimodal sentiment analysis technology can comprehensively utilize information from multiple modalities such as vision, hearing and text, and significantly improve the accuracy and effectiveness of sentiment analysis by mining the information complementarity and potential associations between different modalities. It has become an important trend in current sentiment analysis research. However, data from different modalities such as text, speech, image and video have significant differences in acquisition methods, data dimensions and key features. In addition, how to achieve semantic alignment between cross-modal data, effectively suppress redundant information or background noise in visual and speech data, and fuse cross-modal features with high quality, and mine potential associations between modal data and achieve information complementarity have become technical bottlenecks in sentiment analysis. Therefore, it is urgent to develop a teacher-student sentiment analysis method that integrates cross-modal data, fully considers and overcomes the shortcomings of existing technologies in sentiment analysis, so as to improve the accuracy of sentiment analysis. Summary of the invention

[0004] The technology to be solved by the present invention is to overcome the shortcomings of the existing technology and provide a teacher-student sentiment analysis method that is efficient and has accurate classification results and integrates cross-modal data.

[0005] The technical solution adopted to solve the above technical problems is:

[0006] (1) Divide the data set

[0007] The public benchmark dataset CMU-MOSEI is divided into training set, validation set, and test set according to the ratio of 7:1:2, and the closed-eye frame data is excluded.

[0008] (2) Extracting multimodal features

[0009] For visual modality data, use the Facet tool to extract facial features from the video as follows: v :

[0010]

[0011] Among them, n represents the sequence length, and facial feature s v Including facial landmarks, facial expressions, and facial action units; for auditory modality data, use COVAREP software to extract audio features as follows a :

[0012]

[0013] Audio Features a Including Mel frequency cepstral coefficients, pitch, sound / non-sound paragraph segmentation features, glottal parameters, peak slope parameters and maximum dispersion quotient; for text modal data, use SentiwordNet tool to obtain word-level sentiment polarity sequence I s :

[0014]

[0015] Using the Stanford Part-of-Speech Tagging Tool to Obtain Part-of-Speech Sequence I pos :

[0016]

[0017] Use the SentiLARE pre-trained model to analyze the lexical structure and obtain its token index sequence I t :

[0018]

[0019] (3) Converting cross-modal features

[0020] Use the clustering method to cluster the original visual feature frames in the test set and convert the visual element features into visual features S according to the formula v , the original auditory feature frame is clustered and converted into auditory feature S a :

[0021]

[0022] i=(0,1,…,len-1)

[0023] j=(0,1,…,K-1)

[0024] Among them, len represents the total frame length, represents the i-th frame of the auditory modality, represents the i-th frame of the visual modality, The auditory representation takes the jth cluster, Indicates taking the jth visual cluster, K represents the total number of clusters, a represents the auditory modality, and v represents the visual modality.

[0025] The word-level sentiment polarity sequence I s , Part of Speech Sequence I pos , token index sequence I t As the embedding input, the embedding layer of the Bert model is used to generate the speech text feature S l as follows:

[0026] S l =B1(I t )+B2(I pos )+B2(I s )

[0027] Among them, B1 represents the embedding layer of the Bert model, and B2 represents the normal embedding layer.

[0028] (4) Constructing a sentiment classification network

[0029] The sentiment classification network is composed of a multimodal feature interactive fusion network and a sentiment classifier in series.

[0030] (5) Training sentiment classification network

[0031] 1) Construct loss function

[0032] The loss function is constructed as follows

[0033]

[0034] Among them, δ represents a parameter, a real number with a minimum value of 1.0, and y represents a real value. Represents the predicted value.

[0035] 2) Training sentiment classification network

[0036] The training set is input into the sentiment classification network for training. The input video feature dimension is 35, the audio feature dimension is 74, and the text feature dimension is 768. The data training batch is 64, the data validation and test batch is 128, the weight parameters β and γ range from [0.1, 0.9], and β + γ = 1. The number of fully connected dense layers in the enhanced embedding gate and the enhanced gate is 1. Adam is used as the optimizer of the network model, the learning rate is 4e-5, and the forward propagation and back propagation are repeated. The weights and biases of the network are updated and trained to the loss function. convergence.

[0037] (6) Verify the sentiment classification network

[0038] Input the validation set into the trained sentiment classification network to verify the effect of the network.

[0039] (7) Testing the sentiment classification network

[0040] The test set is input into the trained sentiment classification network to perform sentiment classification detection and output the sentiment analysis results of teachers and students.

[0041] In step (4) of constructing the sentiment classification network of the present invention, the multimodal feature interactive fusion network is composed of a first interactive fusion module and a second interactive fusion module connected in series.

[0042] The first interactive fusion module of the present invention is composed of an audiovisual embedding layer, a normalization layer, a cross-modal self-attention layer, an embedding enhancement gate, and a linear layer connected in series in sequence; the second interactive fusion module is composed of a cross-modal interaction module, an attention pooling layer, and an enhancement gate connected in series in sequence; the output ends of the normalization layer and the embedding enhancement gate are connected to the input of the second interactive fusion module.

[0043] The construction method of the first interactive fusion module of the present invention is:

[0044] The embedded enhancement feature E and the speech-text fusion feature h after supplementing the audio-visual information are determined as follows: g , the multimodal feature H of the first complementary self-attention g :

[0045] E=Gate(V v→l ; V a→l )

[0046] h g =S l ⊕E

[0047] H g =gate(h g )

[0048] Among them, V v→l represents the output of the visual self-attention layer, V a→l represents the output of the auditory self-attention layer, ⊕ represents element-by-element addition, ; represents a cascade operation, Gate(·) is an embedding enhancement gate, and gate(·) represents a linear layer mapping operation.

[0049] The construction method of the second interactive fusion module of the present invention is:

[0050] The high-level latent correlation multimodal feature G and multimodal sentiment representation X are determined as follows:

[0051] X=(γ⊙h g )⊕(β⊙G)

[0052]

[0053] Among them, v, W, b are learning parameters; GATE(·) represents the enhanced gate operation, and ⊙ represents the element-by-element multiplication. The cross-modal interaction module finally outputs the local speech feature H l , audio local features H a , visual local features H v , verbal cross-extension feature G l→g , visual interactive promotion feature G v→g , auditory interactive promotion feature G a→g Determine as follows:

[0054]

[0055] Among them, l v→l represents the output of the visual normalization layer, l a→l represents the output of the auditory normalization layer, represents a symmetric interaction unit of speech, represents the auditory symmetric interaction unit, represents the visual symmetry interaction unit, G l→g , G v→g , G a→g Splicing to get G g , G g Represents the global cross-promotion feature.

[0056] The beneficial effects of the present invention are as follows:

[0057] Since the multimodal feature interactive fusion network of the present invention adopts the first interactive fusion module and the second interactive fusion module, the first interactive fusion module interactively fuses the speech features with the visual and auditory features to form the speech text fusion features and the multimodal features of the first complementary self-attention after supplementing the visual and auditory information; the second interactive fusion module fully explores the intrinsic correlation between the multimodal features of the first complementary self-attention and the speech text features and the audio-visual embedded features to obtain high-level potential multimodal features; the multimodal features of the first complementary self-attention and the high-level potential multimodal features are weightedly fused to generate efficient fusion features, which are input into the sentiment classifier for classification. The present invention has the advantages of cross-modal data information complementarity and accurate sentiment classification results, and is suitable for sentiment classification of teachers and students in real education and teaching scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a schematic diagram of a flow chart of the present invention.

[0059] Figure 2 It is a structural diagram of the multimodal feature interactive fusion network. DETAILED DESCRIPTION

[0060] The present invention will be further described in detail below in conjunction with the accompanying drawings and examples, but the present invention is not limited to the following embodiments.

[0061] Example 1

[0062] exist Figure 1 In the embodiment, the teacher-student sentiment analysis method integrating cross-modal data is composed of the following steps:

[0063] (1) Divide the data set

[0064] The public benchmark dataset CMU-MOSEI is divided into training set, validation set, and test set according to the ratio of 7:1:2, and the closed-eye frame data is excluded.

[0065] (2) Extracting multimodal features

[0066] For visual modality data, use the Facet tool to extract facial features from the video as follows: v :

[0067]

[0068] Among them, n represents the sequence length, and facial feature s v Including facial landmarks, facial expressions, and facial action units; for auditory modality data, use COVAREP software to extract audio features related to human emotions and intonation as follows a :

[0069] Audio Features a Including Mel frequency cepstral coefficients, pitch, sound / non-sound paragraph segmentation features, glottal parameters, peak slope parameters and maximum dispersion quotient; for text modal data, use SentiwordNet tool to obtain word-level sentiment polarity sequence I s :

[0070]

[0071] Using the Stanford Part-of-Speech Tagging Tool to Obtain Part-of-Speech Sequence I pos :

[0072]

[0073] Use the SentiLARE pre-trained model to analyze the lexical structure and obtain its token index sequence I t :

[0074]

[0075] (3) Converting cross-modal features

[0076] Use the clustering method to cluster the original visual feature frames in the test set and convert the visual element features into visual features S according to the formula v, the original auditory feature frame is clustered and converted into auditory feature S a :

[0077]

[0078] i=(0,1,…,len-1)

[0079] j=(0,1,…,K-1)

[0080] Among them, len represents the total frame length, represents the i-th frame of the auditory modality, represents the i-th frame of the visual modality, The auditory representation takes the jth cluster, Indicates taking the jth visual cluster, K represents the total number of clusters, a represents the auditory modality, and v represents the visual modality.

[0081] The word-level sentiment polarity sequence I s , Part of Speech Sequence I pos , token index sequence I t As the embedding input, the embedding layer of the Bert model is used to generate the speech text feature S l as follows:

[0082] S l =B1(I t )+B2(I pos )+B2(I s )

[0083] Among them, B1 represents the embedding layer of the Bert model, and B2 represents the normal embedding layer.

[0084] (4) Constructing a sentiment classification network

[0085] The sentiment classification network is composed of a multimodal feature interactive fusion network and a sentiment classifier in series.

[0086] Figure 2 The structural diagram of the multimodal feature interactive fusion network is given. Figure 2 In the embodiment, the multimodal feature interactive fusion network is composed of a first interactive fusion module and a second interactive fusion module connected in series.

[0087] The first interactive fusion module is composed of an audiovisual embedding layer, a normalization layer, a cross-modal self-attention layer, an embedding enhancement gate, and a linear layer connected in series in sequence; the second interactive fusion module is composed of a cross-modal interaction module, an attention pooling layer, and an enhancement gate connected in series in sequence; the output ends of the normalization layer and the embedding enhancement gate are connected to the input of the second interactive fusion module.

[0088] The construction method of the first interactive fusion module in this embodiment is:

[0089] The embedded enhancement feature E and the speech-text fusion feature h after supplementing the audio-visual information are determined as follows: g , the multimodal feature H of the first complementary self-attention g :

[0090] E=Gate(V v→l ; V a→l )

[0091] h g =S l ⊕E

[0092] H g =gate(h g )

[0093] Among them, V v→l represents the output of the visual self-attention layer, V a→l represents the output of the auditory self-attention layer, ⊕ represents element-by-element addition, ; represents a cascade operation, Gate(·) is an embedding enhancement gate, and gate(·) represents a linear layer mapping operation.

[0094] The construction method of the second interactive fusion module of this embodiment is:

[0095] The high-level latent correlation multimodal feature G and multimodal sentiment representation X are determined as follows:

[0096] X=(γ⊙h g )⊕(β⊙G)

[0097]

[0098] Among them, v, W, b are learning parameters; GATE(·) represents the enhanced gate operation, and ⊙ represents the element-by-element multiplication. The cross-modal interaction module finally outputs the local speech feature H l , audio local features H a , visual local features H v , verbal cross-extension feature G l→g , visual interactive promotion feature G v→g , auditory interactive promotion feature G a→g Determine as follows:

[0099]

[0100] Among them, l v→l represents the output of the visual normalization layer, l a→l represents the output of the auditory normalization layer, represents a symmetric interaction unit of speech, represents the auditory symmetric interaction unit, represents the visual symmetry interaction unit, G l→g, G v→g , G a→g Splicing to get G g , G g Represents the global cross-promotion feature.

[0101] The multi-modal feature interactive fusion network of the present invention adopts a first interactive fusion module and a second interactive fusion module. The first interactive fusion module interactively fuses speech features with visual and auditory features to form speech-text fusion features and multi-modal features of initial complementary self-attention after supplementing visual and auditory information; the second interactive fusion module fully explores the intrinsic correlation between the multi-modal features of initial complementary self-attention and speech-text features and audio-visual embedded features to obtain high-level potential multi-modal features; the multi-modal features of initial complementary self-attention and the high-level potential multi-modal features are weightedly fused to generate efficient fusion features, which are input into a sentiment classifier for classification, thereby improving the accuracy of the sentiment classification results.

[0102] (5) Training sentiment classification network

[0103] 1) Construct loss function

[0104] The loss function is constructed as follows

[0105]

[0106] Wherein, δ represents a parameter, a real number with a minimum value of 1.0. In this embodiment, δ is 1.0. y represents a real value. Represents the predicted value.

[0107] 2) Training sentiment classification network

[0108] The training set is input into the sentiment classification network for training. The input video feature dimension is 35, the audio feature dimension is 74, the text feature dimension is 768, the data training batch is 64, the data verification and test batch is 128, the weight parameters β and γ range is [0.1, 0.9], and β + γ = 1. The β value range of this embodiment is 0.3, and the γ value range is 0.7. The number of fully connected dense layers in the enhanced embedding gate and the enhanced gate is 1. Adam is used as the optimizer of the network model, the learning rate is 4e-5, and the forward propagation and back propagation are repeated. The weights and biases of the network are updated and trained to the loss function convergence.

[0109] (6) Verify the sentiment classification network

[0110] Input the validation set into the trained sentiment classification network to verify the effect of the network.

[0111] (7) Testing the sentiment classification network

[0112] The test set is input into the trained sentiment classification network to perform sentiment classification detection and output the sentiment analysis results of teachers and students.

[0113] Complete the teacher-student sentiment analysis method that integrates cross-modal data.

[0114] Example 2

[0115] The teacher-student sentiment analysis method for integrating cross-modal data in this embodiment consists of the following steps:

[0116] (1) Divide the data set

[0117] This step is the same as in Example 1.

[0118] (2) Extracting multimodal features

[0119] This step is the same as in Example 1.

[0120] (3) Converting cross-modal features

[0121] This step is the same as in Example 1.

[0122] (4) Constructing a sentiment classification network

[0123] This step is the same as in Example 1.

[0124] (5) Training sentiment classification network

[0125] 1) Construct loss function

[0126] The loss function is constructed as follows

[0127]

[0128] Wherein, δ represents a parameter, and its minimum value is a real number of 1.0. In this embodiment, the value of δ is 10.0. y represents a real value. Represents the predicted value.

[0129] 2) Training sentiment classification network

[0130] The training set is input into the sentiment classification network for training. The input video feature dimension is 35, the audio feature dimension is 74, the text feature dimension is 768, the data training batch is 64, the data verification and test batch is 128, the weight parameters β and γ range is [0.1, 0.9], and β + γ = 1. The β value range of this embodiment is 0.1, and the γ value range is 0.9. The number of fully connected dense layers in the enhanced embedding gate and the enhanced gate is 1. Adam is used as the optimizer of the network model, the learning rate is 4e-5, and the forward propagation and back propagation are repeated. The weights and biases of the network are updated and trained to the loss function convergence.

[0131] The other steps are the same as those in Example 1. The teacher-student sentiment analysis method integrating cross-modal data is completed.

[0132] Example 3

[0133] The teacher-student sentiment analysis method for integrating cross-modal data in this embodiment consists of the following steps:

[0134] (1) Divide the data set

[0135] This step is the same as in Example 1.

[0136] (2) Extracting multimodal features

[0137] This step is the same as in Example 1.

[0138] (3) Converting cross-modal features

[0139] This step is the same as in Example 1.

[0140] (4) Constructing a sentiment classification network

[0141] This step is the same as in Example 1.

[0142] (5) Training sentiment classification network

[0143] 1) Construct loss function

[0144] The loss function is constructed as follows

[0145]

[0146] Wherein, δ represents a parameter, a real number with a minimum value of 1.0. In this embodiment, the value of δ is 5.0. y represents a real value. Indicates prediction.

[0147] 2) Training sentiment classification network

[0148] The training set is input into the sentiment classification network for training. The input video feature dimension is 35, the audio feature dimension is 74, the text feature dimension is 768, the data training batch is 64, the data verification and test batch is 128, the weight parameters β and γ range is [0.1, 0.9], and β + γ = 1. The β value range of this embodiment is 0.9, and the γ value range is 0.1. The number of fully connected dense layers in the enhanced embedding gate and the enhanced gate is 1. Adam is used as the optimizer of the network model, the learning rate is 4e-5, and the forward propagation and back propagation are repeated. The weights and biases of the network are updated and trained to the loss function convergence.

[0149] The other steps are the same as those in Example 1. The teacher-student sentiment analysis method integrating cross-modal data is completed.

[0150] To verify the beneficial effects of the present invention, the inventors conducted a comparative experiment using the teacher-student sentiment analysis method for integrating cross-modal data in Example 1 of the present invention (hereinafter referred to as Example 1) and W. Yu, H. Xu, Z. Yuan, and J. Wu, Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis, (in AAAI, vol. 35, no. 12, 2021, pp. 10790–10 797.) (hereinafter referred to as comparative experiment 1), and constructed two types of evaluations: binary classification and regression. The classification results are evaluated by the binary classification accuracy Acc-2 and the weighted F1 score. For regression, the recognition performance is evaluated by the mean absolute error MAE and the Pearson correlation Corr. The experimental results are shown in Table 1, in which the left side of the table shows the binary classification results of negative and non-negative, and the right side shows the classification results of negative and positive.

[0151] Table 1 Experimental results of Example 1 method and comparative experiment

[0152] Experimental Group MAE Corr Accuracy (%) Weighted F1 score Comparative experiment 1 0.536 0.764 80.75 / 84.57 81.28 / 84.56 Example 1 0.534 0.789 82.59 / 86.85 83.10 / 86.86

[0153] As can be seen from Table 1, compared with comparative experiment 1, the mean absolute error of Example 1 is improved by 0.002, the Pearson coefficient is improved by 0.025, the accuracy is improved by 1.83% and 2.28% respectively, and the weighted F1 score is improved by 1.82 and 2.3 respectively. The scores of various indicators of Example 1 are significantly improved.

Claims

1. A teacher-student sentiment analysis method integrating cross-modal data, characterized in that It consists of the following steps: (1) Divide the data set The public benchmark dataset CMU-MOSEI was divided into training set, validation set, and test set according to 7:1:2, and the closed-eye frame data was removed; (2) Extracting multimodal features For visual modality data, use the Facet tool to extract facial features from the video as follows: v : Among them, n represents the sequence length, and facial feature s v Including facial landmarks, facial expressions, and facial action units; for auditory modality data, use COVAREP software to extract audio features as follows a : Audio Features a Including Mel frequency cepstral coefficients, pitch, sound / non-sound paragraph segmentation features, glottal parameters, peak slope parameters and maximum dispersion quotient; for text modal data, use SentiwordNet tool to obtain word-level sentiment polarity sequence I s : Using the Stanford Part-of-Speech Tagging Tool to Obtain Part-of-Speech Sequence I pos : Use the SentiLARE pre-trained model to analyze the lexical structure and obtain its token index sequence I t : (3) Converting cross-modal features Use the clustering method to cluster the original visual feature frames in the test set and convert the visual element features into visual features S according to the formula v , the original auditory feature frame is clustered and converted into auditory feature S a : Among them, len represents the total frame length, represents the i-th frame of the auditory modality, represents the i-th frame of the visual modality, The auditory representation takes the jth cluster, Indicates taking the jth visual cluster, K represents the total number of clusters, a represents the auditory modality, and v represents the visual modality; The word-level sentiment polarity sequence I s , Part of Speech Sequence I pos , token index sequence I t As the embedding input, the embedding layer of the Bert model is used to generate the speech text feature S l as follows: S l =B1(I t )+B2(I pos )+B2(I s ) Among them, B1 represents the embedding layer of the Bert model, and B2 represents the common embedding layer; (4) Constructing a sentiment classification network The sentiment classification network is composed of a multimodal feature interactive fusion network and a sentiment classifier in series; (5) Training sentiment classification network 1) Construct loss function The loss function is constructed as follows Among them, δ represents a parameter, a real number with a minimum value of 1.0, and y represents a real value. represents the predicted value; 2) Training sentiment classification network The training set is input into the sentiment classification network for training. The input video feature dimension is 35, the audio feature dimension is 74, and the text feature dimension is 768. The data training batch is 64, the data validation and test batch is 128, the weight parameters β and γ range from [0.1, 0.9], and β + γ = 1. The number of fully connected dense layers in the enhanced embedding gate and the enhanced gate is 1. Adam is used as the optimizer of the network model, the learning rate is 4e-5, and the forward propagation and back propagation are repeated. The weights and biases of the network are updated and trained to the loss function. convergence; (6) Verify the sentiment classification network Input the validation set into the trained sentiment classification network to verify the effect of the network. (7) Testing the sentiment classification network The test set is input into the trained sentiment classification network to perform sentiment classification detection and output the sentiment analysis results of teachers and students.

2. The teacher-student sentiment analysis method for integrating cross-modal data according to claim 1 is characterized by: In step (4) of constructing the sentiment classification network, the multimodal feature interactive fusion network is composed of a first interactive fusion module and a second interactive fusion module connected in series.

3. The method for analyzing teacher-student sentiment by integrating cross-modal data according to claim 2 is characterized in that: The first interactive fusion module is composed of an audiovisual embedding layer, a normalization layer, a cross-modal self-attention layer, an embedding enhancement gate, and a linear layer connected in series in sequence; the second interactive fusion module is composed of a cross-modal interaction module, an attention pooling layer, and an enhancement gate connected in series in sequence; the output ends of the normalization layer and the embedding enhancement gate are connected to the input of the second interactive fusion module.

4. The teacher-student sentiment analysis method for integrating cross-modal data according to claim 2 or 3, characterized in that: The construction method of the first interactive fusion module is: The embedded enhancement feature E and the speech-text fusion feature h after supplementing the audio-visual information are determined as follows: g , the multimodal feature H of the first complementary self-attention g : E=Gate(V v→l ;V a→l ) H g =gate(h g ) Among them, V v→l represents the output of the visual self-attention layer, V a→l represents the output of the auditory self-attention layer, Indicates Element addition, ; represents a cascade operation, Gate(·) is an embedded enhancement gate, and gate(·) represents a linear layer mapping operation.

5. The method for analyzing teacher-student sentiment by integrating cross-modal data according to claim 2 or 3, characterized in that The construction method of the second interactive fusion module is: The high-level latent correlation multimodal feature G and multimodal sentiment representation X are determined as follows: Among them, v, W, b are learning parameters; GATE(·) represents the enhanced gate operation, ⊙ represents the element-by-element multiplication; the cross-modal interaction module finally outputs the local speech feature H l , audio local features H a , visual local features H v , verbal cross-extension feature G l→g , visual interactive promotion feature G v→g , auditory interactive promotion feature G a→g Determine as follows: Among them, l v→l represents the output of the visual normalization layer, l a→l represents the output of the auditory normalization layer, represents a symmetric interaction unit of speech, represents the auditory symmetric interaction unit, represents the visual symmetry interaction unit, G l→g , G v→g , G a→g Splicing to get G g , G g Represents the global cross-promotion feature.

Citation Information

Patent Citations

  • Multi-modal sentiment analysis model fusing sentiment resources

    CN115577161A

  • Semantic sentiment analysis method fusing in-depth features and time sequence models

    US11194972B1

  • Multi-dimensional fine-grained dynamic sentiment analysis method and system

    WO2023065619A1

  • AU2020102150A4