Teacher-student sentiment analysis method fusing cross-modal data

Abstract

A teacher-student sentiment analysis method fusing cross-modal data is composed of the steps of dividing a data set, extracting multi-modal features, converting cross-modal features, constructing a sentiment classification network, training the sentiment classification network, verifying the sentiment classification network and testing the sentiment classification network. The first interactive fusion module of the multi-modal feature interactive fusion network of the present application interacts and fuses speech features with visual and audio features to form speech text fusion features after supplementing visual and audio information and multi-modal features; the second interactive fusion module mines the internal correlation between multi-modal features, speech text features and audio-visual embedded features, obtains high-level potential multi-modal features and multi-modal sentiment representations, and inputs the multi-modal sentiment representations into a sentiment classifier for classification. The present application has the advantages of cross-modal data information complementation, accurate sentiment classification results and the like, and is suitable for teacher-student sentiment classification in an educational teaching scene.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically involving a method for analyzing teacher-student sentiment by fusing cross-modal data. Background Technology

[0002] Multimodal data-driven teacher-student sentiment analysis technology utilizes facial expressions, body postures, vocal characteristics, and spoken language to identify teacher-student emotions during the teaching and learning process. Positive emotional communication helps create good teacher-student relationships, improves teaching quality, and promotes students' all-round development; while negative emotional communication hinders student growth and affects the effectiveness of teaching and learning. Therefore, building positive channels for teacher-student emotional communication is of great significance for promoting the development of education.

[0003] Traditional sentiment analysis techniques primarily rely on single-modal information, such as text, speech, images, or videos, which limits their accuracy. Multimodal sentiment analysis, on the other hand, comprehensively utilizes information from multiple modalities, including visual, auditory, and textual data. By uncovering complementary information and potential correlations between different modalities, it significantly improves the accuracy and effectiveness of sentiment analysis, becoming a major trend in current sentiment analysis research. However, data from different modalities, such as text, speech, images, and videos, differ significantly in acquisition methods, data dimensions, and key features. Furthermore, achieving semantic alignment between cross-modal data, effectively suppressing redundant information or background noise in visual and speech data, and high-quality fusion of cross-modal features to uncover potential correlations and achieve information complementarity between different modalities have become technical bottlenecks in sentiment analysis. Therefore, there is an urgent need to develop a teacher-student sentiment analysis method that integrates cross-modal data, fully considering and overcoming the shortcomings of existing techniques in sentiment analysis, thereby improving its accuracy. Summary of the Invention

[0004] The technical problem to be solved by this invention is to overcome the shortcomings of the prior art and provide an efficient and accurate method for analyzing teacher-student sentiment by fusing cross-modal data.

[0005] The technical solution adopted to solve the above technical problems is:

[0006] (1) Split the dataset

[0007] The publicly available benchmark dataset CMU-MOSEI was divided into training, validation, and test sets in a 7:1:2 ratio, and the closed-eye frame data was removed.

[0008] (2) Extracting multimodal features

[0009] For visual modality data, the Facet tool is used to extract human facial features from videos using the following formula: v :

[0010]

[0011] Where n represents the sequence length, and s represents the facial features. v This includes facial landmarks, facial expressions, and facial motion units; for auditory modal data, audio features were extracted using COVAREP software according to the following formula. a :

[0012]

[0013] Audio features s a This includes Mel-frequency cepstral coefficients, pitch, sound / non-sound segmentation features, glottal parameters, peak slope parameters, and maximum dispersion quotient; for text modal data, the SentiwordNet tool is used to obtain word-level sentiment polarity sequences I. s :

[0014]

[0015] Using the Stanford part-of-speech tagging tool to obtain part-of-speech sequences I pos :

[0016]

[0017] The token index sequence I is obtained using lexical analysis of the SentiLARE pre-trained model. t :

[0018]

[0019] (3) Transformation of cross-modal features

[0020] Clustering methods are used to cluster the original visual feature frames in the test set, transforming the visual element features into visual features S according to the following formula. v Auditory raw feature frames are clustered and transformed into auditory features S a :

[0021]

[0022] i = (0, 1, ..., len-1)

[0023] j = (0, 1, ..., K-1)

[0024] Where len represents the total number of frames. This represents the i-th frame of the auditory modality. This represents the i-th frame of the visual modality. Auditory representation takes the j-th cluster. Let represent the j-th visual cluster, K represent the total number of clusters, a represent the auditory modality, and v represent the visual modality.

[0025] The word-level sentiment polarity sequence I s Part-of-speech sequence I pos Token Index Sequence I t As the embedding input, speech text features S are generated using the embedding layer of the BERT model. l as follows:

[0026] S l =B1(I t )+B2(I pos )+B2(I s )

[0027] Where B1 represents the embedding layer of the BERT model, and B2 represents the ordinary embedding layer.

[0028] (4) Constructing an emotion classification network

[0029] The sentiment classification network consists of a multimodal feature interaction fusion network and a sentiment classifier connected in series.

[0030] (5) Training the sentiment classification network

[0031] 1) Constructing the loss function

[0032] Construct the loss function as follows

[0033]

[0034] Where δ represents the parameter, taking a minimum value of 1.0, and y represents the true value. This represents the predicted value.

[0035] 2) Training the sentiment classification network

[0036] The training set was input into the sentiment classification network for training. The input video features had a dimension of 35, audio features had a dimension of 74, and text features had a dimension of 768. The training batch size was 64, and the validation and test batch sizes were 128. The weight parameters β and γ ranged from [0.1, 0.9], and β + γ = 1. The number of fully connected dense layers in the augmentation embedding gate and augmentation gate was 1. Adam was used as the optimizer for the network model, with a learning rate of 4e-5. Forward and backward propagation were iteratively repeated, and the weights and biases of the network were updated until the loss function was reached. convergence.

[0037] (6) Validating the sentiment classification network

[0038] The validation set was input into the trained sentiment classification network to verify the network's performance.

[0039] (7) Testing the sentiment classification network

[0040] The test set is input into the trained sentiment classification network to perform sentiment classification detection and output the sentiment analysis results of teachers and students.

[0041] In step (4) of the present invention, the construction of the emotion classification network is composed of a first interaction fusion module and a second interaction fusion module connected in series.

[0042] The first interactive fusion module of the present invention is composed of an audiovisual embedding layer and a normalization layer, a cross-modal self-attention layer, an embedding enhancement gate, and a linear layer connected in series; the second interactive fusion module is composed of a cross-modal interactive module, an attention pooling layer, and an enhancement gate connected in series; the outputs of the normalization layer and the embedding enhancement gate are connected to the input of the second interactive fusion module.

[0043] The method for constructing the first interactive fusion module of the present invention is as follows:

[0044] The embedding enhancement feature E and the speech-text fusion feature h after supplementing audiovisual information are determined by the following formula. g The first complementary self-focused multimodal feature H g :

[0045] E = Gate(V) v→l V a→l )

[0046] h g =S l ⊕E

[0047] H g =gate(h g )

[0048] Among them, V v→l V represents the output of the visual self-attention layer. a→l represents the output of the auditory self-attention layer, ⊕ represents element-wise addition, ; represents a cascade operation, Gate(·) is an embedding enhancement gate, and gate(·) represents a linear layer mapping operation.

[0049] The method for constructing the second interactive fusion module of the present invention is as follows:

[0050] The high-level latent association multimodal features G and multimodal sentiment representation X are determined by the following formula:

[0051] X=(γ⊙h g )⊕(β⊙G)

[0052]

[0053] Where v, W, and b are the learning parameters; GATE(·) represents the enhancement gate operation, and ⊙ represents successive multiplication. The cross-modal interaction module ultimately outputs the local speech features H. l Local audio features H a Visual local features H v Verbal interaction promotion features G l→g Visual interaction promotion features G v→g Auditory interaction promotion features G a→g Determine by the following formula:

[0054]

[0055] Among them, l v→l Indicates the output of the visual normalization layer, l a→l This represents the output of the auditory normalization layer. Represents a symmetrical interactive unit of speech. Represents an auditory symmetrical interaction unit. G represents a visually symmetrical interactive unit. l→g G v→g G a→g G is obtained by splicing g G g This indicates a global interactive promotion feature.

[0056] The beneficial effects of this invention are as follows:

[0057] Because the multimodal feature interaction fusion network of this invention employs a first interaction fusion module and a second interaction fusion module, the first interaction fusion module interacts and fuses speech features with visual and auditory features respectively to form speech-text fusion features supplemented with visual and auditory information and initial complementary self-focused multimodal features; the second interaction fusion module fully explores the intrinsic correlation between the initial complementary self-focused multimodal features and speech-text features and audiovisual embedded features to obtain high-level latent multimodal features; the initial complementary self-focused multimodal features and high-level latent multimodal features are weighted and fused to generate efficient fusion features, which are then input into the sentiment classifier for classification. This invention has advantages such as information complementarity across modal data and accurate sentiment classification results, and is suitable for sentiment classification of teachers and students in real educational and teaching scenarios. Attached Figure Description

[0058] Figure 1 This is a flowchart of the present invention.

[0059] Figure 2 This is a schematic diagram of the structure of a multimodal feature interaction fusion network. Detailed Implementation

[0060] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but the present invention is not limited to the following embodiments.

[0061] Example 1

[0062] exist Figure 1 In this embodiment, the teacher-student sentiment analysis method that integrates cross-modal data consists of the following steps:

[0063] (1) Split the dataset

[0064] The publicly available benchmark dataset CMU-MOSEI was divided into training, validation, and test sets in a 7:1:2 ratio, and the closed-eye frame data was removed.

[0065] (2) Extracting multimodal features

[0066] For visual modality data, the Facet tool is used to extract human facial features from videos using the following formula: v :

[0067]

[0068] Where n represents the sequence length, and s represents the facial features. v This includes facial landmarks, facial expressions, and facial movement units; for auditory modal data, COVAREP software was used to extract audio features related to human emotion and intonation using the following formula. a :

[0069] Audio features s a This includes Mel-frequency cepstral coefficients, pitch, sound / non-sound segmentation features, glottal parameters, peak slope parameters, and maximum dispersion quotient; for text modal data, the SentiwordNet tool is used to obtain word-level sentiment polarity sequences I. s :

[0070]

[0071] Using the Stanford part-of-speech tagging tool to obtain part-of-speech sequences I pos :

[0072]

[0073] The token index sequence I is obtained using lexical analysis of the SentiLARE pre-trained model. t :

[0074]

[0075] (3) Transformation of cross-modal features

[0076] Clustering methods are used to cluster the original visual feature frames in the test set, transforming the visual element features into visual features S according to the following formula. vAuditory raw feature frames are clustered and transformed into auditory features S a :

[0077]

[0078] i = (0, 1, ..., len-1)

[0079] j = (0, 1, ..., K-1)

[0080] Where len represents the total number of frames. This represents the i-th frame of the auditory modality. This represents the i-th frame of the visual modality. Auditory representation takes the j-th cluster. Let represent the j-th visual cluster, K represent the total number of clusters, a represent the auditory modality, and v represent the visual modality.

[0081] The word-level sentiment polarity sequence I s Part-of-speech sequence I pos Token Index Sequence I t As the embedding input, speech text features S are generated using the embedding layer of the BERT model. l as follows:

[0082] S l =B1(I t )+B2(I pos )+B2(I s )

[0083] Where B1 represents the embedding layer of the BERT model, and B2 represents the ordinary embedding layer.

[0084] (4) Constructing an emotion classification network

[0085] The sentiment classification network consists of a multimodal feature interaction fusion network and a sentiment classifier connected in series.

[0086] Figure 2 A schematic diagram of the multimodal feature interaction fusion network is given. Figure 2 In this embodiment, the multimodal feature interaction fusion network is composed of a first interaction fusion module and a second interaction fusion module connected in series.

[0087] The first interactive fusion module consists of an audiovisual embedding layer, a normalization layer, a cross-modal self-attention layer, an embedding enhancement gate, and a linear layer connected in series. The second interactive fusion module consists of a cross-modal interactive module, an attention pooling layer, and an enhancement gate connected in series. The outputs of the normalization layer and the embedding enhancement gate are connected to the input of the second interactive fusion module.

[0088] The method for constructing the first interactive fusion module in this embodiment is as follows:

[0089] The embedding enhancement feature E and the speech-text fusion feature h after supplementing audiovisual information are determined by the following formula. g The first complementary self-focused multimodal feature H g :

[0090] E = Gate(V) v→l V a→l )

[0091] h g =S l ⊕E

[0092] H g =gate(h g )

[0093] Among them, V v→l V represents the output of the visual self-attention layer. a→l represents the output of the auditory self-attention layer, ⊕ represents element-wise addition, ; represents a cascade operation, Gate(·) is an embedding enhancement gate, and gate(·) represents a linear layer mapping operation.

[0094] The method for constructing the second interactive fusion module in this embodiment is as follows:

[0095] The high-level latent association multimodal features G and multimodal sentiment representation X are determined by the following formula:

[0096] X=(γ⊙h g )⊕(β⊙G)

[0097]

[0098] Where v, W, and b are the learning parameters; GATE(·) represents the enhancement gate operation, and ⊙ represents successive multiplication. The cross-modal interaction module ultimately outputs the local speech features H. l Local audio features H a Visual local features H v Verbal interaction promotion features G l→g Visual interaction promotion features G v→g Auditory interaction promotion features G a→g Determine by the following formula:

[0099]

[0100] Among them, l v→l Indicates the output of the visual normalization layer, l a→l This represents the output of the auditory normalization layer. Represents a symmetrical interactive unit of speech. Represents an auditory symmetrical interaction unit. G represents a visually symmetrical interactive unit. l→gG v→g G a→g G is obtained by splicing g G g This indicates a global interactive promotion feature.

[0101] The multimodal feature interaction fusion network of this invention employs a first interaction fusion module and a second interaction fusion module. The first interaction fusion module interacts and fuses speech features with visual and auditory features respectively to form speech-text fusion features supplemented with visual and auditory information and initial complementary self-focused multimodal features. The second interaction fusion module fully explores the intrinsic correlation between the initial complementary self-focused multimodal features and speech-text features and audiovisual embedding features to obtain high-level latent multimodal features. The initial complementary self-focused multimodal features and high-level latent multimodal features are weighted and fused to generate efficient fusion features, which are input into the sentiment classifier for classification, thereby improving the accuracy of sentiment classification results.

[0102] (5) Training the sentiment classification network

[0103] 1) Constructing the loss function

[0104] Construct the loss function as follows

[0105]

[0106] Where δ represents a parameter, with a minimum value of 1.0 (in this embodiment, δ is 1.0), and y represents the true value. This represents the predicted value.

[0107] 2) Training the sentiment classification network

[0108] The training set is input into the sentiment classification network for training. The input video features have a dimension of 35, audio features have a dimension of 74, and text features have a dimension of 768. The training batch size is 64, and the validation and test batch size is 128. The weight parameters β and γ range from [0.1, 0.9], and β + γ = 1. In this embodiment, the β value ranges from 0.3, and the γ value ranges from 0.7. The number of fully connected dense layers in the reinforcement embedding gate and reinforcement gate is 1. Adam is used as the optimizer for the network model, with a learning rate of 4e-5. Forward and backward propagation are repeated repeatedly, and the weights and biases of the network are updated until the loss function is reached. convergence.

[0109] (6) Validating the sentiment classification network

[0110] The validation set was input into the trained sentiment classification network to verify the network's performance.

[0111] (7) Testing the sentiment classification network

[0112] The test set is input into the trained sentiment classification network to perform sentiment classification detection and output the sentiment analysis results of teachers and students.

[0113] A method for analyzing teacher-student sentiment by integrating cross-modal data was developed.

[0114] Example 2

[0115] The teacher-student sentiment analysis method that integrates cross-modal data in this embodiment consists of the following steps:

[0116] (1) Split the dataset

[0117] The steps are the same as in Example 1.

[0118] (2) Extracting multimodal features

[0119] The steps are the same as in Example 1.

[0120] (3) Transformation of cross-modal features

[0121] The steps are the same as in Example 1.

[0122] (4) Constructing an emotion classification network

[0123] The steps are the same as in Example 1.

[0124] (5) Training the sentiment classification network

[0125] 1) Constructing the loss function

[0126] Construct the loss function as follows

[0127]

[0128] Where δ represents a parameter, with a minimum value of 1.0 (in this embodiment, δ is 10.0), and y represents the true value. This represents the predicted value.

[0129] 2) Training the sentiment classification network

[0130] The training set is input into the sentiment classification network for training. The input video features have a dimension of 35, audio features have a dimension of 74, and text features have a dimension of 768. The training batch size is 64, and the validation and test batch size is 128. The weight parameters β and γ range from [0.1, 0.9], and β + γ = 1. In this embodiment, the β value ranges from 0.1, and the γ value ranges from 0.9. The number of fully connected dense layers in the augmentation embedding gate and augmentation gate is 1. Adam is used as the optimizer for the network model, with a learning rate of 4e-5. Forward and backward propagation are repeated repeatedly, and the weights and biases of the network are updated until the loss function is reached. convergence.

[0131] The other steps are the same as in Example 1. This completes the method for analyzing teacher-student sentiment by integrating cross-modal data.

[0132] Example 3

[0133] The teacher-student sentiment analysis method that integrates cross-modal data in this embodiment consists of the following steps:

[0134] (1) Split the dataset

[0135] The steps are the same as in Example 1.

[0136] (2) Extracting multimodal features

[0137] The steps are the same as in Example 1.

[0138] (3) Transformation of cross-modal features

[0139] The steps are the same as in Example 1.

[0140] (4) Constructing an emotion classification network

[0141] The steps are the same as in Example 1.

[0142] (5) Training the sentiment classification network

[0143] 1) Constructing the loss function

[0144] Construct the loss function as follows

[0145]

[0146] Where δ represents a parameter, with a minimum value of 1.0 (a real number), and in this embodiment, δ is 5.0. y represents the true value. Indicates a prediction.

[0147] 2) Training the sentiment classification network

[0148] The training set is input into the sentiment classification network for training. The input video features have a dimension of 35, audio features have a dimension of 74, and text features have a dimension of 768. The training batch size is 64, and the validation and test batch size is 128. The weight parameters β and γ range from [0.1, 0.9], and β + γ = 1. In this embodiment, the β value ranges from 0.9, and the γ value ranges from 0.1. The number of fully connected dense layers in the augmentation embedding gate and augmentation gate is 1. Adam is used as the optimizer for the network model, with a learning rate of 4e-5. Forward and backward propagation are repeated repeatedly, and the weights and biases of the network are updated until the loss function is reached. convergence.

[0149] The other steps are the same as in Example 1. This completes the method for analyzing teacher-student sentiment by integrating cross-modal data.

[0150] To verify the beneficial effects of this invention, the inventors conducted a comparative experiment using the teacher-student sentiment analysis method integrating cross-modal data from Embodiment 1 of this invention (hereinafter referred to as Embodiment 1) and W.Yu, H.Xu, Z.Yuan, and J.Wu, Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis (in AAAI, vol.35, no.12, 2021, pp.10790–10797.) (hereinafter referred to as Comparative Experiment 1). Two types of evaluation were constructed: binary classification and regression. The classification results were evaluated using the binary classification accuracy (Acc-2) and weighted F1 score. For regression, the recognition performance was evaluated using the mean absolute error (MAE) and Pearson correlation (Corr). The experimental results are shown in Table 1, where the left side of the table represents the binary classification results for negative and non-negative data, and the right side represents the classification results for negative and positive data.

[0151] Table 1. Experimental results of the method and comparative experiment in Example 1.

[0152] experimental group MAE Corr Accuracy (%) Weighted F1 score Comparative Experiment 1 0.536 0.764 80.75 / 84.57 81.28 / 84.56 Example 1 0.534 0.789 82.59 / 86.85 83.10 / 86.86

[0153] As shown in Table 1, compared with Comparative Experiment 1, Example 1 showed a significant improvement in all indicators, with the mean absolute error increasing by 0.002, the Pearson coefficient increasing by 0.025, the accuracy increasing by 1.83% and 2.28% respectively, and the weighted F1 score increasing by 1.82 and 2.3 respectively.

Claims

1. A method for analyzing teacher-student sentiment by integrating cross-modal data, characterized in that... It consists of the following steps: (1) Split the dataset The publicly available benchmark dataset CMU-MOSEI was divided into training, validation, and test sets in a 7:1:2 ratio, and the closed-eye frame data was removed. (2) Extracting multimodal features For visual modality data, the Facet tool is used to extract human facial features from videos using the following formula: v : Where n represents the sequence length, and s represents the facial features. v This includes facial landmarks, facial expressions, and facial motion units; for auditory modal data, audio features were extracted using COVAREP software according to the following formula. a : Audio features s a This includes Mel-frequency cepstral coefficients, pitch, sound / non-sound segmentation features, glottal parameters, peak slope parameters, and maximum dispersion quotient; for text modal data, the SentiwordNet tool is used to obtain word-level sentiment polarity sequences I. s : Using the Stanford part-of-speech tagging tool to obtain part-of-speech sequences I pos : The token index sequence I is obtained using lexical analysis of the SentiLARE pre-trained model. t : (3) Transformation of cross-modal features Clustering methods are used to cluster the original visual feature frames in the test set, transforming the visual element features into visual features S according to the following formula. v Auditory raw feature frames are clustered and transformed into auditory features S a : Where len represents the total number of frames. This represents the i-th frame of the auditory modality. This represents the i-th frame of the visual modality. Auditory representation takes the j-th cluster. This indicates that the j-th visual cluster is selected, K represents the total number of clusters, a represents the auditory modality, and v represents the visual modality. The word-level sentiment polarity sequence I s Part-of-speech sequence I pos Token Index Sequence I t As the embedding input, speech text features S are generated using the embedding layer of the BERT model. l as follows: S l =B1(I t )+B2(I pos )+B2(I s ) Where B1 represents the embedding layer of the BERT model, and B2 represents the ordinary embedding layer; (4) Constructing an emotion classification network The sentiment classification network consists of a multimodal feature interaction fusion network and a sentiment classifier connected in series. (5) Training the sentiment classification network 1) Constructing the loss function Construct the loss function as follows Where δ represents the parameter, taking a minimum value of 1.0, and y represents the true value. Indicates the predicted value; 2) Training the sentiment classification network The training set was input into the sentiment classification network for training. The input video features had a dimension of 35, audio features had a dimension of 74, and text features had a dimension of 768. The training batch size was 64, and the validation and test batch sizes were 128. The weight parameters β and γ ranged from [0.1, 0.9], and β + γ = 1. The number of fully connected dense layers in the augmentation embedding gate and augmentation gate was 1. Adam was used as the optimizer for the network model, with a learning rate of 4e-5. Forward and backward propagation were iteratively repeated, and the weights and biases of the network were updated until the loss function was reached. convergence; (6) Validating the sentiment classification network The validation set is input into the trained sentiment classification network to verify the network's performance; (7) Testing the sentiment classification network The test set is input into the trained sentiment classification network to perform sentiment classification detection and output the sentiment analysis results of teachers and students.

2. The teacher-student sentiment analysis method integrating cross-modal data according to claim 1, characterized in that: In step (4), the multimodal feature interaction fusion network is constructed by connecting the first interaction fusion module and the second interaction fusion module in series.

3. The teacher-student sentiment analysis method integrating cross-modal data according to claim 2, characterized in that: The first interactive fusion module is composed of an audiovisual embedding layer, a normalization layer, a cross-modal self-attention layer, an embedding enhancement gate, and a linear layer connected in series. The second interactive fusion module is composed of a cross-modal interactive module, an attention pooling layer, and an enhancement gate connected in series. The outputs of the normalization layer and the embedding enhancement gate are connected to the input of the second interactive fusion module.

4. The teacher-student sentiment analysis method that integrates cross-modal data according to claim 2 or 3, characterized in that: The construction method of the first interactive fusion module is as follows: The embedding enhancement feature E and the speech-text fusion feature h after supplementing audiovisual information are determined by the following formula. g The first complementary self-focused multimodal feature H g : E=Gate(V v→l ;V a→l ) H g =gate(h g ) Among them, V v→l V represents the output of the visual self-attention layer. a→l This represents the output of the auditory self-attention layer. Indicates each Adding elements together; indicates a cascading operation. Gate(·) is an embedding enhancement gate, and gate(·) represents a linear layer mapping operation.

5. The teacher-student sentiment analysis method based on the fusion of cross-modal data as described in claim 2 or 3, characterized in that... The construction method of the second interactive fusion module is as follows: The high-level latent association multimodal features G and multimodal sentiment representation X are determined by the following formula: Where v, W, and b are the learning parameters; GATE(·) represents the enhancement gate operation, and ⊙ represents progressive multiplication; the cross-modal interaction module ultimately outputs the local speech features H. l Local audio features H a Visual local features H v Verbal interaction promotion features G l→g Visual interaction promotion features G v→g Auditory interaction promotion features G a→g Determine by the following formula: Among them, l v→l Indicates the output of the visual normalization layer, l a→l This represents the output of the auditory normalization layer. Represents a symmetrical interactive unit of speech. Represents an auditory symmetrical interaction unit. G represents a visually symmetrical interactive unit. l→g G v→g G a→g G is obtained by splicing g G g This indicates a global interactive promotion feature.

Images