Ideological and political classroom interaction analysis method and system based on multi-modal fusion

Abstract

The invention relates to the technical field of classroom interaction analysis, in particular to an ideological and political classroom interaction analysis method and system based on multi-modal fusion. The method comprises the following steps: S1, collecting data from different modes in an ideological and political classroom; s2, extracting characteristics related to ideological and political teaching from data of different modals through a self-attention mechanism architecture; s3, fusing the features of different sources related to ideological and political teaching by using a convolutional neural network; and S4, using the fused features to establish an interaction analysis model based on a random forest model to analyze the effectiveness of ideological and political classroom interaction. According to the invention, data of three different modes of video, audio and text are fused, so that the method can comprehensively capture various interactive information in an ideological and political classroom, including teaching modes of teachers, emotional reactions of students and content quality of classroom discussion; the effectiveness of ideological and political classroom interaction can be evaluated by using an advanced machine learning technology system.

Description

Technical Field

[0001] The present invention relates to the technical field of classroom interaction analysis, and in particular to an ideological and political classroom interaction analysis method and system based on multimodal fusion. Background Art

[0002] Traditional ideological and political classroom interaction evaluation methods often rely on teachers' subjective judgment or simple questionnaires. Such methods are difficult to comprehensively and objectively evaluate the effectiveness of classroom interaction. Especially in the context of the big data era, it is difficult to process a large amount of classroom data manually. Relying on only one type of data (such as video or audio) to analyze classroom interaction may miss important information. For example, video can show students' facial expressions and body language, but cannot capture the content of students' speeches; audio can record students' speeches, but cannot directly show students' non-verbal behaviors. Ideological and political courses are one of the important ways to cultivate students' correct values. The quality of their teaching directly affects students' growth and development. In order to improve the teaching quality of ideological and political courses, a method that can deeply analyze classroom interactions, discover problems in a timely manner, and put forward improvement suggestions is needed. Therefore, a method and system for analyzing ideological and political classroom interactions based on multimodal fusion is provided. Summary of the invention

[0003] The purpose of the present invention is to provide an ideological and political classroom interaction analysis method and system based on multimodal fusion to solve the limitations of traditional evaluation methods and the insufficiency of single modality data mentioned in the above background technology.

[0004] To achieve the above object, the present invention aims to provide an ideological and political classroom interaction analysis method based on multimodal fusion, comprising: S1. Collect data from different modes in ideological and political classes; S2, extracting ideological and political teaching-related features from data of different modalities through the self-attention mechanism architecture; S3, using convolutional neural networks to fuse features from different sources related to ideological and political education; S4. Use the fused features to establish an interactive analysis model based on the random forest model to analyze the effectiveness of ideological and political classroom interaction.

[0005] As a further improvement of the present technical solution, in S1, the data of different modes in the ideological and political classroom include: teaching videos, audios and texts of the ideological and political classroom.

[0006] As a further improvement of the technical solution, in S2, the features related to ideological and political teaching are extracted from data of different modalities through the self-attention mechanism architecture, including the following steps: S2.1, preprocess the data of different modalities; S2.2, use residual network to extract features of video frames, and use temporal convolutional network to capture dynamic information in the video; S2.3, using Mel-frequency cepstral coefficients, convert the audio signal into a feature vector and input the audio features into the self-attention module to capture the relationship between the audio clips; S2.4. Use language model embedding to obtain word embedding representation of the text, and input the word embedding into the self-attention module to capture the relationship between sentences and sentences.

[0007] As a further improvement of the technical solution, in S2.2, a residual network is used to extract features of video frames, and a temporal convolutional network is used to capture dynamic information in the video, including the following steps: S2.21, cropping the video frame to a size of 224x224, scaling the video frame, keeping the aspect ratio unchanged, and normalizing the video frame; S2.22. Use the pre-trained residual network model and input the pre-processed video frames into the residual network to extract the features of the middle layer; S2.23, the feature vectors extracted from the residual network are arranged in time order to form a three-dimensional tensor; S2.24. Use temporal convolutional networks to process time series data; S2.25. Use causal convolution to ensure that the output of the current time step depends only on the previous time step and set the dilation factor. S2.26. Input the feature vector sequence extracted from the residual network into the temporal convolutional network, and learn dynamic information through training, especially the changes in the teacher's facial expressions when explaining and the reactions of the students when listening to the class.

[0008] As a further improvement of the technical solution, in S2.3, using Mel-frequency cepstral coefficients to convert the audio signal into a feature vector includes the following steps: S2.31, pre-emphasize the original audio signal to enhance the high-frequency information; S2.32, dividing the audio signal into a series of short time frames; S2.33, applying a Hamming window to each frame and performing a fast Fourier transform on each frame to obtain a frequency domain representation; S2.34, applying a set of triangular Mel filters in the frequency domain; S2.35, taking the logarithm of the energy output by each filter, and performing discrete cosine transform on the logarithmic energy to obtain a set of new coefficients, which are Mel-frequency cepstrum coefficients; S2.36. Normalize the extracted Mel-frequency cepstral coefficient features.

[0009] As a further improvement of the technical solution, in S2.35, the discrete cosine transform is: ; in, Indicates Energy characteristics of frequency bands; Indicates The logarithm of the energy of the Mel filter; represents the number of Mel filters; Represents the index of the Mel filter; Indices representing the mel-frequency cepstral coefficients.

[0010] As a further improvement of the technical solution, in S3, features from different sources related to ideological and political education are fused using a convolutional neural network, including the following steps: S3.1, normalize the features of different modes; S3.2, concatenating the normalized features of different modalities together to form a new feature vector; S3.3, input the concatenated feature vector into the input layer of the convolutional neural network; S3.4, extract the feature map by convolving the fused features through the convolution layer. The convolution layer extracts the local pattern in the feature map by sliding the window and increases the nonlinearity through the activation function; S3.5, after the convolution operation, the maximum pooling method is used to reduce the spatial dimension of the feature map; S3.6 flattens all feature maps into a single feature vector, and integrates the feature vectors through a fully connected layer to form a higher-level abstract representation; S3.7. The feature vector processed by the fully connected layer is passed to the output layer of the convolutional neural network model to output the final feature fusion data.

[0011] As a further improvement of the technical solution, in S3.5, the maximum pooling method is: ; in, Indicates that the feature map after the maximum pooling operation is at position The value on Represents the coordinate offset within the pooling window; Indicates the offset in the row direction; Indicates the offset in the column direction; Indicates the size of the pooling window; Represents the original feature map.

[0012] As a further improvement of the technical solution, in S4, the fused features are used to establish an interactive analysis model based on a random forest model to analyze the effectiveness of ideological and political classroom interaction, including the following steps: S4.1. Define labels to measure the effectiveness of ideological and political classroom interaction, and use the fused features and labels as a dataset, dividing the dataset into a training set and a test set; S4.2. Set key parameters and fit the random forest model using the training set data; S4.3. Use the grid search method to adjust the model's hyperparameters, find the optimal parameter combination, and evaluate the performance of the random forest model on the test set; S4.4. Analyze the prediction results of the random forest model and verify whether the random forest model can effectively evaluate the effectiveness of ideological and political classroom interaction.

[0013] On the other hand, the present invention provides an ideological and political classroom interaction analysis system based on multimodal fusion, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any one of the ideological and political classroom interaction analysis methods based on multimodal fusion described above.

[0014] Compared with the prior art, the present invention has the following beneficial effects: 1. This ideological and political classroom interaction analysis method and system based on multimodal fusion can comprehensively capture various interactive information in the ideological and political classroom, including the teacher's teaching method, students' emotional response, and the content quality of classroom discussions, by integrating data from three different modes: video, audio, and text. Using advanced machine learning techniques such as self-attention mechanism, residual network, temporal convolutional network, and random forest model, the system can automatically extract features related to teaching quality and evaluate the effectiveness of ideological and political classroom interaction based on this. Compared with traditional single-modal analysis, this method can more accurately reflect the teaching interaction status in a real classroom environment, thereby helping education administrators and teachers improve teaching strategies.

[0015] 2. The ideological and political classroom interaction analysis method and system based on multimodal fusion can identify and quantify interactive behaviors in ideological and political classrooms, and can also provide intelligent feedback through the established interactive analysis model. Through the automated evaluation of the effectiveness of classroom interaction, educators can obtain immediate feedback information and understand which teaching links need to be improved and which aspects are doing well. In addition, this method can also help identify students who actively participate in discussions and possible silent groups, so as to take targeted measures to stimulate the learning enthusiasm of all students and further improve the teaching effect of ideological and political courses. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1The figure is a flow chart of the overall method of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] Example 1: Please refer to Figure 1 As shown, this embodiment provides an ideological and political classroom interaction analysis method based on multimodal fusion, including the following steps: S1. Collect data from different modes in ideological and political classes; In this embodiment, data of different modes in the ideological and political classroom include: teaching videos of the ideological and political classroom (interactions between teachers and students in the ideological and political class), audio (sounds of speeches in class) and text (records of students' speeches in class or discussions about classroom topics).

[0019] S2, extracting ideological and political teaching-related features from data of different modalities through the self-attention mechanism architecture; In this embodiment, the self-attention mechanism can help the model focus on the most relevant parts of ideological and political teaching when processing multimodal data, which means that it can filter out key information that is truly related to teaching interaction from a large amount of classroom data; in the ideological and political classroom environment, there is a complex interactive relationship between video, audio and text data, and the self-attention mechanism can help the model learn the interdependence and correlation between these modalities, so as to more accurately understand the nature of classroom interaction; the self-attention mechanism allows the model to assign different weights to each element in the input sequence, which means that it can dynamically adjust the attention of different features according to the importance of specific tasks, which is particularly useful for analyzing classroom interactions because the importance of interactions in different time periods or scenarios may be different; extracting features related to ideological and political teaching from data of different modalities through the self-attention mechanism architecture includes the following steps: S2.1. Preprocess data of different modalities to ensure that they are suitable for input into the self-attention mechanism architecture. Video data should highlight the images of teachers explaining ideological and political content, audio data should clearly record the ideological and political themes in the dialogue between teachers and students, and text data should contain students’ discussions or notes on ideological and political knowledge points. S2.2. Use the residual network (ResNet) to extract the features of the video frame and use the temporal convolutional network (TCN) to capture the dynamic information in the video; Among them, ResNet is designed to deal with image classification problems. Its purpose is to overcome the gradient vanishing and gradient exploding problems in deep networks, and at the same time, it can learn high-level features in images. In the ideological and political classroom video analysis, ResNet is used to extract static features in each frame of the image, such as the teacher's facial expressions, gestures, and classroom environment; TCN is a network structure specially designed to process sequence data. Its original intention was to capture dynamic features in time series data. In the ideological and political classroom video analysis, TCN is used to capture dynamic changes between frames, such as teacher's movement changes, students' emotional reactions, etc.; ResNet introduces a residual learning framework, which makes information and gradients easier to propagate forward and backward by adding jump connections, thereby solving the gradient vanishing and gradient exploding problems in deep network training. This feature enables the model to be trained deeper, so as to learn more complex features; TCN uses causal convolution and dilated convolution to process time series data, which can capture information of different time spans and maintain invariance to the length of the input sequence. This allows the model to effectively capture changes that occur over a longer period of time in the video; a residual network (ResNet) is used to extract the features of the video frames, and a temporal convolutional network (TCN) is used to capture dynamic information in the video, including the following steps: S2.21, crop the video frame to a size of 224x224, scale the video frame, keep the aspect ratio unchanged, and normalize the video frame (convert the pixel value to the [0,1] interval or subtract the mean); S2.22. Use the pre-trained residual network model and input the pre-processed video frame into the residual network to extract the features of the middle layer. Extract the features of the penultimate layer, that is, remove the classification head of the last layer. The output feature is a vector of fixed length, such as 512 dimensions or higher. S2.23, the feature vectors extracted from the residual network are arranged in time order to form a three-dimensional tensor T×C×H, where T is the time step, C is the feature dimension, and H is the height of the feature (i.e., the height is 1); S2.24. Use a temporal convolutional network (TCN) to process time series data. TCN consists of multiple convolutional layers, each followed by batch normalization and an activation function (such as ReLU). S2.25. Use causal convolution to ensure that the output of the current time step depends only on the previous time step, and set the dilation factor to capture information of different time spans. Use multiple dilation factors to form a multi-scale temporal receptive field. S2.26, input the feature vector sequence extracted from the residual network into the temporal convolutional network, and learn dynamic information through training, especially the changes in the teacher's expression when explaining and the reactions of the students when listening to the class; S2.3, use Mel-frequency cepstral coefficients (MFCC) to convert the audio signal into a feature vector, and input the audio features into the self-attention module to capture the relationship between the audio clips; Among them, MFCC is a feature extraction method widely used in speech recognition and audio processing. Its main purpose is to convert the original audio signal into a set of feature vectors that can represent the speech content. In the analysis of ideological and political classroom interaction, MFCC can help extract audio features related to classroom interaction, such as intonation changes, speaking speed, etc.; the original audio signal usually has a high dimension. Direct processing is not only computationally expensive, but also easy to cause model overfitting. By extracting features through MFCC, high-dimensional audio signals can be converted into a set of lower-dimensional feature vectors, simplifying the subsequent processing process; MFCC is designed based on the characteristics of the human auditory system. It maps audio signals to the Mel frequency scale, which is closer to the way the human ear perceives sound. Therefore, MFCC features can better capture important information in human speech; using Mel frequency cepstral coefficients (MFCC) to convert audio signals into feature vectors includes the following steps: S2.31, pre-emphasize the original audio signal to enhance the high-frequency information, because the human ear is more sensitive to high frequencies; S2.32, divide the audio signal into a series of short time frames, the frame length is between 20-30 milliseconds, and the frame shift (the time interval between adjacent frames) is approximately half of the frame length; S2.33, applying a Hamming window to each frame to reduce the discontinuity of the frame edge, and performing a fast Fourier transform on each frame to obtain a frequency domain representation, so that the spectrum obtained can more accurately reflect the energy distribution of the signal at each frequency; S2.34, applying a set of triangular Mel filters in the frequency domain, which are uniformly distributed on the Mel frequency scale, to extract the spectral envelope; S2.35. Take the logarithm of the energy output by each filter. This step is to linearize the perception, and perform discrete cosine transform on the logarithmic energy to obtain a new set of coefficients, namely Mel-frequency cepstrum coefficients; Furthermore, the discrete cosine transform can concentrate the energy of the signal on a few coefficients, which means that after the logarithmic energy is subjected to the discrete cosine transform, most of the energy is concentrated in the first few coefficients, and the following coefficients usually contain less energy. This energy concentration characteristic is conducive to compressing data and reducing the dimension of the feature vector while retaining the most important information; the discrete cosine transform can extract a set of new coefficients from the logarithmic energy spectrum. These coefficients represent the characteristics of the audio signal in the frequency domain. These characteristics can better reflect the frequency components and distribution of the speech signal, which helps to distinguish different speech signals; the discrete cosine transform is: ; in, Indicates Energy characteristics of frequency bands; Indicates The energy logarithm of a Mel filter is obtained by taking the logarithm of the energy value after passing through the Mel filter bank; represents the number of Mel filters; Represents the index of the Mel filter; Indicates the index of the Mel-frequency cepstral coefficient; S2.36, normalizing the extracted Mel-frequency cepstral coefficient features; S2.4. Use Embedding Language Model (ELMo) to obtain word embedding representation of text, and input the word embedding into the self-attention module to capture the relationship between sentences and within sentences. ELMo is a context-sensitive word embedding method, which is learned from a large amount of text data through a bidirectional language model (BiLM). BiLM also considers the information of the context before and after the word, so that the generated word embedding can reflect the different meanings of the word in different contexts. ELMo generates a vector representation containing contextual information for each word through a multi-layer bidirectional LSTM network. In the text analysis of ideological and political classes, the word embedding generated by ELMo can more accurately capture the semantic information in students' speeches or discussions. By inputting these embeddings into the self-attention module, we can further discover the word relationship within the sentence and the logical connection between sentences.

[0020] S3, using convolutional neural networks to fuse features from different sources related to ideological and political education; In this embodiment, in the analysis of ideological and political classroom interaction, different data sources (such as video, audio, and text) each carry different information. The purpose of using CNN for feature fusion is to combine the information in these heterogeneous data sources to generate a comprehensive feature representation so as to more comprehensively reflect the situation of classroom interaction. By fusing the features of different modalities, the deficiencies of single-modal data in certain aspects can be made up, thereby improving the accuracy and reliability of classroom interaction analysis. For example, video can capture the expressions and body movements of teachers and students, while audio can record the content of the conversation, and text contains students' thinking and feedback. Combining this information can more accurately evaluate the quality of classroom interaction. CNN can automatically learn different levels of features in the input data through multi-layer convolution and pooling operations. For ideological and political classroom interaction analysis, this means that high-level abstract features can be extracted from the original data, and these features can better reflect the essence of classroom interaction. The convolutional neural network is used to fuse the features of different sources related to ideological and political teaching, including the following steps: S3.1. Normalize the features of different modalities. This is because the features of different modalities may have different scale ranges. Normalization can make the features at the same order of magnitude to avoid some features dominating the learning process of the model. S3.2. Concatenate the normalized features of different modalities together to form a new feature vector. If the video feature is an A-dimensional vector, the audio feature is a B-dimensional vector, and the text feature is a D-dimensional vector, then these three vectors can be simply concatenated into an A+B+D-dimensional feature vector. S3.3, input the concatenated feature vector into the input layer of the convolutional neural network; S3.4. The fused features are convolved through a series of convolutional layers to extract feature maps. The convolutional layer extracts local patterns in the feature map by sliding windows and adds nonlinearity through the activation function ReLU (the ReLU function remains linear (slope is 1) on the positive half axis and is a constant 0 on the negative half axis. This nonlinear mapping enables the model to capture nonlinear relationships in the data). The convolutional layer can help capture the complex interactions between different modal features; S3.5, after the convolution operation, the maximum pooling method is used to reduce the spatial dimension of the feature map; Furthermore, in convolutional neural networks (CNNs), as the convolutional layers deepen, the size of the feature map may become very large, which will lead to increased consumption of computing resources. Maximum pooling can effectively reduce the size of the feature map, thereby reducing the input dimension of subsequent layers and reducing the computational burden. The maximum pooling operation selects the maximum value in the local area, which can make the feature map invariant to the translation of the input to a certain extent. This means that even if the input image moves in a small range, the key information in the feature map can still be retained, thereby enhancing the robustness of the model. The maximum pooling method is: ; in, Indicates that the feature map after the maximum pooling operation is at position The value on Represents the coordinate offset within the pooling window; Indicates the offset in the row direction; Indicates the offset in the column direction; Indicates the size of the pooling window, which is a two-dimensional rectangular area; represents the original feature map; S3.6 flattens all feature maps into a single feature vector, and integrates the feature vectors through a fully connected layer to form a higher-level abstract representation. These high-level features can better express the relationship between data; S3.7. The feature vector processed by the fully connected layer is passed to the output layer of the convolutional neural network model to output the final feature fusion data.

[0021] S4. Use the fused features to establish an interactive analysis model based on the random forest model to analyze the effectiveness of ideological and political classroom interaction; In this embodiment, random forest is an integrated learning method that makes a final prediction by constructing multiple decision trees and combining their results. This method can reduce the risk of overfitting that may be generated by a single decision tree, thereby improving the robustness and generalization ability of the model; random forest can evaluate the importance of each feature, which is very useful for understanding which features have the greatest impact on the effectiveness of classroom interaction. By sorting the importance of features, teachers and education managers can be guided on which aspects to focus on to improve teaching results; random forest can handle the problem of class imbalance well, even if the number of samples in some categories is much less than that in other categories, the model can still give reasonable predictions; using the fused features to establish an interactive analysis model based on the random forest model to analyze the effectiveness of ideological and political classroom interaction, including the following steps: S4.1. Define labels to measure the effectiveness of ideological and political classroom interaction. The labels include the level of understanding of ideological and political theory, the quality of classroom discussion, and students' cognition of national policies. The fused features and labels are used as a dataset, and the dataset is divided into a training set and a test set. S4.2. Set key parameters, including the number of trees, maximum depth, maximum number of features for finding the best split point when splitting nodes, etc., and fit the random forest model using the training set data. At this stage, each tree will be trained based on a subset of samples and a subset of features to reduce the risk of overfitting. S4.3. Use the grid search method to adjust the model's hyperparameters (first define a grid of hyperparameters, i.e., a set of possible values ​​for each hyperparameter, then evaluate the model's performance by traversing all possible combinations, and finally select the best performing set of hyperparameters), find the optimal parameter combination, and evaluate the performance of the random forest model on the test set; S4.4. Analyze the prediction results of the random forest model and verify whether the random forest model can effectively evaluate the effectiveness of ideological and political classroom interaction (define the evaluation indicator as accuracy to measure the performance of the model, so as to determine whether the model's classification of the effectiveness of classroom interaction is accurate).

[0022] Embodiment 2: This embodiment provides an ideological and political classroom interaction analysis system based on multimodal fusion, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any one of the ideological and political classroom interaction analysis methods based on multimodal fusion described above.

[0023] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only preferred examples of the present invention, and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A method for analyzing ideological and political classroom interaction based on multimodal fusion, characterized in that: The following steps are involved: S1. Collect data from different modes in ideological and political classes; S2, extracting ideological and political teaching-related features from data of different modalities through the self-attention mechanism architecture; S3, using convolutional neural networks to fuse features from different sources related to ideological and political education; S4. Use the fused features to establish an interactive analysis model based on the random forest model to analyze the effectiveness of ideological and political classroom interaction.

2. The method for analyzing ideological and political classroom interaction based on multimodal fusion according to claim 1 is characterized by: In S1, the data of different modes in the ideological and political classroom include: teaching videos, audios and texts of the ideological and political classroom.

3. The method for analyzing ideological and political classroom interaction based on multimodal fusion according to claim 2 is characterized by: In S2, the features related to ideological and political teaching are extracted from data of different modalities through the self-attention mechanism architecture, including the following steps: S2.1, preprocess the data of different modalities; S2.2, use residual network to extract features of video frames, and use temporal convolutional network to capture dynamic information in the video; S2.3, using Mel-frequency cepstral coefficients, convert the audio signal into a feature vector and input the audio features into the self-attention module to capture the relationship between the audio clips; S2.

4. Use language model embedding to obtain word embedding representation of the text, and input the word embedding into the self-attention module to capture the relationship between sentences and sentences.

4. The method for analyzing ideological and political classroom interaction based on multimodal fusion according to claim 3 is characterized by: In S2.2, a residual network is used to extract features of video frames, and a temporal convolutional network is used to capture dynamic information in the video, including the following steps: S2.21, cropping the video frame to a size of 224x224, scaling the video frame, keeping the aspect ratio unchanged, and normalizing the video frame; S2.

22. Use the pre-trained residual network model and input the pre-processed video frames into the residual network to extract the features of the middle layer; S2.23, the feature vectors extracted from the residual network are arranged in time order to form a three-dimensional tensor; S2.

24. Use temporal convolutional networks to process time series data; S2.

25. Use causal convolution to ensure that the output of the current time step depends only on the previous time step and set the dilation factor. S2.

26. Input the feature vector sequence extracted from the residual network into the temporal convolutional network, and learn dynamic information through training, especially the changes in the teacher's facial expressions when explaining and the reactions of the students when listening to the class.

5. The method for analyzing ideological and political classroom interaction based on multimodal fusion according to claim 4 is characterized by: In S2.3, the audio signal is converted into a feature vector using Mel-frequency cepstral coefficients, including the following steps: S2.31, pre-emphasize the original audio signal to enhance the high-frequency information; S2.32, dividing the audio signal into a series of short time frames; S2.33, applying a Hamming window to each frame and performing a fast Fourier transform on each frame to obtain a frequency domain representation; S2.34, applying a set of triangular Mel filters in the frequency domain; S2.35, taking the logarithm of the energy output by each filter, and performing discrete cosine transform on the logarithmic energy to obtain a set of new coefficients, which are Mel-frequency cepstrum coefficients; S2.

36. Normalize the extracted Mel-frequency cepstral coefficient features.

6. The method for analyzing ideological and political classroom interaction based on multimodal fusion according to claim 5 is characterized by: In S2.35, the discrete cosine transform is: ； in, Indicates Energy characteristics of each frequency band; Indicates The logarithm of the energy of the Mel filter; represents the number of Mel filters; Represents the index of the Mel filter; Indices representing the mel-frequency cepstral coefficients.

7. The method for analyzing ideological and political classroom interaction based on multimodal fusion according to claim 6 is characterized by: In S3, a convolutional neural network is used to fuse features from different sources related to ideological and political education, including the following steps: S3.1, normalize the features of different modes; S3.2, concatenating the normalized features of different modalities together to form a new feature vector; S3.3, input the concatenated feature vector into the input layer of the convolutional neural network; S3.4, extract the feature map by convolving the fused features through the convolution layer. The convolution layer extracts the local pattern in the feature map by sliding the window and increases the nonlinearity through the activation function; S3.5, after the convolution operation, the maximum pooling method is used to reduce the spatial dimension of the feature map; S3.6 flattens all feature maps into a single feature vector, and integrates the feature vectors through a fully connected layer to form a higher-level abstract representation; S3.

7. The feature vector processed by the fully connected layer is passed to the output layer of the convolutional neural network model to output the final feature fusion data.

8. The method for analyzing ideological and political classroom interaction based on multimodal fusion according to claim 7 is characterized by: In S3.5, the maximum pooling method is: ； in, Indicates that the feature map after the maximum pooling operation is at position The value on Represents the coordinate offset within the pooling window; Indicates the offset in the row direction; Indicates the offset in the column direction; Indicates the size of the pooling window; Represents the original feature map.

9. The method for analyzing ideological and political classroom interaction based on multimodal fusion according to claim 8 is characterized by: In S4, the fused features are used to establish an interactive analysis model based on a random forest model to analyze the effectiveness of ideological and political classroom interaction, including the following steps: S4.

1. Define labels to measure the effectiveness of ideological and political classroom interaction, and use the fused features and labels as a dataset, dividing the dataset into a training set and a test set; S4.

2. Set key parameters and fit the random forest model using the training set data; S4.

3. Use the grid search method to adjust the model's hyperparameters, find the optimal parameter combination, and evaluate the performance of the random forest model on the test set; S4.

4. Analyze the prediction results of the random forest model and verify whether the random forest model can effectively evaluate the effectiveness of ideological and political classroom interaction.

10. An ideological and political classroom interactive analysis system based on multimodal fusion, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The processor executes a computer program to implement the ideological and political classroom interaction analysis method based on multimodal fusion as described in any one of claims 1 to 9.

Images