Intelligent teaching interaction feedback method and system driven by multi-modal sentiment analysis

Abstract

The invention relates to an intelligent teaching interaction feedback method and system driven by multi-modal sentiment analysis. The method comprises the following steps: firstly, acquiring multi-modal data; and carrying out separation and fusion on the multi-modal data by using a hybrid neural network model to generate a comprehensive emotion vector. And then, performing feature extraction on the comprehensive emotion vector, capturing a dynamic emotion sequence by means of a long-short term memory network, and comparing the dynamic emotion sequence with a preset threshold to obtain a current emotion state. Based on the current emotional state, a real-time emotional change curve is mapped by using a generative adversarial network, key time nodes are extracted from the real-time emotional change curve, and then a dynamic feedback parameter vector is generated. And finally, optimizing the dynamic feedback parameter vector by using a reinforcement learning algorithm and correcting a quantitative index to obtain an optimized emotional state evaluation result. The above steps can reflect the emotional state of the student more truly, realize dynamic tracking, accurate evaluation and optimized feedback of the emotional state of the student, and significantly improve the teaching interaction effect.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and particularly relates to an intelligent teaching interaction feedback method and system driven by multi-modal sentiment analysis. Background Art

[0002] With the development of artificial intelligence technology, there has emerged an intelligent teaching interaction feedback technology driven by multi-modal sentiment analysis. In the current era of booming digital education, online education and intelligent teaching systems have become important components in the field of education. Traditional teaching interaction feedback methods are mostly based on teachers' subjective observations and simple learning performance evaluations, and cannot comprehensively and accurately capture the emotional changes and needs of students during the learning process. With the rapid development of artificial intelligence technology, the ability to process multi-modal data has been continuously improved, providing a new way to deeply understand the learning status of students. However, existing intelligent teaching systems still have many deficiencies in sentiment analysis and interaction feedback. On the one hand, most systems can only process single-modal data, making it difficult to comprehensively analyze students' multi-faceted emotional information and resulting in inaccurate judgments of students' emotional states; on the other hand, even though some systems attempt multi-modal analysis, they lack depth and effectiveness in data fusion, emotion recognition algorithms, and optimization of feedback strategies, and cannot provide personalized and accurate teaching interaction feedback based on students' real-time emotional changes, making it difficult to meet the growing high-quality teaching needs. Summary of the Invention

[0003] Based on this, in view of the above technical problems, it is necessary to provide an intelligent teaching interaction feedback method and system driven by multi-modal sentiment analysis that can achieve dynamic tracking, accurate evaluation, and optimized feedback of students' emotional states.

[0004] In a first aspect, the present application provides an intelligent teaching interaction feedback method driven by multi-modal sentiment analysis, including:

[0005] Obtain multi-modal data; the multi-modal data includes video data, audio data, and text data.

[0006] Separate the multi-modal data using a hybrid neural network model, and assign weights to each modality for fusion to generate a comprehensive sentiment vector.

[0007] Extract features from the comprehensive sentiment vector, use a long short-term memory network to capture the dynamic sentiment sequence and compare it with a preset threshold to obtain the current emotional state.

[0008] Based on the current emotional state, use a generative adversarial network to map it into a real-time emotional change curve, and extract key time nodes to generate a dynamic feedback parameter vector.

[0009] The dynamic feedback parameter vector is optimized by using a reinforcement learning algorithm, and the quantization index is corrected to obtain an optimized emotional state evaluation result.

[0010] In one embodiment, a hybrid neural network model is used to separate multimodal data, and weights are assigned to each modality for fusion to generate a comprehensive emotion vector, including:

[0011] The multimodal data is input into the hybrid neural network model for separation to obtain a multi-dimensional feature vector; the multimodal data includes a video stream, an audio stream, and a text stream.

[0012] The video stream is processed for time synchronization to generate a sequence of facial key frames aligned with the audio stream and input into a three-dimensional convolutional layer to obtain a facial expression feature vector.

[0013] The audio stream is processed for frame-by-frame noise reduction to generate a set of speech segments including speech fundamental frequency parameters and input into a bidirectional long short-term memory network to obtain a speech intonation feature vector.

[0014] The text stream is processed for semantic dependency extraction to generate a syntactic tree structure with part-of-speech tagging and input into a graph attention network to obtain a text semantic feature vector.

[0015] According to the dimensionality differences of the facial expression feature vector, the speech intonation feature vector, and the text semantic feature vector, a fusion weight matrix is obtained by using a dynamic weight allocation algorithm.

[0016] Based on the fusion weight matrix, the multi-dimensional feature vector is weighted and concatenated to obtain a comprehensive emotion vector.

[0017] In one embodiment, feature extraction is performed on the comprehensive emotion vector, and a long short-term memory network is used to capture the dynamic emotion sequence and compare it with a preset threshold to obtain the current emotional state, including:

[0018] According to the preset time window length, the comprehensive emotion vector is segmented in time series to obtain dynamic emotion sequence segments.

[0019] The dynamic emotion sequence segments are input into the long short-term memory network, and the hidden layer states of the dynamic emotion sequence segments are extracted.

[0020] A classification confidence vector is generated according to the hidden layer state; the classification confidence vector includes the probability distribution of each emotion category.

[0021] Based on the preset threshold, the probabilities of each emotion category in the classification confidence vector are judged. If the probability of an emotion category is higher than the preset threshold of the corresponding category, the current emotion state category is determined to generate the current emotion state label; if it is lower than the preset threshold, the comprehensive emotion vector is re-extracted.

[0022] Associate and store the current emotional state label with the dynamic emotional sequence segment to obtain the current emotional state.

[0023] In one embodiment, use a generative adversarial network to map based on the current emotional state to a real-time emotional change curve, and extract key time nodes to generate a dynamic feedback parameter vector, including:

[0024] Reduce the dimension of the current emotional state by using principal component analysis combined with linear discriminant analysis to generate a low-dimensional emotional feature matrix.

[0025] Input the low-dimensional emotional feature matrix into the generative adversarial network generator to obtain an emotional change curve sequence.

[0026] Use the curve slope mutation detection algorithm to extract the emotional change curve sequence to obtain a set of key time nodes.

[0027] Input the set of key time nodes into the decision tree algorithm to generate a feedback parameter vector according to the preset branching rules.

[0028] Perform calculations based on the feedback parameter vector and linear discriminant analysis to obtain a discriminant weight matrix.

[0029] Perform matrix fusion operation on the discriminant weight matrix and the feedback parameter vector to obtain a fused dynamic feedback parameter vector; the dynamic feedback parameter vector is used to update the parameter configuration of the generative adversarial network discriminator.

[0030] In one embodiment, the discriminant weight matrix is calculated by the following formula:

[0031]

[0032] Among them, W represents the discriminant weight matrix, m represents the feature dimension, α i represents the weight coefficient, S b represents the between-class scatter matrix, S ω represents the within-class scatter matrix, λ represents the balance factor, F(t) represents the feedback parameter vector, K represents the number of parameters, β k represents the mapping coefficient, P k (t) represents the basis vector function, γ represents the regularization coefficient, and R(t) represents the residual vector.

[0033] In one embodiment, use the reinforcement learning algorithm to optimize the dynamic feedback parameter vector and correct the quantization index to obtain an optimized emotional state evaluation result, including:

[0034] Obtain the real-time update data of the dynamic feedback parameter vector.

[0035] Calculate the weight gradient value according to the real-time update data using the reinforcement learning policy network.

[0036] Compare the weight gradient value with a preset feedback threshold. If the weight gradient value exceeds the feedback threshold, use a Bayesian network to correct the weight gradient value to obtain the corrected multi-dimensional sentiment feature weight.

[0037] Generate a sentiment state evaluation matrix based on the multi-dimensional sentiment feature weight and perform feature matching with a quantization index database to obtain a feature matching result.

[0038] Obtain the state transition probability distribution in the feature matching result; the state transition probability distribution is used to update the dynamic attenuation coefficient.

[0039] Combine the dynamic attenuation coefficient with the sentiment state evaluation matrix to generate an optimized sentiment state evaluation result.

[0040] In a second aspect, the present application further provides an intelligent teaching interaction feedback system driven by multi-modal sentiment analysis. The system includes:

[0041] A multi-modal data fusion module for obtaining multi-modal data; the multi-modal data includes video data, audio data, and text data; and is further used to separate the multi-modal data using a hybrid neural network model and assign weights to each modality for fusion to generate a comprehensive sentiment vector.

[0042] A sentiment state analysis module for extracting features from the comprehensive sentiment vector, using a long short-term memory network to capture dynamic sentiment sequences and compare them with a preset threshold to obtain the current sentiment state; and is further used to map the current sentiment state using a generative adversarial network into a real-time sentiment change curve, and extract key time nodes to generate a dynamic feedback parameter vector.

[0043] A state evaluation optimization module for optimizing the dynamic feedback parameter vector using a reinforcement learning algorithm and correcting the quantization index to obtain an optimized sentiment state evaluation result.

[0044] In a third aspect, the present application further provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the foregoing method is implemented.

[0045] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the foregoing method is implemented.

[0046] The above intelligent teaching interaction feedback method, system, computer device and storage medium driven by multi-modal sentiment analysis. First, multi-modal data including video data, audio data and text data are acquired. Then, a hybrid neural network model is used to separate these multi-modal data and assign weights to each modality, thereby fusing and generating a comprehensive sentiment vector. Subsequently, feature extraction is performed on the comprehensive sentiment vector, and a long short-term memory network is used to capture the dynamic sentiment sequence, which is compared with a preset threshold to obtain the current sentiment state. Based on the current sentiment state, a generative adversarial network is used to map out the real-time sentiment change curve, key time nodes are extracted from it, and then a dynamic feedback parameter vector is generated. Finally, a reinforcement learning algorithm is used to optimize the dynamic feedback parameter vector and correct the quantization index to obtain an optimized sentiment state evaluation result. The above steps make the collection and analysis of students' sentiment information more comprehensive and accurate, can more truly reflect the students' sentiment status, and realize the dynamic tracking, accurate evaluation and optimized feedback of students' sentiment states. It not only helps teachers understand the learning mood fluctuations of students in a timely manner, but also can provide personalized teaching interaction feedback for students according to the evaluation results, thereby significantly improving the teaching effect and promoting the improvement of students' learning enthusiasm and learning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0048] Figure 1 It is a flowchart of an intelligent teaching interaction feedback method driven by multi-modal sentiment analysis provided by an embodiment of the present invention;

[0049] Figure 2 It is a flowchart of using a reinforcement learning algorithm to optimize the dynamic feedback parameter vector and correct the quantization index to obtain an optimized sentiment state evaluation result provided by an embodiment of the present invention;

[0050] Figure 3 It is a structural block diagram of an intelligent teaching interaction feedback system driven by multi-modal sentiment analysis provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0051] In order to make the purpose, technical solutions and advantages of the present application clearer, the following will further describe the present application in detail in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

[0052] In one embodiment, as Figure 1 shown, the present application provides an intelligent teaching interaction feedback method driven by multimodal sentiment analysis, which may include the following steps:

[0053] Step S101, obtaining multimodal data; the multimodal data includes video data, audio data, and text data.

[0054] Specifically, the multimodal data covers three different forms of data: video data, audio data, and text data. Video data can capture rich visual information such as an individual's facial expressions and body movements, which can often intuitively reflect the individual's emotional state. For example, a smile may represent a positive emotion, while a frown may imply a negative feeling. Audio data contains features such as the intonation, speech rate, and volume of speech. Different intonation changes and speech rate speeds may convey different emotions. For example, the speech rate may increase and the intonation may rise when excited. Text data contains the text content expressed by an individual, and its emotional tendency, whether positive, negative, or neutral, can be inferred through semantic analysis. To obtain this multimodal data, various devices and channels can be used. Video data can be collected through a camera, which can be installed in a suitable position in the teaching scenario to comprehensively record the performance of students; audio data can be collected using a microphone; and text data can be obtained from students' homework, discussion area posts, online chat records, and other sources.

[0055] Step S102, separating the multimodal data using a hybrid neural network model and assigning weights to each modality to fuse and generate a comprehensive sentiment vector.

[0056] Since data of different modalities have different characteristics and structures, it is difficult to directly analyze them. Therefore, they need to be separated first. The hybrid neural network model combines the advantages of multiple neural networks and can adopt appropriate processing methods for data of different modalities. For video data, a convolutional neural network (CNN) may be used to extract spatial features in the image; for audio data, a recurrent neural network (RNN) or its variants (such as LSTM, GRU) can better handle its time series characteristics; for text data, a network with a Transformer architecture performs well in semantic understanding. After separating the features of each modality, assigning weights to each modality is a key step. The importance of data of different modalities in reflecting emotions is not fixed but varies according to the specific scenario and task. For example, in some cases, facial expressions may be more accurate in reflecting emotions than text content, and in this case, the weight of the video modality should be relatively high. By training the model, appropriate weights are determined for each modality based on a large amount of data and actual situations, and then these features with weights are fused to finally generate a comprehensive sentiment vector.

[0057] In step S103, feature extraction is performed on the comprehensive sentiment vector, and a long short-term memory network is used to capture the dynamic sentiment sequence and compare it with a preset threshold to obtain the current sentiment state.

[0058] First, feature extraction is performed on the comprehensive sentiment vector. The purpose is to extract the most representative and discriminative features from this high-dimensional vector for more accurate sentiment analysis. The long short-term memory network (LSTM) can capture the dynamic changes of sentiment over time. Sentiment is not static but fluctuates over time. LSTM can remember past sentiment information and combine current features to predict future sentiment trends. After capturing the dynamic sentiment sequence, it is compared with a preset threshold. The preset threshold is set based on a large amount of experimental data and experience and is used to divide different sentiment categories, such as positive, negative, and neutral. By comparing the size relationship between the dynamic sentiment sequence and the preset threshold, it can be determined which category the current sentiment state belongs to. If the sentiment intensity exceeds the threshold of positive sentiment, the current sentiment state is determined to be positive; if it is lower than the threshold of negative sentiment, it is determined to be negative; and if it is between the two, it is neutral. This method based on feature extraction and threshold comparison can objectively and accurately judge the current sentiment state.

[0059] In step S104, based on the current sentiment state, a generative adversarial network is used to map it into a real-time sentiment change curve, and key time nodes are extracted to generate a dynamic feedback parameter vector.

[0060] Specifically, a GAN is used to map the current sentiment state into a real-time sentiment change curve. The generator generates a possible sentiment change curve based on the current sentiment state, and the discriminator determines whether the generated curve is real and reasonable. Through continuous game and optimization between the two, the generated sentiment change curve can more accurately reflect the dynamic change process of sentiment over time. Key time nodes are extracted from the generated real-time sentiment change curve. These key time nodes are usually the moments when significant sentiment changes occur, such as when sentiment suddenly changes from positive to negative, or when the sentiment intensity rises or falls sharply. Based on these key time nodes, a dynamic feedback parameter vector is generated. This vector contains key parameters related to sentiment changes, such as the amplitude and speed of sentiment changes.

[0061] In step S105, a reinforcement learning algorithm is used to optimize the dynamic feedback parameter vector and correct the quantization index to obtain an optimized sentiment state evaluation result.

[0062] Specifically, the dynamic feedback parameter vector is used as the initial strategy of the agent. The agent continuously tries different parameter adjustment strategies in the environment and evaluates the advantages and disadvantages of these strategies based on the reward signals given by the environment. The reward signals can be designed according to the actual application scenarios and goals. For example, if the adjusted parameters can more accurately reflect the emotional state, a higher reward is given; otherwise, a lower reward is given. Through continuous learning and adjustment, the agent gradually finds the optimal parameter adjustment strategy, thereby optimizing the dynamic feedback parameter vector. At the same time, correcting the quantization index is also an important link. The quantization index is used to measure various aspects of the emotional state, such as the intensity and stability of emotions. As the dynamic feedback parameter vector is optimized, the quantization index also needs to be corrected accordingly to ensure the accuracy and consistency of the evaluation results. After the optimization of the reinforcement learning algorithm and the correction of the quantization index, the finally obtained optimized emotional state evaluation result can more accurately reflect the true emotional state of the individual.

[0063] The above-mentioned intelligent teaching interaction feedback method driven by multi-modal sentiment analysis first obtains multi-modal data including video data, audio data, and text data. Then, a hybrid neural network model is used to separate these multi-modal data and assign weights to each modality, thereby fusing and generating a comprehensive sentiment vector. Subsequently, feature extraction is performed on the comprehensive sentiment vector, and a long short-term memory network is used to capture the dynamic sentiment sequence, which is compared with a preset threshold to obtain the current emotional state. Based on the current emotional state, a generative adversarial network is used to map out the real-time emotional change curve, key time nodes are extracted from it, and then a dynamic feedback parameter vector is generated. Finally, a reinforcement learning algorithm is used to optimize the dynamic feedback parameter vector and correct the quantization index to obtain an optimized emotional state evaluation result. The above steps make the collection and analysis of students' emotional information more comprehensive and accurate, can more truly reflect the emotional status of students, and realize the dynamic tracking, accurate evaluation, and optimized feedback of students' emotional states.

[0064] In one embodiment, using a hybrid neural network model to separate multi-modal data and assign weights to each modality to fuse and generate a comprehensive sentiment vector may include the following steps:

[0065] Step S201, input the multi-modal data into the hybrid neural network model for separation to obtain a multi-dimensional feature vector; the multi-modal data includes a video stream, an audio stream, and a text stream.

[0066] Step S202, perform time synchronization processing on the video stream to generate a sequence of facial key frames aligned with the audio stream and input it into a three-dimensional convolutional layer to obtain a facial expression feature vector.

[0067] Step S203: Perform frame-by-frame noise reduction processing on the audio stream to generate a set of speech segments including voice fundamental frequency parameters and input them into a bidirectional long short-term memory network to obtain voice intonation feature vectors.

[0068] Step S204: Extract semantic dependency relationships from the text stream to generate a syntactic tree structure with part-of-speech tags and input it into a graph attention network to obtain text semantic feature vectors.

[0069] Step S205: Use the dynamic weight allocation algorithm to obtain a fusion weight matrix based on the dimensional differences of the facial expression feature vectors, voice intonation feature vectors, and text semantic feature vectors.

[0070] Step S206: Perform weighted concatenation on the multi-dimensional feature vectors based on the fusion weight matrix to obtain a comprehensive emotion vector.

[0071] First, input the multi-modal data including video stream, audio stream, and text stream into a hybrid neural network model for separation to obtain multi-dimensional feature vectors. Subsequently, different processing is carried out for data of different modalities. For the video stream, first perform time synchronization processing to generate a sequence of facial key frames aligned with the audio stream, and then input it into a three-dimensional convolutional layer to obtain facial expression feature vectors; the audio stream is processed by frame-by-frame noise reduction to generate a set of speech segments including voice fundamental frequency parameters, and then input into a bidirectional long short-term memory network to obtain voice intonation feature vectors; while the text stream needs to extract semantic dependency relationships to generate a syntactic tree structure with part-of-speech tags and then input into a graph attention network to finally obtain text semantic feature vectors. Then, based on the dimensional differences of the facial expression feature vectors, voice intonation feature vectors, and text semantic feature vectors, use the dynamic weight allocation algorithm to obtain a fusion weight matrix. Finally, perform weighted concatenation on the multi-dimensional feature vectors based on this fusion weight matrix to obtain a comprehensive emotion vector.

[0072] In this embodiment, appropriate processing methods are respectively adopted for the characteristics of data of different modalities, which can fully exploit the emotion information in each modality of data. For example, the three-dimensional convolutional layer processes video key frames to accurately extract facial expression features; the bidirectional long short-term memory network processes the audio stream to effectively capture the emotional changes in voice intonation; the graph attention network analyzes text semantics to accurately grasp the emotional tendency contained in the text. The application of the dynamic weight allocation algorithm fully considers the dimensional differences of different modality feature vectors, and allocates reasonable weights for each modality of data according to the actual situation, so that the finally generated comprehensive emotion vector can more comprehensively and accurately reflect the emotion information contained in the multi-modal data, greatly improving the accuracy and reliability of multi-modal emotion analysis.

[0073] In one embodiment, feature extraction is performed on the comprehensive sentiment vector, and a long short-term memory network is used to capture the dynamic sentiment sequence and compare it with a preset threshold to obtain the current sentiment state, which may include the following steps:

[0074] Step S301, perform time series segmentation on the comprehensive sentiment vector according to the preset time window length to obtain dynamic sentiment sequence segments.

[0075] Step S302, input the dynamic sentiment sequence segments into the long short-term memory network to extract the hidden layer state of the dynamic sentiment sequence segments.

[0076] Step S303, generate a classification confidence vector according to the hidden layer state; the classification confidence vector includes the probability distribution of each sentiment category.

[0077] Step S304, based on the preset threshold, judge the probability of each sentiment category in the classification confidence vector. If the probability of a sentiment category is higher than the preset threshold of the corresponding category, determine the current sentiment state category and generate the current sentiment state label; if it is lower than the preset threshold, re-extract the comprehensive sentiment vector.

[0078] Step S305, associate and store the current sentiment state label with the dynamic sentiment sequence segments to obtain the current sentiment state.

[0079] First, perform time series segmentation on the comprehensive sentiment vector according to the preset time window length to obtain dynamic sentiment sequence segments. Then, input these dynamic sentiment sequence segments into the long short-term memory network (LSTM). Through the powerful ability of LSTM to remember and process time series data, extract the hidden layer state of the dynamic sentiment sequence segments. After that, generate a classification confidence vector according to the hidden layer state, and this vector covers the probability distribution of each sentiment category. Then, based on the preset threshold, judge the probability of each sentiment category in the classification confidence vector. If the probability of a certain sentiment category is higher than its corresponding preset threshold, then the current sentiment state category can be determined and the corresponding current sentiment state label can be generated; if it is lower than the preset threshold, re-extract the comprehensive sentiment vector for further analysis. Finally, associate and store the generated current sentiment state label with the corresponding dynamic sentiment sequence segments to obtain the current sentiment state.

[0080] On the one hand, by segmenting the comprehensive sentiment vector through a time window and using a long short-term memory network to extract the hidden layer state, it is possible to effectively capture the dynamic change information of sentiment in the time dimension, avoid the limitations of analyzing only relying on static data, and more accurately reflect the evolution process of sentiment. On the other hand, judging the classification confidence vector based on a preset threshold and re-extracting and processing the situation below the threshold ensure the accuracy and reliability of the sentiment state judgment. Associating and storing the sentiment state labels and sequence segments provides a detailed data basis for comprehensively analyzing the sentiment change trend and characteristics in the follow-up, which helps to make more targeted and effective decisions according to the sentiment state in multiple fields such as intelligent teaching and customer service, and improve the user experience and service quality.

[0081] In one embodiment, mapping based on the current sentiment state using a generative adversarial network into a real-time sentiment change curve and extracting key time nodes to generate a dynamic feedback parameter vector may include the following steps:

[0082] Step S401, perform dimensionality reduction on the current sentiment state using principal component analysis combined with linear discriminant analysis to generate a low-dimensional sentiment feature matrix.

[0083] Step S402, input the low-dimensional sentiment feature matrix into the generative adversarial network generator to obtain a sequence of sentiment change curves.

[0084] Step S403, use the curve slope mutation detection algorithm to extract the sequence of sentiment change curves to obtain a set of key time nodes.

[0085] Step S404, input the set of key time nodes into the decision tree algorithm to generate a feedback parameter vector according to the preset branching rules.

[0086] Step S405, perform calculations based on the feedback parameter vector and linear discriminant analysis to obtain a discriminant weight matrix.

[0087] Step S406, perform matrix fusion operation on the discriminant weight matrix and the feedback parameter vector to obtain a fused dynamic feedback parameter vector; the dynamic feedback parameter vector is used to update the parameter configuration of the generative adversarial network discriminator.

[0088] Specifically, first, a method combining principal component analysis (PCA) and linear discriminant analysis (LDA) is used to reduce the dimension of the current emotional state data, thereby generating a low-dimensional emotional feature matrix. This can reduce the data dimension while retaining key emotional information, reducing the subsequent computational complexity. Then, the generated low-dimensional emotional feature matrix is input into the generator of the generative adversarial network (GAN), and the characteristics of the generator are used to generate a sequence of emotional change curves, which can intuitively show the dynamic changes of emotions over time. Subsequently, the curve slope mutation detection algorithm is used to screen and extract the generated sequence of emotional change curves, find the points with relatively significant emotional changes from them, and then obtain a set of key time nodes. After obtaining the set of key time nodes, it is input into the decision tree algorithm, and a feedback parameter vector is generated according to the preset branching rules. This vector contains important information related to the key nodes of emotional changes. Then, based on the feedback parameter vector and linear discriminant analysis, calculations are performed again to obtain a discriminant weight matrix. Finally, a matrix fusion operation is performed on the discriminant weight matrix and the feedback parameter vector to obtain a fused dynamic feedback parameter vector. This vector is used to update the parameter configuration of the discriminator of the generative adversarial network, enabling the discriminator to better judge the subsequent generated emotional data.

[0089] At the data processing level, the combination of principal component analysis and linear discriminant analysis not only effectively reduces the data dimension and improves the computational efficiency, but also highlights the key features in the emotional data. The sequence of emotional change curves generated by the generative adversarial network presents the emotional dynamic changes in an intuitive form, facilitating understanding and analysis. The application of the curve slope mutation detection algorithm and the decision tree algorithm accurately locates the key time nodes and generates a feedback parameter vector, providing strong support for further optimizing the model. In terms of model optimization, by updating the parameter configuration of the discriminator of the generative adversarial network with the dynamic feedback parameter vector, the model can continuously adapt to new data, improve the analysis and judgment ability of emotional data, and thus provide more accurate and valuable analysis results in application scenarios related to emotional analysis, such as intelligent teaching interaction, customer emotion monitoring, etc., and assist in the formulation of relevant decisions.

[0090] In one embodiment, the discriminant weight matrix can be calculated by the following formula:

[0091]

[0092] where W represents the discriminant weight matrix, m represents the feature dimension, α i represents the weight coefficient, S b represents the between-class scatter matrix, S ω represents the within-class scatter matrix, λ represents the balance factor, F(t) represents the feedback parameter vector, K represents the number of parameters, β k represents the mapping coefficient, Pk (t) represents the basis vector function, γ represents the regularization coefficient, and R(t) represents the residual vector.

[0093] In this embodiment, by integrating various key factors such as the between-class scatter matrix and the within-class scatter matrix, this formula fully considers the distribution characteristics of the data and the class differences, and can accurately measure the importance of different features in classification. In practical applications, for example, in an intelligent teaching interaction feedback system driven by multi-modal sentiment analysis, an accurate discriminant weight matrix helps to more precisely evaluate the emotional state of students, thereby providing strong support for the adjustment of teaching strategies. At the same time, the introduction of parameters such as the balance factor and the regularization coefficient enhances the stability and robustness of the model, enabling it to adapt to complex and changing data environments, effectively improving the overall performance and reliability of the system, and showing strong application potential in multiple fields such as education and customer service.

[0094] In one of the embodiments, as Figure 2 shown, using a reinforcement learning algorithm to optimize the dynamic feedback parameter vector and correct the quantization index, the optimized emotional state evaluation result can include the following steps:

[0095] Step S501, obtain the real-time updated data of the dynamic feedback parameter vector.

[0096] Step S502, calculate the weight gradient value according to the real-time updated data using the reinforcement learning policy network.

[0097] Step S503, compare the weight gradient value with a preset feedback threshold. If the weight gradient value exceeds the feedback threshold, use a Bayesian network to correct the weight gradient value to obtain the corrected multi-dimensional emotional feature weight.

[0098] Step S504, generate an emotional state evaluation matrix according to the multi-dimensional emotional feature weight and perform feature matching with the quantization index database to obtain a feature matching result.

[0099] Step S505, obtain the state transition probability distribution in the feature matching result; the state transition probability distribution is used to update the dynamic decay coefficient.

[0100] Step S506, combine the dynamic decay coefficient with the emotional state evaluation matrix to generate the optimized emotional state evaluation result.

[0101] Specifically, first, obtain the real-time updated data of the dynamic feedback parameter vector. These real-time updated data are the basis of the entire optimization process, and they reflect the latest information on the change of emotional state over time. Then, based on these real-time updated data, calculate the weight gradient value by means of a reinforcement learning policy network. The weight gradient value reflects the influence degree of parameter change on the objective function and is an important basis for adjusting the model parameters. Subsequently, compare the calculated weight gradient value with a preset feedback threshold. If the weight gradient value exceeds the feedback threshold, it means that the current parameter adjustment may be too drastic and will affect the stability of the model. At this time, use a Bayesian network to correct the weight gradient value to obtain the corrected multi-dimensional emotional feature weight. Based on the corrected multi-dimensional emotional feature weight, generate an emotional state evaluation matrix, which comprehensively considers the information of multiple emotional feature dimensions. Then, match the features of the emotional state evaluation matrix with the quantization index database to obtain the feature matching result. In the feature matching result, obtain the state transition probability distribution, which can reflect the transfer possibility of the emotional state between different moments and is used to update the dynamic decay coefficient. Finally, combine the dynamic decay coefficient with the emotional state evaluation matrix to generate the optimized emotional state evaluation result.

[0102] In this embodiment, by calculating the weight gradient value through real-time updated data and a reinforcement learning policy network, the model can dynamically adjust parameters according to the latest emotional data, improving the adaptability and accuracy of the model. The correction of the weight gradient value by the Bayesian network enhances the stability of the model and avoids model fluctuations caused by excessive parameter adjustment. Teachers can timely adjust teaching strategies based on these more accurate evaluation results, such as adjusting the difficulty and rhythm of teaching content, or giving students more targeted encouragement and guidance, thereby improving teaching effects and promoting the improvement of students' learning experience and learning achievements.

[0103] In one of the embodiments, as Figure 3 shown, the present application also provides an intelligent teaching interaction feedback system driven by multi-modal sentiment analysis. The system may include:

[0104] A multi-modal data fusion module 601, configured to obtain multi-modal data; the multi-modal data includes video data, audio data, and text data; and is further configured to separate the multi-modal data by using a hybrid neural network model and allocate weights to each modality for fusion to generate a comprehensive emotional vector.

[0105] An emotional state analysis module 602, configured to extract features from the comprehensive emotional vector, capture the dynamic emotional sequence by using a long short-term memory network and compare it with a preset threshold to obtain the current emotional state; and is further configured to map the current emotional state to a real-time emotional change curve by using a generative adversarial network and extract key time nodes to generate a dynamic feedback parameter vector.

[0106] The state evaluation optimization module 603 is used to optimize the dynamic feedback parameter vector and modify the quantitative index using the reinforcement learning algorithm to obtain the optimized emotional state evaluation result.

[0107] In the aforementioned intelligent teaching interactive feedback system driven by multimodal sentiment analysis, the multimodal data fusion module acquires multimodal data, including video, audio, and text data. It then uses a hybrid neural network model to separate these data, assigning weights to each modal data based on its characteristics. Finally, it fuses these data to generate a comprehensive sentiment vector, thereby integrating the sentiment information within the multimodal data. The sentiment state analysis module, based on the comprehensive sentiment vector, first performs feature extraction, employs a long short-term memory network to capture dynamic sentiment sequences, and then compares them with a preset threshold to determine the current sentiment state. Simultaneously, based on this state, a generative adversarial network is used to map a real-time sentiment change curve. Dynamic feedback parameter vectors are generated by extracting key time nodes, thereby exploring patterns in sentiment change and providing data support for subsequent optimization. The state evaluation and optimization module receives the dynamic feedback parameter vector, optimizes it using a reinforcement learning algorithm, and modifies quantitative indicators to ultimately obtain optimized sentiment state assessment results. These steps make the collection and analysis of student sentiment information more comprehensive and accurate, more realistically reflecting students' emotional states and enabling dynamic tracking, precise assessment, and optimized feedback of students' sentiment states.

[0108] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0109] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the multimodal sentiment analysis-driven intelligent teaching interactive feedback method and system as described above are implemented.

[0110] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0111] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The components described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present disclosure solution. A person of ordinary skill in the art can understand and implement it without creative efforts.

[0112] The above embodiments only represent several implementation manners of the embodiments of the present application. The descriptions are relatively specific and detailed, but they should not be construed as limiting the patent scope of the embodiments of the application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the embodiments of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the embodiments of the present application.

Claims

1. A multi-modal sentiment analysis-driven intelligent teaching interaction feedback method, characterized in that, The method includes: Obtaining multimodal data; the multimodal data includes video data, audio data, and text data; Separating the multimodal data using a hybrid neural network model and assigning weights to each modality for fusion to generate a comprehensive sentiment vector; Performing feature extraction on the comprehensive sentiment vector, using a long short-term memory network to capture the dynamic sentiment sequence and comparing it with a preset threshold to obtain the current sentiment state; Based on the current sentiment state, using a generative adversarial network to map it into a real-time sentiment change curve, extracting key time nodes to generate a dynamic feedback parameter vector; Using a reinforcement learning algorithm to optimize the dynamic feedback parameter vector and correct the quantization index to obtain an optimized sentiment state evaluation result.

2. The method according to claim 1, characterized in that, The separating the multimodal data using a hybrid neural network model and assigning weights to each modality for fusion to generate a comprehensive sentiment vector includes: Inputting the multimodal data into a hybrid neural network model for separation to obtain multi-dimensional feature vectors; the multimodal data includes a video stream, an audio stream, and a text stream; Performing time synchronization processing on the video stream to generate a sequence of facial key frames aligned with the audio stream and inputting it into a three-dimensional convolutional layer to obtain a facial expression feature vector; Performing frame-by-frame noise reduction processing on the audio stream to generate a set of speech segments including voice fundamental frequency parameters and inputting it into a bidirectional long short-term memory network to obtain a speech intonation feature vector; Performing semantic dependency relationship extraction on the text stream to generate a syntactic tree structure with part-of-speech tags and inputting it into a graph attention network to obtain a text semantic feature vector; Using a dynamic weight assignment algorithm to obtain a fusion weight matrix according to the dimensionality differences of the facial expression feature vector, speech intonation feature vector, and text semantic feature vector; Based on the fusion weight matrix, performing weighted splicing on the multi-dimensional feature vectors to obtain a comprehensive sentiment vector.

3. The method according to claim 1, characterized in that, The performing feature extraction on the comprehensive sentiment vector, using a long short-term memory network to capture the dynamic sentiment sequence and comparing it with a preset threshold to obtain the current sentiment state includes: Performing time series segmentation on the comprehensive sentiment vector according to a preset time window length to obtain dynamic sentiment sequence segments; Inputting the dynamic sentiment sequence segments into a long short-term memory network to extract the hidden layer states of the dynamic sentiment sequence segments; Generating a classification confidence vector according to the hidden layer states; the classification confidence vector includes the probability distributions of each sentiment category; Based on a preset threshold, judging the probability of each sentiment category in the classification confidence vector. If the probability of the sentiment category is higher than the preset threshold of the corresponding category, determining the current sentiment state category to generate a current sentiment state label; if it is lower than the preset threshold, re-extracting the comprehensive sentiment vector; Associating and storing the current sentiment state label with the dynamic sentiment sequence segments to obtain the current sentiment state.

4. The method according to claim 1, wherein The mapping the current sentiment state into a real-time sentiment change curve using a generative adversarial network based on the current sentiment state, extracting key time nodes to generate a dynamic feedback parameter vector includes: Performing dimensionality reduction on the current sentiment state using principal component analysis combined with linear discriminant analysis to generate a low-dimensional sentiment feature matrix; Input the low-dimensional emotion feature matrix into the generator of the generative adversarial network to obtain a sequence of emotion change curves; Use the curve slope mutation detection algorithm to extract the sequence of emotion change curves to obtain a set of key time nodes; Input the set of key time nodes into the decision tree algorithm to generate a feedback parameter vector according to the preset branching rules; Perform calculations based on the feedback parameter vector and the linear discriminant analysis to obtain a discriminant weight matrix; Perform matrix fusion operation on the discriminant weight matrix and the feedback parameter vector to obtain a fused dynamic feedback parameter vector; the dynamic feedback parameter vector is used to update the parameter configuration of the discriminator of the generative adversarial network.

5. The method according to claim 4, characterized in that, The discriminant weight matrix is calculated through the following formula: Among them, W represents the discrimination weight matrix, m represents the feature dimension, α i represents the weight coefficient, S b represents the between-class scatter matrix, S ω represents the within-class scatter matrix, λ represents the balance factor, F(t) represents the feedback parameter vector, K represents the number of parameters, β k represents the mapping coefficient, P k (t) represents the basis vector function, γ represents the regularization coefficient, and R(t) represents the residual vector.

6. The method according to claim 1, characterized in that, Optimizing the dynamic feedback parameter vector and correcting the quantization index by using the reinforcement learning algorithm to obtain an optimized emotion state evaluation result, including: Obtain the real-time update data of the dynamic feedback parameter vector; Calculate the weight gradient value according to the real-time update data by using the reinforcement learning policy network; Compare the weight gradient value with a preset feedback threshold. If the weight gradient value exceeds the feedback threshold, use the Bayesian network to correct the weight gradient value to obtain a corrected multi-dimensional emotion feature weight; Generate an emotion state evaluation matrix according to the multi-dimensional emotion feature weight and perform feature matching with the quantization index database to obtain a feature matching result; Obtain the state transition probability distribution in the feature matching result; the state transition probability distribution is used to update the dynamic decay coefficient; Combine the dynamic decay coefficient with the emotion state evaluation matrix to generate an optimized emotion state evaluation result.

7. An intelligent teaching interaction feedback system driven by multimodal sentiment analysis, characterized in that, The system includes: A multi-modal data fusion module, configured to obtain multi-modal data; the multi-modal data includes video data, audio data, and text data; and is further configured to separate the multi-modal data by using a hybrid neural network model and assign weights to each modality for fusion to generate a comprehensive emotion vector; An emotion state analysis module, configured to extract features from the comprehensive emotion vector, capture the dynamic emotion sequence by using a long short-term memory network and compare it with a preset threshold to obtain the current emotion state; and is further configured to map the current emotion state to a real-time emotion change curve by using a generative adversarial network, extract key time nodes to generate a dynamic feedback parameter vector; A state evaluation optimization module, configured to optimize the dynamic feedback parameter vector and correct the quantization index by using a reinforcement learning algorithm to obtain an optimized emotion state evaluation result.

8. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images