A teaching video sentiment analysis method and system based on multi-modal analysis

Abstract

This invention discloses a method and system for sentiment analysis of teaching videos based on multimodal analysis. The method includes: acquiring classroom teaching videos and processing them to obtain a raw dataset suitable for sentiment analysis; inputting the raw data from the raw dataset into a multimodal sentiment analysis model to obtain a target dataset; performing event identification based on the target data in the target dataset to obtain multiple event results; and, based on the event results, constructing a multimodal teaching sentiment interaction visual analysis system to perform sentiment evolution analysis on individual teachers and students, sentiment association analysis on student groups, and teacher-student teaching sentiment interaction analysis to obtain sentiment analysis results for the teaching videos. This invention can focus on the emotional changes of teachers and students, integrate multimodal data, improve the accuracy of analysis in complex situations, and can be widely applied in the field of computer technology.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and system for sentiment analysis of teaching videos based on multimodal analysis. Background Technology

[0002] Emotional engagement in teaching and learning is a crucial factor influencing students' learning status and outcomes. In recent years, affective computing has become one of the most active research topics in education, attracting widespread attention from academia and industry. With the development of big data and artificial intelligence technologies, affective computing can provide technical support for identifying, analyzing, and understanding the emotional states of teachers and students in teaching and learning.

[0003] In the field of sentiment analysis for instructional videos, the following main problems exist:

[0004] (1) The importance of teachers' emotions has been overlooked. Current emotion analysis research mainly focuses on students' emotional changes, paying less attention to teachers' emotional expression and interaction during the teaching process. However, teachers' emotions play a key role in shaping the classroom atmosphere and improving teaching effectiveness.

[0005] (2) Insufficient integration of multimodal data. Existing multimodal sentiment data visualization methods use a limited range of modal information and often focus on single-dimensional sentiment analysis, failing to fully demonstrate the synergistic relationship of multimodal data in different contexts.

[0006] (3) Limitations of Emotional Event Recognition Models. Most current emotional event recognition models are based on unimodal analysis and do not fully consider the interaction between multiple factors such as context, time, and emotion. At the same time, these models are insufficient in handling the time synchronization and consistency of multimodal data, which affects the accuracy of analysis in complex situations. Summary of the Invention

[0007] The main objective of this invention is to propose a method and system for emotion analysis of teaching videos based on multimodal analysis, which can focus on the emotional changes of teachers and students, integrate multimodal data, and improve the accuracy of analysis in complex situations.

[0008] To achieve the above objectives, one aspect of this invention proposes a method for sentiment analysis of teaching videos based on multimodal analysis, comprising the following steps:

[0009] Acquire classroom teaching videos and process them to obtain raw datasets that can be used for sentiment analysis;

[0010] The raw data from the original dataset is input into the multimodal sentiment analysis model to obtain the target dataset;

[0011] Event identification is performed based on the target data in the target dataset to obtain multiple event results;

[0012] Based on the event results, a multimodal teaching emotion interaction visual analysis system was built to conduct emotion evolution analysis of individual teachers and students, emotion association analysis of student groups, and teacher-student teaching emotion interaction analysis, thereby obtaining the emotion analysis results of the teaching video.

[0013] In some embodiments, acquiring classroom teaching videos and processing them to obtain a raw dataset that can be used for sentiment analysis includes the following steps:

[0014] The acquired classroom teaching videos are preprocessed to extract student facial information, teaching audio information, and teaching text information. The teaching audio information is stored in mp3 / wav files, and the teaching text information is stored in txt files.

[0015] Extract text information from mp3 / wav files and save it as a txt file;

[0016] Align the text messages along the timeline.

[0017] In some embodiments, inputting the raw data from the original dataset into a multimodal sentiment analysis model to obtain the target dataset includes the following steps:

[0018] Students' facial information is input into the Face Emptiness Detection model in the multimodal sentiment analysis model to obtain students' facial sentiment data;

[0019] The teaching text information is input into the Text Emot ion Detection model in the multimodal sentiment analysis model to obtain the teacher's text sentiment data;

[0020] The teaching audio information is input into the Audio Emptiness Detection model in the multimodal sentiment analysis model to obtain the teacher's audio sentiment data;

[0021] The target dataset is constructed based on the student facial emotion data, the teacher text emotion data, and the teacher audio emotion data.

[0022] In some embodiments, the step of inputting student facial information into the Face Emotion Detection model in a multimodal sentiment analysis model to obtain student facial emotion data specifically involves:

[0023] Load each frame of the classroom teaching video, use the Haar feature classifier to detect faces, label each face that appears in the video, analyze the video frame by frame, complete identity matching by measuring the vector distance between the detected students and the pre-labeled people's images, perform sentiment analysis on each detected face region, capture the dominant sentiment in the analysis results, and output the facial sentiment of different students in each frame.

[0024] In some embodiments, the step of inputting teaching text information into the Text Emotion Detection model in a multimodal sentiment analysis model to obtain teacher text sentiment data specifically involves:

[0025] The input text is converted to lowercase, and extra spaces and punctuation are removed to standardize the text format; the text is broken down into individual words; each word is searched to determine its sentiment score in the VADER sentiment dictionary; the scores are adjusted based on context and grammatical features; and the scores of all words are aggregated to calculate the overall sentiment score of the text.

[0026] In some embodiments, the step of inputting teaching audio information into the AudioEmotionDetection model in a multimodal emotion analysis model to obtain teacher audio emotion data specifically involves:

[0027] The system loads input audio data, performs format conversion, and segments the audio samples into sequence data, inputting them into the model in a manner suitable for sequence processing. It preprocesses the audio data, using a convolutional neural network to extract local features, and implements temporal distributed convolution operations through a TimeDistributed wrapper, ensuring that each audio frame at each time step can be processed by convolution. An LSTM layer is used to capture the temporal dependencies in the audio sequence data, identifying dynamic changes in emotional information. The cross-entropy loss function is used to calculate the error in the model's prediction results, and the SGD optimizer is used to optimize the parameters. Finally, it generates audio emotion recognition results, providing corresponding emotion labels for each audio sample and additional information on the emotion probability distribution as needed.

[0028] In some embodiments, the step of performing event identification based on target data in the target dataset to obtain multiple event results further includes the following steps:

[0029] All the obtained emotional information was weighted and distributed, with student facial emotional data accounting for 70%, teacher text emotional data accounting for 15%, and teacher audio emotional data accounting for 15%. Emotional labels were then assigned to each frame.

[0030] During the identification process, teaching events are divided into four categories: attention time, classroom climax, abnormal state, and normal state;

[0031] Among them, focus time refers to the time when more than 50% and more than 10 students in the classroom are in a neutral or negative emotional state for more than 5 seconds, while the teacher is in a neutral or negative emotional state.

[0032] For classroom climax: When the teacher's emotions are neutral or positive, more than 50% of all identified students and more than 10 students in the classroom are in a positive emotional state for more than 5 seconds.

[0033] Abnormal states: When the teacher is in a negative emotional state or when all identified students in the classroom are simultaneously in 5 or more different emotional states for more than 5 seconds;

[0034] For normal conditions: this refers to class time other than focused time, peak classroom moments, and abnormal conditions.

[0035] In some embodiments, the step of conducting emotional evolution analysis of individual teachers and students, emotional association analysis of student groups, and emotional interaction analysis of teachers and students based on the event results, and obtaining the emotional analysis results of the teaching video by building a multimodal teaching emotional interaction visual analysis system, includes the following steps:

[0036] A visualization section is set up in the first area of ​​the classroom teaching video to display the overall emotional data of the entire lesson;

[0037] A graph display section is set up in the second area of ​​the classroom teaching video to show the emotional changes of each character over time;

[0038] A multi-column information card section is set up in the third area of ​​the classroom teaching video to display various emotions identified in the video.

[0039] Another aspect of this invention provides a sentiment analysis system for teaching videos based on multimodal analysis, comprising:

[0040] The first module is used to acquire classroom teaching videos and process them to obtain a raw dataset that can be used for sentiment analysis;

[0041] The second module is used to input the raw data from the original dataset into the multimodal sentiment analysis model to obtain the target dataset;

[0042] The third module is used to perform event identification based on the target data in the target dataset and obtain multiple event results.

[0043] The fourth module is used to conduct emotional evolution analysis of individual teachers and students, emotional association analysis of student groups, and emotional interaction analysis of teachers and students based on the event results by building a multimodal teaching emotional interaction visual analysis system, so as to obtain the emotional analysis results of the teaching video.

[0044] To achieve the above objectives, another aspect of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0045] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0046] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0047] The embodiments of this invention include at least the following beneficial effects: This invention provides a method and system for sentiment analysis of teaching videos based on multimodal analysis. This scheme acquires classroom teaching videos and processes them to obtain a raw dataset suitable for sentiment analysis; the raw data in the raw dataset is input into a multimodal sentiment analysis model to obtain a target dataset; event identification is performed based on the target data in the target dataset to obtain multiple event results; based on the event results, a multimodal teaching sentiment interaction visual analysis system is built to perform sentiment evolution analysis on individual teachers and students, sentiment association analysis on student groups, and teacher-student teaching sentiment interaction analysis to obtain the sentiment analysis results of the teaching videos. The embodiments of this invention can focus on the emotional changes of teachers and students, integrate multimodal data, and improve the accuracy of analysis in complex situations. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0049] Figure 2 This is a flowchart of the overall steps provided in the embodiments of the present invention;

[0050] Figure 3 This is a flowchart illustrating the specific implementation steps provided in the embodiments of the present invention;

[0051] Figure 4 This is a flowchart of the multimodal sentiment analysis model provided in this embodiment of the invention;

[0052] Figure 5 This is the visualization process for the first area provided in the embodiments of the present invention;

[0053] Figure 6 This is the visualization process for the second area provided in this embodiment of the invention;

[0054] Figure 7 This is the visualization process for the third area provided in the embodiments of the present invention;

[0055] Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0057] It is understood that the terms “first,” “second,” etc., used in this invention may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, first information may also be referred to as second information without departing from the scope of embodiments of the invention, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to determination” as used herein may be interpreted as “when…” or “when…” or “in response to determination.”

[0058] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0060] Before providing a detailed description of the embodiments of the present invention, some related technologies involved in the embodiments of the present invention will be described first, as follows:

[0061] Video teaching: Video teaching is a method of education and training using video content. This format can include recorded lectures, online courses, teaching demonstrations, etc. Video teaching provides dynamic visual and auditory information, helping students better understand and master knowledge.

[0062] Emotional event recognition: Emotional event recognition refers to identifying human emotional states or emotional events by analyzing video or audio content. This typically involves computer vision and natural language processing technologies to extract emotional features from various information sources such as facial expressions, voice tone, and body language.

[0063] Visual analytics: Visual analytics is a major research area of ​​data visualization, a science that uses interactive visual interfaces for analysis and reasoning. It connects human intelligence with machine intelligence, allowing uniquely human strengths to be fully utilized in the analytical process. Humans can interact with machines through visual views, intuitively and efficiently transforming massive amounts of information into knowledge and reasoning, allowing them to participate in the main analysis and decision-making processes, discover visual knowledge, and acquire valuable information.

[0064] The present invention relates to a method and system for sentiment analysis of teaching videos based on multimodal analysis, and pertains to the field of computer technology. The method for sentiment analysis of teaching videos based on multimodal analysis provided in this invention can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal may be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited thereto. The server may be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server may also be a node server in a blockchain network. The software may be an application implementing the sentiment analysis method for teaching videos based on multimodal analysis, but is not limited to the above forms.

[0065] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0066] like Figure 1 The diagram shown is a schematic representation of an implementation environment provided by an embodiment of the present invention. (Refer to...) Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected via a network, either wirelessly or via a wired connection, to complete data transmission and exchange.

[0067] Server 101 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0068] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0069] Terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc. It can also be a vehicle-mounted terminal of the various device types described above, but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment of the invention does not impose any limitations.

[0070] Exemplary based on Figure 1The implementation environment shown in this embodiment of the invention provides a method for sentiment analysis of teaching videos based on multimodal analysis. The following description uses the application of this method for sentiment analysis of teaching videos based on multimodal analysis in server 101 as an example. It can be understood that this method can also be applied to terminal 102.

[0071] Reference Figure 2 , Figure 2 The flowchart illustrates a multimodal analysis-based sentiment analysis method for instructional videos applied to a server, as provided in this embodiment of the invention. The execution subject of this method can be any of the aforementioned computer devices (including servers or terminals). (Refer to...) Figure 2 The method may include the following steps:

[0072] Acquire classroom teaching videos and process them to obtain raw datasets that can be used for sentiment analysis;

[0073] The raw data from the original dataset is input into the multimodal sentiment analysis model to obtain the target dataset;

[0074] Event identification is performed based on the target data in the target dataset to obtain multiple event results;

[0075] Based on the event results, a multimodal teaching emotion interaction visual analysis system was built to conduct emotion evolution analysis of individual teachers and students, emotion association analysis of student groups, and teacher-student teaching emotion interaction analysis, thereby obtaining the emotion analysis results of the teaching video.

[0076] In some embodiments, acquiring classroom teaching videos and processing them to obtain a raw dataset that can be used for sentiment analysis includes the following steps:

[0077] The acquired classroom teaching videos are preprocessed to extract student facial information, teaching audio information, and teaching text information. The teaching audio information is stored in mp3 / wav files, and the teaching text information is stored in txt files.

[0078] Extract text information from mp3 / wav files and save it as a txt file;

[0079] Align the text messages along the timeline.

[0080] In some embodiments, inputting the raw data from the original dataset into a multimodal sentiment analysis model to obtain the target dataset includes the following steps:

[0081] Students' facial information is input into the Face Emptiness Detection model in the multimodal sentiment analysis model to obtain students' facial sentiment data;

[0082] The teaching text information is input into the Text Emot ion Detection model in the multimodal sentiment analysis model to obtain the teacher's text sentiment data;

[0083] The teaching audio information is input into the Audio Emptiness Detection model in the multimodal sentiment analysis model to obtain the teacher's audio sentiment data;

[0084] The target dataset is constructed based on the student facial emotion data, the teacher text emotion data, and the teacher audio emotion data.

[0085] In some embodiments, the step of inputting student facial information into the Face Emotion Detection model in a multimodal sentiment analysis model to obtain student facial emotion data specifically involves:

[0086] Load each frame of the classroom teaching video, use the Haar feature classifier to detect faces, label each face that appears in the video, analyze the video frame by frame, complete identity matching by measuring the vector distance between the detected students and the pre-labeled people's images, perform sentiment analysis on each detected face region, capture the dominant sentiment in the analysis results, and output the facial sentiment of different students in each frame.

[0087] In some embodiments, the step of inputting teaching text information into the Text Emotion Detection model in a multimodal sentiment analysis model to obtain teacher text sentiment data specifically involves:

[0088] The input text is converted to lowercase, and extra spaces and punctuation are removed to standardize the text format; the text is broken down into individual words; each word is searched to determine its sentiment score in the VADER sentiment dictionary; the scores are adjusted based on context and grammatical features; and the scores of all words are aggregated to calculate the overall sentiment score of the text.

[0089] In some embodiments, the step of inputting teaching audio information into the AudioEmotionDetection model in a multimodal emotion analysis model to obtain teacher audio emotion data specifically involves:

[0090] The system loads input audio data, performs format conversion, and segments the audio samples into sequence data, inputting them into the model in a manner suitable for sequence processing. It preprocesses the audio data, using a convolutional neural network to extract local features, and implements temporal distributed convolution operations through a TimeDistributed wrapper, ensuring that each audio frame at each time step can be processed by convolution. An LSTM layer is used to capture the temporal dependencies in the audio sequence data, identifying dynamic changes in emotional information. The cross-entropy loss function is used to calculate the error in the model's prediction results, and the SGD optimizer is used to optimize the parameters. Finally, it generates audio emotion recognition results, providing corresponding emotion labels for each audio sample and additional information on the emotion probability distribution as needed.

[0091] In some embodiments, the step of performing event identification based on target data in the target dataset to obtain multiple event results further includes the following steps:

[0092] Users can assign weights to the obtained emotional data themselves. In this embodiment, the weight of student facial emotional data is set at 70%, the weight of teacher text emotional data is set at 15%, and the weight of teacher audio emotional data is set at 15%. Emotional tags are assigned to each frame according to this ratio.

[0093] During the identification process, teaching events are divided into four categories: attention time, classroom climax, abnormal state, and normal state;

[0094] Specifically, focus time refers to the period during which, when the teacher's emotional state is neutral or positive, more than 50% and more than 10 students in the classroom maintain a neutral or lower emotional state for more than 5 seconds (Note: Users can set the "50%", "10", and "5" values ​​themselves. The settings provided here are based on the data obtained in this embodiment and are considered appropriate).

[0095] For classroom climax: When the teacher's emotions are neutral or positive, more than 50% and more than 10 students identified in the classroom are in a positive emotional state for more than 5 seconds (Note: The data "50%", "10", and "5" can be set by the user. The settings given here are only based on the data obtained in this embodiment and are considered suitable).

[0096] For abnormal states: when the teacher is in a negative emotional state or when all identified students in the classroom are simultaneously in 5 or more different emotional states, and the duration exceeds 5 seconds (Note: the "5 states" and "5 seconds" data can be set by the user. Here, only a more suitable setting is given based on the data obtained in this embodiment).

[0097] Normal state: refers to classroom time excluding focused time, classroom climax, and abnormal state.

[0098] In some embodiments, the step of conducting emotional evolution analysis of individual teachers and students, emotional association analysis of student groups, and emotional interaction analysis of teachers and students based on the event results, and obtaining the emotional analysis results of the teaching video by building a multimodal teaching emotional interaction visual analysis system, includes the following steps:

[0099] A visualization section is set up in the first area of ​​the classroom teaching video to display the overall emotional data of the entire lesson;

[0100] A graph display section is set up in the second area of ​​the classroom teaching video to show the emotional changes of each character over time;

[0101] A multi-column information card section is set up in the third area of ​​the classroom teaching video to display various emotions identified in the video.

[0102] The specific implementation process of this invention will be described in detail below using a specific application scenario as an example:

[0103] To address the problems existing in the prior art, the purpose of this invention is to provide a method for sentiment analysis of teaching videos based on multimodal analysis, characterized by:

[0104] (1) Multimodal sentiment analysis and event recognition. This invention innovatively considers the emotional changes of both teachers and students from multiple perspectives. By performing unified time-synchronized analysis of students' facial expressions, teachers' audio emotions, and the emotional information in the language text, it achieves comprehensive event recognition of teaching videos. The system utilizes advanced data visualization technology to construct a sentiment analysis platform for teaching videos, providing intuitive and reliable sentiment analysis data support for classroom effectiveness evaluation.

[0105] (2) Innovative Visualization Design Facilitates Multimodal Data Presentation. This technology has designed a set of novel visualization charts to efficiently present key aspects of sentiment analysis and the results of multimodal data analysis. Through precise visual expression, the presentation effect of multimodal data is improved, ensuring the intuitiveness and reliability of data analysis results, and providing a scientific basis for educational management and classroom optimization.

[0106] refer to Figure 3 The multimodal analysis-based sentiment analysis method for teaching videos according to embodiments of the present invention may include the following steps:

[0107] Step 1: Data processing.

[0108] This project uses classroom teaching videos as the raw audio and video data.

[0109] First, the original teaching videos are preprocessed, which involves extracting student facial information, teaching audio information, and teaching text information from the original video data.

[0110] Step 1: Input the initial audio and video data, i.e. the original teaching video.

[0111] Step 2: Extract student facial expression information (directly obtained from the original video file), audio information (stored in mp3 / wav files), and text information (stored in txt files) from the original teaching video. Convert the original teaching video to mp3 / wav files for later use, and extract the text information from the mp3 / wav files and save it as a txt file.

[0112] Step 3: Align the text information with the timeline.

[0113] Step 2: Multimodal Sentiment Analysis Model

[0114] refer to Figure 4 In this embodiment of the invention, the processed student facial information, teaching text information, and teaching audio information are input into a multimodal sentiment analysis model to obtain student facial sentiment data, teacher text sentiment data, and teacher audio sentiment data, respectively.

[0115] Face Emptiness Detection Model:

[0116] Main steps:

[0117] Step 1: Process the video frame by frame: Load the video and read each frame of the video until the video ends.

[0118] Step 2: Face detection: Use the Haar feature classifier to detect faces.

[0119] Step 3: Character labeling: Manually label each face that appears in the video to facilitate identity matching in the next step.

[0120] Step 4: Identity matching: Analyze the video frame by frame and complete the identity matching by measuring the vector distance between the detected students and the images of pre-labeled people.

[0121] Step 5: Sentiment Analysis: Perform sentiment analysis on each detected face region, capture the dominant sentiment in the analysis results, and output the facial sentiment of different students in each frame.

[0122] Text Emulation Detection Model: Using the VADER model.

[0123] Main steps:

[0124] Step 1: Text preprocessing: Convert the input text to lowercase, remove extra spaces and punctuation marks to standardize the text format.

[0125] Step 2: Tokenization: Break the text down into individual words for later processing.

[0126] Step 3: Sentiment Dictionary Lookup: Look up each word to determine its sentiment score in the VADER sentiment dictionary. Sentiment scores are categorized as positive, negative, and neutral, typically represented by a range from -1 to 1.

[0127] Step 4: Sentiment Intensity Adjustment: Adjust scores based on context and grammatical features. For example, "best" will receive a higher positive score than "good"; negative words (such as "no") will reverse the sentiment score of the following words.

[0128] Step 5: Sentiment Score Summary: Summarize the scores of all words to calculate the overall sentiment score of the text. VADER will calculate positive, negative, and neutral scores separately and generate a composite score, which is usually between -1 and 1, representing the overall sentiment tendency.

[0129] Step 6: Output Results: Output positive, negative, neutral scores and overall score. Users can judge the sentiment of the text based on these scores.

[0130] Audio Emotion Detection Model:

[0131] Main steps:

[0132] Step 1: Audio Data Loading: Load the input audio data, perform format conversion, segment the audio samples into sequence data, and input them into the model in a manner suitable for sequence processing.

[0133] Step 2: Data Preprocessing and Augmentation: Preprocess the audio data to improve the model's generalization ability and reduce the risk of overfitting.

[0134] Step 3: Feature extraction. Use a convolutional neural network (CNN) to extract local features of the audio. Implement time-distributed convolution operation through TimeDistributed wrapping so that audio frames at each time step can be processed by convolution.

[0135] Step 4: Temporal Dependency Modeling: Use LSTM layers to capture temporal dependencies in audio sequence data and identify dynamic changes in emotional information to obtain more accurate emotional expression.

[0136] Step 5: Sentiment Classification and Optimization: The cross-entropy loss function is used to calculate the error of the model prediction results, and the SGD optimizer (stochastic gradient descent) is used to optimize the parameters.

[0137] Step 6: Output sentiment labels: The model generates audio sentiment recognition results, provides corresponding sentiment labels for each audio sample, and can provide additional information such as sentiment probability distribution as needed.

[0138] Step 3: Event identification.

[0139] Event identification is performed using three aspects: student facial emotion data, teacher text emotion data, and teacher audio emotion data.

[0140] The main steps are as follows:

[0141] Step 1: Emotional Information Preprocessing: The user assigns weights to all the obtained emotional information, focusing primarily on student emotions. Therefore, the weights are allocated as follows: student facial emotion data accounts for 70%, teacher text emotion data accounts for 15%, and teacher audio emotion data accounts for 15%. Emotional labels are then assigned to each frame.

[0142] Step 2: Emotional Event Identification: In this identification, we categorize teaching events into four types: attention span, classroom climax, abnormal state, and normal state.

[0143] Attention span: When the teacher's emotional state is neutral or positive, a certain percentage (user-defined) and a certain number (user-defined) of all identified students in the classroom maintain a neutral or lower emotional state for a duration exceeding a certain number of seconds (user-defined). Two segments exceeding 5 seconds with an interval of less than 2 seconds can be considered the same event. During this event, students can be considered to be actively participating in the class and engaging in serious learning.

[0144] Classroom Climax: When the teacher's emotions are neutral or positive, a certain percentage (user-defined) and a certain number (user-defined) of all identified students in the classroom maintain a positive emotional state for a duration exceeding a certain number of seconds (user-defined). Two segments exceeding 5 seconds with an interval of less than 2 seconds can be considered the same event. In this event, students' emotions are considered to have been influenced by the teacher, leading to active and engaged participation in interaction or group discussions.

[0145] Abnormal state: When the teacher is in a negative emotional state or when all identified students in the classroom are simultaneously in five or more different emotional states, and the duration exceeds a certain number of seconds (user-defined). Two segments exceeding 5 seconds with an interval of less than 2 seconds can be considered the same event. This event can be attributed to most students being distracted and inattentive, or to the teacher experiencing emotional fluctuations and uttering words with negative connotations.

[0146] Normal state: Class time outside of periods of focused attention, peak classroom moments, and abnormal situations is considered a normal teaching process, i.e., a normal state.

[0147] Step 4: System Setup.

[0148] Visual Analysis of Teacher-Student Emotional Interaction in Teaching. Through the analysis of audio and video data, algorithm design, visualization design, and human-computer interaction exploration, a multimodal visual analysis system for teaching emotional interaction is built. This system aims to analyze the evolution of individual teacher and student emotions, the emotional correlation of student groups, and the interaction of teacher-student teaching emotions, thereby diagnosing teaching emotions, exploring teaching patterns, and improving teaching effectiveness.

[0149] In terms of system presentation, there is a column below the video that intuitively displays the overall emotional data for the entire lesson, such as... Figure 5 As shown, a stacked chart displays the emotional proportion of a single subject throughout the lesson. Different colored stacked bars represent different emotions, and the corresponding subject's portrait and name are labeled next to them. A sorting option is provided, allowing users to select a certain emotion and sort the subjects in descending order according to the proportion of that emotion, providing a preliminary analysis of each subject's individual performance throughout the video. A pie chart uses different colors to display the proportion of various emotions for all subjects, providing a preliminary overall overview analysis.

[0150] The upper right column displays the information using a line graph, such as... Figure 6 As shown, multiple data curves are plotted on a timeline to illustrate the emotional changes of various individuals over time. Options are provided to filter the curves (one or more) to display the curves of interest. The d3-brush allows users to easily specify time periods of interest within the data visualization interface using mouse manipulation. Users can then explore the changes in specific objects and emotions within a given timeframe.

[0151] The column at the bottom right is as follows Figure 7As shown, the system uses multi-column data cards to display various emotions identified in the video. For each event, the system provides users with a preliminary understanding of the event through the occurrence time and word cloud. On the far right is an image showing the degree of emotional correlation, which includes the degree of connection between facial expressions, tone of voice, content and emotion, including the shift and duration of the dominant emotion during the event. This is a key part of the system's multimodal recognition and visualization.

[0152] In summary, this method constructs a teaching video sentiment analysis system using multimodal sentiment analysis technology, comprehensively analyzing the emotions of two important roles: teachers and students. This provides a basis for analyzing and evaluating classroom effectiveness, enhancing the reliability and rationality of such analysis and evaluation.

[0153] This invention has the following characteristics:

[0154] 1. In the emotion analysis of teaching videos, the emotions of students and teachers should be considered in a unified manner from a temporal perspective.

[0155] 2. Multimodal sentiment analysis: Student sentiment is obtained by analyzing students' facial expressions, and teacher sentiment is obtained by analyzing the audio and text information of the teacher's lecture. Combining the different dimensions of sentiment from both provides the possibility for analyzing and evaluating classroom effectiveness.

[0156] 3. Event Recognition: Starting with students' facial emotion data and teachers' text and audio emotion data, users can set weights for the multi-dimensional emotions of both to identify different events, providing the possibility for more in-depth analysis and evaluation of classroom effectiveness.

[0157] 4. Sentiment Data Visualization System: Displays the obtained sentiment data as various charts and provides interactive exploration, making the analysis results more intuitive.

[0158] Compared with the prior art, the present invention has the following advantages:

[0159] Existing multimodal sentiment data visualization methods utilize relatively limited modal information and often focus more on student-centric sentiment analysis. This technology, however, employs multimodal sentiment analysis, analyzing sentiment data from three aspects (facial, audio, and text). Furthermore, it innovatively incorporates a teacher-student collaborative analysis feature, enabling simultaneous sentiment identification for both teachers and students. The obtained sentiment data is then visualized, making the results more intuitive and allowing users to interactively explore and view them as needed. Additionally, the technology proposes segmenting teaching events, providing users with a clearer understanding of changes in classroom pace.

[0160] Another aspect of this invention provides a sentiment analysis system for teaching videos based on multimodal analysis, comprising:

[0161] The first module is used to acquire classroom teaching videos and process them to obtain a raw dataset that can be used for sentiment analysis;

[0162] The second module is used to input the raw data from the original dataset into the multimodal sentiment analysis model to obtain the target dataset;

[0163] The third module is used to perform event identification based on the target data in the target dataset and obtain multiple event results.

[0164] The fourth module is used to conduct emotional evolution analysis of individual teachers and students, emotional association analysis of student groups, and emotional interaction analysis of teachers and students based on the event results by building a multimodal teaching emotional interaction visual analysis system, so as to obtain the emotional analysis results of the teaching video.

[0165] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0166] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned multimodal analysis-based emotional analysis method for instructional videos. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0167] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0168] Please see Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0169] The processor 801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0170] The memory 802 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 802 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801 to implement the teaching video sentiment analysis method based on multimodal analysis of the embodiments of this invention.

[0171] The 803 input / output interface is used to implement information input and output.

[0172] The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0173] Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804);

[0174] The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0175] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for sentiment analysis of instructional videos based on multimodal analysis.

[0176] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0177] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0178] It should be noted that in various specific embodiments of the present invention, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of the present invention require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user-related data for the normal operation of the embodiments of the present invention acquired.

[0179] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0180] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0181] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0182] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0183] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0184] It should be understood that in this invention, "at least one (item)" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0185] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0186] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0187] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0188] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0189] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present invention should be within the scope of the claims of the present invention.

Claims

1. A sentiment analysis method for instructional videos based on multimodal analysis, characterized in that, Includes the following steps: Acquire classroom teaching videos and process them to obtain raw datasets that can be used for sentiment analysis; The raw data from the original dataset is input into the multimodal sentiment analysis model to obtain the target dataset; Event identification is performed based on the target data in the target dataset to obtain multiple event results; Based on the event results, a multimodal teaching emotional interaction visual analysis system was built to conduct emotional evolution analysis of individual teachers and students, emotional association analysis of student groups, and emotional interaction analysis of teachers and students, thereby obtaining the emotional analysis results of the teaching videos. The process of inputting the raw data from the original dataset into the multimodal sentiment analysis model to obtain the target dataset includes the following steps: Students' facial information is input into the Face Emotion Detection model in the multimodal sentiment analysis model to obtain students' facial sentiment data; The teaching text information is input into the Text Emotion Detection model in the multimodal sentiment analysis model to obtain the teacher's text sentiment data; The teaching audio information is input into the Audio Emotion Detection model in the multimodal sentiment analysis model to obtain the teacher's audio sentiment data; The target dataset is constructed based on the student facial emotion data, the teacher text emotion data, and the teacher audio emotion data; The step of identifying events based on target data in the target dataset to obtain multiple event results also includes the following steps: All the obtained emotional information was weighted and distributed, with student facial emotional data accounting for 70%, teacher text emotional data accounting for 15%, and teacher audio emotional data accounting for 15%. Emotional labels were then assigned to each frame. During the identification process, teaching events are divided into four categories: attention time, classroom climax, abnormal state, and normal state; Among them, focus time refers to the time when more than 50% and more than 10 students in the classroom are in a neutral or negative emotional state for more than 5 seconds, while the teacher is in a neutral or negative emotional state. For classroom climax: When the teacher's emotions are neutral or positive, more than 50% of all identified students and more than 10 students in the classroom are in a positive emotional state for more than 5 seconds. Abnormal states: When the teacher is in a negative emotional state or when all identified students in the classroom are simultaneously in 5 or more different emotional states for more than 5 seconds; For normal state: refers to class time other than focused time, class climax, and abnormal state; Based on the event results, a multimodal teaching emotion interaction visual analysis system is built to conduct emotion evolution analysis on individual teachers and students, emotion association analysis on student groups, and teacher-student teaching emotion interaction analysis to obtain the emotion analysis results of the teaching video, including the following steps: A visualization section is set up in the first area of ​​the classroom teaching video to display the overall emotional data of the entire lesson; A graph display section is set up in the second area of ​​the classroom teaching video to show the emotional changes of each character over time; A multi-column information card section is set up in the third area of ​​the classroom teaching video to display various emotions identified in the video.

2. The method for sentiment analysis of teaching videos based on multimodal analysis according to claim 1, characterized in that, The process of acquiring classroom teaching videos and processing them to obtain a raw dataset suitable for sentiment analysis includes the following steps: The acquired classroom teaching videos are preprocessed to extract student facial information, teaching audio information, and teaching text information. The teaching audio information is stored in mp3 / wav files, and the teaching text information is stored in txt files. Extract text information from mp3 / wav files and save it as a txt file; Align the text messages along the timeline.

3. The method for sentiment analysis of teaching videos based on multimodal analysis according to claim 1, characterized in that, The process of inputting student facial information into the Face Emotion Detection model within the multimodal sentiment analysis model to obtain student facial emotion data specifically involves: Load each frame of the classroom teaching video, use the Haar feature classifier to detect faces, label each face that appears in the video, analyze the video frame by frame, complete identity matching by measuring the vector distance between the detected students and the pre-labeled people's images, perform sentiment analysis on each detected face region, capture the dominant sentiment in the analysis results, and output the facial sentiment of different students in each frame.

4. The method for sentiment analysis of teaching videos based on multimodal analysis according to claim 1, characterized in that, The process of inputting teaching text information into the Text Emotion Detection model within the multimodal sentiment analysis model to obtain teacher text sentiment data specifically involves: Convert the input text to lowercase, remove extra spaces and punctuation marks to standardize the text format; break the text into individual words; look up each word and determine its sentiment score in the VADER sentiment dictionary; Scores are adjusted based on context and grammatical features; scores for all words are aggregated to calculate the overall sentiment score for the text. The process of inputting teaching audio information into the Audio Emotion Detection model within the multimodal emotion analysis model to obtain teacher audio emotion data specifically involves: The system loads input audio data, performs format conversion, and segments the audio samples into sequence data, inputting them into the model in a manner suitable for sequence processing. It preprocesses the audio data, using a convolutional neural network to extract local features, and implements temporal distributed convolution operations through TimeDistributed wrapping, ensuring that audio frames at each time step can be processed by convolution. An LSTM layer is used to capture temporal dependencies in the audio sequence data, identifying dynamic changes in emotional information. The cross-entropy loss function is used to calculate the error in the model's prediction results, and the SGD optimizer is used to optimize the parameters. Finally, it generates audio emotion recognition results, providing corresponding emotion labels for each audio sample and additional information on the emotion probability distribution as needed.

5. A system for implementing the multimodal analysis-based sentiment analysis method for instructional videos as described in any one of claims 1-4, characterized in that, include: The first module is used to acquire classroom teaching videos and process them to obtain a raw dataset that can be used for sentiment analysis; The second module is used to input the raw data from the original dataset into the multimodal sentiment analysis model to obtain the target dataset; The third module is used to perform event identification based on the target data in the target dataset and obtain multiple event results. The fourth module is used to conduct emotional evolution analysis of individual teachers and students, emotional association analysis of student groups, and emotional interaction analysis of teachers and students based on the event results by building a multimodal teaching emotional interaction visual analysis system, so as to obtain the emotional analysis results of the teaching video.

6. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 4.

Images