Adaptive Teaching Strategy Adjustment Method Based on Sentiment Analysis, Computer Device

Abstract

The present application discloses an adaptive teaching strategy adjustment method and a computer device based on sentiment analysis. Among them, the method includes: obtaining a target sentiment feature data stream according to multi-modal student sentiment data, and then extracting visual deep features, audio deep features, and physiological deep features from it; processing visual attention weights, audio attention weights, physiological attention weights, visual deep features, audio deep features, and physiological deep features according to a preset fusion strategy to obtain multi-dimensional fusion features, analyzing this feature to obtain basic sentiment recognition information of individual students; analyzing the basic sentiment recognition information through the constructed personalized sentiment model of each individual student to obtain corresponding sentiment state evaluation information; generating a personalized teaching adjustment strategy based on the constructed teaching knowledge graph and sentiment state evaluation information to dynamically adjust the actual teaching plan. This method can achieve dynamic adjustment and meet personalized teaching needs.

Description

Technical Field

[0001] The present disclosure relates to the field of intelligent education technology, and particularly to an adaptive teaching strategy adjustment method and a computer device based on sentiment analysis. Background Art

[0002] In the field of education, improving teaching effectiveness by timely adjusting teaching strategies has been an important issue that educators have long been concerned about; in recent years, with the development of artificial intelligence technology, intelligent education systems have been increasingly widely used in the teaching process.

[0003] Currently, common teaching strategy adjustment methods mainly rely on explicit assessment means such as classroom quizzes and exercises to judge students' learning effects. For example, evaluating students' mastery of knowledge points through classroom quizzes, or judging teaching effectiveness based on the completion of after-class assignments. These means focus on examining students' mastery of knowledge, while ignoring students' emotional experiences during the learning process.

[0004] More advanced solutions use computer vision technology to collect data such as students' expressions and postures in real time, and analyze students' language expressions in combination with speech recognition technology to evaluate students' learning states. The system analyzes and processes the collected data based on preset rules to provide simple teaching suggestions for teachers. However, these solutions only evaluate students' learning states from limited aspects such as expressions, postures, and language expressions, and fail to comprehensively and deeply understand students' emotional states. Different students have different personalities, learning styles, and emotional response patterns. Existing analysis methods often adopt unified standards and models, without fully considering these individual differences, and are unable to dynamically optimize adjustment strategies according to actual teaching effects, making it difficult to meet the needs of personalized teaching. For example, the same expression or posture may have different meanings for different students, but the system may make the same interpretation, resulting in misjudgment of students' emotional states. Summary of the Invention

[0005] In view of this, the embodiments of the present disclosure provide an adaptive teaching strategy adjustment method and a computer device based on sentiment analysis, which can solve problems existing in the prior art such as single evaluation method, failure to consider students' individual differences, inability to dynamically optimize according to actual teaching effects, and difficulty in meeting the needs of personalized teaching.

[0006] In a first aspect, an embodiment of the present disclosure provides a method for adjusting an adaptive teaching strategy based on sentiment analysis, including: preprocessing the collected multi-modal student sentiment data to obtain a target sentiment feature data stream; extracting visual deep features, audio deep features, and physiological deep features from the target sentiment feature data stream; processing the preset visual attention weight, audio attention weight, physiological attention weight, the visual deep features, the audio deep features, and the physiological deep features according to a preset fusion strategy to obtain multi-dimensional fusion features; analyzing the multi-dimensional fusion features using a preset multi-task learning framework to obtain basic sentiment recognition information of individual students; constructing a personalized sentiment model based on the historical data of each individual student; analyzing the basic sentiment recognition information based on the personalized sentiment model to obtain corresponding sentiment state evaluation information; generating a personalized teaching adjustment strategy based on the constructed teaching knowledge graph and the sentiment state evaluation information; and dynamically adjusting the actual teaching plan according to the teaching adjustment strategy.

[0007] In a second aspect, an embodiment of the present disclosure further provides a system for adjusting an adaptive teaching strategy based on sentiment analysis, including: a collection module for preprocessing the collected multi-modal student sentiment data to obtain a target sentiment feature data stream; a deep feature extraction module for extracting visual deep features, audio deep features, and physiological deep features from the target sentiment feature data stream; a multi-dimensional fusion feature acquisition module for processing the preset visual attention weight, audio attention weight, physiological attention weight, the visual deep features, the audio deep features, and the physiological deep features according to a preset fusion strategy to obtain multi-dimensional fusion features; an identification module for analyzing the multi-dimensional fusion features using a preset multi-task learning framework to obtain basic sentiment recognition information of individual students; a personalized sentiment model construction module for constructing a personalized sentiment model based on the historical data of each individual student; a sentiment state evaluation information acquisition module for analyzing the basic sentiment recognition information based on the personalized sentiment model to obtain corresponding sentiment state evaluation information; a teaching adjustment strategy generation module for generating a personalized teaching adjustment strategy based on the constructed teaching knowledge graph and the sentiment state evaluation information; and a dynamic adjustment module for dynamically adjusting the actual teaching plan according to the teaching adjustment strategy.

[0008] In a third aspect, an embodiment of the present disclosure further provides a computer device, adopting the following technical solution: The computer device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method for adjusting an adaptive teaching strategy based on sentiment analysis as described in any one of the above.

[0009] In a fourth aspect, embodiments of the present disclosure further provide a computer-readable storage medium storing computer instructions for causing a computer to execute the adaptive teaching strategy adjustment method based on sentiment analysis described in any one of the above.

[0010] In a fifth aspect, embodiments of the present disclosure further provide a computer program product including a computer program / instructions, which when executed by a processor implement the steps of the method described in any one of the above.

[0011] The adaptive teaching strategy adjustment method based on sentiment analysis disclosed in this application can more accurately identify the emotional state of students through multi-modal data fusion, deep feature extraction, and construction of a personalized emotion model, providing a reliable basis for teaching adjustment; generating personalized teaching adjustment strategies based on the emotional state assessment information of students and the teaching knowledge graph can meet the different needs of each student, improve the pertinence and effectiveness of teaching; timely adjusting the teaching plan to adapt to the emotional changes of students can improve students' learning enthusiasm and participation, and promote students' active learning; dynamically adjusting the teaching plan and continuously optimizing the teaching process can improve the teaching quality and efficiency, and maximize the teaching effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] To more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below.

[0013] Figure 1 It is a schematic flowchart of the adaptive teaching strategy adjustment method based on sentiment analysis provided by the embodiments of the present disclosure.

[0014] Figure 2 For Figure 1 it is a schematic flowchart of the method for extracting visual deep features, audio deep features, and physiological deep features from the target emotion feature data stream in

[0015] Figure 3 For Figure 1 it is a schematic flowchart of the method for obtaining multi-dimensional fusion features in

[0016] Figure 4 For Figure 1 it is a schematic flowchart of the method for obtaining the basic emotion recognition information of individual students in

[0017] Figure 5 For Figure 1 it is a schematic flowchart of the method for constructing a personalized emotion model in

[0018] Figure 6A structural schematic diagram of a computer device provided by an embodiment of the present disclosure. Detailed implementation manners

[0019] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0020] Referring to Figure 1 , the present application discloses an adaptive teaching strategy adjustment method based on sentiment analysis, including:

[0021] S100, preprocess the collected multi-modal student sentiment data to obtain a target sentiment feature data stream. In this embodiment, the multi-modal student sentiment data includes facial expression video stream data, body movement video stream data, classroom voice data, and physiological parameter data. Through this step, a normalized target sentiment feature data stream can be output.

[0022] S200, extract visual deep features, audio deep features, and physiological deep features from the target sentiment feature data stream. In this embodiment, the dimensions of the visual deep features, audio deep features, and physiological deep features are the same. Specifically, the visual deep features include facial features and pose features; the audio deep features include several time-frequency domain features; the physiological deep features include several physiological features. The deep features extracted through this step can more comprehensively and accurately reflect the internal structure and semantic information of the data, which helps to improve the accuracy of sentiment recognition. The extracted deep features have stronger generalization ability and can maintain good performance in different data sets and scenarios.

[0023] S300, process the preset visual attention weights, audio attention weights, physiological attention weights, visual deep features, audio deep features, and physiological deep features according to a preset fusion strategy to obtain multi-dimensional fusion features. By fusing features of different modalities, the complementary nature of each modality data can be fully utilized to improve the accuracy and reliability of sentiment recognition; the setting of attention weights can be adjusted according to the importance of different modalities to highlight the modality information that contributes more to sentiment recognition. The multi-dimensional fusion features can, to a certain extent, reduce the limitations of single-modal data and improve the robustness of the model.

[0024] S400, analyze the multi-dimensional fusion features using a preset multi-task learning framework to obtain the basic sentiment recognition information of individual students. Multi-task learning can share underlying features, reduce the number of model parameters, and improve learning efficiency; the association between different tasks can promote each other and improve the accuracy of sentiment recognition; the basic sentiment recognition information includes not only sentiment categories but also sentiment intensities, providing a richer basis for subsequent personalized teaching adjustment.

[0025] S500, construct a personalized emotion model based on the historical data of each individual student. The emotional responses and learning characteristics of each student are different. The personalized emotion model can better adapt to the individual differences of students and improve the accuracy of emotion recognition. Based on the personalized emotion model, teaching adjustment strategies that better meet the needs of each student can be provided to improve the teaching effect. As students learn and grow, the historical data is continuously updated, and the personalized emotion model can also be dynamically adjusted accordingly to better reflect the emotional changes of students.

[0026] S600, analyze the basic emotion recognition information based on the personalized emotion model to obtain the corresponding emotion state evaluation information. The personalized emotion model can combine the historical data and individual characteristics of students to conduct a more in-depth analysis of the basic emotion recognition information and provide more comprehensive and accurate emotion state evaluation information. By evaluating the emotion state, potential emotional problems of students, such as long-term negative emotion states, can be discovered and measures can be taken in a timely manner for intervention. The emotion state evaluation information can more accurately reflect the learning state and needs of students and provide a more reliable basis for subsequent teaching adjustment strategies.

[0027] S700, generate personalized teaching adjustment strategies based on the constructed teaching knowledge graph and emotion state evaluation information. The rich knowledge in the teaching field can provide a comprehensive and accurate basis for the generation of teaching adjustment strategies. Combining the emotion state evaluation information, the generated teaching adjustment strategies can be customized for each student's specific situation to improve the pertinence and effectiveness of teaching. The knowledge in the teaching knowledge graph can be continuously updated and expanded to support the innovation of teaching adjustment strategies and promote the improvement and optimization of teaching methods.

[0028] S800, dynamically adjust the actual teaching plan according to the teaching adjustment strategies. Dynamically adjusting the actual teaching plan can be adjusted in a timely manner according to the emotional state and learning needs of students to improve the adaptability and flexibility of teaching. By continuously optimizing the teaching plan, the learning enthusiasm and learning effect of students can be improved, thereby improving the teaching quality. The personalized teaching adjustment strategies can better meet the individual differences of students and promote the all-round development of students.

[0029] The adaptive teaching strategy adjustment method based on sentiment analysis disclosed in this application can more accurately identify the emotional states of students through multi-modal data fusion, deep feature extraction, and the construction of a personalized emotion model, providing a reliable basis for teaching adjustment; based on the emotional state assessment information of students and the teaching knowledge graph, personalized teaching adjustment strategies are generated, which can meet the different needs of each student, improve the pertinence and effectiveness of teaching; timely adjustment of the teaching plan to adapt to the emotional changes of students can improve students' learning enthusiasm and participation, and promote students' active learning; dynamically adjusting the teaching plan and continuously optimizing the teaching process can improve the teaching quality and efficiency, and maximize the teaching effect.

[0030] For the method of obtaining the target emotional feature data stream in S100, it specifically includes:

[0031] A100, collecting the facial expression video stream data and body movement video stream data of students through the arranged image acquisition device. Specifically, the facial expression and body movement video stream data of students are collected in real time through the high-definition cameras arranged in the classroom.

[0032] Furthermore, different types of cameras are selected according to the actual scenario and requirements; if it is in a classroom environment, a wide-angle high-definition network camera can be selected, which can cover a large area and ensure that the collected video has high clarity, facilitating subsequent analysis of students' facial expressions and body movements. The number and position of cameras are reasonably arranged according to the size and shape of the classroom. Generally speaking, for a small classroom, 2-3 cameras may be sufficient to cover the entire space; while for a large classroom or lecture hall, more cameras are needed. The cameras can be installed at the four corners of the classroom or at high positions in the front and rear of the classroom to ensure that the faces and body movements of each student can be clearly captured. Furthermore, a face detection algorithm can be used to extract the facial region images of each student, and the coordinate information of 68 key feature points can be obtained using facial feature point localization technology. In addition, the sitting postures, hand movements and other body movements of students in the video stream can be analyzed through a human pose estimation model.

[0033] A200, collecting classroom voice data through the arranged microphone array. Among them, the classroom voice data includes teacher voice and student voice.

[0034] A300, collecting the physiological parameter data of students through wearable devices. Specifically, the physiological parameter data such as students' heart rate and galvanic skin response are collected through wearable devices. Among them, the wearable devices can be smart bracelets, smart watches, etc.

[0035] Through the solution disclosed in steps A100 - A300, multi-modal data can be comprehensively obtained. The emotions of students can be expressed in various ways such as facial expressions, body movements, voices, and physiological reactions. Collecting facial expression video stream data, body movement video stream data, classroom voice data, and physiological parameter data can capture students' emotional information from multiple dimensions, providing a rich data basis for subsequent accurate emotional analysis. Different modalities of data are complementary in emotional expression. For example, facial expressions may directly reflect the current emotional state of students, while changes in physiological parameters may reveal potential emotional fluctuations. Combining multi-modal data can reduce the limitations brought by single-modal data and improve the accuracy and reliability of emotion recognition.

[0036] A400. Perform motion compensation and image enhancement processing on both the facial expression video stream data and the body movement video stream data to obtain target video data.

[0037] Specifically, when collecting facial expression and body movement video stream data, the movement of students may cause image instability, affecting the accuracy of subsequent feature extraction. Motion compensation can process the image sequence to eliminate image displacement and jitter caused by movement, making the facial expressions and body movements in the image clearer and more stable, facilitating the accurate extraction of relevant features. At the same time, motion compensation helps maintain the continuity between video frames, making the features of facial expressions and body movements more coherent in time, which is very important for analyzing the dynamic changes of students' emotions and can more accurately capture the development and transformation process of emotions. Further, appropriate video processing tools can be used to read the video stream data containing facial expressions and body movements into the processing system. For example, in the Python programming environment, the OpenCV library is often used to open video files. Then, perform a frame-by-frame reading operation on the imported video stream, decompose the continuous video stream into a series of independent image frames, and store these image frames for subsequent processing of each frame.

[0038] The motion compensation processing stage includes: 1) Feature point detection and tracking. Specifically, a suitable algorithm is selected from various feature point detection algorithms (such as SIFT, SURF, ORB, etc.) to detect feature points in the image frame. Taking the ORB algorithm as an example, it can detect feature points in the first frame image and calculate their descriptors. The optical flow method (such as the Lucas - Kanade optical flow method) is used to track the motion of feature points between adjacent image frames. By matching the descriptors of feature points in adjacent frames, corresponding feature point pairs are found, and then the motion information of the feature points is obtained. 2) Affine transformation and motion compensation. Specifically, according to the tracked feature point pairs, the affine transformation matrix between adjacent frames is calculated. This matrix describes the transformation relationship of the image frame in terms of translation, rotation, scaling, etc. Then, the previous frame image is transformed according to the calculated affine transformation matrix to achieve motion compensation, making the motion between adjacent frames smoother. The image enhancement processing stage includes: 1) Brightness and contrast adjustment. Specifically, the brightness and contrast of the image are adjusted by using the method of linear transformation. By setting appropriate brightness gain (alpha) and contrast offset (beta) values, the visual effect of the image becomes clearer and more distinct, highlighting the details of facial expressions and body movements. 2) Histogram equalization. Specifically, for grayscale images, the global histogram equalization technique can be used to enhance the overall contrast of the image. The gray - value distribution of the image is readjusted so that the brightness range of the image is wider and the details are more obvious. When global histogram equalization may cause over - enhancement in some regions of the image, the adaptive histogram equalization method is adopted. It performs histogram equalization in local regions of the image, which can better preserve the detail information of the image and avoid the problem of over - enhancement. 3) Sharpening processing. Specifically, the Laplacian sharpening method is used to enhance the edge information of the image; by defining a specific convolution kernel and performing a convolution operation on the image, the edges of the objects in the image are highlighted, making the contours of facial expressions and body movements clearer and making the image look sharper. A series of image frames after motion compensation and image enhancement processing are recombined into a complete video file according to the original video frame rate and resolution, using a suitable video coding format (such as the MP4 format). In this way, the processed target video data is obtained, in which facial expressions and body movements are clearer and the motion is smoother, facilitating subsequent analysis and application.

[0039] A500 performs noise reduction and sound source localization processing on classroom voice data, extracts effective voice segments using the voice activity detection algorithm, and obtains the target audio data.

[0040] There may be various background noises in the classroom environment, such as the conversations of other students, the noises outside the classroom, etc. These noises will interfere with the quality of the speech signal and affect the accuracy of sentiment analysis. Through noise reduction processing, the background noise can be effectively removed, making the speech signal more pure and highlighting the speech information of the students. The pure speech signal is more conducive to extracting accurate speech features, such as pitch, timbre, speech rate, etc. These speech features are closely related to the emotional state of the students. Through noise reduction processing, the accuracy of speech feature extraction can be improved, thus providing a more reliable basis for emotion recognition. In the classroom environment, multiple students may be speaking at the same time. Sound source localization processing can determine the source of each speech signal, that is, the identity of the speaker, which is very important for analyzing the emotional states of different students and can accurately correspond the speech information to the specific students. Through sound source localization, the speech signals of different speakers can be separated to avoid mutual interference of speech signals, which helps to more clearly analyze the speech features and emotional states of each student and improve the accuracy of emotion recognition. The classroom speech data may contain a large number of silent periods or meaningless noises. These invalid data will increase the computational complexity of subsequent processing. The voice activity detection algorithm can automatically identify the effective speech segments, remove the silent and noisy parts, reduce the processing volume of invalid data, and improve the processing efficiency; extracting the effective speech segments can make the subsequent analysis more focused on the key speech information of the students and avoid being interfered by invalid data, which helps to more accurately analyze the speech content and emotional states of the students and improve the accuracy of emotion recognition.

[0041] Furthermore, the method for obtaining target audio data may include: 1) Data preparation: Before processing classroom speech data, the speech data needs to be loaded into the processing system first. It is necessary to ensure that the format of the speech data (such as common WAV, MP3, etc.) can be supported by the processing tools used. If the data is in MP3 format, considering that many speech processing libraries support the WAV format more natively and stably, it may be necessary to convert it to the WAV format first. 2) Noise reduction processing: The core purpose of noise reduction is to remove the background noise in the classroom speech data, thereby improving the clarity of the speech. Two common noise reduction methods include spectral subtraction and Wiener filtering. Spectral subtraction is an effective and simple method for noise reduction in the frequency domain. It works based on the statistical characteristics of the noise. First, estimate the noise spectrum from the spectrum of the noisy speech, and then subtract this estimated noise spectrum from the noisy speech spectrum to achieve noise reduction. The specific operation is to first determine that the first few seconds of the audio are a pure noise segment, obtain this noise data, calculate its spectrum and find the average value to get the average noise spectrum. Then perform a short-time Fourier transform on the entire audio to obtain the amplitude spectrum and phase spectrum of the audio. Subtract the average noise spectrum from the audio amplitude spectrum, and then perform an inverse short-time Fourier transform in combination with the phase spectrum to finally obtain the noise-reduced audio. Wiener filtering is an optimal filtering method based on the minimum mean square error criterion. While suppressing noise, it can retain the information of the speech signal to the greatest extent. By calling relevant functions, Wiener filtering can be applied to the audio data to achieve noise reduction.

[0042] 3) Sound source localization processing; The role of sound source localization is to determine the source location of the sound in the classroom, which is very helpful for analyzing classroom interaction. The time difference of arrival (TDOA) based on a microphone array is a common sound source localization method, and its specific steps are as follows: Use an array composed of multiple microphones to collect classroom speech data and record the time information when each microphone receives the sound; Calculate the TDOA by comparing the time differences of the same sound signal received by different microphones. The cross-correlation function can be used to estimate this time difference; Based on the calculated TDOA and the geometric structure of the microphone array, use algorithms such as triangulation to calculate the location of the sound source. The specific calculation method will vary depending on the layout of the microphone array (such as linear array, circular array, etc.). 4) Voice activity detection (VAD); The function of voice activity detection is to identify the valid speech segments in the audio data and remove the silent and noisy segments. Two common VAD algorithms include energy-based VAD and machine learning-based VAD; The energy-based VAD algorithm determines the speech and silent parts by calculating the energy of the audio signal. First, the audio is framed, then the energy of each frame is calculated, and an energy threshold is set. The part with energy higher than the threshold is determined as speech, and the part lower than the threshold is determined as silence; For machine learning-based VAD, a pre-trained machine learning model (such as a model based on recurrent neural network, long short-term memory network, etc.) can be used for voice activity detection. There are some open-source libraries that provide pre-trained VAD models, and the audio can be subjected to voice activity detection by calling these models.

[0043] 5) Extract valid speech segments, that is, extract the valid speech segments in the audio data according to the results of voice activity detection; Taking the result of energy-based VAD as an example, when the start frame of speech is detected, it is marked until the end frame of speech is detected, and the audio segments during this period are extracted. Finally, all the extracted speech segments are spliced into a complete audio. 6) Save the target audio data, that is, save the extracted valid speech segments as an audio file for subsequent use. An appropriate audio format and saving tool can be selected to save the processed audio to a specified location.

[0044] A600, filter and denoise the physiological parameter data to obtain the target parameter data; The target video data, target audio data, and target parameter data associated with time series constitute the target emotional feature data stream.

[0045] The physiological parameter data collected by wearable devices may be interfered by various external factors, such as body movement, electromagnetic interference, etc. Filtering and denoising processing can remove these interference signals, making the physiological parameter data more smooth and accurate in reflecting the physiological state of students; accurate physiological parameter data is very important for analyzing the emotional state of students. Filtering and denoising processing can improve the accuracy of physiological feature extraction, such as the extraction of features like heart rate variability and galvanic skin response, thus providing a more reliable basis for emotion recognition.

[0046] The method for obtaining the target emotional feature data stream disclosed in A100 - A600 pre - processes multi - modal data, removes noise, interference and invalid information in the data, improves the quality and usability of the data, and provides a good data foundation for subsequent emotion analysis and feature extraction; various processing methods in the pre - processing process, such as motion compensation, image enhancement, noise reduction, sound source localization, etc., can highlight the key features in the data, making the extracted features more accurate and effective, which is beneficial to improving the accuracy and reliability of emotion recognition; removing invalid data and reducing interference information can reduce the computational amount and complexity of subsequent processing, improve processing efficiency, and enable the entire emotion analysis system to run more quickly and accurately; correlating the pre - processed target video data, target audio data and target parameter data according to time sequence can comprehensively consider the change relationship of different modal data in time, and more comprehensively and accurately analyze the emotional state of students and its dynamic change process. Further, time synchronization and data alignment can also be performed on the target emotional feature data stream, and the continuous data stream can be segmented into sample segments of a fixed time length by using the sliding window method; outliers can be removed through data cleaning.

[0047] Refer to Figure 2 , the method of "extracting visual deep features, audio deep features, and physiological deep features from the target emotional feature data stream" in S200 specifically includes:

[0048] S210, extract facial features from the target video data in the target emotional feature data stream by using a pre - trained VGGFace network.

[0049] Among them, VGGFace is a deep convolutional neural network specifically designed for face recognition tasks based on the VGG-16 architecture. VGG-16 is a very classic convolutional neural network structure with multiple convolutional layers and pooling layers, which can automatically learn different levels of feature representations from images. VGGFace is trained on a large-scale face dataset and can learn rich face feature patterns. Pre-training means that the model has been pre-trained on a large-scale dataset and has learned some general feature representations. In this embodiment, the VGGFace network is trained with face feature knowledge, and the VGGFace network that meets the requirements is used as the pre-trained VGGFace network. That is, the pre-trained VGGFace network has learned face feature knowledge and does not need to train the model from scratch.

[0050] The method for extracting facial features includes: inputting an image containing a face into the pre-trained VGGFace network. The network will perform a series of operations such as convolution and pooling on the image, and finally output a feature vector. This feature vector contains various information about the face, such as facial contours, facial feature shapes, textures, etc., and can be used for subsequent face recognition, expression analysis, etc.

[0051] S220, use ResNet50 to extract pose features from the target video data and target audio data in the target emotion feature data stream.

[0052] Among them, ResNet (Residual Network) is a deep convolutional neural network. It solves the problems of gradient disappearance and gradient explosion during the training process of deep neural networks by introducing residual blocks, enabling the network to be trained deeper. ResNet50 is a specific model in the ResNet series, which contains 50 layers of neural networks. The method for extracting pose features includes: when an image containing human poses is input, ResNet50 will perform feature extraction on the image. Through the convolutional layers and pooling layers of the network, ResNet50 can learn information such as the positions, angles, and relative relationships of different parts of the human body (such as arms, legs, torso, etc.), and finally output a feature vector representing the human pose. These pose features can be used for tasks such as human action recognition and pose estimation.

[0053] S230, improve 1D-CNN using a preset strategy, and extract several time-frequency domain features from the target emotion feature data stream based on the improved 1D-CNN. The time-frequency domain features include at least one of time-frequency domain, fundamental frequency, and formant.

[0054] Among them, the convolutional neural network (CNN) is a deep learning model designed specifically for processing data with grid structures (such as images and audio); the one-dimensional convolutional neural network (1D-CNN) is mainly used to process one-dimensional sequence data, and audio signals are typical one-dimensional time series data. In 1D-CNN, the convolutional layer performs convolutional operations on the input audio sequence by sliding the convolutional kernel, extracting local features; the convolutional kernel is like a small filter that moves continuously on the audio sequence, calculates the local area at each position, and obtains a new feature value; through the operations of multiple convolutional kernels, different types of local features can be extracted.

[0055] Improving the 1D-CNN using a preset strategy means improving the original 1D-CNN. The network structure can be adjusted, that is, increasing or decreasing the number of convolutional layers and pooling layers, changing the size and number of convolutional kernels, etc., to optimize the network's ability to extract audio features. New activation functions can also be introduced, that is, using activation functions more suitable for the characteristics of audio data to help the network better learn and express audio features; regularization methods can also be added, such as Dropout, L1 / L2 regularization, etc., to prevent the network from overfitting and improve the generalization ability of the model.

[0056] Based on the improved 1D-CNN, several time-frequency domain features are extracted from the target emotional feature data stream, including: inputting the audio signal into the improved 1D-CNN, and the network will automatically learn the time-frequency domain features in the audio signal, including time-frequency domain, fundamental frequency, formants, etc.; through operations such as convolution, pooling, and activation, the network will convert the input audio signal into a series of feature vectors, and these feature vectors contain the required acoustic feature information and can be used for subsequent tasks.

[0057] Among them, the time-frequency domain means that the audio signal contains not only information in the time dimension but also information in the frequency dimension. Time-frequency domain analysis is to analyze the audio signal in both the time and frequency dimensions to more comprehensively understand the characteristics of the audio. The fundamental frequency is the basic frequency of vocal cord vibration, which determines the pitch of the sound. In speech, different people have different fundamental frequencies when speaking, and the fundamental frequency of the same person also changes under different emotional states; by extracting the fundamental frequency feature, it can be used for tasks such as speech recognition and emotion analysis. The formant is the frequency region where the energy is relatively concentrated in the speech spectrum, and it is closely related to the timbre and pronunciation method of speech; different vowels and consonants have different formant patterns, so the formant feature can be used for tasks such as phoneme recognition and speech synthesis of speech.

[0058] S240, after initially processing the target parameter data in the target emotional feature data stream using the LSTM network, several physiological features are obtained.

[0059] Specifically, it includes: 1) Data input, that is, inputting the target emotional feature data stream into the LSTM network in chronological order, and the physiological data at each time step is used as an input sample of the network. 2) Feature learning: The LSTM network processes the input data through the gating mechanism, learns the temporal dependence relationship and feature patterns in the data, and the network updates the cell state and hidden state according to the current input and the hidden state at the previous moment. 3) Output result: After being processed through multiple time steps, the LSTM network can output the corresponding physiological features (presented in vector form). For example, for physiological signals (i.e., target parameter data), the long short-term memory network (LSTM) is used for feature extraction, that is, the electrocardiogram signal sequence is input into the LSTM model to obtain several physiological features, and the dimension of the feature vector is 64.

[0060] In the solution disclosed in S210 - S240, by adopting different networks and methods to extract visual deep features, audio deep features, and physiological deep features, the emotional state of students can be comprehensively described from multiple dimensions; different types of features are complementary. For example, facial features can reflect the direct expressions of students, gesture features can reflect body language, and time-frequency domain features and physiological features can provide additional information from the aspects of speech and physiology. Comprehensive utilization of these features can improve the accuracy and reliability of emotion recognition; these deep learning networks can automatically learn the deep features in the data, and these features are more expressive than traditional handcrafted features. They can capture the complex patterns and potential information in the data, thus more accurately reflecting the emotional state of students and providing more powerful support for subsequent emotion analysis and decision-making; different networks and methods can be adjusted and optimized according to different task requirements and data characteristics. This adaptability and flexibility enable the entire feature extraction system to better handle various complex situations and improve the performance and practicality of the system.

[0061] Refer to Figure 3 , for the method of "processing the preset visual attention weight, audio attention weight, physiological attention weight, visual deep feature, audio deep feature, and physiological deep feature according to the preset fusion strategy in S300 to obtain the multi-dimensional fusion feature", that is, the method for obtaining the multi-dimensional fusion feature, specifically includes:

[0062] S310, determine the visual attention weight, audio attention weight, and physiological attention weight. In this embodiment, each weight is preferably preset according to the actual teaching scenario, or the scaled dot-product attention mechanism can be used to calculate the attention weights of each modality feature through learnable query vectors. Specifically, presetting the weights according to the actual teaching scenario can fully consider the importance of each modality feature for sentiment analysis in different teaching scenarios. For example, in some classrooms dominated by teacher lectures, audio information may be more important, and at this time, the audio attention weight can be appropriately increased; while in classrooms with group discussions or practical activities, visual information (such as students' body movements and facial expressions) may be more crucial, and accordingly, the visual attention weight can be increased. This can make the subsequent fusion process more targeted and improve the accuracy of sentiment analysis.

[0063] S320, multiply the visual deep feature, audio deep feature, and physiological deep feature by the visual attention weight, audio attention weight, and physiological attention weight respectively to obtain the visual weighted feature, audio weighted feature, and physiological weighted feature. By multiplying the visual deep feature, audio deep feature, and physiological deep feature by the corresponding attention weights respectively, the features that are more important for sentiment analysis can be highlighted; the features with larger weights will have relatively larger values after multiplication and will receive more attention in subsequent processing; while the features with smaller weights will be weakened accordingly, which can make the model more focused on key information, reduce the interference of irrelevant information, and improve the accuracy of sentiment analysis.

[0064] S330, perform element-wise addition of the visual weighted feature, audio weighted feature, and physiological weighted feature with the visual deep feature, audio deep feature, and physiological deep feature respectively to obtain the visual fusion feature, audio fusion feature, and physiological fusion feature. Performing element-wise addition of the weighted feature and the deep feature can achieve the fusion of feature information. The weighted feature highlights the key information, while the deep feature itself contains rich original feature information; through the addition operation, both the integrity of the original features is retained and the key information is emphasized, making the fused feature have stronger expressive ability. For example, the visual fusion feature contains both the information of students' facial expressions and body movements reflected by the original visual deep feature, and through the weighted operation, the part more relevant to sentiment analysis is highlighted, thus more comprehensively and accurately describing the visual sentiment information of students. This addition-based fusion method avoids the problem of information loss that may be caused by simple feature selection or replacement. The features of each modality are retained and strengthened in the fusion process, and different features complement each other, providing richer and more comprehensive information for subsequent sentiment analysis.

[0065] S340, splice the visual fusion feature, the audio fusion feature, and the physiological fusion feature in the feature dimension to obtain a multi-dimensional fusion feature. Splicing the visual fusion feature, the audio fusion feature, and the physiological fusion feature in the feature dimension can form a multi-dimensional fusion feature vector. This vector integrates the information of the visual, audio, and physiological modalities, comprehensively describes the emotional state of the student from multiple perspectives. The features of different modalities are complementary in emotional expression, and this complementarity can be fully utilized through splicing to improve the accuracy and reliability of emotion analysis. The spliced multi-dimensional fusion feature has a unified data format, facilitating subsequent processing such as classification and clustering. The subsequent model can directly use this fusion feature vector as input for emotion recognition or other related tasks, improving the processing efficiency and performance of the entire system.

[0066] The solutions disclosed in S310 - S340 can fully utilize the complementarity of multi-modal data, highlight key information, comprehensively and accurately describe the emotional state of the student, and thus significantly improve the accuracy of emotion analysis by reasonably determining the attention weights, weighting and fusing the features, and finally splicing the features. Whether presetting the weights or using the scaled dot-product attention mechanism to calculate the weights enables the model to adapt to different teaching scenarios and data characteristics. This flexibility allows the model to work effectively in various complex situations, improving the generalization ability and practicality of the system. The entire fusion process effectively integrates the feature information of each modality to form a more expressive multi-dimensional fusion feature. This optimized feature representation can better reflect the essential features of the student's emotion, providing better input for the subsequent emotion analysis model and facilitating the improvement of the model's performance.

[0067] Furthermore, the specific solution for "using the scaled dot-product attention mechanism to calculate the attention weights of each modality feature through a learnable query vector" is as follows: For the visual deep feature, the audio deep feature, and the physiological deep feature. The visual deep feature can be represented by the matrix where is the length of the visual feature sequence, that is, the number of visual feature vectors, and is the dimension of each visual feature vector; the audio deep feature is represented by the matrix where is the length of the audio feature sequence, and is the dimension of each audio feature vector; the physiological deep feature is represented by the matrix where is the length of the physiological feature sequence, and is the dimension of each physiological feature vector. At the same time, we introduce a learnable query vector , and we will use it later to calculate the attention weights of these three types of features: visual, audio, and physiological.

[0068] The detailed calculation scheme for the attention weights of each modal feature is as follows: Step 1: Initialize the learnable query vector. First, we need to initialize the learnable query vector . Generally, random initialization is adopted, and the Gaussian distribution is commonly used to generate the initial values. The characteristics of the Gaussian distribution make the query vector have randomness and diversity in the initial state, providing a good starting point for the subsequent learning process and allowing the model to have the opportunity to explore different feature matching methods.

[0069] Step 2: Calculate the similarity between each modal feature and the query vector respectively. Visual feature similarity calculation: For each feature vector in the visual deep feature matrix (j = 1, 2, …, ), we calculate its similarity with the query vector through the dot product operation. The dot product operation can measure the directional similarity between two vectors, and the formula is . The obtained similarity score reflects the matching degree between the th visual feature vector and the query vector. The higher the score, the more similar they are. Audio feature similarity calculation: Similarly, for each feature vector in the audio deep feature matrix (j = 1, 2, …, ), the dot product formula is used to calculate its similarity with the query vector . This similarity score reflects the matching situation between the audio feature vector and the query vector. Physiological feature similarity calculation: For each feature vector in the physiological deep feature matrix (j = 1, 2, …, ), its similarity with the query vector is calculated through the dot product , and this score reflects the similarity degree between the physiological feature vector and the query vector.

[0070] Step 3: Scale the similarity scores of each modality respectively. Visual similarity scaling: To avoid the dot product result of the visual feature and the query vector being too large, resulting in unstable gradients, we scale the visual similarity score. The scaling factor is selected as the square root of the feature dimension , and the scaling formula is . This can control the visual similarity score within a suitable range and make the subsequent calculations more stable. Audio similarity scaling: For the similarity score of the audio feature, the same scaling factor is also used for scaling, and the formula is The scaling operation ensures that the audio similarity scores are within the appropriate range, which helps to improve the stability of model training. Physiological similarity scaling: The similarity scores of physiological features are also scaled in the same way, that is, By scaling, the physiological similarity score can better participate in the subsequent calculation process.

[0071] Step 4: Calculate the attention weights of each modality separately. Visual attention weight calculation: Use the softmax function to convert the scaled visual similarity scores into visual attention weights. The softmax function maps the scores to a probability distribution so that the sum of all weights is 1. For visual features, The attention weight calculation formula of feature vectors is The higher the score of the visual feature vector, the greater the corresponding attention weight, which means that the feature will be given a higher importance in subsequent processing. Audio attention weight calculation: For audio features, the softmax function is also used to convert the scaled audio similarity score into audio attention weight. The attention weight calculation formula of an audio feature vector is: This enables the model to assign different importance to different audio features based on how well they match the query vector. Physiological attention weight calculation: For physiological features, the scaled physiological similarity scores are converted to physiological attention weights through a softmax function. The calculation formula of the attention weight of a physiological feature vector is In this way, the model can assign different weights to different physiological features according to their similarity with the query vector.

[0072] Through the above four steps, the visual attention weight, audio attention weight and physiological attention weight can be determined respectively. First, the learnable query vector is initialized, and then the similarity between each modal feature and the query vector is calculated respectively. Then the similarity score is scaled, and finally the scaled similarity score is converted into attention weight using the softmax function. These attention weights can help the model pay more attention to features with higher similarity to the query vector when processing multimodal data, thereby improving the performance of the model and the efficiency of using different modal information.

[0073] Further, before multiplying the visual deep feature, the audio deep feature, and the physiological deep feature by the visual attention weight, the audio attention weight, and the physiological attention weight respectively, it may further include: S201, adaptively and dynamically adjusting the visual attention weight, the audio attention weight, and the physiological attention weight through the determined optimization objective function to obtain the optimized visual attention weight, audio attention weight, and physiological attention weight; S202, multiplying the visual deep feature, the audio deep feature, and the physiological deep feature by the optimized visual attention weight, audio attention weight, and physiological attention weight respectively to obtain the visual weighted feature, audio weighted feature, and physiological weighted feature.

[0074] Among them, adaptively and dynamically adjusting the visual attention weight, the audio attention weight, and the physiological attention weight through the determined optimization objective function to obtain the visual optimized attention weight, audio optimized attention weight, and physiological optimized attention weight includes:

[0075] B100, determining the optimization objective function. The optimization objective function is: ; . Among them, is a hyperparameter used to balance the feature quality and inter-modal complementarity, is the visual initial weight, is the audio initial weight, is the physiological initial weight.

[0076] Further, ; ; . Among them, is the mean of the visual deep feature, is the standard deviation of the visual deep feature, is the mean of the audio deep feature, is the standard deviation of the audio deep feature, is the mean of the physiological deep feature, is the standard deviation of the physiological deep feature.

[0077] B200, randomly setting the visual initial weight, the audio initial weight, and the physiological initial weight; using the stochastic gradient descent method to optimize the optimization objective function, and when the optimization objective function converges or reaches the preset number of iterations, output the optimized visual attention weight, audio attention weight, and physiological attention weight.

[0078] Specifically, calculate the gradient of the objective function with respect to the attention parameter . Update the attention parameter according to the gradient, and the update formula is: . Among them, is the learning rate, for example = 0.001. Repeat the above steps until the objective function converges or the maximum number of iterations is reached. By comprehensively considering the signal-to-noise ratio, temporal continuity, and inter-modal complementarity of features, the quality and value of features can be more comprehensively evaluated, thereby adaptively adjusting feature weights and improving the utilization efficiency of features. By making full use of the advantages of each modality data and combining the dynamic change information of features, the accuracy, stability, and generalization ability of the model can be improved. This solution can be applied to different types of multi-modal data, and can adapt to different application scenarios and requirements by adjusting the parameters of the objective function and optimization algorithm.

[0079] Referring to Figure 4 , for the method of obtaining the basic emotion recognition information of individual students in S400, it specifically includes:

[0080] S410, preset a multi-task learning framework; the multi-task learning framework includes a shared feature encoding layer, and an emotion classification task layer and an intensity prediction task layer both connected to the shared feature encoding layer.

[0081] In this embodiment, the shared feature encoding layer includes a multi-layer perceptron, the emotion classification task layer includes a softmax classifier, and the intensity prediction task layer includes a regressor. In this step, the multi-task learning framework integrates two related tasks of emotion classification and intensity prediction; in traditional single-task learning, each task needs to train a model independently, which consumes a large amount of computing resources and time, while multi-task learning allows the model to learn multiple tasks simultaneously in one training process and share some learning results, thus significantly improving the learning efficiency. For example, the shared feature encoding layer can extract useful features for both the emotion classification and intensity prediction tasks at the same time, avoiding repeated calculations and learning processes.

[0082] There is an inherent association between the two tasks of emotion classification and intensity prediction. For example, different emotion categories are often accompanied by different degrees of emotion intensity. The multi-task learning framework can capture this association, enabling the model to draw on information from each other during the learning process. For example, the information obtained during emotion classification can assist intensity prediction, and vice versa, thereby improving the model's overall understanding ability of students' emotional states.

[0083] S420, input the multi-dimensional fusion features into the shared feature encoding layer, and extract emotion general features through a multi-layer perceptron.

[0084] Specifically, the input multi-dimensional fusion features are first fed into the shared feature encoding layer. The multi-layer perceptron of the shared feature encoding layer performs non-linear transformations on these features. Through the stacking of multiple neuron layers, the general patterns in the features are gradually extracted. In each layer, the neurons receive the output of the previous layer as input, and after weighted summation and activation function processing, they output to the next layer. In this way, the multi-layer perceptron can learn the complex representation of the features, remove the noise and redundant information in the features, and extract the general features that are useful for both emotion classification and intensity prediction. In this embodiment, the shared feature encoding layer adopts a multi-layer perceptron, which has a powerful non-linear mapping ability and can map the multi-dimensional fusion features to a more expressive feature space, extracting the general features that are useful for both emotion classification and intensity prediction. The multi-layer structure of the multi-layer perceptron can learn the feature representations at different levels, gradually abstracting the high-level emotion-related features from the low-level feature combinations.

[0085] The multi-dimensional fusion features may contain a large amount of information, and some of this information may be redundant or not directly helpful for emotion analysis. The multi-layer perceptron can abstract and compress these features, extracting the core information related to emotions, removing the irrelevant noise and redundant features. This can reduce the computational amount of subsequent tasks, improve the processing efficiency and generalization ability of the model. The multi-layer perceptron can learn the general feature representations applicable to both emotion classification and intensity prediction. These general features can reflect the essential features of emotions and are not limited to a specific task. For example, some features such as facial expressions and speech intonations may have an impact on both the emotion category and intensity at the same time. The multi-layer perceptron can effectively extract these features, providing strong support for subsequent tasks.

[0086] S430, input the emotion general features into the emotion classification task layer, and perform emotion classification through a softmax classifier, outputting the probability distribution of the emotion categories.

[0087] Among them, the softmax classifier processes the input features. It first performs a linear transformation on the features to obtain the scores for each emotion category. Then, through the softmax function, these scores are converted into a probability distribution. The formula of the softmax function is: ; where represents the probability of the th emotion category, is the The score of each emotion category, where K is the total number of emotion categories. The final output is the probability distribution of each emotion category, and the category with the highest probability is the predicted emotion category. In this embodiment, the emotion classification task layer uses a softmax classifier, which can convert the output of the model into the probability distribution of each emotion category; the property of the softmax function makes the sum of the probabilities of all categories equal to 1, which conforms to the basic definition of probability and is convenient for intuitively judging the likelihood of a student being in each emotion category. The probability distribution of the emotion categories output by the softmax classifier can quantify the likelihood of a student being in each emotion category, making the result of emotion recognition more intuitive and accurate. Teachers or relevant personnel can judge the most likely emotion state of a student and the relative likelihood of various emotion states based on these probability values. For example, when the emotion classification result of a certain student shows that the probability of "happy" is 0.8, the probability of "angry" is 0.1, and the probability of "sad" is 0.1, it can be clearly known that the student is probably in a happy emotion state. In actual sentiment analysis, there are usually multiple emotion categories, such as happy, angry, sad, surprised, etc. The softmax classifier can handle this multi-classification problem well. By calculating the probability of each category, a reasonable probability value is assigned to each emotion category, thus achieving accurate emotion classification.

[0088] S440, input the general emotion features into the intensity prediction task layer, and perform emotion intensity prediction through a regressor, and output the corresponding intensity prediction value.

[0089] Among them, the probability distribution of the emotion categories and the corresponding intensity prediction values constitute the basic emotion recognition information for the corresponding student individual. Specifically, the regressor processes the input features and converts the features into a continuous numerical value through a linear or non-linear mapping relationship. This numerical value is the predicted emotion intensity value, and finally the regressor directly outputs the predicted emotion intensity value. Among them, common regressors can be linear regression models, neural network regression models, etc. In this embodiment, the intensity prediction task layer uses a regressor, which can predict the intensity value of the emotion according to the input general emotion features; the regression model can learn the continuous mapping relationship between the features and the emotion intensity, so as to more accurately output the predicted value of the emotion intensity.

[0090] In this application, in addition to knowing the emotional category of students, it is also very important to understand the intensity of emotions. The intensity of emotions can reflect the intensity of students' emotions, which is of great significance for judging students' emotional states and taking corresponding intervention measures. For example, for the emotion of "anger", when the intensity is weak, simple comfort may be sufficient, while when the intensity is strong, more timely and effective intervention measures may be required; the intensity prediction value output by the regressor can provide more comprehensive information for emotional analysis, helping teachers or relevant personnel make more accurate decisions. The intensity of emotions is a dynamically changing indicator, and the regressor can predict the change of the intensity of emotions in real time according to the input features, which enables the model to track the dynamic change process of students' emotional states, timely detect the fluctuations of students' emotions, and provide a basis for personalized teaching and psychological counseling.

[0091] Referring to Figure 5 , for the construction method of the personalized emotion model in S500, it includes:

[0092] S510, obtaining the historical data of each student individual within a preset period; the historical data includes historical emotion data, personal characteristics, and learning styles, and the historical emotion data includes historical emotion categories and corresponding intensity information; the personal characteristics include age, gender, and personality traits.

[0093] Among them, the learning style includes question-solving habits and new knowledge learning habits.

[0094] In this step, the historical data covers multiple aspects such as historical emotion data, personal characteristics, and learning styles. The historical emotion data can directly reflect the past emotional states of students, including emotion categories and intensities, allowing the model to understand the changing rules of students' emotions; age, gender, and personality traits in personal characteristics will affect students' emotional expression methods and emotional responses to different situations; learning styles, such as question-solving habits and new knowledge learning habits, are closely related to students' emotional experiences during the learning process; integrating these multi-dimensional data can more comprehensively and meticulously depict the emotional characteristics of each student, providing a rich information basis for constructing a personalized emotion model. There are significant differences among different students in terms of emotions, personal characteristics, and learning styles. By collecting the historical data of each student, these individual differences can be captured. For example, extroverted students may express positive emotions more directly, while introverted students may express emotions more implicitly; students with different learning styles will also have different emotional responses when facing learning difficulties. This difference information is crucial for constructing a personalized emotion model that can accurately reflect the unique emotional patterns of each student.

[0095] S520, training the initial model using a general emotion dataset to obtain the target emotion model; the initial model includes a convolutional neural network model or a recurrent neural network model.

[0096] Among them, the general sentiment dataset is a collection of a large amount of text or speech data with sentiment labels. These data are widely collected from different sources, with a certain degree of universality and representativeness, and can be used to train and evaluate sentiment analysis models. Using such a dataset to train an initial model (such as a convolutional neural network model or a recurrent neural network model) allows the model to quickly learn general sentiment patterns and features. These general knowledge are the commonalities of many individual sentiment expressions and play an important guiding role in subsequent processing of each student's personalized data. For example, the model can learn the general associations between certain facial expressions, speech intonations and specific emotions. Pre-training on the general dataset can make the parameters of the model converge to a relatively stable state, reducing the impact of random initialization during subsequent personalized training. This helps to improve the training efficiency and stability of the model and avoid overfitting or unstable training during the personalized training process.

[0097] S530, freeze the parameters of the first F1 layers of the target sentiment model, use the historical data as the training set, train the parameters of the latter F2 layers of the target sentiment model, update the parameters of the latter F2 layers through the backpropagation algorithm, and record the trained target sentiment model as the personalized sentiment model.

[0098] Among them, F1 + F2 = F0, 0 < F2 < F0 / 2, and F0 is the total number of neural network layers of the target sentiment model.

[0099] In this step, freezing the parameters of the first F1 layers of the target sentiment model means that the general sentiment patterns and features learned by these layers are retained. The first F1 layers usually extract some underlying and universal features, such as the basic contours of facial expressions, the basic rhythms of speech, etc. These general features are helpful for understanding various sentiment expressions. By freezing the parameters of these layers, it can be ensured that the model will not lose these important general knowledge during the personalized training process.

[0100] Training the parameters of the post-F2 layer of the target emotion model enables personalized adjustment of the model based on the historical data of each student; the post-F2 layer is usually responsible for processing more advanced and specific features and decisions. Since there are differences in the emotional patterns of different students, by using the historical data of each student to train the post-F2 layer, the model can learn the unique emotional characteristics and patterns of that student, thus achieving personalized adaptation for each student. For example, the model can adjust the sensitivity to the specific emotional expressions of a certain student according to the student's historical data. Only training the parameters of the post-F2 layer and satisfying 0 < F2 < F0 / 2 (where F0 is the total number of neural network layers of the target emotion model) greatly reduces the number of parameters to be trained. This not only reduces the computational cost and time overhead of training but also reduces the risk of overfitting. Because the historical data volume of each student is relatively limited, if all the parameters of the entire model are trained, it is easy to cause the model to overfit these limited data. Only training some parameters can enable the model to maintain a certain generalization ability while learning personalized features.

[0101] In the solution disclosed in S510 - S530, by comprehensively considering the multi-dimensional historical data of each student and combining general emotional knowledge for personalized training, the personalized emotion model can more accurately identify the emotional state of each student. It utilizes both general emotional patterns and fully considers the individual differences of students, thereby improving the accuracy and reliability of emotion recognition; the personalized emotion model can adapt to the unique emotional expressions and change rules of each student; in practical applications, different students may have different emotional reactions when facing the same learning situation. The personalized emotion model can make targeted adjustments according to the historical data of each student, better meet the needs of different students, and improve the adaptability and practicality of the model; accurate personalized emotion recognition can provide more valuable information for educators to help them understand the emotional state and needs of each student. Teachers can adjust teaching strategies in a timely manner according to the emotional changes of students, provide personalized learning support and psychological counseling, and promote the learning and development of students.

[0102] For the method of generating personalized teaching adjustment strategies in S700, it includes:

[0103] C100, constructing a hierarchical teaching knowledge graph based on educational domain expert knowledge and curriculum standards; the teaching knowledge graph includes the knowledge point level, the teaching method level, and the learning activity template level.

[0104] At the knowledge point level, the ontology modeling method is adopted to define the core concepts and their relationships, and clarify the prerequisite dependencies, difficulty levels, and importance degrees of knowledge points; at the teaching method level, a method library containing various teaching methods is established, and the applicable scenarios, expected effects, and precautions of each method are marked; a learning activity template library is constructed to define typical activity forms and activity parameters, and the typical activity forms include question scenarios, group discussion scenarios, and practical operation scenarios. In this embodiment, the teaching knowledge graph covers the knowledge point level, the teaching method level, and the learning activity template level, and presents the knowledge in the education field in a hierarchical manner; at the knowledge point level, the definitions, associations, and hierarchical relationships of each knowledge point in the course are clarified, enabling teachers and students to clearly understand the overall framework and context of the knowledge. For example, in a mathematics course, the connections between different knowledge sections such as algebra and geometry and the sub-knowledge points within them can be clearly shown.

[0105] The teaching method level and the learning activity template level provide diverse choices and references for teaching; the teaching method level lists various teaching methods applicable to different knowledge points and teaching objectives, such as the lecture method, the discussion method, the inquiry method, etc.; the learning activity template level contains specific learning activity designs, such as group discussion activities, experimental operation activities, etc. This multi-dimensional knowledge graph provides comprehensive guidance for the teaching process, helping teachers flexibly select teaching methods and design learning activities according to the actual situation.

[0106] C200, according to the teaching knowledge graph and the emotional state assessment information, determines the knowledge mastery degree and classroom progress information of the student group for different knowledge points.

[0107] According to the knowledge mastery degree, classroom progress information, and emotional state assessment information, determine the state space in the reinforcement learning algorithm; the state space includes the state vector s, , where represents the mastery degree of the th knowledge point, represents the classroom progress, represents the emotional state. Combining the teaching knowledge graph and the emotional state assessment information can accurately determine the knowledge mastery degree and classroom progress information of the student group for different knowledge points; the knowledge mastery degree reflects the students' understanding and application abilities of each knowledge point, and the classroom progress information reflects the progress of teaching; at the same time, the emotional state assessment information can reflect the students' emotional experiences during the learning process, such as whether they are active and whether they are confused, etc.; by integrating these information, teachers can comprehensively understand the students' learning status and discover the problems and difficulties existing in the students' learning process.

[0108] Determine the knowledge mastery level, classroom progress information, and emotional state assessment information as the state space in the reinforcement learning algorithm, providing a scientific basis for subsequent teaching decisions; the reinforcement learning algorithm can learn the optimal teaching strategy based on this state information to improve teaching effectiveness. For example, when a student has a low mastery level of a certain knowledge point and is in a negative mood, the reinforcement learning algorithm can recommend more suitable teaching methods and activities based on this state information to help the student improve their learning interest and comprehension ability.

[0109] C300, screen out teaching methods, corresponding activity arrangements, and time allocation information suitable for the current teaching situation in the teaching knowledge graph according to the knowledge mastery level and classroom progress information.

[0110] Screen out teaching methods, corresponding activity arrangements, and time allocation information suitable for the current teaching situation in the teaching knowledge graph according to the knowledge mastery level and classroom progress information, which makes teaching decisions more targeted and enables the selection of the most suitable teaching resources based on the actual learning situation and teaching progress of students. For example, if a student has difficulty understanding an abstract knowledge point, the teacher can screen out relevant activity arrangements using visual teaching methods from the knowledge graph and allocate time reasonably for teaching to help the student better understand the knowledge point.

[0111] C400, generate personalized teaching adjustment strategies based on the teaching methods, corresponding activity arrangements, and time allocation information. The personalized teaching adjustment strategies generated based on the screened teaching methods, activity arrangements, and time allocation information can fully consider the unique needs and characteristics of each student or group of students; different students have differences in knowledge mastery level, learning style, and emotional state, and the personalized teaching adjustment strategies can be customized according to these differences to provide the most suitable learning path and support for each student, thereby improving the learning effect and satisfaction of students. As the teaching process progresses, the knowledge mastery level, classroom progress, and emotional state of students will constantly change, and the personalized teaching adjustment strategies generated based on the teaching knowledge graph and emotional state assessment information can be updated and optimized in real time, adjusting the teaching methods and activity arrangements in a timely manner according to the new state information to adapt to the dynamic changes in the teaching situation and ensure that teaching always remains efficient and targeted.

[0112] In the solution disclosed in C100 - C400, by constructing a teaching knowledge graph and comprehensively considering the emotional state assessment information, accurate learning situation analysis and targeted teaching decisions can be achieved, thereby improving the quality and effect of teaching; teachers can select the most suitable teaching methods and activities according to the actual situation of students to help students better master knowledge and improve learning ability; the personalized teaching adjustment strategy can meet the unique needs of each student, give full play to the advantages of students, make up for the deficiencies of students, and promote the personalized development of students. Students can improve their learning interest and enthusiasm and better achieve their learning goals when studying in a learning environment suitable for themselves; the teaching decision-making process based on the teaching knowledge graph and reinforcement learning algorithm is more scientific and efficient. Teachers can quickly and accurately screen out suitable teaching resources, reasonably arrange teaching time and activities, and avoid the process of blindly trying different teaching methods in traditional teaching, thus improving teaching efficiency.

[0113] In another embodiment, for the method of generating the personalized teaching adjustment strategy in S700, it includes:

[0114] D100, construct a hierarchical teaching knowledge graph according to the knowledge of education domain experts and curriculum standards; the teaching knowledge graph includes the knowledge point level, teaching method level, and learning activity template level. At the knowledge point level, the ontology modeling method can be used to define the core concepts and their relationships, and they are represented in the form of RDF triples (Entity1, Relation, Entity2), such as (Concept A, Prerequisite Dependency, Concept B). At the same time, attributes such as prerequisite dependencies, difficulty levels, and importance degrees are marked for the knowledge points. Further, all knowledge points can be extracted from the curriculum standards and expert knowledge, classified and numbered. For example, for the mathematics curriculum, the knowledge points can be divided into categories such as algebra, geometry, and statistics; determine the relationships between knowledge points, such as sequence, inclusion relationship, causal relationship, etc. A directed graph can be used to represent these relationships, with nodes representing knowledge points and edges representing relationships.

[0115] At the teaching method level, a method library containing various teaching methods can be established, covering various teaching methods such as the lecture method, discussion method, case teaching method, project-based learning, etc. Each method is clearly marked with its applicable scenarios, expected effects, and precautions, that is, each teaching method can be classified and numbered, analyze the teaching methods suitable for each knowledge point, and establish a mapping relationship between knowledge points and teaching methods. For example, knowledge points with strong theoreticality may be suitable for the lecture method, while knowledge points with strong practicality may be suitable for the project-based learning method.

[0116] Construct a learning activity template library, a learning activity template library can be constructed, including typical activity forms such as questioning, group discussion, practical operation, etc., and define activity parameters such as duration and number of participants.

[0117] In this step, it covers three levels: knowledge points, teaching methods, and learning activity templates, providing a systematic and comprehensive framework for teaching; teachers can clearly understand the structure of the knowledge system, clarify the relationships between various knowledge points, and at the same time obtain a variety of teaching methods and learning activity templates to choose from, which helps to formulate a scientific and reasonable teaching plan; constructed based on the knowledge of education domain experts and curriculum standards, it ensures the standardization and authority of teaching content, which enables teaching activities to closely follow the curriculum standards, ensures that the knowledge learned by students meets educational requirements, and improves the consistency of teaching quality.

[0118] D200 determines the knowledge mastery level and classroom progress information of the student group for different knowledge points according to the teaching knowledge graph and emotional state assessment information. By combining the teaching knowledge graph and emotional state assessment information, it can accurately understand the mastery situation of the student group for different knowledge points and the actual progress of the classroom. Teachers can thereby discover the weak links in students' learning, timely adjust the teaching focus and rhythm, avoid the teaching content being too difficult or too easy, and improve the pertinence of teaching; the knowledge mastery level and classroom progress information are important bases for subsequently determining the state space of the reinforcement learning algorithm, screening teaching resources, and generating teaching strategies, providing a clear direction for teaching adjustment.

[0119] D300 determines the state space in the reinforcement learning algorithm according to the knowledge mastery level, classroom progress information, and emotional state assessment information. According to the definition of the state space, the state vector s, , where represents the mastery level of the th knowledge point, represents the classroom progress, which can be expressed by the number of completed knowledge points or the percentage of learning activity progress, represents the emotional state, which can be expressed by the emotional score. Quantifying the knowledge mastery level, classroom progress, and emotional state using a state vector enables the reinforcement learning algorithm to process this information. This quantification method helps the algorithm to more accurately understand the current teaching state and provides accurate data support for subsequent decisions.

[0120] Specifically, all possible state combinations can be used as the state space. For example, if there are n knowledge points, each knowledge point has m possible values for the mastery level, the classroom progress has p max possible values, and the emotional state has e max possible values, then the size of the state space is m n ×p max ×e max. The state space comprehensively considers the factors of knowledge, progress, and emotion, fully reflects the actual situation in the teaching process, can capture the dynamic changes in students' learning states, and makes the teaching adjustment strategy more in line with the actual teaching needs.

[0121] D400, according to the knowledge mastery level and classroom progress information, screen out teaching methods, corresponding activity arrangements, and time allocation information suitable for the current teaching situation in the teaching knowledge graph.

[0122] Screen out teaching methods, activity arrangements, and time allocation information in the teaching knowledge graph according to the knowledge mastery level and classroom progress information to ensure that the selected teaching resources match the current teaching situation. This can improve the utilization efficiency of teaching resources, avoid waste of resources, and make teaching activities more efficient. The teaching knowledge graph contains rich teaching methods and learning activity templates. Teachers can select the most suitable resources according to different teaching states to meet the learning needs of different students and promote the personalized development of students. Specifically, according to the students' knowledge mastery level and classroom progress, search for teaching methods suitable for the current knowledge points and teaching stages in the teaching knowledge graph. For example, if students have a low mastery level of a certain knowledge point, more detailed and basic teaching methods such as the lecture method may need to be selected. According to the selected teaching method, search for the corresponding learning activity template in the teaching knowledge graph. For example, if the discussion method is selected, the corresponding learning activity template may be the classroom discussion activity template. According to the teaching method and activity arrangement, combined with the classroom progress and teaching objectives, determine the time allocation for each teaching activity. For example, for a 45-minute class, if the lecture method and the classroom discussion method are selected, the lecture method can be allocated 25 minutes and the classroom discussion method can be allocated 20 minutes.

[0123] D500, according to the teaching method, corresponding activity arrangement, and time allocation information, determine the action space in the reinforcement learning algorithm. Among them, each action can be represented as a triple , where represents the teaching method, represents the activity arrangement, represents the time allocation. Represent the teaching method, activity arrangement, and time allocation information as a triple as the action space. For example, if there are M teaching methods, L activity arrangements, and T time allocation schemes, then the size of the action space is M×L×T. This enables the reinforcement learning algorithm to clearly identify the available teaching decision options. The algorithm can search and select in this action space to find the optimal teaching strategy; the clear representation of the action space enables the algorithm to more conveniently evaluate and compare different teaching combinations, improving the decision-making efficiency and accuracy of the algorithm.

[0124] D600, configure the reward function. Specifically, the reward function is as follows: ; . Among them, is the improvement in the degree of knowledge mastery after performing the action , is the improvement degree of the emotional state, is the change in classroom participation, , 、 are the corresponding weight coefficients. In this step, the configured reward function comprehensively considers the improvement in the degree of knowledge mastery, the improvement of the emotional state, and the change in classroom participation, and comprehensively measures the effect of teaching activities. This prompts the algorithm to not only focus on students' knowledge learning but also pay attention to students' emotional experience and classroom participation when selecting teaching actions, cultivating students' positive learning attitudes and good learning habits. By setting different weight coefficients, the importance of each factor in the reward function can be flexibly adjusted according to teaching objectives and actual needs. For example, if more attention is paid to students' emotional experience, the weight coefficient of the improvement degree of the emotional state can be appropriately increased, thereby guiding the algorithm to select teaching strategies that are more conducive to students' emotional development.

[0125] D700, train the determined reinforcement learning algorithm based on the reward function, and record the trained reinforcement learning algorithm as the target learning algorithm. In this embodiment, the reinforcement learning algorithm is a double Q-learning network architecture including an evaluation network and a target network. Among them, the evaluation network is used to predict the action value, and the target network is used to generate the target Q value. The value update formula is: ; among them, represents the action value estimation of taking action a in state s, is the learning rate, is the immediate reward, is the discount factor, is after performing the action , when the agent transfers to the new state , the maximum value among the action value estimations corresponding to all possible actions , is the updated value of .

[0126] The specific training process includes: 1) Environment interaction: The agent selects an action to execute in the current state , and the environment returns the new state and reward R; 2) Action value estimation: The evaluation network predicts the action value estimation Q(s,a) of taking action a in state s; 3) Target Q-value generation: The target network generates the action value estimations corresponding to all possible actions in the new state s′ and takes the maximum value ; 4) Value update: Update the parameters of the evaluation network according to the value update formula; 5) Target network update: Regularly copy the parameters of the evaluation network to the target network to maintain the stability of the target network. When the preset number of training rounds is reached or the reward value converges to a certain extent, stop training.

[0127] Training using a double Q-learning network architecture can effectively reduce the estimation bias of the algorithm, improve the learning efficiency and stability of the algorithm; By continuously interacting and learning with the environment, the algorithm can gradually find the optimal teaching strategy and improve the teaching effect. The reinforcement learning algorithm has strong adaptability and can adjust the teaching strategy in real time according to the changes in the teaching state. In the actual teaching process, the learning situation and emotional state of students will change continuously, and the algorithm can respond to these changes in a timely manner and make appropriate teaching decisions.

[0128] D800, obtain the target action corresponding to the target state based on the target learning algorithm. Specifically, an improved Monte Carlo tree search method can be used to explore the strategy space, and each search includes four stages: selection, expansion, simulation, and backtracking. Further, input the current target state s into the trained target learning algorithm (i.e., the evaluation network), and the evaluation network predicts the action values of all possible actions in the target state, and selects the action with the largest action value as the target action.

[0129] D900, obtain the teaching method, activity arrangement, and time allocation information corresponding to the target action; Integrate the teaching method, corresponding activity arrangement, and time allocation information to generate a personalized teaching adjustment strategy. Specifically, according to the target action, extract the corresponding teaching method, activity arrangement, and time allocation information from the action space; Integrate the teaching method, activity arrangement, and time allocation information to form a detailed personalized teaching adjustment strategy. For example, the teaching method is the lecture method, the activity arrangement is to explain knowledge points in class, and the time allocation is 30 minutes; Then the teaching method is the discussion method, the activity arrangement is to have a group discussion on relevant cases, and the time allocation is 15 minutes.

[0130] In D800 - D900, the target action is obtained based on the target learning algorithm, and the corresponding teaching methods, activity arrangements and time allocation information are integrated to generate personalized teaching adjustment strategies. This strategy can be customized according to the specific situation of each student or student group to meet the learning needs of different students and promote the personalized development of students; personalized teaching adjustment strategies can more accurately meet students' learning needs, improve students' learning interest and participation, thereby improving teaching quality and efficiency. At the same time, through the optimization decision of the reinforcement learning algorithm, the limitations of subjective decision-making in traditional teaching can be avoided, making the teaching process more scientific and reasonable. In summary, based on the constructed teaching knowledge graph and emotional state evaluation information, the reinforcement learning algorithm is used to generate personalized teaching adjustment strategies, which can comprehensively and accurately grasp the teaching situation, optimize teaching decisions, improve teaching quality and efficiency, and promote students' personalized development.

[0131] The method disclosed in the present application collects multimodal student emotional data, including visual, audio and physiological data, performs preprocessing and deep feature extraction, and obtains multi-dimensional fusion features according to a preset fusion strategy. The comprehensive analysis of this multimodal data can capture the students' emotional state more comprehensively and deeply, and can provide richer and more accurate emotional information compared to the analysis of a single data source; extracting deep visual features, deep audio features and deep physiological features can dig out more essential emotional features, avoid the limitations of surface data, and thus more accurately identify the students' emotional state; constructing a personalized emotional model based on the historical data of each individual student, fully considering the individual differences of students, and the model can be based on the students. The unique emotional patterns and historical expressions of students are used to analyze the basic emotion recognition information, and the emotion state evaluation information that is more in line with the actual situation of the students is obtained; based on the teaching knowledge map and the emotion state evaluation information, personalized teaching adjustment strategies are generated, and teaching plans can be tailored for each student according to their specific emotional state and learning situation, to meet the personalized learning needs of students and improve the teaching effect; the plan can dynamically adjust the actual teaching plan according to the actual teaching effect. As the teaching process progresses, the emotional data of students are continuously collected and analyzed, the emotion state evaluation information is updated in real time, and new teaching adjustment strategies are generated accordingly, so that the teaching process can always adapt to the changes in students' learning status and improve the adaptive ability of the system.

[0132] The prior art analyzes students' emotional states rather one-sidedly, often relying on a single data source only; while the method disclosed in this application can comprehensively analyze students' emotional states from multiple perspectives by collecting multi-modal student emotional data and performing fusion processing, avoiding the limitations of single data. At the same time, a personalized emotional model is constructed based on the historical data of individual students, fully considering the unique emotional patterns and individual differences of each student, and more accurate emotional state assessment information that conforms to the actual situation of students can be obtained. The teaching strategy adjustment suggestions in the prior art are often preset fixed solutions and are difficult to meet the needs of personalized teaching; while the method disclosed in this application generates personalized teaching adjustment strategies based on the emotional state assessment information and the teaching knowledge graph, and can provide targeted teaching suggestions according to the specific emotional state, knowledge mastery situation and learning progress of each student, improving the effectiveness and adaptability of teaching strategies.

[0133] The prior art system has insufficient self-adaptive ability and cannot dynamically optimize and adjust strategies according to the actual teaching effect; the method disclosed in this application can collect and analyze students' emotional data in real time, dynamically generate new teaching adjustment strategies according to the changes in the emotional state assessment information, and adjust the actual teaching plan accordingly. This real-time and dynamic adjustment mechanism enables the teaching process to adapt to the changes in students' learning states in a timely manner, improves the self-adaptive ability of the system, and ensures the continuous optimization of the teaching effect. Through comprehensive emotional analysis, personalized assessment and teaching strategies, and dynamic self-adaptive adjustment ability, this solution effectively solves the problems existing in the prior art and provides a more scientific and effective method for improving the teaching effect.

[0134] On the other hand, the present application discloses an adaptive teaching strategy adjustment system based on sentiment analysis, which is used to execute the adaptive teaching strategy adjustment method based on sentiment analysis disclosed in the first aspect of the present application, including: a collection module, configured to preprocess the collected multi-modal student sentiment data to obtain a target sentiment feature data stream; a deep feature extraction module, configured to extract visual deep features, audio deep features, and physiological deep features from the target sentiment feature data stream; a multi-dimensional fusion feature acquisition module, configured to process the preset visual attention weight, audio attention weight, physiological attention weight, visual deep features, audio deep features, and physiological deep features according to a preset fusion strategy to obtain multi-dimensional fusion features; an identification module, configured to analyze the multi-dimensional fusion features using a preset multi-task learning framework to obtain basic sentiment identification information of individual students; a personalized sentiment model construction module, configured to construct a personalized sentiment model based on the historical data of each individual student; a sentiment state evaluation information acquisition module, configured to analyze the basic sentiment identification information based on the personalized sentiment model to obtain corresponding sentiment state evaluation information; a teaching adjustment strategy generation module, configured to generate a personalized teaching adjustment strategy based on the constructed teaching knowledge graph and the sentiment state evaluation information; and a dynamic adjustment module, configured to dynamically adjust the actual teaching plan according to the teaching adjustment strategy.

[0135] The computer device according to an embodiment of the present disclosure includes a memory and a processor. The memory is used to store non-temporary computer-readable instructions. Specifically, the memory may include one or more computer program products, and the computer program products may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the computer device to execute desired functions.

[0136] As Figure 6 It is a schematic structural diagram of a computer device provided by an embodiment of the present disclosure. It shows a schematic structural diagram of a computer device suitable for implementing the computer device in the embodiment of the present disclosure. Figure 6 The shown computer device is only an example, and should not bring any limitation to the functions and usage scope of the embodiment of the present disclosure. As Figure 6As shown, the computer device may include a processor (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage device into a random access memory (RAM). In the RAM, various programs and data required for the operation of the computer device are also stored. The processor, ROM, and RAM are connected to each other via a bus. An input / output interface is also connected to the bus. For a detailed description of this embodiment, reference can be made to the corresponding descriptions in the foregoing embodiments, which will not be elaborated herein.

Claims

1. An adaptive teaching strategy adjustment method based on sentiment analysis, characterized in that, Including: Preprocess the collected multi-modal student emotion data to obtain a target emotion feature data stream; Extract visual deep features, audio deep features, and physiological deep features from the target emotion feature data stream; Process the preset visual attention weights, audio attention weights, physiological attention weights, the visual deep features, the audio deep features, and the physiological deep features according to a preset fusion strategy to obtain multi-dimensional fusion features. Further including: determining the visual attention weights, audio attention weights, and physiological attention weights; multiplying the visual deep features, the audio deep features, and the physiological deep features by the visual attention weights, the audio attention weights, and the physiological attention weights respectively to obtain visual weighted features, audio weighted features, and physiological weighted features; adding the visual weighted features, the audio weighted features, and the physiological weighted features to the visual deep features, the audio deep features, and the physiological deep features respectively by element to obtain visual fusion features, audio fusion features, and physiological fusion features; splicing the visual fusion features, the audio fusion features, and the physiological fusion features in the feature dimension to obtain multi-dimensional fusion features; Use a preset multi-task learning framework to analyze the multi-dimensional fusion features to obtain basic emotion recognition information of individual students; Construct a personalized emotion model based on the historical data of each individual student; Analyze the basic emotion recognition information based on the personalized emotion model to obtain corresponding emotion state evaluation information; Generate a personalized teaching adjustment strategy based on the constructed teaching knowledge graph and the emotion state evaluation information; Dynamically adjust the actual teaching plan according to the teaching adjustment strategy; Among them, the determining the visual attention weights, audio attention weights, and physiological attention weights includes: determining an optimization objective function; The optimized objective function is as follows: ; ; ; ; is the initial visual weight, is the initial audio weight, is the initial physiological weight, is the mean of the visual deep features, is the standard deviation of the visual deep features, is the mean of the audio deep features, is the standard deviation of the audio deep features, is the mean of the physiological deep features, is the standard deviation of the physiological deep features, is the time continuity index of the visual deep features, is the time continuity index of the audio deep features, is the time continuity index of the physiological deep features, is the adjustment coefficient for the complementarity between different features, is the complementarity index between the visual deep features and the audio deep features, is the complementarity index between the visual deep features and the physiological deep features, is the complementarity index between the audio deep features and the physiological deep features; Randomly set the visual initial weight, audio initial weight, and physiological initial weight; use the stochastic gradient descent method to optimize the optimization objective function, and when the optimization objective function converges or reaches a preset number of iterations, output the optimized visual attention weights, audio attention weights, and physiological attention weights.

2. The method for adjusting an adaptive teaching strategy based on sentiment analysis according to claim 1, wherein The multi-modal student emotion data includes facial expression video stream data, limb movement video stream data, classroom voice data, and physiological parameter data; The preprocessing the collected multi-modal student emotion data to obtain a target emotion feature data stream includes: Collect the facial expression video stream data and limb movement video stream data of students through the arranged image acquisition device; Collect classroom voice data through the arranged microphone array; Collect the physiological parameter data of students through wearable devices; Perform motion compensation and image enhancement processing on both the facial expression video stream data and the limb movement video stream data to obtain target video data; Perform noise reduction and sound source localization processing on the classroom voice data, and use a voice activity detection algorithm to extract effective voice segments to obtain target audio data; Perform filtering and denoising processing on the physiological parameter data to obtain target parameter data; The target video data, the target audio data, and the target parameter data associated with the time series constitute a target emotional feature data stream.

3. The method for adjusting an adaptive teaching strategy based on sentiment analysis according to claim 1, wherein The dimensions of the visual deep feature, the audio deep feature, and the physiological deep feature are the same; The visual deep feature includes a facial feature and a pose feature; The audio deep feature includes a plurality of time-frequency domain features; The physiological deep feature includes a plurality of physiological features; Extracting the visual deep feature, the audio deep feature, and the physiological deep feature from the target emotional feature data stream includes: Extracting the facial feature from the target video data in the target emotional feature data stream by using a pre-trained VGGFace network; Extracting the pose feature from the target video data and the target audio data in the target emotional feature data stream by using ResNet50; Improving 1D-CNN by using a preset strategy, and extracting a plurality of time-frequency domain features from the target emotional feature data stream based on the improved 1D-CNN, where the time-frequency domain features include at least one of a time-frequency domain, a fundamental frequency, and a formant; Extracting features from the target parameter data in the target emotional feature data stream by using an LSTM network to obtain a plurality of physiological features.

4. The method for adjusting an adaptive teaching strategy based on sentiment analysis according to claim 1, characterized in that Using a preset multi-task learning framework to analyze the multi-dimensional fusion feature to obtain the basic emotional recognition information of the student individual, including: Presetting a multi-task learning framework; The multi-task learning framework includes a shared feature encoding layer, and an emotion classification task layer and an intensity prediction task layer both connected to the shared feature encoding layer; The shared feature encoding layer includes a multi-layer perceptron, the emotion classification task layer includes a softmax classifier, and the intensity prediction task layer includes a regressor; Inputting the multi-dimensional fusion feature into the shared feature encoding layer, and extracting the emotion general feature through the multi-layer perceptron; Inputting the emotion general feature into the emotion classification task layer, and performing emotion classification through the softmax classifier to output the probability distribution of the emotion category; Inputting the emotion general feature into the intensity prediction task layer, and performing emotion intensity prediction through the regressor to output the corresponding intensity prediction value; Among them, the probability distribution of the emotion category and the corresponding intensity prediction value constitute the basic emotional recognition information of the corresponding student individual.

5. The method for adjusting an adaptive teaching strategy based on sentiment analysis according to claim 1, wherein Constructing a personalized emotion model based on the historical data of each student individual includes: Obtaining the historical data of each student individual within a preset period; the historical data includes historical emotion data, personal characteristics, and learning styles, the historical emotion data includes historical emotion categories and corresponding intensity information; the personal characteristics include age, gender, and personality traits; Training an initial model by using a general emotion data set to obtain a target emotion model; the initial model includes a convolutional neural network model or a recurrent neural network model; Freeze the parameters of the first F1 layers of the target emotion model. Use the historical data as the training set to train the parameters of the subsequent F2 layers of the target emotion model. Update the parameters of the subsequent F2 layers through the backpropagation algorithm, and denote the trained target emotion model as the personalized emotion model; where F1 + F2 = F0, 0 < F2 < F0 / 2, and F0 is the total number of neural network layers of the target emotion model.

6. The method for adjusting an adaptive teaching strategy based on sentiment analysis according to claim 1, wherein Based on the constructed teaching knowledge graph and the emotion state evaluation information, generate a personalized teaching adjustment strategy, including: Construct a hierarchical teaching knowledge graph according to the knowledge of education domain experts and curriculum standards; the teaching knowledge graph includes a knowledge point level, a teaching method level, and a learning activity template level; According to the teaching knowledge graph and the emotion state evaluation information, determine the knowledge mastery level and classroom progress information of the student group for different knowledge points; Determine the state space in the reinforcement learning algorithm according to the knowledge mastery level, the classroom progress information, and the emotional state assessment information; the state space includes a state vector s, , where represents the mastery level of the -th knowledge point, represents the classroom progress, represents the emotional state; According to the knowledge mastery level and the classroom progress information, screen out teaching methods, corresponding activity arrangements, and time allocation information suitable for the current teaching situation in the teaching knowledge graph; Generate a personalized teaching adjustment strategy according to the teaching methods, corresponding activity arrangements, and the time allocation information.

7. The method for adjusting an adaptive teaching strategy based on sentiment analysis according to claim 1, characterized in that Based on the constructed teaching knowledge graph and the emotion state evaluation information, generate a personalized teaching adjustment strategy, including: Construct a hierarchical teaching knowledge graph according to the knowledge of education domain experts and curriculum standards; the teaching knowledge graph includes a knowledge point level, a teaching method level, and a learning activity template level; According to the teaching knowledge graph and the emotion state evaluation information, determine the knowledge mastery level and classroom progress information of the student group for different knowledge points; According to the knowledge mastery level, the classroom progress information, and the emotion state evaluation information, determine the state space in the reinforcement learning algorithm; According to the knowledge mastery level and the classroom progress information, screen out teaching methods, corresponding activity arrangements, and time allocation information suitable for the current teaching situation in the teaching knowledge graph; According to the teaching methods, corresponding activity arrangements, and the time allocation information, determine the action space in the reinforcement learning algorithm; Configure a reward function; train the determined reinforcement learning algorithm based on the reward function, and denote the trained reinforcement learning algorithm as the target learning algorithm; The reinforcement learning algorithm is a double Q-learning network architecture including an evaluation network and a target network; Obtain the target action corresponding to the target state based on the target learning algorithm; Obtain the teaching methods, activity arrangements, and time allocation information corresponding to the target action; Integrate the teaching methods, corresponding activity arrangements, and the time allocation information to generate a personalized teaching adjustment strategy.

8. A computer device, characterized in that, The computer device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the adaptive teaching strategy adjustment method based on emotion analysis according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to execute the method for adjusting an adaptive teaching strategy based on sentiment analysis according to any one of claims 1-7.

Images