Deep learning-based methods, systems, and media for analyzing learning sentiment.

Abstract

This invention discloses a method, system, and medium for analyzing the emotional intensity of learning based on deep learning. The method involves: acquiring facial expression images of students during learning, extracting effective facial regions to obtain and save the expression image to be identified; constructing a three-dimensional state-space emotional model of the learning expression; analyzing the expression image to be identified to obtain the student's emotional intensity, specifically: inputting the expression image to be identified into an emotional classification module, extracting feature vectors for emotional classification, and obtaining the emotion of the expression to be identified; using an emotional projection module to calculate the Euclidean distance between the emotion of the expression to be identified and the standard emotion in the three-dimensional state-space emotional coordinate system of the learning expression, matching the true emotion of the expression to be identified; and judging the emotional intensity of the expression to be identified in an emotional intensity judgment module. This invention, based on the three-dimensional state-space emotional model of the learning expression, innovatively introduces Maxout neurons to increase the model depth, fundamentally solving the problem of three-dimensional gradient vanishing in the model.

Description

Technical Field

[0001] This invention belongs to the technical field of image recognition and emotion computing, specifically relating to a method, system, and medium for analyzing the level of learning emotion based on deep learning, applicable to the field of analyzing the level of student learning emotion in the teaching process. Background Technology

[0002] The analysis of students' emotional levels has historically relied on traditional emotion models, which are primarily built using three methods: dictionaries, machine learning, and deep learning. Early emotion models were built and analyzed mainly based on modal data input, including text, video images, and speech. This approach summarized and categorized emotions to form an emotion dictionary for lookup. However, with the increasing prevalence of distance learning and personalized instruction, traditional emotion models face significant challenges in terms of the accuracy of matching emotions to facial expressions, the clarity of representing the positional relationships of various emotions, and the reasonableness of classifying different levels of emotions. Therefore, emotion model building methods based on advanced algorithms such as deep learning are widely used, especially in the fields of educational management and analysis.

[0003] Currently, student learning emotion models are mainly divided into one-dimensional, two-dimensional, three-dimensional, and four-dimensional forms of representation based on various dimensions. Specific emotion models include the Prachk Circle model, the OCC model, and the PAD emotion model. However, these models all have problems to varying degrees, such as poor correspondence between emotions and facial expressions and imprecise division of the degree and relationship of various emotions. At the level of emotion model building methods, one approach is based on convolutional neural networks (CNNs). This method primarily utilizes the characteristics of CNNs to improve the accuracy of the correspondence between students' emotions and expressions. However, it is limited by the increase in neural network depth, thus failing to solve the problems of overfitting and gradient vanishing in the training model. Another approach is based on long short-term memory (LSTM) networks, which to some extent optimizes the problems of overfitting and gradient vanishing in the timeline, but gradient vanishing still exists in the spatial dimension. A third approach is a hybrid approach based on CNNs and LSTM networks, which achieves a balance between the accuracy of the correspondence between students' emotions and expressions and the problems of overfitting and gradient vanishing in the training model. However, it cannot effectively balance the clarity of the representation of various emotional positional relationships and the rationality of different levels of emotion classification. Therefore, it is very important and meaningful to build a more accurate, scientific, reasonable, and efficient student learning expression emotion model based on deep learning algorithms. Summary of the Invention

[0004] The main objective of this invention is to overcome the shortcomings and deficiencies of existing technologies. Addressing three main problems of traditional emotion models—low accuracy in the correspondence between emotions and facial expressions, poor clarity in representing the positional relationships of various emotions, and poor rationality in classifying different levels of emotions—this invention provides a deep learning-based method, system, and medium for analyzing the degree of emotion. Based on an established three-dimensional state-space emotion model for learning facial expressions, this invention innovatively introduces Maxout neurons, thereby increasing the depth of the three-dimensional state-space emotion model for learning facial expressions and fundamentally solving the problem of three-dimensional gradient diffusion in the model. Furthermore, it introduces the concept of an emotion classifier, enabling effective classification of the emotional states of learned facial expressions, thereby further enhancing the model's generalization ability.

[0005] To achieve the above objectives, the present invention provides a method for analyzing learning sentiment based on deep learning, comprising the following steps:

[0006] Collect images of students' facial expressions while they are learning, extract the effective facial regions, obtain the images of the expressions to be recognized, and save them.

[0007] Construct a three-dimensional state space emotion model for learning facial expressions, including an emotion classification module, an emotion projection module, and an emotion degree judgment module;

[0008] Based on the three-dimensional state-space emotion model of learned facial expressions, the image of the facial expression to be recognized is analyzed to obtain the student's level of emotional learning, specifically:

[0009] The image of the expression to be identified is input into the emotion classification module. The feature vector of the image of the expression to be identified is extracted using average pooling. Then, emotion classification is performed based on Maxout neurons to obtain the emotion of the expression to be identified.

[0010] The emotion projection module is used to calculate the Euclidean distance between the emotion of the expression to be identified and the standard emotion in the three-dimensional state space emotion coordinate system of the learned expression, and to match the real emotion of the expression to be identified.

[0011] Based on the true emotion of the expression to be identified, the emotion level judgment module determines the emotion level of the expression to be identified by calculating the Euclidean distance between the emotion of the expression to be identified and the true emotion.

[0012] As a preferred technical solution, the emotion classification module is constructed based on a feature extraction network, and Maxout neurons are introduced as classifiers to classify the feature vectors extracted by the feature extraction network.

[0013] The emotion projection module is established based on the three-dimensional state space coordinate system of learned facial expressions;

[0014] The steps for establishing the three-dimensional state space emotion coordinate system for learning facial expressions are as follows:

[0015] Suppose there are N standard affective types for student learning, and each standard affective type is divided into M affective levels. Then the number of affective states for student learning is N. M ;

[0016] A three-dimensional emotional coordinate system for learning expressions is established, with standard emotional type, emotional level, and student learning emotional state as the x, y, and z axes, respectively.

[0017] Based on the students' learning emotional states and the Markov state transition matrix, construct the emotional state transition matrix A, denoted as:

[0018]

[0019] Among them, a ij Let represent the probability that the emotional state of student i will change to the emotional state of student j, and n and m represent the number of emotional states of students. In the emotional state transition matrix, the sum of the probabilities of each row is 1, and the sum of the probabilities of each column is 1.

[0020] As a preferred technical solution, the learned three-dimensional state space emotion model is trained using the gradient descent method, and the training steps are as follows:

[0021] The pre-prepared raw sample dataset with pre-labeled sentiment tags is preprocessed and divided into training and test sets;

[0022] T training samples are randomly drawn with replacement from the training set samples;

[0023] Introduce a text classification model and train T Softmax classifiers, which will serve as T weak classifiers.

[0024] Collect OOBt data corresponding to T training samples, and combine it with the OOBt data output by the weak classifiers to calculate the reliability Rit of the T weak classifiers on different categories.

[0025] Obtain the T trained weak classifiers and their corresponding weights;

[0026] Each neuron in the Maxout neuron consists of multiple different activated neurons, and each output is the maximum value among the activated neurons. The gradient P of the Maxout neuron during gradient descent training is calculated as follows:

[0027]

[0028] Among them, z i represents the activation value of the i-th activated neuron, and max represents the maximum activation value set.

[0029] As a preferred technical solution, obtaining the emotion of the facial expression to be identified specifically involves:

[0030] The image of the expression to be recognized is input into the emotion classification module. The emotion classification module divides the image of the expression to be recognized into a grid of fixed size, sets the pixel value in the grid to the average value of all pixels in the grid, and then performs average pooling to extract the feature vector of the image of the expression to be recognized.

[0031] The feature vector of the expression image to be identified is input into the Maxout neuron for emotion classification to obtain the emotion of the expression to be identified.

[0032] As a preferred technical solution, the matching of the true emotion of the expression to be identified specifically includes:

[0033] The feature vector of emotion extracted from the emotion classification module is represented as Y = (y1, y2, ..., y...). n ) T ;

[0034] The feature vector of standard emotion in the emotion projection module is represented as: X = (x1, x2, ..., x n ) T ;

[0035] Calculate the Euclidean distance between the emotion of the expression to be recognized and the standard emotion, and map the emotion of the expression to be recognized onto the three-dimensional state space emotion coordinate system of the learned expression. The formula is as follows:

[0036] D = ||XY||

[0037] Where n represents the dimension;

[0038] The minimum value is selected from all calculated Euclidean distances, and the standard sentiment corresponding to the minimum value is obtained as the true sentiment of the expression to be identified.

[0039] As a preferred technical solution, the emotion level judgment module establishes a spherical model based on the standard emotion type and corresponding emotion level, on the basis of learning the three-dimensional state space emotion coordinate system of facial expressions. The spherical model takes a specific emotion level under the standard emotion type as the standard emotion state, and takes the coordinates of the standard emotion state in the three-dimensional state space emotion coordinate system of facial expressions as the origin O of the spherical geometry of the spherical model, and takes the emotion level corresponding to the standard emotion state as the radius r of the sphere.

[0040] When the true emotion of the expression to be identified is obtained, the coordinates of the emotion of the expression to be identified in the emotion coordinate system of the three-dimensional state space of the learned expression are mapped to the spherical model, and the Euclidean distance between the emotion of the expression to be identified and the true emotion is calculated in the spherical model of the true emotion.

[0041] Let the feature vector of the emotion of the expression to be recognized in the spherical model be Y = (y1, y2, ..., y3). n ) T Then, the Euclidean distance between the emotion of the expression to be identified and the true emotion is calculated as: D = ||XY||.

[0042] As a preferred technical solution, the determination of the emotional level of the expression to be identified specifically involves:

[0043] The obtained Euclidean distance D is compared with the set Euclidean distance coefficient:

[0044] When the Euclidean distance is 0 ≤ D < r1, the learning emotion level of the expression to be recognized is strong.

[0045] When the Euclidean distance r1≤D<r2, the learning of the emotion of the expression to be recognized is relatively strong;

[0046] When the Euclidean distance r2≤D<r3, the learning emotional level of the expression to be recognized is moderate.

[0047] When the Euclidean distance r3≤D<r4, the learning of the emotional level of the expression to be recognized is relatively weak.

[0048] When the Euclidean distance r4≤D≤r5, the emotional level of the expression to be recognized is weak.

[0049] As a preferred technical solution, the emotion level judgment module also introduces an emotion entropy value to evaluate changes in learning emotions. The calculation formula is as follows:

[0050] L=(a 11 *loga 11 +a 12 *loga 12 +…+a ij loga ij )*(-B)

[0051] Where L represents the emotional entropy value of the expression to be recognized, and a ij Let B represent the probability that the learning emotional state of the i-th student changes to the learning emotional state of the j-th student, where B is a constant.

[0052] When the emotional entropy value of the expression to be identified is greater than the set emotional entropy threshold, it indicates that the emotional state of the expression to be identified changes rapidly, and that the student's emotional state is relatively unstable and the corresponding emotional state is easy to shift; otherwise, it indicates that the student's emotional state is relatively stable and the corresponding emotional state shifts slowly.

[0053] In another aspect, the present invention provides a learning sentiment analysis system based on deep learning, which is applied to the above-mentioned learning sentiment analysis method based on deep learning, including a data acquisition and processing module, a model building module, and an analysis and recognition module;

[0054] The acquisition and processing module is used to acquire facial expression images of students during learning, extract valid facial regions, obtain the expression image to be recognized, and save it.

[0055] The model building module is used to build a three-dimensional state space emotion model for learning facial expressions, including an emotion classification module, an emotion projection module, and an emotion degree judgment module;

[0056] The analysis and recognition module is used to analyze the facial expression image to be recognized based on the three-dimensional state space emotion model of the learned expression, and to obtain the student's learning emotion level, specifically:

[0057] The image of the expression to be identified is input into the emotion classification module. The feature vector of the image of the expression to be identified is extracted using average pooling. Then, emotion classification is performed based on Maxout neurons to obtain the emotion of the expression to be identified.

[0058] The emotion projection module is used to calculate the Euclidean distance between the emotion of the expression to be identified and the standard emotion in the three-dimensional state space emotion coordinate system of the learned expression, and to match the real emotion of the expression to be identified.

[0059] Based on the true emotion of the expression to be identified, the emotion level judgment module determines the emotion level of the expression to be identified by calculating the Euclidean distance between the emotion of the expression to be identified and the true emotion.

[0060] In another aspect, the present invention provides a computer-readable storage medium storing a program that, when executed by a processor, implements the aforementioned deep learning-based learning sentiment analysis method.

[0061] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0062] Compared with existing technologies (such as a method for building an emotion model based on a convolutional neural network, a method for building an emotion model based on a long short-term memory network, and a method for building an emotion model based on a hybrid of convolutional neural networks and long short-term memory networks), the method described in this invention significantly improves the accuracy of indicators such as recognition and classification of student learning expressions under the same training dataset and experimental environment. In particular, regarding the problem of three-dimensional gradient vanishing in the model, this invention increases the model depth by introducing Maxout neurons, thereby solving this problem at its source and improving the performance of the model.

[0063] In terms of the accuracy of student facial expression recognition, the method described in this invention improves the accuracy of the location information of the corresponding recognized emotion in the three-dimensional emotion model by approximately 10% compared to existing technologies (such as a method for establishing an emotion model based on a convolutional neural network, a method for establishing an emotion model based on a long short-term memory network, and a method for establishing an emotion model based on a hybrid of convolutional neural networks and long short-term memory networks).

[0064] In terms of the accuracy of student facial expression classification, the method described in this invention improves the accuracy by approximately 5% compared to existing technologies (such as a method for establishing an emotion model based on a convolutional neural network, a method for establishing an emotion model based on a long short-term memory network, and a method for establishing an emotion model based on a hybrid of convolutional neural networks and long short-term memory networks) by preprocessing the facial expression feature vectors based on average pooling technology and accurately classifying them based on Maxout neurons. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is a flowchart of a deep learning-based learning sentiment analysis method in an embodiment of the present invention;

[0067] Figure 2 This is a schematic diagram of the process for obtaining students' learning emotion levels in an embodiment of the present invention;

[0068] Figure 3 This is a schematic diagram of the three-dimensional state space emotion coordinate system for learning facial expressions in an embodiment of the present invention;

[0069] Figure 4 This is a schematic diagram of a spherical model in an embodiment of the present invention;

[0070] Figure 5(a) is a curve comparing the accuracy of student learning emotion recognition in an embodiment of the present invention;

[0071] Figure 5(b) is a curve comparing the accuracy of student learning emotion classification in an embodiment of the present invention;

[0072] Figure 6 This is a schematic diagram of the structure of a deep learning-based learning sentiment analysis system in an embodiment of the present invention;

[0073] Figure 7 This is a schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. Detailed Implementation

[0074] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0075] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0076] like Figure 1 , 2 As shown, this embodiment of the deep learning-based learning sentiment analysis method includes the following steps:

[0077] S1. Collect facial expression images of students during learning, extract the effective facial regions, obtain the expression images to be recognized, and save them; the effective facial regions mainly include the areas near the students' eyes, eyebrows, mouth, and other facial features. In the actual extraction process, the facial expression images are quickly captured by relevant camera equipment, and the areas near the facial features are divided and extracted to obtain the expression images to be recognized and saved.

[0078] S2. Construct a three-dimensional state space emotion model for learning facial expressions, including an emotion classification module, an emotion projection module, and an emotion degree judgment module;

[0079] The emotion classification module is built on a feature extraction network and introduces Maxout neurons as classifiers to classify the feature vectors extracted by the feature network, thereby achieving accurate classification of various emotions, solving the gradient vanishing problem, increasing the depth of the training network, and solving the model generalization problem.

[0080] This invention uses different student learning emotional states as x, y, z coordinates, with the corresponding coordinate axis units representing emotional intensity levels, to establish a three-dimensional state space coordinate system for learning expressions. Based on this, an emotion projection module is constructed. Therefore, the representation of different student learning emotions on the coordinate axes should be complex, only indicating a predominance of one emotion. For example, a standard pleasant emotion type might be a point in the three-dimensional state space coordinate system that leans towards a certain coordinate axis. The specific steps for establishing the three-dimensional state space coordinate system for learning expressions are as follows:

[0081] Suppose there are N standard affective types for students learning, and each standard affective type is divided into M affective levels. Then the number of affective states that students learn is N. M N M This represents multiple combinations corresponding to different standard emotion types and emotion levels;

[0082] A three-dimensional emotional coordinate system for learning expressions is established, with standard emotional types, emotional levels, and students' learning emotional states as the x, y, and z axes, respectively. Figure 3 As shown;

[0083] The three-dimensional state space emotional coordinate system for learning facial expressions established in this embodiment intuitively reflects the trajectory of emotional state transfer. Since the emotional state change process reflected by students learning facial expressions can be regarded as the transfer from one data point to another in the three-dimensional state space emotional coordinate system for learning facial expressions, the corresponding emotional transfer process is defined as Markov emotional state transfer. Therefore, by constructing an emotional state transfer matrix A, the transfer process between different emotional states is mathematically represented. The emotional state transfer matrix A is expressed as:

[0084]

[0085] Among them, a ij Let represent the probability that the emotional state of student i will change to the emotional state of student j, and n and m represent the number of emotional states of students. In the emotional state transition matrix, the sum of the probabilities of each row is 1, and the sum of the probabilities of each column is 1.

[0086] Based on the learning of the three-dimensional state space of facial expressions and the emotional coordinate system, the emotional degree judgment module establishes a spherical model according to the standard emotional type and the corresponding emotional degree. Specifically, the specific emotional level under the standard emotional type is taken as the standard emotional state, and the coordinates of the standard emotional state in the learning of the three-dimensional state space of facial expressions are taken as the origin point O of the spherical geometry of the spherical model. The emotional degree corresponding to the standard emotional state is taken as the radius r of the sphere.

[0087] In this embodiment, the three-dimensional state space emotion model for learning facial expressions is trained using gradient descent. The training steps are as follows:

[0088] The pre-prepared raw sample dataset with pre-labeled sentiment tags is preprocessed and divided into training and test sets;

[0089] T training samples are randomly drawn with replacement from the training set samples;

[0090] We introduce a text classification model from existing methods and train T Softmax classifiers, which are then used as T weak classifiers.

[0091] Collect standard OOBt data corresponding to T training samples, and combine it with the OOBt data output by the weak classifiers to calculate the reliability Rit of the T weak classifiers on different categories, thereby improving the efficiency and accuracy of training.

[0092] Obtain the T trained weak classifiers and their corresponding weights.

[0093] Since each neuron in a Maxout neuron is composed of multiple different activated neurons, and each output is the maximum value among the activated neurons, the gradient P of the Maxout neuron during gradient descent training is calculated as follows:

[0094]

[0095] Where P is the output gradient of the Maxout neuron, z i represents the activation value of the i-th activated neuron, and max represents the maximum activation value set;

[0096] As can be seen from the formula, the output of the corresponding Maxout neuron sets the gradient value of its output activation unit to 1, and the gradient values ​​of the rest to 0. Therefore, in the actual training process, the Maxout neuron can solve the diffusion problem by generating continuous gradients. The solution of the diffusion problem by the Maxout neuron further increases the depth of the training network, thereby realizing the fine classification of the emotions corresponding to the students' learning expressions and solving the generalization problem of traditional models.

[0097] S3. Based on the three-dimensional state space emotion model of learned facial expressions, the image of the facial expression to be recognized is analyzed to obtain the student's level of emotional learning, specifically:

[0098] S31. Input the image of the expression to be identified into the emotion classification module, use average pooling to extract the feature vector of the image of the expression to be identified, and then perform emotion classification based on Maxout neurons to obtain the emotion of the expression to be identified.

[0099] The image of the expression to be recognized is input into the emotion classification module. The emotion classification module divides the image of the expression to be recognized into a network according to a fixed size (usually the segmentation size is 2*2 or 8*8), sets the pixel value in the grid to the average value of all pixels in the grid, and then performs average pooling to extract the feature vector of the image of the expression to be recognized.

[0100] The feature vector of the facial expression image to be identified is input into the Maxout neuron for emotion classification to obtain the emotion of the facial expression to be identified.

[0101] S32. Use the emotion projection module to calculate the Euclidean distance between the emotion of the expression to be identified and the standard emotion in the three-dimensional state space emotion coordinate system of the learned expression, and match the real emotion of the expression to be identified.

[0102] Based on the learning of a three-dimensional state space emotion coordinate system for facial expressions, the emotion of the expression to be identified is projected onto the three-dimensional coordinate system, thus intuitively and clearly obtaining the positional relationship between the emotion of the expression to be identified and the standard emotion. In practice, the potential relationship between the emotion of the expression to be identified and the standard emotion is mainly judged based on the Euclidean distance between different emotions. The standard emotion with the closest Euclidean distance is identified as the closest relationship, that is, the true emotion of the expression to be identified. Specifically:

[0103] Let the feature vector of the emotion of the expression to be recognized be represented as Y = (y1, y2, ..., y3). n ) T The feature vector of standard sentiment is represented as: X = (x1, x2, ..., x... n ) T Then, the Euclidean distance between the emotion of the expression to be identified and all standard emotions is calculated using the following formula:

[0104] D = ||XY||

[0105] Where n represents the dimension;

[0106] The minimum value is selected from all calculated Euclidean distances, and the standard sentiment corresponding to the minimum value is obtained as the true sentiment of the expression to be identified.

[0107] S33. Based on the true emotion of the expression to be identified, the emotional degree of the expression to be identified is determined in the emotion degree judgment module by calculating the Euclidean distance between the emotion of the expression to be identified and the true emotion.

[0108] When the true emotion of the expression to be identified is obtained, the coordinates of the emotion of the expression to be identified in the emotion coordinate system of the three-dimensional state space of the learned expression are mapped to the spherical model, and the Euclidean distance between the emotion of the expression to be identified and the true emotion is calculated in the spherical model of the true emotion.

[0109] Let the feature vector of the emotion of the expression to be recognized in the spherical model be Y = (y1, y2, ..., y3). n ) T Then, the Euclidean distance between the emotion of the expression to be identified and the true emotion is calculated as follows:

[0110] D = ||XY||.

[0111] The obtained Euclidean distance D is compared with the set Euclidean distance coefficient:

[0112] When the Euclidean distance is 0 ≤ D < r1, the learning emotion level of the expression to be recognized is strong.

[0113] When the Euclidean distance r1≤D<r2, the learning of the emotion of the expression to be recognized is relatively strong;

[0114] When the Euclidean distance r2≤D<r3, the learning emotional level of the expression to be recognized is moderate.

[0115] When the Euclidean distance r3≤D<r4, the learning of the emotional level of the expression to be recognized is relatively weak.

[0116] When the Euclidean distance r4≤D≤r5, the emotional level of the expression to be recognized is weak.

[0117] To further objectively evaluate and measure students' emotional state performance in the three-dimensional state space emotion model of learning expressions, this invention introduces an emotional entropy value into the emotion level judgment module to evaluate changes in learning emotions. The calculation formula is as follows:

[0118] L=(a 11 *loga 11 +a 12 *loga 12 +…+a ij loga ij )*(-B)

[0119] Where L represents the emotional entropy value of the expression to be recognized, and a ij Let B represent the probability that the learning emotional state of the i-th student changes to the learning emotional state of the j-th student, where B is a constant.

[0120] The following rules are followed in the corresponding emotional state evaluation process: when the emotional entropy value of the expression to be identified is greater than the set emotional entropy threshold, it indicates that the emotional state of the expression to be identified changes rapidly, and that the student's emotional state is relatively unstable and the corresponding emotional state is easy to shift; otherwise, it indicates that the student's emotional state is relatively stable and the corresponding emotional state shifts slowly. In this embodiment, the set emotional entropy threshold is 0.7.

[0121] To illustrate the effectiveness of this method in detail, this embodiment uses five student learning emotions—pleasure, confusion, surprise, neutrality, and fatigue—as a dataset to specifically implement the deep learning-based learning emotion analysis method and its effects proposed in this invention:

[0122] The specific experimental environment configuration for this embodiment is shown in the table below:

[0123]

[0124] Table 1

[0125] The specific steps are as follows:

[0126] A. Use camera equipment to capture students' facial expressions while learning, save the effective facial areas, obtain the expression images to be recognized and save them. At the same time, based on five standard emotion datasets of pleasure, confusion, surprise, neutrality and fatigue, divide the emotion levels into 10 levels and establish a three-dimensional state space emotion model of learning expressions.

[0127] B. Input the image of the expression to be identified into the three-dimensional state space emotion model of the learned expression, use the emotion classification module to extract the feature vector of the image of the expression to be identified, and perform emotion classification based on Maxout neurons to obtain the emotion of the expression to be identified.

[0128] C. Based on the feature vector of the expression image to be identified, project it into the three-dimensional state space emotion coordinate system of the learned expression, calculate the Euclidean distance between the emotion of the expression to be identified and the standard emotion, and match the real emotion of the expression to be identified.

[0129] D. Locate the standard emotion sphere model representing the true emotion of the expression to be identified, such as... Figure 4 As shown, the Euclidean distance between the emotion of the expression to be identified and the true emotion is calculated in the spherical model to determine the degree of emotion of the expression to be identified.

[0130] Taking a pleasant emotional state as an example, the following criteria are set for determining the learning emotional level of the facial expression to be recognized in this emotional state:

[0131] ① When the Euclidean distance is between 0 ≤ D < 0.24, the corresponding expression to be recognized has a strong learning sentiment.

[0132] ② When the Euclidean distance is between 0.24 and D < 0.46, the corresponding expression to be recognized has a stronger learning emotional level;

[0133] ③ When the Euclidean distance is between 0.46 and D < 0.66, the learning emotional level of the expression to be recognized is moderate;

[0134] ④ When the Euclidean distance is between 0.66 and D < 0.76, the learning of the emotional level of the expression to be recognized is relatively weak;

[0135] ⑤ When the Euclidean distance is between 0.76 ≤ D ≤ 1, the learned emotional level of the expression to be recognized is weak;

[0136] The classic convolutional neural network model, the classic long short-term memory network model, and the classic hybrid model of convolutional neural network and long short-term memory network were used as comparative analysis models to compare and verify the accuracy of students' learning emotional states and classification accuracy under different models; the corresponding experimental results curves are shown in Figure 5(a) and Figure 5(b).

[0137] The results show that:

[0138] 1. The method of this invention achieves a stable recognition accuracy of approximately 82.3% when the number of training iterations is sufficiently large and gradually stabilizes. This is approximately 20.1%, 20.3%, and 10.4% higher than the classic convolutional neural network model, the classic long short-term memory network model, and the classic hybrid model of convolutional neural network and long short-term memory network, respectively, demonstrating a significant advantage in recognition accuracy.

[0139] 2. In terms of the accuracy of students learning facial expression and emotion classification, when the number of training times is large enough and gradually stabilizes, the corresponding classification accuracy is stable at about 81.4%. Compared with the classic convolutional neural network model, the classic long short-term memory network model, and the classic hybrid model of convolutional neural network and long short-term memory network, it improves by about 17.5%, 16.4%, and 9.3% respectively, showing a significant advantage in emotion classification accuracy.

[0140] 3. Compared with the classic convolutional neural network model, the classic long short-term memory network model, and the classic hybrid model of convolutional neural network and long short-term memory network, the method of the present invention can more intuitively and clearly reflect the relationship between various emotions, and can more specifically and quantitatively reflect the emotional intensity of various emotions.

[0141] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0142] Based on the same ideas as the deep learning-based learning sentiment analysis method in the above embodiments, this invention also provides a deep learning-based learning sentiment analysis system, which can be used to execute the aforementioned deep learning-based learning sentiment analysis method. For ease of explanation, the structural diagrams of the deep learning-based learning sentiment analysis system embodiments only show the parts relevant to the embodiments of this invention. Those skilled in the art will understand that the illustrated structures do not constitute a limitation on the device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0143] like Figure 6 As shown, another embodiment of the present invention provides a learning sentiment analysis system based on deep learning, including a data acquisition and processing module, a model building module, and an analysis and recognition module;

[0144] The acquisition and processing module is used to acquire facial expression images of students during learning, extract valid facial regions, obtain the expression image to be recognized, and save it.

[0145] The model building module is used to build a three-dimensional state space emotion model of learned facial expressions, including an emotion classification module, an emotion projection module, and an emotion degree judgment module;

[0146] The analysis and recognition module is used to analyze the facial expression images to be recognized based on the learned three-dimensional state space emotion model, and to obtain the student's emotional level of learning. Specifically:

[0147] The image of the expression to be identified is input into the emotion classification module. The feature vector of the image of the expression to be identified is extracted using average pooling. Then, emotion classification is performed based on Maxout neurons to obtain the emotion of the expression to be identified.

[0148] The emotion projection module is used to calculate the Euclidean distance between the emotion of the expression to be identified and the standard emotion in the three-dimensional state space emotion coordinate system of the learned expression, and to match the real emotion of the expression to be identified.

[0149] Based on the true emotion of the expression to be identified, the emotion level judgment module determines the emotion level of the expression to be identified by calculating the Euclidean distance between the emotion of the expression to be identified and the true emotion.

[0150] It should be noted that the deep learning-based learning sentiment analysis system of the present invention corresponds one-to-one with the deep learning-based learning sentiment analysis method of the present invention. The technical features and beneficial effects described in the embodiments of the deep learning-based learning sentiment analysis method are applicable to the embodiments of the deep learning-based learning sentiment analysis system. For details, please refer to the description in the embodiments of the method of the present invention, which will not be repeated here.

[0151] Furthermore, in the implementation of the deep learning-based learning sentiment analysis system described above, the logical division of each program module is merely illustrative. In practical applications, the functions described above can be assigned to different program modules as needed, for example, for the sake of hardware configuration requirements or software implementation convenience. That is, the internal structure of the deep learning-based learning sentiment analysis system can be divided into different program modules to complete all or part of the functions described above.

[0152] like Figure 7 As shown, in one embodiment, a computer-readable storage medium is provided, storing a program in a memory. When the program is executed by a processor, it implements the deep learning-based learning sentiment analysis method, specifically:

[0153] Collect images of students' facial expressions while they are learning, extract the effective facial regions, obtain the images of the expressions to be recognized, and save them.

[0154] Construct a three-dimensional state space emotion model for learning facial expressions, including an emotion classification module, an emotion projection module, and an emotion degree judgment module;

[0155] Based on the three-dimensional state-space emotion model of learned facial expressions, the image of the facial expression to be recognized is analyzed to obtain the student's level of emotional learning, specifically:

[0156] The image of the expression to be identified is input into the emotion classification module. The feature vector of the image of the expression to be identified is extracted using average pooling. Then, emotion classification is performed based on Maxout neurons to obtain the emotion of the expression to be identified.

[0157] The emotion projection module is used to calculate the Euclidean distance between the emotion of the expression to be identified and the standard emotion in the three-dimensional state space emotion coordinate system of the learned expression, and to match the real emotion of the expression to be identified.

[0158] Based on the true emotion of the expression to be identified, the emotion level judgment module determines the emotion level of the expression to be identified by calculating the Euclidean distance between the emotion of the expression to be identified and the true emotion.

[0159] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0160] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0161] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A learning emotion degree analysis method based on deep learning, characterized by, The method comprises the following steps: Collect facial expression images of students during learning, extract effective facial regions, obtain an expression image to be recognized, and save the expression image to be recognized; A three-dimensional state space emotion model of learning expression is constructed, including an emotion classification module, an emotion projection module, and an emotion degree judgment module; Based on the three-dimensional state space emotion model of learning expression, the emotion degree of students during learning is obtained by analyzing the expression image to be recognized, specifically as follows: The expression image to be recognized is input into the emotion classification module, the feature vector of the expression image to be recognized is extracted by using average pooling, and emotion classification is performed based on Maxout neurons to obtain the emotion of the expression image to be recognized; The emotion projection module is used to calculate the Euclidean distance between the emotion of the expression image to be recognized and the standard emotion in the three-dimensional state space emotion coordinate system of learning expression, and the true emotion of the expression image to be recognized is matched; According to the true emotion of the expression image to be recognized, the emotion degree of the expression image to be recognized is judged by calculating the Euclidean distance between the emotion of the expression image to be recognized and the true emotion in the emotion degree judgment module; The emotion classification module is constructed based on a feature extraction network, and Maxout neurons are introduced as a classifier to classify the feature vector extracted by the feature extraction network; The emotion projection module is established based on the three-dimensional state space coordinate system of learning expression; The three-dimensional state space emotion coordinate system of learning expression is established by taking the standard emotion type, the emotion level, and the learning emotion state of students as the x, y, and z axes, respectively; Suppose that there are N standard emotional types for students to learn, and each standard emotional type is divided into M emotional levels, then the number of emotional states for students to learn is N M ; The emotion state transition matrix A is constructed according to the learning emotion state of students and the Markov state transition matrix, and is expressed as: The three-dimensional state space emotion model of learning expression is trained by using the gradient descent method, and the training steps are as follows: ， wherein, a ij denotes the i th student learning emotional state transition to the j th student learning emotional state, n , m denotes the number of student emotional states; in the emotional state transition matrix, the probability sum of each row is 1, and the probability sum of each column is 1. 2.The deep learning-based learning sentiment degree analysis method of claim 1, wherein, The original sample data set with labeled emotion labels prepared in advance is preprocessed, and is divided into a training set and a test set; T training samples are randomly drawn with replacement from the training set samples; A text classification model is introduced, and T Softmax classifiers are trained as T weak classifiers; The OOBt data corresponding to the T training samples are collected, and the reliability Rit of the T weak classifiers on different categories is calculated based on the OOBt data output by the weak classifiers; The T trained weak classifiers and the corresponding weights are obtained; max Each neuron in the Maxout neuron is composed of multiple different activation neurons, and each output corresponds to the maximum value in the activation neurons. The gradient of the Maxout neuron during gradient descent training P The calculation formula is: ， in, z i Indicates the first i The activation value of an activated neuron. The emotion of the expression image to be recognized is obtained by inputting the expression image to be recognized into the emotion classification module, performing grid segmentation on the expression image to be recognized according to a fixed size in the emotion classification module, setting the pixel value in the grid to the average value of all pixels in the grid, and performing average pooling to extract the feature vector of the expression image to be recognized; This indicates the maximum activation value set. 3.The deep learning-based learning sentiment degree analysis method of claim 2, wherein, The feature vector of the expression image to be recognized is input into the Maxout neurons for emotion classification to obtain the emotion of the expression image to be recognized. The true emotion of the expression image to be recognized is matched by calculating the Euclidean distance between the emotion of the expression image to be recognized and the standard emotion, and mapping the emotion of the expression image to be recognized to the three-dimensional state space emotion coordinate system of learning expression, and the formula is D = ||X-Y||, wherein n represents the dimension. ​ 4.The deep learning-based learning sentiment degree analysis method of claim 3, wherein, ​ The feature vector of the emotion of the expression to be recognized extracted in the emotion classification module is represented as Y ( y 1, y 2,…, y n ) T ; The feature vector of the standard emotion in the emotion projection module is represented as: X ( x 1, x 2,…, x n ) T ; ​ The minimum value is filtered from all the calculated Euclidean distances, and the standard emotion corresponding to the minimum value is obtained as the real emotion of the to-be-identified expression. 5.The deep learning-based learning sentiment degree analysis method according to claim 4, characterized in that, The emotion degree judging module establishes a spherical model according to the standard emotion type and the corresponding emotion degree on the basis of learning the three-dimensional state space emotion coordinate system; the spherical model takes the specific emotion level under the standard emotion type as the standard emotion state, and takes the coordinate of the standard emotion state in the three-dimensional state space emotion coordinate system as the spherical geometry center origin of the spherical model , and takes the emotion degree corresponding to the standard emotion state as the spherical radius . When the real emotion of the to-be-identified expression is obtained, the coordinates of the emotion of the to-be-identified expression in the emotion coordinate system of the learning expression three-dimensional state space are mapped into the spherical model, and the Euclidean distance between the emotion of the to-be-identified expression and the real emotion is calculated in the spherical model of the real emotion. Let the feature vector of the emotion of the expression to be identified in the spherical model be Y ( y 1, y 2,…, y n ) T The Euclidean distance calculation formula of the emotion of the expression to be identified and the real emotion is D = ||X-Y||. 6.The deep learning-based learning sentiment degree analysis method according to claim 5, characterized in that, The emotion degree of the to-be-identified expression is determined, and specifically: The obtained Euclidean distance D is compared with a set Euclidean distance coefficient: When the Euclidean distance 0 ≤ D < r 1, the learning emotional degree of the expression to be identified is strong. When the Euclidean distance r 1 ≤ D < r 2, the learning emotion degree of the to-be-identified expression is stronger. When the Euclidean distance r 2 ≤ D < r 3, the learning emotion degree of the expression to be identified is moderate. When the Euclidean distance r 3 ≤ D < r 4, the learning emotion degree of the expression to be identified is weak. When the Euclidean distance r 4 ≤ D ≤ r 5, then the learning emotion degree of the expression to be identified is weak. 7.The learning-based sentiment degree analysis method based on deep learning according to claim 6, wherein, The emotion degree judgment module further introduces an emotion entropy value for evaluating the change of the learning emotion, and the calculation formula is: ， wherein, L represents the emotional entropy value of the expression to be recognized, a ij represents the probability of the emotional state of the i first student changing to the emotional state of the j second student, B is a constant; When the emotion entropy value of the to-be-identified expression is greater than a set emotion entropy threshold value, it indicates that the emotion state of the to-be-identified expression changes rapidly, which indicates that the learning emotion state of the student at this time is relatively unstable, and the corresponding emotion state is easy to transfer; otherwise, it indicates that the learning emotion state of the student at this time is relatively stable, and the corresponding emotion state is slow to transfer.

8. A learning emotion degree analysis system based on deep learning, characterized by The deep learning-based learning emotion degree analysis method according to any one of claims 1-7 comprises a collection and processing module, a model construction module, and an analysis and recognition module. The collection and processing module is configured to collect facial expression images of a student during learning, extract an effective facial region, obtain a to-be-identified expression image, and save the to-be-identified expression image. The model construction module is configured to construct a learning expression three-dimensional state space emotion model, including an emotion classification module, an emotion projection module, and an emotion degree judgment module. The analysis and recognition module is configured to analyze the to-be-identified expression image based on the learning expression three-dimensional state space emotion model, and obtain the learning emotion degree of the student, and specifically: The to-be-identified expression image is input into the emotion classification module, the feature vector of the to-be-identified expression image is extracted using average pooling, and emotion classification is performed based on a Maxout neuron to obtain the emotion of the to-be-identified expression. The emotion projection module is used to calculate the Euclidean distance between the emotion of the to-be-identified expression and the standard emotion in the emotion coordinate system of the learning expression three-dimensional state space, and match the real emotion of the to-be-identified expression. According to the real emotion of the to-be-identified expression, the emotion degree of the to-be-identified expression is determined by calculating the Euclidean distance between the emotion of the to-be-identified expression and the real emotion in the emotion degree judgment module.

9. A computer readable storage medium storing a program, characterized in that, The program is executed by the processor to implement the deep learning-based learning emotion degree analysis method according to any one of claims 1-7.

Images