A method for emotion analysis in classroom teaching

Abstract

The present invention discloses a method for emotion analysis in classroom teaching, belonging to the technical field of image processing. First, the present invention pre-collects the frontal facial images of students facing the blackboard and real-time collects the facial images of students in the classroom teaching; then, according to the key points of the frontal and real-time facial images, the turning angle and the looking-up angle of the real-time image are calculated, and at the same time, the left-eye deviation value and the right-eye deviation value of the real-time image are obtained according to the eye regions of the two; then, a time series sequence of posture change degrees is constructed based on these data at consecutive moments; finally, the angles, deviation values, and the time series sequence of posture change degrees at consecutive moments are input into the time series processing neural network to obtain the classroom interest value, so as to accurately evaluate the emotional state of students in classroom teaching and improve the accuracy of teaching situation evaluation.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly relates to a method for emotion analysis in classroom teaching. Background Art

[0002] Under the background of current educational informatization and intelligence, the evaluation of classroom teaching quality has always been a key issue of concern in the education field. Traditional classroom teaching evaluation methods mainly rely on teachers' subjective judgments and limited manual observations, lacking means for objectively, real-time, and accurately evaluating students' learning states. At the same time, traditional classroom teaching monitoring technologies are mostly limited to simple attendance rate statistics and surface behavior observations, making it difficult to deeply and accurately capture students' learning engagement.

[0003] Existing technical solutions for analyzing classroom situations based on students' facial recognition can attempt to analyze learning states by capturing facial images and extracting key points. However, relying solely on facial feature recognition, it is difficult to accurately judge students' learning interests and concentration levels. Moreover, existing technologies have deficiencies in obtaining dynamic change characteristics of students' postures and cannot comprehensively reflect the real states of students in the classroom. Therefore, there is a problem of low accuracy in evaluating teaching situations in existing technologies. Summary of the Invention

[0004] Aiming at the above deficiencies in the existing technology, a method for emotion analysis in classroom teaching provided by the present invention solves the problem of low accuracy in evaluating teaching situations in the existing technology.

[0005] In order to achieve the above invention objective, the technical solution adopted by the present invention is: a method for emotion analysis in classroom teaching, including the following sub-steps:

[0006] S1. Pre-collect an image of a student facing the blackboard to obtain a frontal facial image, and collect real-time facial images of students in classroom teaching;

[0007] S2. Obtain the turning angle and looking-up angle of the real-time facial image according to the key points on the frontal facial image and the real-time facial image;

[0008] S3. Obtain the left-eye deviation value and right-eye deviation value of the real-time facial image according to the eye regions on the frontal facial image and the real-time facial image;

[0009] S4. Construct a time series of posture change degrees according to the turning angle, looking-up angle, left-eye deviation value, and right-eye deviation value at consecutive multiple moments;

[0010] S5. Obtain the interest value of the classroom based on the time series processing neural network according to the looking-up angle, turning angle, left-eye deviation value, and right-eye deviation value at consecutive multiple moments, as well as the time series of posture change degrees.

[0011] Furthermore, S2 includes the following sub-steps:

[0012] S21, extracting key points from the frontal facial image and the real-time facial image respectively, wherein the key points include: the outer point of the left eye, the outer point of the right eye, and the tip of the nose;

[0013] S22, respectively obtaining center coordinates of a plurality of key points on the frontal facial image and the real-time facial image;

[0014] S23, calculating the head turning angle according to the difference between the center coordinates of the key points of the real-time facial image and the center coordinates of the key points of the frontal facial image;

[0015] S24, constructing a reference plane normal vector based on multiple key points of the frontal facial image;

[0016] S25, constructing a real-time plane normal vector according to multiple key points of the real-time facial image;

[0017] S26. Calculate the head-up angle according to the real-time plane normal vector and the reference plane normal vector.

[0018] Furthermore, the formula for calculating the head turning angle in S23 is: , where R t is the head turning angle at time t, arctan2 is the four-quadrant inverse tangent function, x o is the horizontal coordinate of the center of the key point of the frontal facial image, y o is the vertical coordinate of the key point center of the frontal facial image, x t is the horizontal coordinate of the key point center of the real-time facial image at time t, y t is the vertical coordinate of the center of the key point of the real-time facial image at time t, and t is the number of the moment.

[0019] Furthermore, the formula for calculating the head-up angle in S26 is: , where θ t is the head-up angle at time t, arccos is the inverse cosine function, h t is the real-time plane normal vector at time t, h o is the normal vector of the reference plane, | | is the modulo operation, and t is the time sequence.

[0020] Furthermore, S3 includes the following sub-steps:

[0021] S31, extracting the iris region and the sclera region on the frontal facial image and the real-time facial image respectively;

[0022] S32, on the frontal facial image or the real-time facial image, stitching the iris region and the sclera region to obtain the eyeball region;

[0023] S33. Extract the geometric center coordinates of the eye region to obtain the eye center;

[0024] S34. Extract the geometric center coordinates of the iris region to obtain the iris center;

[0025] S35. Calculate the Euclidean distance between the iris center and the eye center on the frontal face image to obtain the reference distance;

[0026] S36. Calculate the Euclidean distance between the iris center and the eye center on the real-time face image to obtain the real-time distance;

[0027] S37. Obtain the left-eye deviation value and the right-eye deviation value according to the real-time distance and the reference distance of the same eye.

[0028] Further, the formula for obtaining the left-eye deviation value in S37 is: , where γ L,t is the left-eye deviation value at the t-th moment, d L,t is the real-time distance of the left eye at the t-th moment, d L,o is the reference distance of the left eye, and D is the Euclidean distance between the outer lateral points of the left and right eyes of the same person;

[0029] The formula for obtaining the right-eye deviation value is: , where γ R,t is the right-eye deviation value at the t-th moment, d R,t is the real-time distance of the right eye at the t-th moment, d R,o is the reference distance of the right eye.

[0030] Further, for the same moment, respectively determine whether the absolute value of the head-turning angle is less than the head-turning angle threshold, whether the absolute value of the head-up angle is less than the head-up angle threshold, whether the absolute value of the left-eye deviation value is less than the deviation threshold, and whether the absolute value of the right-eye deviation value is less than the deviation threshold. If all four conditions are met, the number of parameters that meet the conditions is recorded as 4. If only three conditions are met, the number of parameters that meet the conditions is recorded as 3. If only two conditions are met, the number of parameters that meet the conditions is recorded as 2. If only one condition is met, the number of parameters that meet the conditions is recorded as 1. If all four conditions are not met, the number of parameters that meet the conditions is recorded as 0. Take the number of parameters that meet the conditions as the pose change degree at this moment, and form a sequence of pose change degrees for multiple consecutive moments to obtain the pose change degree time series.

[0031] Further, the time series processing neural network in S5 includes: a fully connected unit, a first LSTM layer, a second LSTM layer, a time series feature fusion layer, a time series dilated convolution unit, and an output layer;

[0032] The fully connected unit includes a plurality of fully connected layers, and each fully connected layer is used to process the turning angle, the looking-up angle, the left-eye deviation value, and the right-eye deviation value at one moment;

[0033] The input end of the first LSTM layer is connected to the output end of the fully connected unit; the input end of the second LSTM layer is used to input the time series of the attitude change degree; the input ends of the time series feature fusion layer are respectively connected to the output end of the first LSTM layer and the output end of the second LSTM layer, and its output end is connected to the input end of the time series dilated convolution unit; the output end of the time series dilated convolution unit is connected to the input end of the output layer; the output end of the output layer is used as the output end of the time series processing neural network.

[0034] Further, the expression of the time series feature fusion layer is: , , , where R is the output of the time series feature fusion layer, G1 is the output of the first LSTM layer, G2 is the output of the second LSTM layer, g 1,1 is the output of the 1st cell unit in the first LSTM layer, g 1,n is the output of the nth cell unit in the first LSTM layer, g 1,N is the output of the Nth cell unit in the first LSTM layer, g 2,1 is the output of the 1st cell unit in the second LSTM layer, g 2,n is the output of the nth cell unit in the second LSTM layer, g 2,N is the output of the Nth cell unit in the second LSTM layer, n is a positive integer, N is the sequence length, is the convolutional layer, is element-wise multiplication.

[0035] Further, the time series dilated convolution unit includes: the first convolutional layer, the second convolutional layer, the third convolutional layer, the fourth convolutional layer, the first dilated convolutional layer, the second dilated convolutional layer, the max pooling layer, and the adder A1;

[0036] The input ends of the first convolutional layer are respectively connected to the input ends of the second convolutional layer, the third convolutional layer, and the fourth convolutional layer, and serve as the input end of the time series dilated convolution unit;

[0037] The output end of the first convolutional layer is connected to the input end of the first dilated convolutional layer; the output end of the second convolutional layer is connected to the input end of the second dilated convolutional layer; the output end of the third convolutional layer is connected to the input end of the max pooling layer; the input ends of the adder A1 are respectively connected to the output end of the first dilated convolutional layer, the output end of the second dilated convolutional layer, the output end of the max pooling layer, and the output end of the fourth convolutional layer, and its output end serves as the output end of the time series dilated convolution unit.

[0038] In summary, the beneficial effects of the present invention are as follows:

[0039] 1. By pre-collecting frontal facial images and comparing them with real-time facial images, and combining key point and eye region data, the present invention can accurately obtain multi-dimensional information such as turning angle, looking-up angle, and binocular deviation value. Compared with traditional single facial feature recognition, the comprehensiveness of data collection is greatly improved; in terms of dynamic evaluation, a time series of pose change degrees is constructed to effectively capture the dynamic change characteristics of students' poses, realizing dynamic monitoring of students' classroom states.

[0040] 2. The present invention inputs the looking-up angles, turning angles, left eye deviation values, and right eye deviation values at multiple moments, as well as the time series of pose change degrees, into a time series processing neural network. The time series of pose change degrees is used to enhance the corresponding features of the looking-up angles, turning angles, left eye deviation values, and right eye deviation values at multiple moments, making up for the defect of isolated single feature data information, thereby enhancing the time series correlation and feature complementarity of the data. Through fusion enhancement, the neural network can capture richer and more representative information on students' classroom states, improving the accuracy of time series processing neural network evaluation and solving the problem of low accuracy in teaching situation evaluation in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flowchart of an emotion analysis method for classroom teaching;

[0042] Figure 2 is a schematic structural diagram of a time series processing neural network;

[0043] Figure 3 is a schematic structural diagram of a fully connected unit;

[0044] Figure 4 is a schematic structural diagram of a time series dilated convolution unit. DETAILED DESCRIPTION OF THE INVENTION

[0045] The following describes the specific embodiments of the present invention to facilitate those skilled in the art of the present technology to understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art of the present technology, as long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concept of the present invention are within the scope of protection.

[0046] As Figure 1 shown, an emotion analysis method for classroom teaching includes the following sub-steps:

[0047] S1. Pre-acquire an image of a student facing the blackboard to obtain a frontal face image, and acquire the real-time face image of the student in classroom teaching;

[0048] S2. According to the key points on the frontal face image and the real-time face image, obtain the turning angle and the looking-up angle of the real-time face image;

[0049] S3. According to the eye regions on the frontal face image and the real-time face image, obtain the left-eye deviation value and the right-eye deviation value of the real-time face image;

[0050] S4. According to the turning angles, looking-up angles, left-eye deviation values and right-eye deviation values at consecutive multiple moments, construct a time series of pose change degrees;

[0051] S5. According to the looking-up angles, turning angles, left-eye deviation values and right-eye deviation values at consecutive multiple moments, and the time series of pose change degrees, based on a time series processing neural network, obtain the interest value of the classroom.

[0052] In this embodiment, S2 includes the following sub-steps:

[0053] S21. Extract key points from the frontal face image and the real-time face image respectively, where the key points include: the outer left-eye point, the outer right-eye point and the tip-of-nose point;

[0054] S22. Take the central coordinates of multiple key points on the frontal face image and the real-time face image respectively;

[0055] S23. Calculate the turning angle according to the gap between the central coordinates of the key points of the real-time face image and the central coordinates of the key points of the frontal face image;

[0056] S24. Construct a normal vector of the reference plane according to multiple key points of the frontal face image;

[0057] S25. Construct a normal vector of the real-time plane according to multiple key points of the real-time face image;

[0058] S26. Calculate the looking-up angle according to the normal vector of the real-time plane and the normal vector of the reference plane.

[0059] In this embodiment, the outer left-eye point, the outer right-eye point and the tip-of-nose point can be obtained through a facial feature point detection algorithm (such as Dlib, OpenCV tool or Mediapipe, etc.).

[0060] In this embodiment, the formula for calculating the turning angle in S23 is: , where R t is the turning angle at the t-th moment, arctan2 is the four-quadrant arctangent function, x o is the abscissa of the central coordinate of the key point of the frontal face image, yo is the vertical coordinate of the center of the key point of the frontal face image, x t is the horizontal coordinate of the center of the key point of the real-time face image at the t-th moment, y t is the vertical coordinate of the center of the key point of the real-time face image at the t-th moment, and t is the number of the moment.

[0061] In this embodiment, the formula for calculating the head-up angle in S26 is: , where θ t is the head-up angle at the t-th moment, arccos is the inverse cosine function, h t is the normal vector of the real-time plane at the t-th moment, h o is the normal vector of the reference plane, | | is the modulus operation, and t is the number of the moment.

[0062] Reference plane normal vector: , , ; Real-time plane normal vector: , , ; where, RL o is the vector from the right eye to the left eye in the frontal face image, NL o is the vector from the tip of the nose to the left eye in the frontal face image, RL t is the vector from the right eye to the left eye in the real-time face image at the t-th moment, NL t is the vector from the tip of the nose to the left eye in the real-time face image at the t-th moment, x R,o is the horizontal coordinate of the outer point of the right eye in the frontal face image, y R,o is the vertical coordinate of the outer point of the right eye in the frontal face image, x L,o is the horizontal coordinate of the outer point of the left eye in the frontal face image, y L,o is the vertical coordinate of the outer point of the left eye in the frontal face image, x R,t is the horizontal coordinate of the outer point of the right eye in the real-time face image at the t-th moment, y R,t is the vertical coordinate of the outer point of the right eye in the real-time face image at the t-th moment, x L,t is the horizontal coordinate of the outer point of the left eye in the real-time face image at the t-th moment, y L,t is the vertical coordinate of the outer point of the left eye in the real-time face image at the t-th moment, x N,o is the horizontal coordinate of the tip of the nose point in the frontal face image, y N,o is the vertical coordinate of the tip of the nose point in the frontal face image, x N,t is the horizontal coordinate of the tip of the nose point in the real-time face image at the t-th moment, y N,t is the vertical coordinate of the tip of the nose point in the real-time face image at the t-th moment, is the vector cross product.

[0063] The present invention takes a person's face as a two-dimensional plane, extracts the central coordinates of multiple key points (the central abscissa is equal to the average of the abscissas of the multiple key points, and the central ordinate is equal to the average of the ordinates of the multiple key points), calculates the turning angle according to the gap between the central coordinates of the key points in the real-time face image and the central coordinates of the key points in the frontal face image, and obtains the head-up angle through the normal vector of the reference plane and the normal vector of the real-time plane, so as to reflect the posture activities of students through head-up and turning.

[0064] In this embodiment, S3 includes the following sub-steps:

[0065] S31. Respectively extract the iris region and the sclera region on the frontal face image and the real-time face image;

[0066] S32. On the frontal face image or the real-time face image, splice the iris region and the sclera region to obtain the eyeball region;

[0067] S33. Extract the geometric center coordinates of the eyeball region to obtain the eyeball center;

[0068] S34. Extract the geometric center coordinates of the iris region to obtain the iris center;

[0069] S35. On the frontal face image, calculate the Euclidean distance between the iris center and the eyeball center to obtain the reference distance;

[0070] S36. On the real-time face image, calculate the Euclidean distance between the iris center and the eyeball center to obtain the real-time distance;

[0071] S37. According to the real-time distance and the reference distance of the same eye, obtain the left-eye deviation value and the right-eye deviation value.

[0072] The present invention extracts the pixel values of the pixel points in the iris region and the sclera region (the white part of the eye region) from the frontal face image, calculates the similarity between the pixel values of each pixel point on the real-time face image and the pixel values of the pixel points in the iris region, marks the pixel points with a similarity greater than the similarity threshold as suspected iris pixel points, and the connected region formed by each suspected iris pixel point is the iris region; calculates the similarity between the pixel values of each pixel point on the real-time face image and the pixel values of the pixel points in the sclera region, marks the pixel points with a similarity greater than the similarity threshold as suspected sclera pixel points, and the connected region formed by each suspected sclera pixel point is the sclera region.

[0073] In this embodiment, S33 is specifically: extract the edge contour pixel points of the eyeball region, that is, the contour where the eyeball region contacts the eyelid, and calculate the geometric center of each edge contour pixel point of the eyeball region to obtain the eyeball center.

[0074] Specifically, S34 is as follows: Extract the edge contour pixel points of the iris region, that is, the contour of the circle where the iris region contacts the sclera region, and calculate the geometric center of each edge contour pixel point of the iris region to obtain the iris center.

[0075] In this embodiment, the formula for obtaining the left-eye deviation value in S37 is: , where γ L,t is the left-eye deviation value at the t-th moment, d L,t is the real-time distance of the left eye at the t-th moment, d L,o is the reference distance of the left eye, and D is the Euclidean distance between the outer lateral point of the left eye and the outer lateral point of the right eye of the same person's eyes;

[0076] The formula for obtaining the right-eye deviation value is: , where γ R,t is the right-eye deviation value at the t-th moment, d R,t is the real-time distance of the right eye at the t-th moment, d R,o is the reference distance of the right eye.

[0077] The present invention takes the distance of the iris center in the eyeball region of the frontal face image as a reference, then extracts the relative distance of the iris center in the eyeball region of the real-time face image, and takes the difference between the two to measure the deviation of the iris center.

[0078] In this embodiment, for the same moment, respectively judge whether the absolute value of the turning angle is less than the turning angle threshold, whether the absolute value of the looking-up angle is less than the looking-up angle threshold, whether the absolute value of the left-eye deviation value is less than the deviation threshold, and whether the absolute value of the right-eye deviation value is less than the deviation threshold. If all four conditions are met, the number of parameters that meet the conditions is recorded as 4. If only three conditions are met, the number of parameters that meet the conditions is recorded as 3. If only two conditions are met, the number of parameters that meet the conditions is recorded as 2. If only one condition is met, the number of parameters that meet the conditions is recorded as 1. If all four conditions are not met, the number of parameters that meet the conditions is recorded as 0. Take the number of parameters that meet the conditions as the attitude change degree at this moment, and form a sequence of the attitude change degrees at multiple consecutive moments to obtain the attitude change degree time series.

[0079] The attitude change degree can intuitively reflect the concentration level of students in class. A high attitude change degree (close to 4) indicates that the student's posture is relatively stable and is more likely to be focused on the teaching content. A low attitude change degree (close to 0) means that the student's posture changes greatly and the possibility of being distracted is high. Forming a sequence of the attitude change degrees at multiple consecutive moments to obtain the attitude change degree time series can show the dynamic process of the student's posture change from the time dimension. It can be used to analyze the change trend of the concentration of students in the entire class.

[0080] In this embodiment, the turning angle threshold and the looking-up angle threshold are set to π / 12, that is, 15°. 15° is a relatively small angle range. Taking this as the threshold means allowing a certain degree of natural posture changes for students. The deviation threshold is set to , where is the distance from the iris center of the same eye on the frontal face image to the outer point of the eye. There are differences in the eye structures of different students. By setting the threshold based on the distance related to their own eye structures (d max ), it can better adapt to individual differences. The turning angle threshold, the looking-up angle threshold, and the deviation threshold are not limited to those described in this embodiment and can be adjusted according to experiments or experience.

[0081] As Figure 2 shown, the time series processing neural network in S5 includes: a fully connected unit, a first LSTM layer, a second LSTM layer, a time series feature fusion layer, a time series dilated convolutional unit, and an output layer;

[0082] The fully connected unit includes multiple fully connected layers, and each fully connected layer is used to process the turning angle, the looking-up angle, the left eye deviation value, and the right eye deviation value at one moment;

[0083] The input end of the first LSTM layer is connected to the output end of the fully connected unit; the input end of the second LSTM layer is used to input the time series of posture change degrees; the input ends of the time series feature fusion layer are respectively connected to the output end of the first LSTM layer and the output end of the second LSTM layer, and its output end is connected to the input end of the time series dilated convolutional unit; the output end of the time series dilated convolutional unit is connected to the input end of the output layer; the output end of the output layer is used as the output end of the time series processing neural network.

[0084] One cell unit in the first LSTM layer processes the output of one fully connected layer, and one cell unit in the second LSTM layer processes one element in the time series of posture change degrees.

[0085] As Figure 3 shown, the fully connected unit includes: multiple fully connected layers, and each fully connected layer processes the turning angle, the looking-up angle, the left eye deviation value, and the right eye deviation value at one moment. The specific expression is: , where y t is the output of the fully connected layer, R t is the turning angle at the t-th moment, θ t is the looking-up angle at the t-th moment, γ L,t is the left eye deviation value at the t-th moment, γ R,t is the right eye deviation value at the t-th moment, w R,t is the weight of R t , w θ,t is the weight of θ t is the weight ofγL,t γ L,t The weight, w γR,t γ R,t The weight of b t is the bias for the data at time t.

[0086] Single-dimensional data is insufficient to accurately assess student status. Therefore, each fully connected layer integrates data from four dimensions to generate comprehensive features. The first LSTM layer then processes the long-term dependencies within the time series data. The second LSTM layer inputs the posture change time series to further mine the temporal information within the sequence. The first LSTM layer receives the output of the fully connected units and processes the posture features of the head turn angle, head tilt angle, and deviation value integrated at each moment, focusing on learning and encoding specific posture information in the temporal dimension. The second LSTM layer inputs the posture change time series, focusing on the overall temporal patterns and long-term dependencies of the posture change. Both layers capture posture-related information from different perspectives. Feature multiplication and fusion enable them to complement each other, integrating information from different perspectives to form a more comprehensive feature representation. The time-dilated convolutional unit then observes the posture change time series with a wider "field of view" to mine more complex temporal patterns.

[0087] The interest value is a quantitative result output after a time series processing neural network performs a fusion analysis on multi-dimensional posture data (including head turning angle, head raising angle, binocular deviation value) and time series features (time series sequence of posture change). It is used to reflect the students' concentration level in class and their interest level in the teaching content.

[0088] In this embodiment, the output layer adopts a fully connected layer, and the calculation formula of the class interest value is: , y is the interest value of the class, s m Output the mth eigenvalue of the temporal expansion convolution unit, w m is the mth weight in the fully connected layer, b m is the mth bias in the fully connected layer, L is the number of eigenvalues output by the time series dilation convolution unit, and m is a positive integer. In this embodiment, the expression of the time series feature fusion layer is: , , , where R is the output of the temporal feature fusion layer, G1 is the output of the first LSTM layer, G2 is the output of the second LSTM layer, and g 1,1 is the output of the first cell in the first LSTM layer, g 1,n is the output of the nth cell in the first LSTM layer, g 1,N is the output of the Nth cell in the first LSTM layer, g 2,1 is the output of the first cell in the second LSTM layer, g2,n is the output of the n-th cell in the second LSTM layer, g 2,N is the output of the N-th cell in the second LSTM layer, where n is a positive integer and N is the sequence length. is the convolutional layer is element-wise multiplication.

[0089] The present invention performs feature transformation on the outputs of the first LSTM layer and the second LSTM layer through convolution, and then fuses the two by element-wise multiplication. The element-wise multiplication operation can highlight the features of common concern in the outputs of the two layers.

[0090] Such as Figure 4 shown, the temporal dilation convolutional unit includes: a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a first dilated convolutional layer, a second dilated convolutional layer, a max pooling layer, and an adder A1;

[0091] The input end of the first convolutional layer is respectively connected to the input ends of the second convolutional layer, the third convolutional layer, and the fourth convolutional layer, and serves as the input end of the temporal dilation convolutional unit;

[0092] The output end of the first convolutional layer is connected to the input end of the first dilated convolutional layer; the output end of the second convolutional layer is connected to the input end of the second dilated convolutional layer; the output end of the third convolutional layer is connected to the input end of the max pooling layer; the input ends of the adder A1 are respectively connected to the output end of the first dilated convolutional layer, the output end of the second dilated convolutional layer, the output end of the max pooling layer, and the output end of the fourth convolutional layer, and its output end serves as the output end of the temporal dilation convolutional unit.

[0093] In this embodiment, the weights and biases in the temporal processing neural network can be trained by the gradient descent method.

[0094] In this embodiment, the kernel sizes of the first convolutional layer, the second convolutional layer, the third convolutional layer, and the fourth convolutional layer are , the kernel size of the first dilated convolutional layer is , the dilation rate is 1, the kernel size of the second dilated convolutional layer is , the dilation rate is 2. The pooling window of the max pooling layer is .

[0095] Different dilation rates enable the model to extract features from different scales. The layer with a small dilation rate focuses on local and detailed pose features, and the layer with a large dilation rate captures more global and macroscopic pose patterns. Combining them can obtain a multi-scale pose feature representation.

[0096] By pre-collecting frontal facial images and comparing them with real-time facial images, and combining key point and eye region data, the present invention can accurately obtain multi-dimensional information such as turning angle, looking-up angle, and binocular deviation value. Compared with traditional single facial feature recognition, it greatly improves the comprehensiveness of data collection; in terms of dynamic evaluation, a time series of posture change degrees is constructed to effectively capture the dynamic change characteristics of students' postures, realizing the dynamic monitoring of students' classroom states.

[0097] The present invention inputs the looking-up angles, turning angles, left-eye deviation values, and right-eye deviation values at multiple moments, as well as the time series of posture change degrees into a time series processing neural network. The time series of posture change degrees is used to enhance the corresponding features of the looking-up angles, turning angles, left-eye deviation values, and right-eye deviation values at multiple moments, making up for the defect of isolated single-feature data information, thereby enhancing the time series correlation and feature complementarity of the data. Through the fusion enhancement, the neural network can capture richer and more representative information on students' classroom states, improving the accuracy of the time series processing neural network evaluation, and solving the problem of low accuracy in teaching situation evaluation existing in the prior art.

[0098] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for emotion analysis in classroom teaching, characterized in that, It includes the following steps: S1. Pre-collect an image of a student facing the blackboard to obtain a frontal facial image, and collect the real-time facial image of the student in classroom teaching; S2. According to the key points on the frontal facial image and the real-time facial image, obtain the turning angle and the looking-up angle of the real-time facial image; S3. According to the eye regions on the frontal facial image and the real-time facial image, obtain the left-eye deviation value and the right-eye deviation value of the real-time facial image; S4. According to the turning angles, looking-up angles, left-eye deviation values and right-eye deviation values at consecutive multiple moments, construct a time series of attitude change degrees; In S4, for the same moment, respectively determine whether the absolute value of the turning angle is less than the turning angle threshold, whether the absolute value of the looking-up angle is less than the looking-up angle threshold, whether the absolute value of the left-eye deviation value is less than the deviation threshold, and whether the absolute value of the right-eye deviation value is less than the deviation threshold. If all four conditions are satisfied, the number of parameters that meet the conditions is recorded as 4. If only three conditions are satisfied, the number of parameters that meet the conditions is recorded as 3. If only two conditions are satisfied, the number of parameters that meet the conditions is recorded as 2. If only one condition is satisfied, the number of parameters that meet the conditions is recorded as 1. If all four conditions are not satisfied, the number of parameters that meet the conditions is recorded as 0. Take the number of parameters that meet the conditions as the attitude change degree at this moment, and form a sequence of the attitude change degrees at consecutive multiple moments to obtain a time series of attitude change degrees; S5. According to the looking-up angles, turning angles, left-eye deviation values and right-eye deviation values at consecutive multiple moments, and the time series of attitude change degrees, based on a time series processing neural network, obtain the interest value of the classroom.

2. The emotional analysis method for classroom teaching according to claim 1, wherein S2 includes the following steps: S21. Extract key points from the frontal facial image and the real-time facial image respectively, where the key points include: the outer point of the left eye, the outer point of the right eye and the tip of the nose; S22. Take the central coordinates of multiple key points on the frontal facial image and the real-time facial image respectively; S23. Calculate the turning angle according to the gap between the central coordinates of the key points of the real-time facial image and the central coordinates of the key points of the frontal facial image; S24. Construct a normal vector of the reference plane according to multiple key points of the frontal facial image; S25. Construct a normal vector of the real-time plane according to multiple key points of the real-time facial image; S26. Calculate the looking-up angle according to the normal vector of the real-time plane and the normal vector of the reference plane.

3. The emotional analysis method for classroom teaching according to claim 2, wherein The formula for calculating the rotation angle in S23 is as follows: , where R t is the rotation angle of the turntable at the t-th moment, arctan2 is the four-quadrant arctangent function, x o is the abscissa of the center of the key point of the frontal face image, y o is the ordinate of the center of the key point of the frontal face image, x t is the abscissa of the center of the key point of the real-time face image at the t-th moment, y t is the ordinate of the center of the key point of the real-time face image at the t-th moment, and t is the number of the moment.

4. The emotional analysis method for classroom teaching according to claim 2, wherein The formula for calculating the head-up angle in S26 is as follows: , where θ t is the head-up angle at the t-th moment, arccos is the inverse cosine function, h t is the real-time plane normal vector at the t-th moment, h o is the reference plane normal vector, | | is the modulus operation, and t is the number of the moment.

5. The method for emotion analysis in classroom teaching according to claim 1, wherein S3 includes the following steps: S31. Extract the iris region and the sclera region on the frontal facial image and the real-time facial image respectively; S32. On the frontal facial image or the real-time facial image, splice the iris region and the sclera region to obtain the eyeball region; S33. Extract the geometric center coordinates of the eyeball region to obtain the eyeball center; S34. Extract the geometric center coordinates of the iris region to obtain the iris center; S35. On the frontal facial image, calculate the Euclidean distance between the iris center and the eyeball center to obtain the reference distance; S36. On the real-time facial image, calculate the Euclidean distance between the iris center and the eyeball center to obtain the real-time distance; S37. According to the real-time distance and the reference distance of the same eye, obtain the left-eye deviation value and the right-eye deviation value.

6. The emotional analysis method for classroom teaching according to claim 5, wherein The formula for obtaining the left-eye deviation value in S37 is: , where γ L,t is the left-eye deviation value at the t-th moment, d L,t is the real-time distance of the left eye at the t-th moment, d L,o is the reference distance of the left eye, and D is the Euclidean distance between the outer lateral point of the left eye and the outer lateral point of the right eye of the same person; The formula for obtaining the right-eye deviation value is: , where γ R,t is the right-eye deviation value at the t-th moment, d R,t is the real-time distance of the right eye at the t-th moment, and d R,o is the reference distance of the right eye.

7. The method for emotion analysis in classroom teaching according to claim 1, wherein The time series processing neural network in S5 includes: a fully connected unit, a first LSTM layer, a second LSTM layer, a time series feature fusion layer, a time series dilated convolutional unit, and an output layer; The fully connected unit includes multiple fully connected layers, and each fully connected layer is used to process the turning angle, the looking-up angle, the left eye deviation value, and the right eye deviation value at a moment; The input end of the first LSTM layer is connected to the output end of the fully connected unit; the input end of the second LSTM layer is used to input the time series of the attitude change degree; the input ends of the time series feature fusion layer are respectively connected to the output end of the first LSTM layer and the output end of the second LSTM layer, and its output end is connected to the input end of the time series dilated convolutional unit; the output end of the time series dilated convolutional unit is connected to the input end of the output layer; the output end of the output layer serves as the output end of the time series processing neural network.

8. The method for emotional analysis of classroom teaching according to claim 7, wherein The expression of the temporal feature fusion layer is as follows: , , , where R is the output of the temporal feature fusion layer, G1 is the output of the first LSTM layer, G2 is the output of the second LSTM layer, g 1,1 is the output of the first cell unit in the first LSTM layer, g 1,n is the output of the nth cell unit in the first LSTM layer, g 1,N is the output of the Nth cell unit in the first LSTM layer, g 2,1 is the output of the first cell unit in the second LSTM layer, g 2,n is the output of the nth cell unit in the second LSTM layer, g 2,N is the output of the Nth cell unit in the second LSTM layer, n is a positive integer, N is the sequence length, is the convolutional layer, is element-wise multiplication.

9. The method for emotion analysis in classroom teaching according to claim 7, wherein The time series dilated convolutional unit includes: a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a first dilated convolutional layer, a second dilated convolutional layer, a max pooling layer, and an adder A1; The input ends of the first convolutional layer are respectively connected to the input end of the second convolutional layer, the input end of the third convolutional layer, and the input end of the fourth convolutional layer, and serve as the input end of the time series dilated convolutional unit; The output end of the first convolutional layer is connected to the input end of the first dilated convolutional layer; the output end of the second convolutional layer is connected to the input end of the second dilated convolutional layer; the output end of the third convolutional layer is connected to the input end of the max pooling layer; the input ends of the adder A1 are respectively connected to the output end of the first dilated convolutional layer, the output end of the second dilated convolutional layer, the output end of the max pooling layer, and the output end of the fourth convolutional layer, and its output end serves as the output end of the time series dilated convolutional unit.

Images