A learner sentiment analysis method and device based on cross-time-and-space graph representation

Abstract

The application discloses a learner emotion analysis method and device based on a cross-time-space graph representation, and the method comprises the following steps: dividing a plurality of input learner expression video frames into a plurality of subblocks of equal size and without overlapping, and performing graph node feature embedding processing on the subblocks to obtain initial features of the graph nodes; constructing a cross-time-space nearest neighbor graph structure of each subblock; classifying output emotion space-time graph features of a learner emotion cross-time-space graph representation module to predict the emotion category of the input learner expression video frame; the application can divide the local area of the learner expression video frame without using additional auxiliary information, so that the cross-time-space graph convolution network can directly process the learner expression video frame data; in addition, the application can obtain the space-time correlation graph representation between different regions and different video frames of the learner expression through the cross-time-space graph construction and the graph convolution operator with correlation perception, and can be widely applied to the technical field of computers.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology, and in particular to a learner emotion analysis method and device based on cross-spatiotemporal graph representation. BACKGROUND

[0002] The emotional state of a learner is a real inner experience and corresponding external performance generated in the process of learning cognitive activities, and is one of the key factors for improving the quality and efficiency of education and teaching in human-computer collaboration. In the modern education scene of deep integration of man-machine-objects, accurate analysis of the emotions of learners can provide accurate data for teachers to obtain the learning state of learners in time, and thus promote the improvement of the teaching strategy of teachers in the classroom, and drive the innovation of high-quality classroom teaching.

[0003] Current emotion analysis technologies for learner expressions mainly include two kinds of deep neural network models based on convolutional neural network (CNN) and Transformer. Among them, the method based on CNN is the most widely used framework in current emotion analysis applications, and typical representative methods include DLP-CNN, RAN, FAN and MA-Net, etc. However, these CNN-based emotion analysis methods have defects such as limited receptive field range of convolution operator, which leads to the inability of the analysis model to obtain the structured relationship between the local regions of facial expressions, and thus affects the prediction accuracy of the learner's emotions. In view of the problems existing in the CNN-based emotion analysis method, the Transformer-based emotion analysis method obtains the relationship features of different regions of the learner's face through the self-attention mechanism, and typical representative methods include FER-VT, Former-DFER, etc. However, the Transformer-based emotion analysis method needs to calculate the relationship weights between all local regions, so the model will obtain redundant or noisy relationship features, and in addition, it will also lead to the calculation amount of the algorithm to increase exponentially with the increase of the number of local regions. Overall, the existing learner emotion analysis technologies based on CNN or Transformer are limited by their internal mechanisms, and it is difficult to model fine-grained cross-spatiotemporal structured representation for learner emotions, and thus it is difficult to adapt to actual online or offline teaching scenarios, and the related technical problems need to be solved. SUMMARY

[0004] Therefore, the embodiments of the present application provide a learner emotion analysis method and device based on cross-spatiotemporal graph representation, which can effectively improve the prediction accuracy of learner emotion analysis.

[0005] An aspect of the embodiments of the present application provides a learner emotion analysis method based on cross-spatiotemporal graph representation, comprising:

[0006] The input learner's facial expression video frames are divided into several equal-sized and non-overlapping sub-blocks, and the sub-blocks are subjected to graph node feature embedding processing to obtain the initial features of the graph nodes.

[0007] Based on the initial features of the graph nodes, construct the spatiotemporal nearest neighbor graph for each sub-block;

[0008] Based on the spatiotemporal nearest neighbor graph, the output learner expression graph representation of the learner emotion spatiotemporal graph representation is classified to predict the emotion category of the input learner expression video frame.

[0009] Optionally, the method further includes:

[0010] Acquire students' learning context videos through visual perception devices such as terminal cameras;

[0011] Face detection and face alignment are performed on raw video frames containing learners' facial expressions using a face object detector.

[0012] The resolution of the facial expression video frames after face alignment is scaled and the width and height are uniformly set to obtain the learner's facial expression video frames.

[0013] The learner's facial expression video frames are used as input data for the model.

[0014] Optionally, dividing the input learner facial expression video frames into several equal-sized, non-overlapping sub-blocks includes:

[0015] The first feature extraction block of the deep residual network is used to perform preliminary processing on the learner's facial expression video frames of the original input to obtain a first feature map with a width and height of half that of the original image;

[0016] The second feature map of the second stage is extracted using the second feature extraction block of the deep residual network;

[0017] The third feature map of the third stage is extracted using the third feature extraction block of the deep residual network;

[0018] The fourth feature map of the third stage is extracted using the fourth feature extraction block of the deep residual network;

[0019] The fifth feature map of the third stage is extracted using the fifth feature extraction block of the deep residual network;

[0020] The initial features of the graph nodes are determined based on the first feature map, the second feature map, the third feature map, the fourth feature map, and the fifth feature map.

[0021] Optionally, the step of performing feature embedding processing on the sub-blocks to obtain the initial features of the graph nodes includes:

[0022] The second feature map, the third feature map, and the fourth feature map are transformed into a target dimension feature space by feature mapping, and the fifth feature map is subjected to dimension-invariant feature linear transformation.

[0023] The transformed second, third, fourth, and fifth feature maps are then concatenated.

[0024] The feature map obtained by the concatenation operation is input into the feature fusion function to obtain the initial features of the graph nodes containing multi-scale learner facial expression feature information.

[0025] Optionally, constructing a spatiotemporal nearest neighbor graph for each sub-block based on the initial features of the graph nodes includes:

[0026] Before constructing a spatiotemporal nearest neighbor graph for each graph node, the initial features of the graph node are transformed once.

[0027] After performing the convolution operation on the associative perceptual graph, a feature transformation is performed on all graph nodes to obtain a set of feature vectors as the graph node set;

[0028] For any layer of graph nodes in the graph node set, construct an expanded k-nearest neighbor graph;

[0029] Each graph node is treated as the center point of a local nearest neighbor graph. For any center node of a local nearest neighbor graph, graph convolution is performed on the local graph.

[0030] By using the spatiotemporal correlation perception graph node aggregation function, the spatiotemporal correlation characteristics between different nodes can be obtained;

[0031] The output of the spatiotemporal correlation sensing graph node aggregation function is used as the input of the graph update function to obtain the updated node features. After performing spatiotemporal correlation sensing graph convolution and graph node feature transformation operations, a multilayer perceptron is used to perform feature mapping processing on all updated graph node features to obtain the graph node feature set, thus completing the construction of the spatiotemporal nearest neighbor graph for each sub-block.

[0032] Optionally, in the step of classifying the output learner expression spatiotemporal graph features representing the learner's emotion, the classification expression is:

[0033]

[0034] in, These are the learner's facial expression category labels predicted by the classifier; The spatiotemporal graph representation of the i-th learner's facial expression video frame is shown; AvgPool is the average pooling operator; the pooled global features of the learner's facial expression are shown. As input to the classifier; Φ(·;Θ Φ ) contains learnable parameter Θ Φ The feedforward neural network; LogSoftmax is the activation function.

[0035] Optionally, the method further includes:

[0036] The cross-entropy loss function with a labeled distribution smoothing regularization term is used as the training objective function for the learner sentiment analysis model;

[0037] The objective function is optimized using a gradient descent method to complete the training of the learner sentiment analysis model.

[0038] Another aspect of this invention provides a learner sentiment analysis device based on cross-spatiotemporal graph representation, comprising:

[0039] The first module is used to divide the input learner video frames into several equal-sized and non-overlapping sub-blocks, and to perform graph node feature embedding processing on the sub-blocks to obtain the initial features of the graph nodes.

[0040] The second module is used to construct a spatiotemporal nearest neighbor graph for each sub-block based on the initial characteristics of the graph nodes.

[0041] The third module is used to classify the output expression spatiotemporal graph features represented by the learner's emotional spatiotemporal graph based on the spatiotemporal nearest neighbor graph, and predict the emotional category of the input learner video frame.

[0042] Another aspect of the present invention provides an electronic device, including a processor and a memory;

[0043] The memory is used to store programs;

[0044] The processor executes the program to implement the method described above.

[0045] Another aspect of this invention provides a computer-readable storage medium storing a program that is executed by a processor to implement the methods described above.

[0046] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0047] In embodiments of this invention, the input learner video frame is divided into several equal-sized, non-overlapping sub-blocks, and the sub-blocks are subjected to graph node feature embedding processing to obtain initial features of the graph nodes. Based on the initial features of the graph nodes, a spatiotemporal nearest neighbor graph is constructed for each sub-block. Based on the spatiotemporal nearest neighbor graph, the output facial expression spatiotemporal graph features of the learner emotion spatiotemporal graph representation are classified to predict the emotion category of the input learner video frame. This invention can perform local region division of learner facial expression video frames without using additional auxiliary information, enabling the spatiotemporal graph convolutional network to directly process learner facial expression video frame data. In addition, this invention obtains spatiotemporal relational graph representations between different regions and different video frames of learner expressions through spatiotemporal graph construction and graph convolution operators with spatiotemporal correlation awareness. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 The overall process flowchart provided for embodiments of the present invention.

[0050] Figure 2 This is a schematic diagram of learner emotion cross-temporal graph representation learning provided in an embodiment of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] To address the problems existing in the prior art, one aspect of this invention provides a learner sentiment analysis method based on cross-spatiotemporal graph representation, comprising:

[0053] The input learner's facial expression video frames are divided into several equal-sized and non-overlapping sub-blocks, and the sub-blocks are subjected to graph node feature embedding processing to obtain the initial features of the graph nodes.

[0054] Based on the initial characteristics of the graph nodes, construct the spatiotemporal nearest neighbor graph structure for each sub-block;

[0055] Based on the cross-temporal nearest neighbor graph, the cross-temporal graph features of learner expressions output by the cross-temporal graph representation learning module are classified to predict the emotion category of the input learner expression video frames.

[0056] Optionally, the method further includes:

[0057] Acquire students' learning context videos through visual perception devices such as terminal cameras;

[0058] Face detection and face alignment are performed on the original learner expression video frames containing human faces using a face object detector.

[0059] The resolution of the face image after face alignment is scaled and the width and height are uniformly set to obtain the learner's expression video frames.

[0060] The learner's facial expression video frames are used as input data for the model.

[0061] Optionally, dividing the input learner facial expression video frames into several equal-sized, non-overlapping sub-blocks includes:

[0062] The first feature extraction block of the deep residual network is used to perform preliminary processing on the original input learner expression image to obtain a first feature map with a width and height of half that of the original image;

[0063] The second feature map of the second stage is extracted using the second feature extraction block of the deep residual network;

[0064] The third feature map of the third stage is extracted using the third feature extraction block of the deep residual network;

[0065] The fourth feature map of the third stage is extracted using the fourth feature extraction block of the deep residual network;

[0066] The fifth feature map of the third stage is extracted using the fifth feature extraction block of the deep residual network;

[0067] The initial features of the graph nodes are determined based on the first feature map, the second feature map, the third feature map, the fourth feature map, and the fifth feature map.

[0068] Optionally, the step of performing graph node feature embedding processing on the sub-blocks to obtain the initial features of the graph nodes includes:

[0069] The second feature map, the third feature map, and the fourth feature map are transformed into a target dimension feature space by feature mapping, and the fifth feature map is subjected to dimension-invariant feature linear transformation.

[0070] The transformed second, third, fourth, and fifth feature maps are then concatenated.

[0071] The feature map obtained by the concatenation operation is input into the feature fusion function to obtain the initial features of the graph nodes containing multi-scale learner facial expression feature information.

[0072] Optionally, constructing a spatiotemporal nearest neighbor graph for each sub-block based on the initial features of the graph nodes includes:

[0073] Before constructing a spatiotemporal nearest neighbor graph for each graph node, the initial features of the graph node are transformed once.

[0074] After performing the correlation-aware spatiotemporal graph convolution operation, a feature transformation is performed on all graph nodes to obtain a set of feature vectors as the graph node set;

[0075] For any layer of graph nodes in the graph node set, construct an expanded k-nearest neighbor graph;

[0076] Each graph node is treated as the center point of a local nearest neighbor graph. For any center node of a local nearest neighbor graph, graph convolution is performed on the local graph.

[0077] By using the node aggregation function of the association perception graph, the association relationship features between different nodes can be obtained;

[0078] The output of the association-aware graph node aggregation function is used as the input of the graph update function to obtain the updated node features;

[0079] After performing the correlation-aware spatiotemporal graph convolution and graph node feature transformation operations, a multilayer perceptron is used to perform feature mapping processing on all updated graph node features to obtain a graph node feature set, thus completing the construction of the spatiotemporal nearest neighbor graph for each sub-block.

[0080] Optionally, in the step of classifying the output learner expression spatiotemporal graph features representing the learner's emotion, the classification expression is:

[0081]

[0082] in, These are the learner's facial expression category labels predicted by the classifier; The graph representation of the i-th learner's facial expression video frame; AvgPool is the average pooling operator; the pooled global feature of the learner's facial expression. As input to the classifier; Φ(·;Θ Φ ) contains learnable parameter Θ Φ The feedforward neural network; LogSoftmax is the activation function.

[0083] Optionally, the method further includes:

[0084] The cross-entropy loss function with a labeled distribution smoothing regularization term is used as the training objective function for the learner sentiment analysis model;

[0085] The objective function is optimized using a gradient descent method to complete the training of the learner sentiment analysis model.

[0086] Another aspect of this invention provides a learner sentiment analysis device based on cross-spatiotemporal graph representation, comprising:

[0087] The first module is used to divide the input learner's facial expression video frames into several equal-sized and non-overlapping sub-blocks, and to perform graph node feature embedding processing on the sub-blocks to obtain the initial features of the graph nodes.

[0088] The second module is used to construct a spatiotemporal nearest neighbor graph for each sub-block based on the initial characteristics of the graph nodes.

[0089] The third module is used to classify the output learner expression cross-temporal graph features represented by the learner emotion cross-temporal graph based on the cross-temporal nearest neighbor graph, and predict the emotion category of the input learner expression video frame.

[0090] Another aspect of the present invention provides an electronic device, including a processor and a memory;

[0091] The memory is used to store programs;

[0092] The processor executes the program to implement the method described above.

[0093] Another aspect of this invention provides a computer-readable storage medium storing a program that is executed by a processor to implement the methods described above.

[0094] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0095] The specific implementation process of the present invention will now be described in detail with reference to the accompanying drawings:

[0096] To address the problems of existing technologies, this invention proposes a learner sentiment analysis method based on cross-spatiotemporal graph representation. On one hand, this method can segment local regions of learner facial expression video frames without using additional auxiliary information, enabling cross-spatiotemporal graph convolutional networks to directly process learner facial expression video frame data. On the other hand, by constructing cross-spatiotemporal graphs and using graph convolution operators with spatiotemporal awareness, spatiotemporal relational graph representations between different regions of learner facial expressions and different video frames are obtained, thereby effectively improving the prediction accuracy of learner sentiment analysis.

[0097] like Figure 1 As shown, the learner sentiment analysis method proposed in this invention mainly includes three parts: (1) a learner expression spatiotemporal segmentation and graph node embedding module; (2) a learner sentiment cross-spatiotemporal graph representation learning module; and (3) a learner sentiment spatiotemporal graph feature classification module. The learner expression spatiotemporal segmentation and graph node embedding module is mainly used to divide the input learner expression video frames into several equal-sized, non-overlapping sub-blocks, and to perform feature embedding processing on these blocks to obtain the initial features of the graph nodes. The cross-spatiotemporal graph representation learning module is mainly used to construct and learn the graph representation of each video frame block through cross-spatiotemporal nearest neighbor graph construction. This module mainly consists of multiple identical basic blocks, each containing a graph node transformation feature layer, a spatiotemporal nearest neighbor graph construction, an association-aware spatiotemporal graph convolutional layer, and a multilayer perceptron. The learner sentiment spatiotemporal graph feature classification module classifies the output spatiotemporal graph features of the learner sentiment cross-spatiotemporal graph representation learning module to predict the sentiment category of the input learner. The specific implementation of each component of the entire learner sentiment analysis method will be described in detail below.

[0098] 1. Learner Emotional Data Acquisition and Preprocessing:

[0099] 1) Sources of learner sentiment data:

[0100] This invention primarily targets learner emotional video data from online or offline classrooms. It acquires students' learning context videos through visual perception devices such as terminal cameras, and uses this learning video scene data as raw learner emotional data for analysis.

[0101] 2) Learner sentiment data preprocessing:

[0102] First, a face detector (e.g., MTCNN) is used to detect faces in the original video frames containing learner expressions and perform face alignment. Then, the resolution of the aligned face video frames is scaled and the width and height are uniformly set to W×H (e.g., 224×224). Finally, these aligned and uniformly sized expression video frames are used as input data for the spatiotemporal graph representation model.

[0103] 2. Learner Facial Expression Spatiotemporal Segmentation and Graph Node Embedding Module:

[0104] This invention extracts features from learner facial expression video data using a spatiotemporal graph convolutional network. Therefore, before performing the spatiotemporal graph convolution operation, the learner's facial expression video frames need to be divided into non-overlapping blocks, and feature embedding processing needs to be performed on these blocks. The embedding vector of each video frame block is used as the initial feature of the graph node. The learner facial expression spatiotemporal block division and graph node embedding module consists of two parts: multi-scale feature extraction of learner facial expressions and multi-scale learner facial expression feature fusion.

[0105] 2-1. Multi-scale feature extraction of learner facial expressions

[0106] This invention utilizes a pre-trained deep residual network as a video frame feature extractor for multi-scale learner facial expression features. Multiple levels of learner facial expression features are obtained at different feature extraction stages of the deep residual network and used as input to a multi-scale learner facial expression feature fusion module. The steps for multi-scale learner facial expression feature extraction are described below:

[0107] Step 1: Use the first feature extraction block B1 of the deep residual network (containing a convolutional kernel with a stride of 2 and a size of 7*7 and a max pooling operator with a stride of 2 and a size of 3*3) to process the original input learner face video frames X∈R. C ×H×W After initial processing, a feature map X1 with width and height equal to half that of the original video frame is obtained, and the number of output channels is C1. The formal description of this feature extraction block is as follows:

[0108] X1 = MaxPool(σ(BN(Conv)) 7*7 (X))))

[0109] Where Conv 7*7 This is a 7x7 convolution operation, BN is the batch normalization operator, σ is the Gaussian error linear unit activation function (GELU), and MaxPool is a 3x3 max pooling operator.

[0110] Step 2: Use the second feature extraction block B2 of the deep residual network to extract the feature map X2 of the second stage. B2 consists of m1 residual blocks ResBlock containing two 3*3 convolutional kernels, and the number of output channels is C2. The formal description of this feature extraction block is as follows:

[0111]

[0112] Where Conv 3*3 For a 3x3 convolution operation, This represents m1 residual blocks.

[0113] Step 3: Use the third feature extraction block B3 of the deep residual network to extract the feature map X3 of the third stage. B3 mainly contains m2 3*3 convolutional layers, with C3 output channels. The formal description of this feature extraction block is as follows:

[0114]

[0115] in, m e One residual block,

[0116] Step 4: Use the fourth feature extraction block B4 of the deep residual network to extract the feature map X4 of the fourth stage. B4 mainly contains m3 3*3 convolutional layers, with an output channel number of C4. The formal description of this feature extraction block is as follows:

[0117]

[0118] in, This represents m3 residual blocks.

[0119] Step 5: Use the fifth feature extraction block B5 of the deep residual network to extract the feature map X5 of the fifth stage. B5 mainly contains m4 3*3 convolutional layers, with an output channel count of C5. The formal description of this feature extraction block is as follows:

[0120]

[0121] in, This represents m4 residual blocks.

[0122] 2-2. Multi-scale learner facial expression feature fusion module

[0123] To obtain finer-grained and more discriminative initial features for graph nodes, this module fuses learner facial expression features at different scales extracted from the aforementioned stages. The steps of multi-scale learner facial expression feature fusion are described below:

[0124] Step 1: Perform feature mapping on the output feature maps of the second to fourth stages of the learner expression feature extractor, thereby transforming the feature maps of the three different levels into the target dimension feature space. This operation is formally expressed as follows:

[0125] X′ s =AvgPool(Conv 1*1 (X s ),s=(2,3,4)

[0126] Among them, Conv 1*1 Here, is a 1x1 convolution operator, s is the index of the feature extraction stage, and AvgPool is the average pooling operator.

[0127] Step 2: Perform a dimension-invariant feature linear transformation on the output feature map of the fifth stage. This operation is formally expressed as follows:

[0128] X′5=Conv 1*1 (X5)

[0129] in,

[0130] Step 3: Concatenate the features from the second to fifth stages after transformation. This operation is formally expressed as follows:

[0131] X c =Concat(X′2,X′3,X′4,X′5)

[0132] Here, Concat represents a feature map concatenation operation based on the channel dimension. C a =C′2+C′3+C′4+C′5.

[0133] Step 4: Input the concatenated feature map into the feature fusion function to obtain the initial feature vector of the graph nodes containing multi-scale facial expression feature information. This operation is formally expressed as follows:

[0134] X f =φ(X c ;Θ φ )

[0135] Wherein, φ(·;Θ φ ) indicates that it contains a learnable parameter Θ φ Multi-scale feature fusion functions (e.g., feedforward neural networks),

[0136] 3. Learner Emotional Cross-Spatiotemporal Graph Representation Learning Module:

[0137] This invention constructs a nearest neighbor graph structure across time and space using a spatiotemporal graph convolutional network, and obtains structured graph association features between different local spatial regions of learner expressions and across time domains. This addresses the problem that traditional learner expression analysis models struggle to flexibly capture long-distance dependencies and cross-video frame associations in learner expressions, thereby improving the prediction accuracy of learner sentiment analysis.

[0138] 3-1. Graph Node Feature Transformation:

[0139] To enhance the diversity of graph node features, this invention maps graph node feature vectors to the same feature domain before and after performing graph convolution operations. The steps of graph node feature transformation are described below:

[0140] Step 1: Before dynamically constructing the kNN graph for all nodes each time, the features of the graph nodes need to be transformed. This operation is formally expressed as follows:

[0141]

[0142] Among them, X l ∈R N×D This represents the feature set of graph nodes in the l-th layer. For those with learnable parameters A fully connected network.

[0143] Step 2: After performing the convolution operation on the association-aware graph, a feature transformation is then performed on all graph nodes. This operation is formally expressed as follows:

[0144]

[0145] Where RGConv represents the association-aware spatiotemporal graph convolution operator, It contains learnable parameters A fully connected network, Z l ∈R N×D This is the output of the l-th layer.

[0146] 3-2. Construction of the nearest neighbor graph across time and space:

[0147] To alleviate the oversmoothing problem of traditional graph convolution operations, this invention employs the Dilated k-Nearest Neighbors (DNN) algorithm to construct a local k-nearest neighbor graph structure for each graph node in all video frames. This graph construction algorithm not only effectively reduces the negative impact of oversmoothing in deep graph convolutional networks, but also increases the receptive field of graph convolution without increasing the size of the graph convolution kernel. Spatiotemporal k-nearest neighbor graph construction can be divided into spatial k-nearest neighbor graph construction and temporal k-nearest neighbor graph construction. The process of constructing the spatial k-nearest neighbor graph within a single video frame is as follows.

[0148] (1): The transformed feature vector set X = {x i ∈R D |i=1,2,…,N} are considered as the set of graph nodes v={v i |i=1,2,…,N}.

[0149] (2): For any graph node v at level l i Construct an expanded k-nearest neighbor graph, assuming {n1,n2,…,n} k×d} for v iGiven the sorted k×d neighbor nodes, select the k nearest neighbors from these k×d neighbor nodes. Then node v i The formal expression for the k nearest neighbor nodes is as follows:

[0150]

[0151] Where d represents the expansion rate.

[0152] Unlike the construction of spatial k-nearest neighbor graphs, the construction of temporal k-nearest neighbor graphs requires calculating the feature distance between nodes in different video frames. The construction process is as follows.

[0153] (1): Calculate the target node v of video frame t i,t All nodes v at the same position in other frames i,t′ High-dimensional feature distance dis(v) i,t ,v i,t′ ).

[0154] (2): From candidate node {v i,t′ Select the k cross-video frame nodes with the smallest distance to the target node from the set |i=1,…,N;t′=1,…,T}.

[0155] Since the spatiotemporal graph construction for each node in different layers is dynamically constructed based on the distance between node feature vectors, the edge and node features of any spatiotemporal graph constructed in layer l are different from the spatiotemporal nearest neighbor graph structure constructed in other layers.

[0156] 3-3. Relational Perception Spatial Graph Convolution:

[0157] Traditional graph convolution typically uses max or average pooling as the aggregation function, which leads to the graph node feature update process focusing only on the most salient neighbor nodes or averaging all neighbor nodes. This results in the inability to accurately capture the correlation between different local regions of the learner's expression. Therefore, this invention uses correlation-aware spatial graph convolution to obtain the relationship between various local regions of the learner's expression, so as to extract more fine-grained spatial correlation features of the learner's expression.

[0158] Step 1: Treat each graph node as the center point of a local kNN graph. For any center node v of a local kNN graph... i The general process of performing graph convolution operations (including spatial graph aggregation and spatial graph update) on this local graph can be formally expressed as:

[0159]

[0160] Where, x i l ∈R DFor node v i The feature vector of the l-th layer, For node v i The set of features of neighboring nodes, g(·; Θ g ) indicates that it contains a learnable parameter Θ g The aggregation function, f(·; Θ f ) contains learnable parameter Θ f The graph feature update function, x i l+1 ∈R D .

[0161] Step 2: By designing an aggregation function for the nodes of the association-aware graph, the association relationship features between different nodes are obtained. This aggregation function can be formally expressed as:

[0162]

[0163] Where h(·;Θ) □ ) indicates that it contains a learnable parameter Θ h The association learning function is used to quantify the importance of each edge in the local graph, x j l -x i l v is the center node of the l-th layer. i and neighbor node v j The edge features between them can model the relationships between graph nodes (i.e., the various local regions of the learner's expression).

[0164] Step 3: Use the output of the graph aggregation function as the input of the graph update function to obtain the updated node features, specifically the center node v of the local kNN graph. i The feature update operation can be formally expressed as:

[0165] x i l+1 =f([x i l ,g(·;Θ g )];Θ f )

[0166] Where [,] represents the feature concatenation operation, used to combine the feature x of the center point of the graph. i l Aggregate features g(·; Θ with spatial domain graph g The features are concatenated using the feature update function f(·Θ). f ) can be any differentiable function (e.g., a feedforward neural network).

[0167] 3-4. Relation-aware temporal graph convolution

[0168] The association-aware temporal graph convolution operation mainly includes two processes: temporal graph feature aggregation and temporal graph feature update.

[0169] Step 1: For the constructed k-nearest neighbor time-domain graph G i First, for any node v in the time-domain graph i,t Perform correlation-aware temporal graph aggregation operation:

[0170]

[0171] in This indicates that it contains learnable parameters. A cross-video frame local region association learning function is used to model the importance of each edge in the temporal nearest neighbor graph. Represents the central node v of the l-th layer i,t and time-series neighbor node v i,t′ The edge features between them can be used to model the relationships between nodes in the time-domain graph, that is, the changes in the same local area of ​​the learner's expression at different times.

[0172] Step 2: Then aggregate the above time-domain graph using the function. The output is used as the input to the time-domain graph update operator:

[0173]

[0174] Where [,] represents the feature concatenation operation, used to combine the features of the graph center point. With time-domain graph aggregation function The output features are concatenated, and the feature update function is used. It can be implemented using any differentiable function.

[0175] By using the aforementioned correlation-perceptual temporal graph aggregation and temporal graph update, we can capture the dynamic changes in learners' facial emotions, thereby obtaining the temporal changes in learners' emotions.

[0176] 3-5. Feature mapping based on multilayer perceptron:

[0177] After performing convolution and graph node feature transformation operations on the associated perceptron graph, this invention employs a multi-layer perceptron to further process the updated graph node features through feature mapping, thereby enhancing the graph node feature transformation capability. This process can be formally expressed as follows:

[0178] X l+1 =μ(Z) l ;Θ μ )+Z l

[0179] Where, μ(·;Θμ ) represents a Θ containing two fully connected layers with learnable parameters. μ A multilayer perceptron, and μ(·;Θ μ The neurons in the hidden layer of ) have a dimension of 4×D, Z l ∈R N×D The feature set of the l-th layer graph nodes is updated by graph convolution and transformed by feature transformation.

[0180] 4. Learner Emotion Spatiotemporal Map Feature Classification Module:

[0181] The output of the last layer, the Lth layer, in the spatiotemporal graph representation learning module is the learner's facial expression graph features. This invention uses a feedforward neural network as a classifier for the learner's facial expression graph features to predict the category of the input learner's facial expression. The classification of learner facial expression spatiotemporal graph features can be formally expressed as:

[0182]

[0183] in, This represents the graph representation of the i-th learner's image, where AvgPool is the average pooling operator, and the pooled result represents the global feature of the learner's facial expression. As input to the classifier, Φ(·;Θ Φ ) contains learnable parameter Θ Φ The feedforward neural network uses LogSoftmax as the activation function. These are the learner's facial expression category labels predicted by the classifier.

[0184] 5. Learner sentiment analysis model training:

[0185] Each module proposed in this invention is differentiable, therefore the entire graph representation learning network model is also end-to-end differentiable. Furthermore, this invention uses a cross-entropy loss function with a labeled distribution smoothing regularization term as the training objective function for the entire model. This loss function can be formally expressed as:

[0186]

[0187] Where N is the batch size of the learner's facial expression video frame samples, C is the number of learner facial expression categories, and q(·) is the label distribution smoothing regularization term, which is formally expressed as:

[0188]

[0189] Where ξ represents the probability numerical hyperparameter, and π(·) is the original label distribution.

[0190] Since the model proposed in this invention is end-to-end fully differentiable, the objective function G can be obtained by using gradient descent-based algorithms (such as stochastic gradient descent).SCE Optimization is performed to complete the training of the model.

[0191] In summary, compared with the prior art, the present invention has the following advantages:

[0192] On the one hand, the analysis model of this invention can segment local regions of learner expression video frames without using additional auxiliary information, enabling the spatiotemporal graph convolutional network to directly process learner expression video frame data. On the other hand, by constructing a spatiotemporal graph and using graph convolution operators with correlation awareness, spatiotemporal correlation graph representations between different regions and different video frames of learner expressions are obtained, thereby effectively improving the prediction accuracy of learner sentiment analysis.

[0193] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0194] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0195] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0196] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0197] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0198] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0199] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0200] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0201] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A learner sentiment analysis method based on cross-spatiotemporal graph representation, characterized in that, include: The input learner's facial expression video frames are divided into several equal-sized and non-overlapping sub-blocks, and the sub-blocks are subjected to graph node feature embedding processing to obtain the initial features of the graph nodes. Based on the initial features of the graph nodes, construct a spatiotemporal nearest neighbor graph for each sub-block; Based on the spatiotemporal nearest neighbor graph, the learner expression graph representation output by the learner emotion spatiotemporal graph representation learning module is classified to predict the emotion category of the input learner expression video frame. The learner emotion spatiotemporal graph representation learning module is used to construct and learn the graph representation of each video frame block. This module is composed of multiple identical basic blocks. Each basic block contains a graph node transformation feature layer, spatiotemporal nearest neighbor graph construction, association-aware spatiotemporal graph convolutional layer, and multilayer perceptron. The specific execution process of the learner emotion cross-temporal graph representation learning module is as follows: Before performing graph convolution operations, the feature vectors of graph nodes are mapped to the same feature domain to complete the graph node feature transformation; a local k-nearest neighbor graph structure is constructed for each graph node in all video frames; the relationship between various local regions of the learner's expression is obtained through correlation-aware spatial domain graph convolution, and the spatial domain correlation features of the learner's expression are extracted; correlation-aware temporal graph convolution processing is performed, including temporal graph feature aggregation and temporal graph feature update; after performing correlation-aware graph convolution, graph node feature transformation operations are performed, and after performing correlation-aware graph convolution and graph node feature transformation operations, a multilayer perceptron is used to perform feature mapping processing on all updated graph node features; The method also includes: The cross-entropy loss function with a labeled distribution smoothing regularization term is used as the training objective function for the learner sentiment analysis model; where, the cross-entropy loss function... The expression is: Where N is the batch size of the learner's facial expression video frame samples, and C is the number of learner facial expression categories. It is a label distribution smoothing regularization term; The objective function is optimized using a gradient descent method to complete the training of the learner sentiment analysis model.

2. The learner sentiment analysis method based on cross-spatiotemporal graph representation according to claim 1, characterized in that, The method further includes: Acquire videos of students' learning scenarios using visual perception devices; A face detector is used to detect faces in learning context videos containing learners' facial expressions and to perform face alignment operations. The resolution of the facial expression video frames after face alignment is scaled and the width and height are uniformly set to obtain the learner's facial expression video frames. The learner's facial expression video frames are used as input data for the model.

3. The learner sentiment analysis method based on cross-spatiotemporal graph representation according to claim 1, characterized in that, In the step of classifying the learner facial expression representations output by the learner emotion cross-temporal graph representation learning module, the classification expression is: in, These are the learner's facial expression category labels predicted by the classifier; Indicates the first A graphical representation of a learner's facial expressions in video frames; The average pooling operator is used; the global facial features of the learner after pooling are... As input to the classifier; For those with learnable parameters Feedforward neural network; This is the activation function.

4. An apparatus for implementing the learner sentiment analysis method based on cross-spatiotemporal graph representation as described in any one of claims 1-3, characterized in that, include: The first module is used to divide the input learner's facial expression video frames into several equal-sized and non-overlapping sub-blocks, and to perform graph node feature embedding processing on the sub-blocks to obtain the initial features of the graph nodes. The second module is used to construct a spatiotemporal nearest neighbor graph for each sub-block based on the initial characteristics of the graph nodes. The third module is used to classify the learner expression cross-temporal graph features output by the learner emotion cross-temporal graph representation learning module based on the cross-temporal nearest neighbor graph, and predict the emotion category of the input learner expression video frame.

5. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 3.

Images