An emotion tracking method, system, apparatus and storage medium

Abstract

The application discloses an emotion tracking method, system and device and a storage medium. First, a video frame image is acquired from a video to be tracked, so as to facilitate subsequent image processing and emotion recognition based on the image of the corresponding frame in the video, and emotion tracking of the video is realized. Then, super-resolution processing of the video frame image is performed through an image super-resolution model, so as to improve the image resolution and assist in improving the recognition accuracy. Finally, an emotion recognition result that is real and accurate and conforms to an actual application scenario is obtained through an emotion recognition model based on a shallow feature extraction module, a deep feature learning module and a classification module, and visualization and tracking adjustment of emotion data are realized. The application can effectively improve the recognition accuracy and realize efficient emotion recognition, and can be widely applied to the technical field of image processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to an emotion tracking method, system, device and storage medium. BACKGROUND

[0002] With the continuous acceleration of education informatization, the integration of artificial intelligence technology and education gradually becomes an indispensable part in the process of education and teaching. Various classroom integration applications take "learner-centered" as the concept, focus on learners in the process of technological innovation, and explore the occurrence of learning process and how to promote the enthusiasm of learners. Academic emotions have a certain influence on the stimulation and promotion of learners' learning cognition, and gradually attract the attention of researchers. With the development of artificial intelligence, relevant education products are gradually produced. On the one hand, these products provide more detailed learner academic emotion information for teachers or managers. On the other hand, due to the limitation of technology, the analyzed results of learner academic emotions are difficult to be widely recognized by teachers, and there are deficiencies in academic emotion analysis and learner academic emotion mining in classroom teaching.

[0003] At present, the measurement method of academic emotion mainly collects the emotional state of learners inside and outside the classroom, and evaluates and judges the emotion by using basic emotion theory. There are two categories of existing measurement methods. One is to use artificial measurement method, which collects the emotional state of learners in the learning process by using academic emotion questionnaire, such as AEQ questionnaire compiled by Pekrun et al., and the widely used “adolescent academic emotion questionnaire” compiled by scholars Dong Yan and Yu Guoliang, but this method usually needs to consume a lot of time and energy, and the scope of the survey object is limited, and it is relatively difficult to measure the academic emotion of learners in primary school and below. The second category is the measurement method based on technology, such as sensor sensing technology based on peripheral physiological measurement, or intelligent technology based on facial, voice and other body information observation system (such as FACS and its variants), which can more dynamically, detailedly and accurately collect the emotional state of learners than the artificial measurement method. Among them, the sensor sensing technology uses sensing devices (portable head-mounted devices, wristband devices, etc.) to detect and record various related physiological parameters (measure heart rate, skin electricity, etc.), extract the characteristic changes hidden in the parameters, and analyze the emotional state of learners, which has good robustness and objectivity, but the stability of some physiological parameters is poor, and is easily disturbed by noise in different environments, and the price of the sensing device is high, which cannot be used in a large range. Intelligent education technology uses expression recognition, speech recognition, behavior recognition and other deep learning models to analyze and judge the academic emotion changes of learners, such as speech recognition model collecting speech signals, analyzing and recognizing the rhythm, tone and some non-verbal pronunciation of the voice, but most of the speech recognition models are for non-streaming speech recognition, and less consider the real-time recognition problem of the model. In addition, in the classroom teaching scene, due to the interference of external factors (low face clarity, light change, etc.), the expression recognition model will have low ecological validity, and the recognition accuracy needs to be improved. Therefore, how to construct a learner academic emotion tracking method with stable results and suitable for classroom teaching scene, and provide more real and accurate analysis data for learners, teachers and managers, and promote the practical research of learner academic emotion analysis and intervention in actual teaching is a problem to be solved. SUMMARY

[0004] Therefore, the embodiments of the present application provide an emotion tracking method, system, device and storage medium, which can effectively improve the recognition accuracy and realize efficient emotion recognition.

[0005] In one aspect, the embodiments of the present application provide an emotion tracking method, comprising:

[0006] obtaining a video frame image of a to-be-tracked video;

[0007] The image super-resolution model is used for super-resolution processing on the video frame image to obtain a target image; wherein the image super-resolution model comprises an encoder, a decoder and a repair network;

[0008] The emotion recognition model is used for analyzing the target image to obtain an emotion recognition result; wherein the emotion recognition model comprises a shallow feature extraction module, a deep feature learning module and a classification module;

[0009] Based on the emotion recognition result, visual data and emotion tracking adjustment data are determined.

[0010] Optionally, the video frame image of the video to be tracked is obtained, comprising:

[0011] The video to be tracked is obtained.

[0012] The video to be tracked is extracted by multi-threading to obtain the video frame image.

[0013] Optionally, the image super-resolution model is used for super-resolution processing on the video frame image to obtain a target image, comprising:

[0014] The latent feature of the video frame image is extracted by the encoder;

[0015] The repair network is used for deep generation operation on the latent feature to obtain prior information; wherein the repair network is constructed based on a pre-trained generative adversarial network; the prior knowledge of the image generation direction in the specified latent space is obtained by pre-training;

[0016] The spatial feature is generated according to the latent feature by the decoder; the pixel attention operation is performed on the spatial feature to obtain the weighted spatial feature;

[0017] The prior information and the weighted spatial feature are fused to obtain the target image.

[0018] Optionally, the prior information and the weighted spatial feature are fused to obtain the target image, comprising:

[0019] The channel separation operation is performed on the prior information to obtain first prior information and second prior information;

[0020] The second prior information and the weighted spatial feature are aggregated in the channel to obtain the aggregated feature;

[0021] The global average pooling and two groups of convolution are performed on the aggregated feature to obtain a scaling factor;

[0022] The two groups of convolution are performed on the aggregated feature to obtain a translation factor;

[0023] The first prior information, the second prior information, the scaling factor and the translation factor are integrated to obtain an output feature;

[0024] Fusion reconstruction processing is performed based on the output features to obtain a target image.

[0025] Optionally, an emotion recognition model is used to analyze the target image to obtain an emotion recognition result, including:

[0026] The target image is preprocessed, and the preprocessed target image is subjected to granular information conversion by a shallow feature extraction module to obtain shallow features; wherein the preprocessing includes rotation rectification processing and illumination normalization processing; and the shallow feature extraction module includes a multi-layer convolution group structure.

[0027] The shallow features are subjected to multiple chain operations by a deep feature learning module to obtain deep features; wherein the deep feature learning module includes a double-path intercommunication residual structure block and a feature embedding structure.

[0028] The deep features are subjected to emotion classification by a classification module using a fully connected neural network to obtain the emotion recognition result; wherein the fully connected neural network has a random inactivation operation.

[0029] Optionally, the chain operation includes:

[0030] The target features are subjected to deep feature learning by a double-path multi-scale residual block to obtain learning features.

[0031] The learning features are subjected to first feature embedding processing by a feature embedding structure to obtain weight values; and the target features and the learning features are subjected to second feature embedding processing based on the weight values to obtain enhanced features.

[0032] The target features of the first chain operation are the shallow features, and the target features of the subsequent chain operations are the enhanced features obtained from the previous chain operation.

[0033] Optionally, the target features are subjected to deep feature learning by a double-path multi-scale residual block to obtain learning features, including:

[0034] The target features are subjected to double-path separable convolution operation based on data sharing, and the results of the double-path separable convolution operation are aggregated to obtain multi-scale features.

[0035] First feature weighting processing is performed based on the multi-scale features to obtain first weighted features; and second feature weighting processing is performed based on the multi-scale features to obtain second weighted features.

[0036] The first weighted features and the second weighted features are combined with random weight integration to obtain target weighted features.

[0037] The target weighted features are subjected to dimension reduction processing to realize identity mapping to obtain the learning features.

[0038] In another aspect, embodiments of the present application provide an emotion tracking system, comprising:

[0039] A first module configured to obtain a video frame image of a video to be tracked;

[0040] A second module configured to perform super-resolution processing on the video frame image using an image super-resolution model to obtain a target image, wherein the image super-resolution model comprises an encoder, a decoder and a repair network;

[0041] A third module configured to analyze the target image using an emotion recognition model to obtain an emotion recognition result, wherein the emotion recognition model comprises a shallow feature extraction module, a deep feature learning module and a classification module;

[0042] A fourth module configured to determine visualization data and emotion tracking adjustment data based on the emotion recognition result.

[0043] In another aspect, embodiments of the present application provide an emotion tracking device, comprising a processor and a memory;

[0044] The memory is configured to store a program;

[0045] The processor executes the program to implement the method as described above.

[0046] In another aspect, embodiments of the present application provide a computer readable storage medium, which stores a program, and the program is executed by a processor to implement the method as described above.

[0047] The present application also discloses a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device execute the method as described above.

[0048] The embodiment of the application first acquires a video frame image of a video to be tracked; an image super-resolution model is used to perform super-resolution processing on the video frame image to obtain a target image; wherein the image super-resolution model comprises an encoder, a decoder and a repair network; an emotion recognition model is used to analyze the target image to obtain an emotion recognition result; wherein the emotion recognition model comprises a shallow feature extraction module, a deep feature learning module and a classification module; based on the emotion recognition result, visualized data and emotion tracking adjustment data are determined. The embodiment of the application first acquires a video frame image from a video to be tracked, which facilitates subsequent image processing and emotion recognition based on the corresponding frame image in the video to realize emotion tracking of the video; then the image super-resolution model is used to perform super-resolution processing on the video frame image to improve the image resolution and assist in improving the recognition accuracy; finally, the emotion recognition model is used to obtain a real and accurate emotion recognition result that fits the actual application scenario based on the shallow feature extraction module, the deep feature learning module and the classification module, and then the visualized and tracked adjustment of the emotion data is realized. The embodiment of the application can effectively improve the recognition accuracy and realize efficient emotion recognition. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0050] Figure 1 A flowchart of an emotion tracking method provided by the embodiment of the present application is shown in the figure.

[0051] Figure 2 A whole framework flowchart of an emotion tracking method provided by the embodiment of the present application is shown in the figure.

[0052] Figure 3 A structure diagram of an image super-resolution model provided by the embodiment of the present application is shown in the figure.

[0053] Figure 4 A diagram of a PA spatial attention mechanism structure provided by the embodiment of the present application is shown in the figure.

[0054] Figure 5 A diagram of an ASFT structure provided by the embodiment of the present application is shown in the figure.

[0055] Figure 6 A structure diagram of an emotion recognition model provided by the embodiment of the present application is shown in the figure.

[0056] Figure 7 A diagram of a multi-layer convolution group structure provided by the embodiment of the present application is shown in the figure.

[0057] Figure 8 A schematic diagram of a dual-path interworking residual structure block provided for an embodiment of the present application is shown in FIG. 1.

[0058] Figure 9 A schematic diagram of a feature embedding structure provided for an embodiment of the present application is shown in FIG. 2.

[0059] Figure 10 A schematic diagram of a process of emotion recognition in a classroom teaching scenario provided for an embodiment of the present application is shown in FIG. 3.

[0060] Figure 11 A schematic diagram of academic emotion visualization provided for an embodiment of the present application is shown in FIG. 4.

[0061] Figure 12 A schematic diagram of an emotion clustering heat map provided for an embodiment of the present application is shown in FIG. 5.

[0062] Figure 13 A schematic diagram of emotion change visualization provided for an embodiment of the present application is shown in FIG. 6.

[0063] Figure 14 A schematic diagram of emotion duration visualization provided for an embodiment of the present application is shown in FIG. 7.

[0064] Figure 15 A schematic diagram of an overall process of academic emotion tracking analysis support provided for an embodiment of the present application is shown in FIG. 8. DETAILED DESCRIPTION

[0065] In order to make the objects, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0066] In one aspect, as shown in FIG. 1, an embodiment of the present application provides an emotion tracking method, comprising: Figure 1

[0067] S100, obtaining a video frame image of a video to be tracked;

[0068] It should be noted that in some embodiments, the method comprises: obtaining a video to be tracked; performing multi-thread extraction on the video to be tracked to obtain a video frame image. It should be further noted that the video frame image is not a specific designation for a specific image, and can include images extracted at each time of the video to be tracked.

[0069] Specifically, data is the basis for model analysis, and for the acquisition of image information, as shown in FIG. 2, the feature embedding structure provided for an embodiment of the present application is shown in FIG. 2. Figure 2 ​As shown, corresponding to the video frame information collection and processing part, in the recognition scene of the learning emotion, the embodiment of the present application can use the high-definition camera in the classroom to collect the learning emotion state video of the learner in the classroom (that is, the to-be-tracked video), and use the threading algorithm to extract the video frame in multiple threads to ensure the consistency of the time node of the analysis data obtained based on each learner. Among them, the image contains a large amount of information associated with the learning emotion recognition. In order to reduce the recognition complexity and reduce unnecessary background interference, the embodiment of the present application can also crop the face of the learner in the image to obtain the face image information of each learner as the video frame image. Correspondingly, the video frame image can include the face image information of each learner at each moment in the learning emotion state video of the learner in the classroom.

[0070] S200, using an image super-resolution model to perform super-resolution processing on the video frame image to obtain a target image;

[0071] It should be noted that in some embodiments, the image super-resolution model includes an encoder, a decoder and a repair network. It should be noted that in some embodiments, the image super-resolution model includes: extracting the latent features of the video frame image through the encoder; using the repair network to perform a deep generation operation on the latent features to obtain prior information; wherein the repair network is constructed based on a pre-trained generative adversarial network; obtaining prior knowledge of the image generation direction in the specified latent space through pre-training; generating spatial features according to the latent features through the decoder; performing pixel attention operation on the spatial features to obtain weighted spatial features; and performing fusion processing on the prior information and the weighted spatial features to obtain the target image. It should be further noted that the target image is not specified for a particular image, and it can include a set of images obtained by performing super-resolution processing on the video frame images extracted at each moment based on the to-be-tracked video. It can be understood that the number of images processed each time is subject to the processing specifications of the image super-resolution model, such as processing one image each time, and obtaining all target images through multiple image super-resolution models. It should be noted that the number of prior information is consistent with the scale of the weighted spatial features, so that the prior information and the weighted spatial features are one-to-one corresponding in the fusion processing.

[0072] In some embodiments, the fusion processing of the prior information and the weighted spatial features to obtain the target image includes: performing channel separation operation on the prior information to obtain first prior information and second prior information; performing channel aggregation on the second prior information and the weighted spatial features to obtain aggregated features; performing global average pooling and two groups of convolution on the aggregated features to obtain a scaling factor; performing two groups of convolution on the aggregated features to obtain a translation factor; performing integration processing based on the first prior information, the second prior information, the scaling factor and the translation factor to obtain output features; and performing fusion reconstruction processing based on the output features to obtain the target image.

[0073] Specifically, due to the fixed position of the camera and the principle of perspective (objects appear larger when closer, smaller when farther away), the acquired facial images of learners in the back rows of the classroom are usually blurry and noisy. To obtain higher-resolution facial images of learners, this invention proposes an image super-resolution model based on a GAN (Generative Adversarial Network), the network structure of which is as follows: Figure 3 As shown, corresponding Figure 2 In the super-resolution processing section, Encoder represents the encoder, Decoder represents the decoder, and Generator represents the repair network. In this embodiment of the invention, StyleGAN2 is used as the repair network. The specific workflow of super-resolution processing is as follows:

[0074] The first step is to extract the cropped low-quality images (i.e., video frame images) from the video frame. LR The data is fed into the image super-resolution model to generate feature priors that match the latent features of the original image.

[0075] ① Extract latent features from low-quality images using the U-Net encoder:

[0076] z = B enc (I LR )

[0077] Among them, B enc denoted as encoder, and z represents the latent feature obtained from the U-Net bottleneck.

[0078] ② Based on the pre-trained StyleGAN2 model, generate prior information containing a large amount of rich attribute information, texture details, and other high-frequency features:

[0079]

[0080] Among them, G StyleGAN2 (·) represents the generator operation in the pre-trained repair network StyleGAN2, MLP(·) represents the multilayer perceptron operation, and θ represents the parameters of each layer in the MLP. It is the i-th prior information generated.

[0081] The second step involves using the U-Net decoder to obtain multi-scale weighted spatial features and then using an attention mechanism to aggregate effective information.

[0082]

[0083] in, This represents the i-th decoder in the U-Net network. This represents the weighted spatial features generated by the i-th decoder. The spatial attention operation is represented by the following network structure: Figure 4As shown. In the PA spatial attention operation, a global average pooling (GAP) is first used for the input feature map to accept feature maps of any scale, followed by a 1x1 convolution (conv1x1) to replace the full connection operation, and the output result is sent into the swish activation function. Then a convolution (conv1x1) with an activation function sigmoid and a convolution kernel of 1 is used to obtain weight coefficients, and finally the weight coefficients are multiplied by the input features to obtain the weighted features (weighted spatial features).

[0084] Thirdly, high-quality video frame images are generated.

[0085] ①The significant weighted spatial feature analysis based on attention designed by the embodiment of the application fuses the weighted spatial features after weighting with prior information, and the module is named ASFT, and the network structure is as shown in Figure 5 The process of one-to-one fusion of prior information and weighted spatial features in a single ASFT is as follows: the prior information is divided into channels (c represents the number of channels of each feature / information) and half of the channels retain the original semantics (i.e., the first prior information) without change, and the prior information on the other half of the channels (i.e., the second prior information) is first aggregated with the weighted spatial features, and then two 1x1 convolutions (conv1x1) are used to obtain a scaling factor and a translation factor to complete affine transformation. It should be noted that before obtaining the scaling factor, the aggregated features are globally averaged pooled (GAP) to obtain a spatial dependence relationship, and the related formula is as follows: Figure 5

[0086]

[0087]

[0088] F ′ =2f cinv1 (f GAP (F u ))

[0089]

[0090] Wherein, f split (·) represents a channel separation operation, and represent the separated prior information (i.e., the first prior information and the second prior information), F u represents the feature after aggregation with the weighted spatial features (i.e., the aggregated feature), F ′ represents the scaling factor, f conv1 (·) represents a 1x1 convolution, 2f conv1 (F u ) represents the translation factor, f GAP ​(·) represents a global average pooling, represents the output feature of the i-th decoder after the integration processing.

[0091] ②The multi-scale features are fused to obtain a reconstructed clear face image (i.e., a target image), and a formula of the fusion reconstruction processing is as follows:

[0092]

[0093] wherein Fuse(·) represents a fusion operation of multiple features, I HR represents a target image after super-resolution.

[0094] Compared with the original learner face image I LR , the face image I HR after super-resolution has more prior information and less noise, which is beneficial to subsequent emotion recognition.

[0095] Due to the characteristics of the GAN model, only fictitious images can be generated, which is not completely applicable to classroom scene images. In order to focus on the face image of the learner in the real teaching scene, the inverse mapping method is used in the embodiment of the present application to obtain the pre-trained prior knowledge, specify the direction of image generation in the latent space, and use the attention mechanism to focus on the effective information in the feature fusion process based on affine change to synthesize real images. On the one hand, it reduces the interference of external factors on the face image; on the other hand, it injects prior information to enhance the face detail texture and generate more robust feature expression.

[0096] S300, using an emotion recognition model to analyze the target image and obtain an emotion recognition result;

[0097] The emotion recognition model includes a shallow feature extraction module, a deep feature learning module, and a classification module. It should be noted that some embodiments include: preprocessing the target image, converting the granularity information of the preprocessed target image through the shallow feature extraction module to obtain shallow features; wherein the preprocessing includes rotation rectification processing and illumination normalization processing; the shallow feature extraction module includes a multi-layer convolution group structure; the shallow feature extraction module is used to perform multiple chain operations on the shallow features to obtain deep features; wherein the deep feature learning module includes a double-path intercommunication residual structure block and a feature embedding structure; the classification module is used to perform emotion classification on the deep features using a fully connected neural network to obtain an emotion recognition result; wherein the fully connected neural network has a random inactivation operation.

[0098] ​In some embodiments, the chain operation includes: performing deep feature learning on the target feature through the dual-path multi-scale residual block to obtain learning features; performing first feature embedding processing on the learning features through the feature embedding structure to obtain weight values; and performing second feature embedding processing on the target feature and the learning features based on the weight values to obtain enhanced features; wherein the target feature of the first chain operation is a shallow feature, and the target feature of the subsequent chain operation is the enhanced features obtained by the last chain operation.

[0099] In some embodiments, the deep feature learning on the target feature through the dual-path multi-scale residual block to obtain learning features includes: performing data sharing based dual-path separable convolution operation on the target feature, and aggregating the results of the dual-path separable convolution operation to obtain multi-scale features; performing first feature weighting processing on the multi-scale features to obtain first weighted features; and performing second feature weighting processing on the multi-scale features to obtain second weighted features; combining the first weighted features and the second weighted features to obtain target weighted features according to random weights; and performing dimension reduction processing on the target weighted features to realize identity mapping to obtain the learning features.

[0100] Specifically, the emotion recognition model is the most important module in the learner academic emotion tracking, and the ability of the model to extract effective features directly affects the final recognition result. In the embodiment of the application, the model is constructed by using the attention mechanism and the multi-scale feature set, which corresponds to Figure 2 The emotion recognition part deeply mines the visual semantic information conveyed by the face image, and the network structure thereof refers to Figure 6 The specific steps of emotion recognition are as follows:

[0101] In the first step, the face image after the preprocessing operation is sent into the shallow feature extraction module to complete the conversion of the image to the granular information.

[0102] ①The input high-definition face image is preprocessed to complete face alignment and normalization processing:

[0103]

[0104] Among them, is a rotation rectification operation, is a homomorphic filtering operation (through which the illumination normalization is realized), ILL HR is the learner face image after preprocessing.

[0105] ②The embodiment of the application can use a 4-layer convolution group structure as the shallow feature extraction module to extract shallow features, as shown in Figure 7 , which gradually converts the input image channel to 512 channels:

[0106] T S = Convn (ILL HR )

[0107] where, Conv n (·) represents a multi-layer convolution group (convolution + maximum pooling), T S represents the extracted shallow features.

[0108] Secondly, the shallow features are input into the stacked network structure block (i.e. the deep feature learning module) to complete the learning of the deep features of the learner's face image: based on the double-path intercommunication residual structure block, the multi-scale features are calculated; based on the attention-based residual block (i.e. the feature embedding structure), the feature embedding is enhanced.

[0109] ①The attention-based multi-scale residual block (double-path intercommunication residual structure block + attention-based residual block) designed by the present application is used to learn deep features from shallow features, and the residual block structure is as shown in the left part of Figure 8 .

[0110] A. Firstly, the upper and lower two paths of the double-path intercommunication residual structure block are used to extract features respectively, and the data is shared between the paths. For the same input, the upper path performs separable convolution operation with a convolution kernel of 3, and the lower path performs separable convolution operation with a convolution kernel of 5. Different features are learned each time, and three layers of separable convolution operation are performed. Among them, except for the first layer, the input of the separable convolution operation of the other layers is the sum of the convolution operation results of the two paths of the previous layer, and after the third layer of separable convolution calculation is completed, the results of the upper and lower two paths are spliced in the channel and sent to the next layer operation. The formula description of the process is as follows:

[0111]

[0112] T L =P i ||Q i

[0113] wherein L is the number of double-path layers, P is the output feature of the i-1 layer in the upper path, Q is the output feature of the i-1 layer in the lower path, f d_conv3 (·) is separable convolution operation with a convolution kernel of 3, f d_conv5 (·) is separable convolution with a convolution kernel of 5, || is Concat operation, and T L is the aggregated multi-scale feature.

[0114] B. Then, the multi-scale features are subjected to attention operation through the feature embedding structure. The channel attention mechanism CA structure designed by the embodiment of the present application is as shown in Figure 8As shown in the right graph, the attention weight of the designed double-path is obtained, the first weighting processing of one path includes: using linear change using MaxPool, then using 1*1 convolution (Conv1*1) to complete the full connection operation, and then using swish activation function to obtain feature weighting coefficient A1, and then point multiplying with input feature (i.e. multi-scale feature) to obtain the first weighted feature; the second weighting processing of the other path includes: using linear change using average pooling (AvgPool) operation, then using 1*1 convolution (Conv1*1) to complete the full connection operation, and then using sigmoid activation function to obtain feature weighting coefficient A2, and then point multiplying with input feature (i.e. multi-scale feature) to obtain the second weighted feature. After obtaining the weighted features of the double paths, the two weighted features are randomly added with weights n1, n2 (n1+n2=1) and integrated, and finally the weighted feature (i.e. target weighted feature) is obtained, and the process is represented as:

[0115] A1=δ swish (f conv1 (f MaxPool (T L )))*T L

[0116] A2=δ sigmoid (f conv1 (f AvgPool (T L )))*T L

[0117] T CA =A1*n1+A2*n2

[0118] Wherein, f MaxPool (·) represents the maximum pooling operation, f AvgPool (·) represents the average pooling operation, δ swish represents the activation function swish, δ sigmoid represents the activation function sigmoid, A1 and A2 represent the attention weighting coefficient, n1, n2 represent the attention path weight. T CA is the target weighted feature after executing channel attention.

[0119] C. The target weighted feature subjected to the attention mechanism is processed by dimension reduction, and identity mapping is completed:

[0120] T RA =δ relu (f conv1 (T CA ))+T S

[0121] Wherein, δ relu represents the activation function Relu, TRA Learning features outputted at last by the deep feature learning module representing the attention mechanism based double-path multi-scale feature learning block.

[0122] ②In order to fill the lost features after convolution, the attention mechanism is used to obtain the salient features, which are then integrated with the shallow features, and the network structure is as shown in Figure 9 .

[0123] The first feature embedding processing step is as follows: firstly, the learning features are subjected to global average pooling (GAP) and reshaping, and the size is changed from (B, C, H, W) to (BxC, 1, 1, 1), and then the weight value m is obtained after two times of dense (two times of dense between which an activation function Relu is introduced) and an activation function (sigmoid). The second feature embedding processing step is as follows: in order to effectively fill the lost information, the shallow features T S are inputted and multiplied with the weight value m, and the output result is multiplied with the salient features to obtain the strengthened features T E (i.e. strengthened features), and the process is represented as:

[0124]

[0125] wherein, is the shallow feature embedding operation (i.e. the first feature embedding processing + the second feature embedding processing), and T E represents the strengthened features after the shallow feature embedding.

[0126] The attention based double-path intercommunication residual structure block and the feature embedding structure are regarded as a module (double network block module), and the shallow features are subjected to n times of chain operation:

[0127]

[0128] wherein, is the i-th double network block module, and T A-E is the final deep feature. The processing procedure of the aforementioned double-path intercommunication residual structure block and the feature embedding structure is taken as an example of the shallow features, i.e. the first chain operation, and the input features of the subsequent chain operation are the strengthened features obtained by the last chain operation.

[0129] Thirdly, the full connection neural network is used for emotion classification. In order to reduce the network calculation amount, the Dropout operation is added between the full connection layer and the softmax activation function:

[0130] S N = δ softmax (λdropout (FC(T A-E )))

[0131] Where FC stands for fully connected operation, and λ dropout It's a Dropout operation, δ softmax It is the activation function softmax, S N It is a sequence containing the results of the model output, which may include emotion recognition results obtained by classifying the corresponding video frame images of each recognition object (such as the learner) at each moment of the video to be tracked.

[0132] Deep learning-based emotion tracking methods are significantly less accurate than those used in laboratory settings due to the influence of external environments. This invention addresses this issue by using a GAN-based super-resolution algorithm to reconstruct learners' classroom facial images in the initial stages of emotion recognition. Simultaneously, to reduce external noise interference, illumination normalization and a dual-path interconnected multi-scale residual structure based on an attention mechanism are employed to fuse prior facial knowledge, improving the emotion recognition model. Separable convolutions are used to reduce redundant parameters, deeply mining learners' emotional features and providing a comprehensive tracking of their academic emotions. To preserve original features and further enrich feature representations, a cascaded structure is used to sequentially fuse shallow features with subsequent multi-scale feature layers. This combination of methods improves the accuracy and robustness of emotion recognition.

[0133] To provide a detailed explanation of the technical solution of the present invention, a classroom teaching scenario is used as an example, and the embodiments of the present invention are further illustrated in conjunction with the accompanying drawings. The following is an explanation of the present invention and should not be regarded as a limitation thereof.

[0134] In a classroom setting, each learner's class time is the same, T. Assuming we take duration s as the breakpoint, we collect learner academic sentiment data for n = T / s time points, i.e., when time t... n At that time, the learner's academic mood was Let there be i types of learning emotions among learners in the classroom, denoted as Q = {Q1, Q2, ..., Q...} i}

[0135] Convolutional neural network-based emotion recognition models are used in the field of online and offline learner emotion recognition to assist in analyzing learners' learning status and academic emotions. However, in real-world scenarios, these models often fail to achieve ideal results due to noise effects such as image clarity and lighting intensity. To address the problem of low efficiency in tracking learners' academic emotions in real-world scenarios caused by external interference, this invention focuses on facial image preprocessing and emotion recognition model structure.

[0136] like Figure 10As shown, firstly, after acquiring video frames from the teaching scene, face extraction is further performed. Before emotion recognition, the face images are preprocessed using a deep neural network. A GAN-based image super-resolution network is used to reconstruct the cropped learner face region image, improving the clarity of the face image. Then, the emotion recognition model is improved by using illumination normalization and a dual-path interconnected multi-scale residual network structure based on an attention mechanism to increase the robustness and accuracy of the emotion recognition algorithm. Finally, the academic emotion representation is obtained.

[0137] S400: Based on emotion recognition results, determine visualization data and emotion tracking adjustment data.

[0138] Based on the emotion recognition results (such as academic emotions), further emotion visualization can be performed. The specific steps are as follows:

[0139] The visualization scheme designed in this invention is as follows: Figure 11 As shown, further steps can be taken through... Figure 12 The categories of academic emotions are summarized and aggregated. Figure 13 The process of changes in overall academic attitude is presented. Figure 14 Describe the temporal distribution of academic emotions at different levels of the target audience. Specifically:

[0140] Figure 12 Scatter plots were used to categorize academic emotions. In the plot, dots represent individual learners, and colors represent academic emotions. Learners were clustered based on their displayed academic emotions. When the mouse hovers over a dot, the learner's information (avatar, name, seat, and current academic emotion) is displayed. This visualization directly reflects the proportion of academic emotions occurring at any given time and identifies learners exhibiting these emotions.

[0141] Figure 13 The overall flow of academic emotions is depicted using connected circles. The horizontal axis represents timestamps, and the vertical axis represents the proportion of academic emotions occurring among all learners. Each academic emotion has a corresponding colored circle. The lines connecting the circles have different widths; the academic emotion that dominates throughout the learning process will use the widest line. This visualization allows for comparison of the changing trends and proportions of different academic emotions from an overall timeline perspective.

[0142] Figure 14The Gantt chart is used to describe the time change state of academic emotions at different object levels. For the group or whole learner level, the time point and time length of all academic emotions in the learning process are visualized. For the individual learner level, the academic emotions in the learning stage are presented. In addition, for the visualization graph at any level, when the mouse hovers over a certain bar, the information (number of times, time length) contained in the academic emotion at that time point will be presented.

[0143] The emotion recognition result of the above emotion recognition model can support the tracking analysis of learner academic emotions, obtain tracking adjustment data (including tracking report and emotion adjustment strategy), and generate visual interpretation according to the information flow generated in the emotion tracking model.

[0144] Suppose that students in a class are learning, the camera records the whole expression change of the students in the class during the learning process, and now the academic emotions of the students in the class need to be tracked, as shown in Figure 15 The overall process is as follows:

[0145] 1. Video frames M = {m1, m2,..., m n} of the learning video V are extracted, and then sent to the combined model constructed by the application. Through the super-resolution based on GAN and the attention emotion recognition model based on double-path multi-scale, the academic emotion sequence S N = {S1, S2,..., S i} is obtained.

[0146] 2. The emotion index S i of each academic emotion Q index is calculated. If S index >1, Q i is a positive emotion, otherwise, it is a negative emotion.

[0147] 3. According to the proportion of academic emotion Q i occurring in the group of learners, the level range R = {R s , R m , R l} of Q i is determined.

[0148] 4. The whole academic emotion and the corresponding information appearing in the learning process of the students in the class are visualized and an analysis report is formed.

[0149] 5. According to the data presented by the model and the experience of teachers, the causes of negative academic emotions of learners are determined from the aspects of internal characteristics of learners external environmental factors and mixed elements Origin = {O in , Oen O mix}

[0150] 6. Adjust strategies based on the causes of negative academic emotions, starting with teacher conversations. Suggestions and guidance Social support We can start with three types of strategies.

[0151] 7. Further follow up and observe changes in learners' academic and emotional states.

[0152] This invention also allows for data- and architecture-based visualization of academic emotions. Using scatter plots, flowcharts, and Gantt charts, it analyzes academic emotions from the perspectives of category, proportion, and spatiotemporal dimensions, displaying the distribution and dynamic changes of academic emotions. The visualization is further enriched by dividing emotions into three cluster levels: individual, group, and overall. Furthermore, the emotion tracking method based on this invention enables academic emotion tracking and analysis based on GAN image super-resolution.

[0153] On the other hand, embodiments of the present invention provide an emotion tracking system, comprising: a first module for acquiring video frame images of a video to be tracked; a second module for performing super-resolution processing on the video frame images using an image super-resolution model to obtain a target image; wherein the image super-resolution model includes an encoder, a decoder, and a repair network; and a third module for analyzing the target image using an emotion recognition model to obtain an emotion recognition result; wherein the emotion recognition model includes a shallow feature extraction module, a deep feature learning module, and a classification module.

[0154] The content of the method embodiments of the present invention is applicable to the system embodiments. The specific functions implemented in the system embodiments are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above methods.

[0155] Another aspect of the present invention provides an emotion tracking device, including a processor and a memory;

[0156] Memory is used to store programs;

[0157] The processor executes the program as described above.

[0158] The content of the method embodiments of the present invention is applicable to the device embodiments. The specific functions implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above methods.

[0159] Another aspect of this invention provides a computer-readable storage medium storing a program that is executed by a processor to implement the method described above.

[0160] The contents of the method embodiments of the present application are applicable to the computer readable storage medium embodiments, the computer readable storage medium embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method.

[0161] The computer program product or computer program of the embodiment of the present application also discloses a computer program product or computer program, which comprises computer instructions stored in a computer readable storage medium. The processor of the computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above method.

[0162] In some alternative embodiments, the functions / operations mentioned in the block diagram can not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, two blocks shown in succession can actually be executed substantially simultaneously or the blocks can sometimes be executed in reverse order. In addition, the embodiments presented and described in the flowcharts of the present application are provided by way of example, with the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operations and logical flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of larger operations are independently executed.

[0163] In addition, although the present application is described in the context of functional modules, it should be understood that one or more of the functions and / or features can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules, unless otherwise specified. It can also be understood that detailed discussion of the actual implementation of each module is unnecessary for understanding the present application. Rather, given the properties, functions and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the routine skill of an engineer. Therefore, those skilled in the art can implement the present application set forth in the claims without undue experimentation using ordinary skill. It can also be understood that the disclosed specific concepts are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the appended claims and their equivalents.

[0164] If the functions are implemented in software, the functions can be stored in or implemented as one or more software modules on a computer-readable medium. The computer-readable medium can include a floppy disk, a CD-ROM, a DVD, a Blu-ray Disc, a Flash memory, a DRAM, a SRAM, a cache, a register, other suitable computer-readable media, or any suitable combination thereof. The computer-readable medium can be encoded with one or more machine-readable instructions executable by a machine, including a processor, a processor core, a microprocessor, a microcontroller, a microcomputer, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or any suitable combination thereof. The software module can include one or more instructions executable by the machine, including instructions for the processor or any portion thereof. The software module can include instructions executable by the machine to cause the machine to perform a process, such as the process described herein.

[0165] The logic and / or steps represented in the flowcharts and / or otherwise described herein, for example, can be embodied in non-transitory computer-readable media, which can be executed by an instruction execution device, apparatus, or equipment such as a computer-based device, a processor-based device, or other device that can fetch the instructions from the instruction execution device, apparatus, or equipment and execute the instructions. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution device, apparatus, or equipment.

[0166] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can also be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via an optical scanner, then compiled, interpreted, or otherwise processed, and stored in a computer memory in a form that can be later executed by a computer. In some embodiments, the computer-readable medium can be a non-transitory computer-readable medium.

[0167] It should be understood that aspects of the application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, various steps or methods can be implemented in software or firmware which is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following techniques can be used to implement the hardware used to implement the described functionality: discrete logic circuitry having logic gates for implementing logic functions upon data signals, application specific integrated circuits having logic gates, field programmable gate arrays (FPGAs), and so on, or a combination thereof.

[0168] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in one or more embodiments or examples.

[0169] Although embodiments of the present application have been shown and described, it would be appreciated by those skilled in the art that changes can be made in these embodiments without departing from the principles and spirit of the application, the scope of which is defined in the claims and their equivalents.

[0170] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. An emotion tracking method, characterized in that, include: Acquire video frame images of the video to be tracked; The video frame images are super-resolution processed using an image super-resolution model to obtain the target image; wherein the image super-resolution model includes an encoder, a decoder, and a repair network; The target image is analyzed using an emotion recognition model to obtain emotion recognition results; wherein, the emotion recognition model includes a shallow feature extraction module, a deep feature learning module, and a classification module; The step of using an emotion recognition model to analyze the target image and obtain emotion recognition results includes: The target image is preprocessed, and the shallow feature extraction module performs granular information conversion on the preprocessed target image to obtain shallow features; wherein, the preprocessing includes rotation rectification and illumination normalization; the shallow feature extraction module includes a multi-layer convolutional group structure; The deep feature learning module performs multiple chain operations on the shallow features to obtain deep features; wherein, the deep feature learning module includes a dual-path interconnected residual structure block and a feature embedding structure; The classification module uses a fully connected neural network to classify the deep features for emotion, thereby obtaining the emotion recognition result; wherein the fully connected neural network has a random deactivation operation. The chained operations include: Deep feature learning is performed on the target features through the dual-path multi-scale residual block to obtain the learned features; The learned features are subjected to a first feature embedding process using the feature embedding structure to obtain weight values; based on the weight values, the target features and the learned features are combined to perform a second feature embedding process to obtain enhanced features. In this process, the target feature of the first chain operation is a shallow feature, and the target feature of subsequent chain operations is the enhanced feature obtained from the previous chain operation. The process of performing deep feature learning on the target features through the dual-path multi-scale residual block to obtain learned features includes: The target features are subjected to a two-path separable convolution operation based on data sharing, and the results of the two-path separable convolution operation are aggregated to obtain multi-scale features; Based on the multi-scale features, a first feature weighting process is performed to obtain a first weighted feature; and, based on the multi-scale features, a second feature weighting process is performed to obtain a second weighted feature; The target weighted feature is obtained by integrating the first weighted feature and the second weighted feature with random weights. The target weighted features are subjected to dimensionality reduction processing to achieve identity mapping and obtain learned features; Based on the emotion recognition results, visualization data and emotion tracking adjustment data are determined.

2. The emotion tracking method according to claim 1, characterized in that, The acquisition of video frame images of the video to be tracked includes: Obtain the video to be tracked; The video to be tracked is extracted using multi-threaded extraction to obtain video frame images.

3. The emotion tracking method according to claim 1, characterized in that, The step of using an image super-resolution model to perform super-resolution processing on the video frame images to obtain the target image includes: The encoder extracts latent features from the video frame images; The repair network is used to perform deep generation operations on the latent features to obtain prior information; wherein, the repair network is constructed based on a pre-trained generative adversarial network; prior knowledge of the image generation direction in a specified latent space is obtained through the pre-training. The decoder generates spatial features based on the latent features; pixel attention operations are then performed on the spatial features to obtain weighted spatial features. The prior information and the weighted spatial features are fused to obtain the target image.

4. The emotion tracking method according to claim 3, characterized in that, The process of fusing the prior information and the weighted spatial features to obtain the target image includes: Perform a channel separation operation on the prior information to obtain first prior information and second prior information; The second prior information is combined with the weighted spatial features through channel aggregation to obtain aggregated features; The aggregation features are subjected to global average pooling and two sets of convolutions to obtain the scaling factor; The aggregation features are subjected to two sets of convolutions to obtain the translation factor; The output features are obtained by integrating the first prior information, the second prior information, the scaling factor, and the translation factor. The target image is obtained by performing fusion and reconstruction processing based on the output features.

5. An emotion tracking system, characterized in that, include: The first module is used to acquire video frame images of the video to be tracked; The second module is used to perform super-resolution processing on the video frame image using an image super-resolution model to obtain the target image; wherein, the image super-resolution model includes an encoder, a decoder and a repair network; The third module is used to analyze the target image using an emotion recognition model to obtain emotion recognition results; wherein, the emotion recognition model includes a shallow feature extraction module, a deep feature learning module, and a classification module; The step of using an emotion recognition model to analyze the target image and obtain emotion recognition results includes: The target image is preprocessed, and the shallow feature extraction module performs granular information conversion on the preprocessed target image to obtain shallow features; wherein, the preprocessing includes rotation rectification and illumination normalization; the shallow feature extraction module includes a multi-layer convolutional group structure; The deep feature learning module performs multiple chain operations on the shallow features to obtain deep features; wherein, the deep feature learning module includes a dual-path interconnected residual structure block and a feature embedding structure; The classification module uses a fully connected neural network to classify the deep features for emotion, thereby obtaining the emotion recognition result; wherein the fully connected neural network has a random deactivation operation. The chained operations include: Deep feature learning is performed on the target features through the dual-path multi-scale residual block to obtain the learned features; The learned features are subjected to a first feature embedding process using the feature embedding structure to obtain weight values; based on the weight values, the target features and the learned features are combined to perform a second feature embedding process to obtain enhanced features. In this process, the target feature of the first chain operation is a shallow feature, and the target feature of subsequent chain operations is the enhanced feature obtained from the previous chain operation. The process of performing deep feature learning on the target features through the dual-path multi-scale residual block to obtain learned features includes: The target features are subjected to a two-path separable convolution operation based on data sharing, and the results of the two-path separable convolution operation are aggregated to obtain multi-scale features; Based on the multi-scale features, a first feature weighting process is performed to obtain a first weighted feature; and, based on the multi-scale features, a second feature weighting process is performed to obtain a second weighted feature; The target weighted feature is obtained by integrating the first weighted feature and the second weighted feature with random weights. The target weighted features are subjected to dimensionality reduction processing to achieve identity mapping and obtain learned features; The fourth module is used to determine visualization data and emotion tracking adjustment data based on the emotion recognition results.

6. An emotion tracking device, comprising a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 4.

Images