Student sentiment analysis method and system based on multimodal dynamic memory large model

Abstract

The present invention discloses a student emotion analysis method and system based on a multimodal dynamic memory large model, belonging to the field of multimodal emotion analysis. The method of the present invention first obtains facial expression images of different students when watching online education courses and their corresponding real emotion state category labels to form a training data set, and trains the multimodal dynamic memory large model based on the data set. During the training process, the image encoder and the text encoder are prompted to be fine-tuned, and the visual aggregation representation generated during the historical training process is stored in the dynamic feature space, and the model parameters are optimized by the cross entropy loss; finally, the facial expression image to be classified is input into the trained multimodal dynamic memory large model, and the emotion state category prediction result of the facial expression image is output. The method of the present invention can be effectively applied to the field of education, which is conducive to teachers understanding the emotional state of students, thereby improving students' participation and learning effect.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal sentiment analysis, and in particular relates to a student sentiment analysis method and system based on a multimodal dynamic memory large model. Background Art

[0002] Student sentiment analysis is a method that uses technology to assess and understand students' emotions and emotional states in a virtual learning environment. It uses artificial intelligence and machine learning technologies to obtain information from multiple data sources, such as students' facial expressions, voice intonation, text input, mouse clicks, and browsing habits. These data are processed by complex algorithms to reveal students' emotional responses during the learning process, such as happiness, confusion, anxiety, concentration, and frustration. The goal of sentiment analysis is to help teachers and educational institutions better understand students' emotional dynamics so that they can intervene and support them in a timely manner. For example, when the system detects that students are confused or frustrated, teachers can provide additional explanations or resources to help them overcome difficulties. Through sentiment analysis, online classes can not only meet students' personalized needs more accurately, but also enhance students' learning experience and enthusiasm, making distance education more humane and effective. It provides great potential and prospects for improving the quality of online education.

[0003] Student sentiment analysis faces many challenges and technical issues. First, the accuracy of emotion recognition is a major problem. Students' emotional expressions vary due to cultural, personality and environmental differences, and emotion recognition based solely on facial expressions or voice features may lead to misjudgment. Second, real-time processing and analysis of large amounts of data places high demands on the system's computing power and response speed to ensure that the system can provide accurate sentiment analysis results in a short period of time. Another important challenge is how to effectively apply sentiment analysis methods to the field of education. Teachers not only need to understand students' emotional states, but also need specific advice and tools to adjust teaching strategies to improve student engagement and learning outcomes. Summary of the invention

[0004] The purpose of the present invention is to solve the problems existing in the prior art and provide a student sentiment analysis method and system based on a multimodal dynamic memory large model.

[0005] In order to achieve the above-mentioned invention object, the present invention specifically adopts the following technical solutions:

[0006] In a first aspect, the present invention provides a student sentiment analysis method based on a multimodal dynamic memory large model, which comprises the following steps:

[0007] S1: Obtain facial expression images of different students when watching online education courses and their corresponding real emotional state category labels;

[0008] S2: The facial expression images are input into the multimodal dynamic memory large model for training. During the training process of the multimodal dynamic memory large model, each facial expression image is first projected by a linear projection layer. Each facial expression image corresponds to a token embedding sequence. The token embedding sequence is added to the spatial position encoding result of the facial expression image and the initialized classification token representation as the input token sequence. The input token sequence is input into the image encoder, and the visual cue learning method is used to cue fine-tune each encoding layer of the image encoder, and finally a visual aggregate representation is obtained. The description related to the facial behavior is processed by the text segmentation tool to form a segmented text sequence, and the segmented text sequence is compared with the identifiable The learned text representation is input into the text encoder together to obtain the text aggregation representation, the visual aggregation representation is multiplied by the transposed text aggregation representation and then outputted through Softmax to obtain the first prediction probability, the visual aggregation representation is input into the output module together with the historical features stored in the dynamic feature space to obtain the visual classification representation, the visual aggregation representation is stored in the dynamic feature space as a new feature, the visual aggregation representation is multiplied by the transposed visual classification representation and then outputted through Softmax to obtain the second prediction probability, the weighted sum of the first prediction probability and the second prediction probability is taken as the final prediction probability, and the cross entropy loss is calculated based on the final prediction probability and the true emotional state category label to update the parameters of the multimodal dynamic memory large model;

[0009] S3: Input the facial expression image to be classified into the trained multimodal dynamic memory large model. When the dynamic feature space reaches the preset capacity threshold, calculate the comprehensive score corresponding to the facial expression image to be classified, and judge whether the dynamic feature space needs to be updated based on the comprehensive score, and finally output the emotional state category prediction result of the facial expression image.

[0010] Based on the above scheme, each step can be implemented in the following preferred specific manner.

[0011] As a preferred embodiment of the first aspect, in step S1, the real emotional state category labels include five categories, namely happiness, confusion, anxiety, concentration and depression.

[0012] As a preferred embodiment of the first aspect, in step S2, the specific process of using the visual cue learning method to perform cue fine-tuning on the qth encoding layer of the image encoder is as follows:

[0013] AS21: Get the classification token representation learned by the (q-1)th encoding layer of the image encoder, multiply the classification token representation by the linear projection matrix to obtain the projected classification token representation, perform layer normalization on the projected classification token representation, and then apply the multi-head self-attention mechanism to obtain the processed classification token representation, and add the processed classification token representation and the projected classification token representation as the summary token of the qth encoding layer;

[0014] AS22: Add the classification token representation to the randomly initialized learnable vector as the first intermediate token representation, perform layer normalization on the first intermediate token representation, and then apply the multi-head self-attention mechanism to obtain the processed first intermediate token representation, and add the processed first intermediate token representation to the first intermediate token representation as the local prompt token of the qth encoding layer;

[0015] AS23: Randomly initialize the global prompt token of the qth encoding layer, obtain the image features output by the (q-1)th encoding layer of the image encoder, concatenate the image features with the local prompt tokens, the global prompt tokens and the summary tokens to form the original visual input representation, normalize the original visual input representation by layer and then apply the pre-trained self-attention mechanism to obtain the processed visual input representation, add the processed visual input representation to the original visual input representation to obtain the original visual output representation, delete the local prompt tokens, the global prompt tokens and the summary tokens from the original visual output representation to obtain the processed visual output representation, normalize the processed visual output representation by layer and process it by a feedforward neural network to obtain the initial image features, and add the initial image features to the processed visual output representation as the image features output by the qth encoding layer of the image encoder.

[0016] As a preferred embodiment of the above-mentioned first aspect, in step S2, the specific process of obtaining the visual aggregate representation is as follows: multiplying the image features output by the last encoding layer of the image encoder with a linear projection matrix to obtain the projected image features, splicing the projected image features corresponding to each facial expression image and then performing an average pooling operation to output the visual aggregate representation.

[0017] As a preferred embodiment of the first aspect above, in step S2, the specific process of generating the visual classification representation by the output module is as follows: taking the visual aggregation representation corresponding to a facial expression image as the query representation, searching for the feature corresponding to the real emotional state category label y of the facial expression image in the dynamic feature space and using it as the key representation and the value representation, processing the query representation through the first projection function to obtain the first projection feature, processing the key representation through the second projection function and transposing it to obtain the second projection feature, processing the value representation through the third projection function to obtain the third projection feature, performing cosine similarity calculation on the first projection feature and the second projection feature to obtain the first similarity feature, taking the first similarity feature as the input variable of the sharpness function, after processing by the sharpness function, obtaining the sharpness feature weight, weighting the third projection feature and the sharpness feature weight in the form of multiplication to obtain the second similarity feature, processing the second similarity feature through the fourth projection function to obtain the element of the yth row in the visual classification representation, processing all facial expression images, and finally forming the visual classification representation.

[0018] As a preferred embodiment of the above-mentioned first aspect, the specific processing process in the projection function is as follows: the features input into the projection function are passed through a fully connected layer in which all parameters are randomly initialized to obtain a first intermediate feature, the first intermediate feature is added to the features input into the projection function to form a residual connection to obtain a second intermediate feature, the second intermediate feature is L2 normalized along the feature dimension to obtain the features output by the projection function.

[0019] As a preferred embodiment of the first aspect, the sharpness function Rd(·) has a functional form as follows:

[0020] Rd(x)=exp(-w r (1-x))

[0021] Where x represents the variable input to the sharpness function; w r Represents a hyperparameter that adjusts sharpness.

[0022] As a preferred embodiment of the first aspect, in step S3, the specific process of determining whether the dynamic feature space needs to be updated based on the comprehensive score is as follows:

[0023] S31: calculating an entropy value based on the first predicted probability corresponding to the visual aggregate representation of the facial expression image to be classified as the current predicted entropy, calculating an entropy value based on the first predicted probability corresponding to each feature in the dynamic feature space as the historical predicted entropy, averaging all historical predicted entropies as the average historical entropy, adding the current predicted entropy, the average historical entropy and a preset constant as the total confidence score, and taking the ratio of the preset confidence score to the total confidence score as the comprehensive score of the facial expression image;

[0024] S32: When the dynamic feature space reaches a preset capacity threshold, if the comprehensive score of the facial expression image to be classified is greater than or equal to the lowest comprehensive score in the dynamic feature space, the visual aggregate representation of the facial expression image to be classified replaces the features corresponding to the lowest comprehensive score in the dynamic feature space to update the dynamic feature space; if the comprehensive score of the facial expression image to be classified is less than the lowest comprehensive score in the dynamic feature space, the dynamic feature space remains unchanged.

[0025] In a second aspect, the present invention provides a student sentiment analysis system based on a multimodal dynamic memory large model, which comprises:

[0026] A data acquisition module is used to obtain facial expression images of different students when watching online education courses and their corresponding real emotional state category labels;

[0027] The model acquisition module is used to input facial expression images into the multimodal dynamic memory large model for training. During the training process of the multimodal dynamic memory large model, each facial expression image is first projected by a linear projection layer, and each facial expression image corresponds to a token embedding sequence. The token embedding sequence is added to the spatial position encoding result of the facial expression image and the initialized classification token representation as the input token sequence, and the input token sequence is input into the image encoder. The visual cue learning method is used to perform cue fine-tuning on each encoding layer of the image encoder, and finally a visual aggregate representation is obtained. The description related to the facial behavior is processed by a text segmentation tool to form a segmented text sequence, and the segmented text sequence is added to the image encoder. The columns are input into the text encoder together with the learnable text representation to obtain the text aggregation representation, the visual aggregation representation is multiplied by the transposed text aggregation representation and then output the first prediction probability through Softmax, the visual aggregation representation is input into the output module together with the historical features stored in the dynamic feature space to obtain the visual classification representation, the visual aggregation representation is stored in the dynamic feature space as a new feature, the visual aggregation representation is multiplied by the transposed visual classification representation and then output the second prediction probability through Softmax, the weighted sum of the first prediction probability and the second prediction probability is taken as the final prediction probability, and the cross entropy loss is calculated based on the final prediction probability and the true emotional state category label to update the multimodal dynamic memory large model parameters;

[0028] The result acquisition module is used to input the facial expression image to be classified into the trained multimodal dynamic memory large model. When the dynamic feature space reaches the preset capacity threshold, the comprehensive score corresponding to the facial expression image to be classified is calculated, and based on the comprehensive score, it is determined whether the dynamic feature space needs to be updated, and finally the emotional state category prediction result of the facial expression image is output.

[0029] In a third aspect, the present invention provides a computer electronic device comprising a memory and a processor;

[0030] The memory is used to store computer programs;

[0031] The processor is used to implement a student sentiment analysis method based on a multimodal dynamic memory large model as described in any of the schemes of the first aspect above when executing the computer program.

[0032] Compared with the traditional student sentiment analysis method, the present invention has the following beneficial effects:

[0033] The present invention makes full use of the contrastive language-image pre-training model and provides a new visual cue learning method. Compared with traditional visual learning strategies, the visual cue learning method of the present invention can more effectively capture and understand visual information, and can be widely used in various downstream computer vision tasks, such as student sentiment analysis. In terms of text prompts, the present invention introduces learnable text representations instead of manually designed prompt words. In addition, the present invention designs a dynamic feature space, which retains the characteristics of historical test data during the test process. Ultimately, the multimodal dynamic memory large model of the present invention can further mine the potential information of facial expression images and further improve the model performance under the sentiment analysis task of student facial expression recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flow chart of the steps of the method of the present invention;

[0035] Figure 2 A schematic diagram of the process of obtaining a visual aggregate representation by the method of the present invention;

[0036] Figure 3 A schematic diagram of the process of obtaining text aggregation representation by the method of the present invention;

[0037] Figure 4 A schematic diagram of the present invention using a visual cue learning method to perform cue fine-tuning on an image encoder;

[0038] Figure 5 A schematic diagram of a process of obtaining a first prediction probability and a second prediction probability according to the present invention;

[0039] Figure 6 A schematic diagram of the process of obtaining the final predicted probability of the present invention;

[0040] Figure 7 It is a system block diagram of the present invention. DETAILED DESCRIPTION

[0041] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0042] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0043] The multimodal large model has significant advantages in solving the accuracy of emotion recognition and the computing power and response speed of real-time processing of large amounts of data, providing a new visual representation learning method. Compared with traditional visual learning strategies, it can capture and understand visual information more effectively. First, the multimodal large model combines image and text data, which can extract information from multimodal data and improve the accuracy of emotion recognition. For example, the multimodal large model can simultaneously analyze students' facial expressions, body language, and text content in classroom interactions to provide more comprehensive emotion recognition. This multimodal analysis can more accurately capture students' emotional states than a single modality because it takes into account more contextual information and behavioral characteristics. In terms of computing power and response speed, the multimodal large model relies on advanced neural network architectures and efficient parallel processing technologies. Existing multimodal large models are usually based on the Transformer architecture, have strong parallel processing capabilities, and can process large amounts of data simultaneously. This enables the multimodal large model to analyze students' emotional states in real time and provide feedback in a short time. The multimodal visual language large model also has adaptive learning capabilities, which can continuously learn and optimize the model from new data to improve the accuracy of emotion recognition and real-time processing capabilities. Through continuous interaction with teachers and students, the multimodal large model can gradually understand and adapt to the emotional expressions of different students, and improve the personalization and accuracy of emotion recognition. Therefore, the multimodal large model can significantly improve the accuracy of sentiment analysis of students in the classroom through multimodal data fusion, efficient parallel processing and adaptive learning. The latest multimodal large models such as the Contrastive Language-Image Pretraining (CLIP) model and the contrastive learning-based dual encoder model ‌ALIGN collected 400 million and 1.8 billion image-text pairs from the Internet, respectively. Then, these multimodal large models are widely used in various downstream computer vision tasks and have shown excellent performance.

[0044] In this embodiment, the contrastive language-image pre-training model is specifically introduced as an example. CLIP uses contrastive learning to understand and associate visual and language information. When predicting emotion, CLIP first converts the input image and text data into high-dimensional feature vectors respectively, and extracts key features through visual and language encoders. Then, these feature vectors are projected into the same embedding space, in which the cosine similarity of the image and text is calculated. CLIP optimizes a contrast loss function so that the correctly matched image and text pairs are closer in the embedding space, while the incorrectly matched pairs are farther. As mentioned above, the present invention hopes to adopt the contrastive language-image pre-training model in multimodal emotion analysis. In order to effectively apply the contrastive language-image pre-training model to the processing of student facial expression data, the present invention considers two important aspects. On the one hand, it is necessary to retain the generalization ability of the backbone of the contrastive language-image pre-training model, that is, to retain the video representation and text representation obtained from the original image encoder and text encoder in CLIP; on the other hand, it must be able to effectively adapt to the field of student emotion analysis. In view of the first aspect, the present invention proposes a novel text prompt learning method to fine-tune the text encoder, and adopts a visual prompt learning method to fine-tune the image encoder to achieve competitive strong supervision performance to better adapt to the emotion analysis task of student facial expression recognition. By fine-tuning the CLIP model with prompts, its adaptability and accuracy in student emotion analysis can be significantly improved. The fine-tuned CLIP model can more accurately capture students' emotional changes, such as happiness, confusion, anxiety, concentration and frustration, and provide timely feedback. In this way, teachers can flexibly adjust teaching methods according to students' immediate emotional changes, thereby providing more customized support and assistance. Fine-tuning can also make CLIP better adapt to students' emotional expressions in different cultural and personality backgrounds, and further improve the accuracy and universality of emotion recognition. In view of the second aspect, the present invention additionally introduces a learnable output module to adapt to the fine-tuned CLIP model.

[0045] like Figure 1 As shown, in a preferred implementation of the present invention, the student sentiment analysis method based on the multimodal dynamic memory large model includes the following steps S1 to S3. The specific implementation process is described below.

[0046] S1: Obtain facial expression images of different students when watching online education courses and their corresponding real emotional state category labels.

[0047] It should be noted that, in this embodiment, facial expression videos of students watching online education courses are obtained from the online education platform. Each facial expression video with a length of N is extracted frame by frame at a fixed interval, and each frame size is a static facial expression image of Y×U, where Y and U are the height and width of the facial expression image, respectively.

[0048] It should be noted that in this embodiment, the emotional state of students when watching online education courses is described by a variety of category labels. These category labels help teachers better understand students' emotions and reactions, so as to effectively adjust teaching strategies and provide support. In this embodiment, the above-mentioned real emotional state category labels can be summarized into the following categories: happiness refers to students' satisfaction and pleasure in learning, which is manifested as smiling, active participation, and confident expression; confusion refers to students' understanding barriers when facing concepts or tasks, which is manifested as frowning, frequent questions, or seeking additional guidance; anxiety reflects tension and uneasiness, which is manifested as students fidgeting, wandering eyes, or nervous speech; concentration indicates that students concentrate their attention, presenting static postures and thoughtful expressions; depression stems from frustration, which is manifested as students' low mood, lack of motivation, and decreased participation. These real emotional state category labels not only help teachers capture students' emotional states in a timely manner, but also help to formulate personalized teaching support measures to improve students' learning experience and effectiveness, making online education more focused and humane.

[0049] S2: The facial expression images are input into the multimodal dynamic memory large model for training. In the training process of the multimodal dynamic memory large model, each facial expression image is first projected by a linear projection layer, and each facial expression image corresponds to a token embedding sequence. The token embedding sequence is added to the spatial position encoding result of the facial expression image and the initialized classification token representation as the input token sequence. The input token sequence is input into the image encoder, and the visual cue learning method is used to cue fine-tune each encoding layer of the image encoder, and finally a visual aggregation representation (such as Figure 2 As shown in the figure, the descriptions related to facial behaviors are processed by the text segmentation tool to form a text sequence after word segmentation. The text sequence after word segmentation and the learnable text representation are input into the text encoder together to obtain the text aggregation representation (as shown in the figure). Figure 3As shown in the figure, the visual aggregation representation is multiplied by the transposed text aggregation representation and then outputted through Softmax to the first prediction probability, the visual aggregation representation is input into the output module together with the historical features stored in the dynamic feature space to obtain the visual classification representation, the visual aggregation representation is stored in the dynamic feature space as a new feature, the visual aggregation representation is multiplied by the transposed visual classification representation and then outputted through Softmax to the second prediction probability, the weighted sum of the first prediction probability and the second prediction probability is taken as the final prediction probability, and the cross entropy loss is calculated based on the final prediction probability and the true emotional state category label to update the multimodal dynamic memory large model parameters.

[0050] In step S2 of this embodiment, each facial expression image is divided into non-overlapping blocks, that is, each facial expression image is divided into a plurality of non-overlapping square patches of size P×P. After obtaining these square patches, this embodiment uses random scaling and cropping, horizontal flipping, random rotation and color jittering to avoid overfitting. The processed square patches are flattened into a set of vectors, and then a linear projection layer is used to project these vectors to form a token embedding sequence.

[0051] It should be noted that, in step S2 of this embodiment, Figure 4 As shown in FIG, the specific process of using the visual cue learning method to perform cue fine-tuning on the qth encoding layer of the image encoder is as follows:

[0052] AS21: Get the classification token representation learned by the (q-1)th encoding layer of the image encoder, multiply the classification token representation by the linear projection matrix to obtain the projected classification token representation, perform layer normalization on the projected classification token representation, and then apply the multi-head self-attention mechanism to obtain the processed classification token representation, and add the processed classification token representation and the projected classification token representation as the summary token of the qth encoding layer;

[0053] AS22: Add the classification token representation to the randomly initialized learnable vector as the first intermediate token representation, perform layer normalization on the first intermediate token representation, and then apply the multi-head self-attention mechanism to obtain the processed first intermediate token representation, and add the processed first intermediate token representation to the first intermediate token representation as the local prompt token of the qth encoding layer;

[0054] AS23: Randomly initialize the global cue token of the qth encoding layer, obtain the image features output by the (q-1)th encoding layer of the image encoder, concatenate the image features with the local cue tokens, the global cue tokens and the summary tokens to form the original visual input representation, normalize the original visual input representation by layer and then apply the pretrained self-attention mechanism to obtain the processed visual input representation, add the processed visual input representation to the original visual input representation to obtain the original visual output representation, delete the local cue tokens, the global cue tokens and the summary tokens from the original visual output representation to obtain the processed visual output representation, normalize the processed visual output representation by layer and process it through a feedforward neural network to obtain the initial image features, and add the initial image features to the processed visual output representation as the image features output by the qth encoding layer of the image encoder.

[0055] It should be noted that in step S2 of this embodiment, the specific process of obtaining the visual aggregate representation is as follows: multiplying the image features output by the last encoding layer of the image encoder with a linear projection matrix to obtain the projected image features, concatenating the projected image features corresponding to each facial expression image, and then performing an average pooling operation to output the visual aggregate representation.

[0056] It should be noted that in step S2 of the present embodiment, a learnable text representation is used as the context of each category descriptor on the text encoder. This process does not require experts to design context words, and allows the multimodal dynamic memory large model to learn the relevant context information of each expression during training. Specifically, the input of the text encoder specifically includes two types of data. The first input data is the result of the description related to facial behavior after being processed by the text segmentation tool Tokenizer. The text segmentation tool is a tool for dividing text into smaller units. It can divide long text into smaller units and then convert the segmented text into an index sequence. The second input data is a class-specific text representation, which is independent of each description. By inputting these two types of data as prompts into the text encoder, a text aggregation representation representing the visual concept can be obtained.

[0057] In general, in step S2 of this embodiment, there are two main goals for prompting learning of the image encoder: 1) utilizing temporal information by introducing information exchange between frames; 2) providing additional parameters to adapt the image encoder to the distribution of facial expression images. To this end, the present invention introduces three additional tokens in each layer of the image encoder, a summary token to summarize the discriminant information between all frames, a local prompt token to convey the discriminant information of each facial expression image to the remaining facial expression images, and a global prompt token to provide learning capabilities to adapt the multimodal dynamic memory large model to the distribution of facial expression images. In the text encoder part, considering that different facial expressions have both common attributes and unique or special attributes at the local behavior level, therefore, unlike simply inputting the name of the facial expression category, the present invention inputs the description related to the facial behavior into the text encoder together with the learnable text representation after being processed by the text segmentation tool, and the text representation here can provide more detailed and accurate information about the specific movement or position of the muscles involved in each expression.

[0058] It should be noted that, in step S2 of this embodiment, Figure 5 As shown in the figure, for the facial expression recognition task, the visual aggregate representation extracted by the image encoder and the text aggregate representation extracted by the text encoder are L2 normalized respectively, and then the visual aggregate representation is multiplied by the transposed text aggregate representation and then outputted through Softmax to obtain the first prediction probability. .

[0059] It should be noted that in step S2 of this embodiment, the specific process of generating the visual classification representation by the output module is as follows: the visual aggregation representation corresponding to a facial expression image is As the query representation, find the true emotional state category label of the facial expression image in the dynamic feature space Corresponding features And use it as the key representation and value representation, process the query representation through the first projection function to obtain the first projection feature, process the key representation through the second projection function and transpose it to obtain the second projection feature, process the value representation through the third projection function to obtain the third projection feature, calculate the cosine similarity between the first projection feature and the second projection feature to obtain the first similarity feature, use the first similarity feature as the input variable of the sharpness function, after processing it with the sharpness function, obtain the sharpness feature weight, weight the third projection feature and the sharpness feature weight in the form of multiplication to obtain the second similarity feature, process the second similarity feature through the fourth projection function to obtain the visual classification representation Middle Elements of a row :

[0060]

[0061] in, , , and denote the first, second, third and fourth projection functions respectively; Indicates cosine similarity calculation; represents the sharpness function; Represents matrix transpose.

[0062] It should be noted that the specific processing process in the above projection function is as follows: the features input into the projection function are passed through a fully connected layer with all parameters randomly initialized to obtain a first intermediate feature, the first intermediate feature is added to the features input into the projection function to form a residual connection to obtain a second intermediate feature, the second intermediate feature is L2 normalized along the feature dimension to obtain the features output by the projection function.

[0063] Furthermore, in this embodiment, the function form of the above projection function is:

[0064]

[0065] in, Represents the features input to the projection function; represents a fully connected layer with all parameters randomly initialized; Represents the L2 normalization operation along the feature dimension.

[0066] Furthermore, in this embodiment, the sharpness function is in the form of:

[0067]

[0068] in, represents the variable input to the sharpness function; Represents a hyperparameter that adjusts sharpness.

[0069] After the above process, the visual classification representation can be obtained , and finally convert it into the required classification prediction probability, that is, multiply the visual aggregation representation with the transposed visual classification representation and then pass Softmax to output the second prediction probability . Finally, if Figure 6 As shown, the weighted sum of the first predicted probability and the second predicted probability is taken as the final predicted probability :

[0070]

[0071] In the formula, Both represent preset weight hyperparameters.

[0072] S3: Input the facial expression image to be classified into the trained multimodal dynamic memory large model. When the dynamic feature space reaches the preset capacity threshold, calculate the comprehensive score corresponding to the facial expression image to be classified, and judge whether the dynamic feature space needs to be updated based on the comprehensive score, and finally output the emotional state category prediction result of the facial expression image.

[0073] It should be noted that, in step S3 of the present embodiment, inspired by the accumulation of knowledge and recollection in the human brain, a study has proposed to introduce an external memory component to allow the storage and retrieval of historical knowledge to facilitate decision-making. Recently, the idea of ​​a memory network has been introduced into the self-adaptation of CLIP. Thus, the present invention introduces a dynamic feature space to support read and write operations on features, thereby optimizing the process of facial recognition for students. In the training process of the multimodal dynamic memory large model, the size of the dynamic feature space is consistent with the size of the facial expression image used for training, which is used to store the visual aggregation representation generated by each iteration round, and is applied as a historical feature to the calculation of the next iteration round. After the multimodal dynamic memory large model is trained, the dynamic feature space of this stage is set to a preset size, and its length is limited, so when the visual aggregation representation stored therein reaches a limit, the dynamic feature space needs to be updated.

[0074] The specific process of determining whether the dynamic feature space needs to be updated based on the comprehensive score mentioned in step S3 will be described in detail below.

[0075] S31: Calculate the entropy value based on the first predicted probability corresponding to the visual aggregate representation of the facial expression image to be classified as the current predicted entropy, calculate the entropy value based on the first predicted probability corresponding to each feature in the dynamic feature space as the historical predicted entropy, average all historical predicted entropies as the average historical entropy, and calculate the current predicted entropy as the average historical entropy. , average historical entropy And the preset constants Add up as the total confidence score, and set the preset confidence score The ratio of the confidence score to the total confidence score is used as the comprehensive score of the facial expression image. :

[0076]

[0077] S32: When the dynamic feature space reaches a preset capacity threshold, if the comprehensive score of the facial expression image to be classified is greater than or equal to the lowest comprehensive score in the dynamic feature space, the visual aggregate representation of the facial expression image to be classified replaces the features corresponding to the lowest comprehensive score in the dynamic feature space to update the dynamic feature space; if the comprehensive score of the facial expression image to be classified is less than the lowest comprehensive score in the dynamic feature space, the dynamic feature space remains unchanged.

[0078] The present invention will now use a specific example to demonstrate the application effect of the student sentiment analysis method based on the multimodal dynamic memory large model described in S1 to S3 in the above embodiments on a specific data set, so as to facilitate understanding of the essence of the present invention.

[0079] Example

[0080] The specific implementation process of the student sentiment analysis method based on the multimodal dynamic memory large model used in this embodiment is as described above and will not be repeated here.

[0081] In order to demonstrate the technical effect of the method proposed in the present invention, it is verified in an actual teaching scenario. The dataset selected by the present invention is an educational dataset, which focuses on online classes in an educational environment and aims to perform student facial expression recognition and emotion analysis. The dataset contains facial expression videos and facial expression images from different students while watching online courses. These videos and images capture the various emotional states displayed by students during the learning process, such as anxiety, concentration, confusion, happiness, and frustration. Each video and image is annotated to indicate the emotional expression of the student at a specific moment, thereby providing training data for subsequent emotion analysis and recognition tasks. In addition, the dataset also contains background information of the students, such as grade, gender, and academic performance, to facilitate the study of the relationship between emotion and learning outcomes.

[0082] The evaluation indicators used are UAR and WAR, which are the most commonly used in the field of facial expression recognition. UAR refers to the average recall rate calculated over all categories. It contributes equally to each category, so it is suitable for situations where the category distribution is uneven. The larger the UAR value, the higher the accuracy of the model in identifying each emotion category, especially when dealing with minority class samples. On the other hand, WAR takes into account the frequency of each category in the data set, and calculates the recall rate by weighted average to ensure that more common categories have a greater impact on the total score. The higher the WAR value, the better the overall performance of the model for the entire data set, especially in accurately identifying high-frequency categories. In general, the larger the values ​​of these two indicators, the better the performance of the model in the emotion recognition task, and the more effective it is in capturing and classifying different facial expressions. A comparison of the method proposed in the present invention with the previous dynamic expression recognition method is shown in Table 1. In Table 1, the implementation processes of the three models, namely, the Transformer-based dynamic facial expression recognition model Former-DFER, the convolutional neural network ResNet-18-based dynamic facial expression recognition model EST, and the visual language large model-based dynamic facial expression recognition model EmoCLIP, all belong to the prior art and will not be elaborated on here.

[0083] Table 1. Facial expression recognition results of different methods on the education dataset

[0084] method UAR WAR Former-DFER 53.69 65.70 EST 53.94 65.85 EmoCLIP 58.04 62.12 The present invention 66.89 75.64

[0085] In addition, the embodiments of the present invention also conduct ablation experiments to better examine the contributions of each module or method proposed in the present invention. In Table 2, Backbone represents the use of the original image encoder and text encoder in the contrastive language-image pre-training model; W / Prompting represents the addition of visual prompt learning for the image encoder on the basis of Backbone, and the introduction of learnable text representation on the text encoder; W / Memory: corresponds to the complete multimodal dynamic memory large model of the present invention.

[0086] Table 2. Ablation experiment results of the present invention on the education dataset

[0087] method UAR WAR Backbone 23.34 20.07 W / Prompting 60.61 72.65 W / Memory 66.89 75.64

[0088] The ablation experiment results show that the proposed method has a significant effect compared with the previous method. The main reason is that the proposed method models the frame-level information and inter-frame relationship of the facial expression video, and uses the characteristics of the historical test samples retained by the dynamic memory network during the test process, so that the multimodal dynamic memory large model can further mine the potential information of the data outside the training data set.

[0089] It should also be noted that the student sentiment analysis method based on the multimodal dynamic memory large model in the above embodiment can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a student sentiment analysis system based on a multimodal dynamic memory large model corresponding to the student sentiment analysis method based on the multimodal dynamic memory large model provided in the above embodiment, such as Figure 7 As shown, it includes:

[0090] A data acquisition module is used to obtain facial expression images of different students when watching online education courses and their corresponding real emotional state category labels;

[0091] The model acquisition module is used to input facial expression images into the multimodal dynamic memory large model for training. During the training process of the multimodal dynamic memory large model, each facial expression image is first projected by a linear projection layer, and each facial expression image corresponds to a token embedding sequence. The token embedding sequence is added to the spatial position encoding result of the facial expression image and the initialized classification token representation as the input token sequence, and the input token sequence is input into the image encoder. The visual cue learning method is used to perform cue fine-tuning on each encoding layer of the image encoder, and finally a visual aggregate representation is obtained. The description related to the facial behavior is processed by a text segmentation tool to form a segmented text sequence, and the segmented text sequence is added to the image encoder. The columns are input into the text encoder together with the learnable text representation to obtain the text aggregation representation, the visual aggregation representation is multiplied by the transposed text aggregation representation and then output the first prediction probability through Softmax, the visual aggregation representation is input into the output module together with the historical features stored in the dynamic feature space to obtain the visual classification representation, the visual aggregation representation is stored in the dynamic feature space as a new feature, the visual aggregation representation is multiplied by the transposed visual classification representation and then output the second prediction probability through Softmax, the weighted sum of the first prediction probability and the second prediction probability is taken as the final prediction probability, and the cross entropy loss is calculated based on the final prediction probability and the true emotional state category label to update the multimodal dynamic memory large model parameters;

[0092] The result acquisition module is used to input the facial expression image to be classified into the trained multimodal dynamic memory large model. When the dynamic feature space reaches the preset capacity threshold, the comprehensive score corresponding to the facial expression image to be classified is calculated, and based on the comprehensive score, it is determined whether the dynamic feature space needs to be updated, and finally the emotional state category prediction result of the facial expression image is output.

[0093] Similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the student sentiment analysis method based on the multimodal dynamic memory large model provided in the above embodiment, which includes a memory and a processor;

[0094] The memory is used to store computer programs;

[0095] The processor is used to implement the student sentiment analysis method based on the multimodal dynamic memory large model in the above embodiment when executing the computer program.

[0096] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0097] It is understandable that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0098] It should also be noted that those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0099] It should also be noted that the data involved in the present invention is obtained with full consent and authorization, and the collection, use and processing of relevant information must comply with relevant laws, regulations and standards of relevant countries and regions.

[0100] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A student sentiment analysis method based on a multimodal dynamic memory large model, characterized in that: The following steps are involved: S1: Obtain facial expression images of different students when watching online education courses and their corresponding real emotional state category labels; S2: The facial expression images are input into the multimodal dynamic memory large model for training. During the training process of the multimodal dynamic memory large model, each facial expression image is first projected by a linear projection layer. Each facial expression image corresponds to a token embedding sequence. The token embedding sequence is added to the spatial position encoding result of the facial expression image and the initialized classification token representation as the input token sequence. The input token sequence is input into the image encoder, and the visual cue learning method is used to cue fine-tune each encoding layer of the image encoder, and finally a visual aggregate representation is obtained. The description related to the facial behavior is processed by the text segmentation tool to form a segmented text sequence, and the segmented text sequence is compared with the identifiable The learned text representation is input into the text encoder together to obtain the text aggregation representation, the visual aggregation representation is multiplied by the transposed text aggregation representation and then outputted through Softmax to obtain the first prediction probability, the visual aggregation representation is input into the output module together with the historical features stored in the dynamic feature space to obtain the visual classification representation, the visual aggregation representation is stored in the dynamic feature space as a new feature, the visual aggregation representation is multiplied by the transposed visual classification representation and then outputted through Softmax to obtain the second prediction probability, the weighted sum of the first prediction probability and the second prediction probability is taken as the final prediction probability, and the cross entropy loss is calculated based on the final prediction probability and the true emotional state category label to update the parameters of the multimodal dynamic memory large model; S3: Input the facial expression image to be classified into the trained multimodal dynamic memory large model. When the dynamic feature space reaches the preset capacity threshold, calculate the comprehensive score corresponding to the facial expression image to be classified, and judge whether the dynamic feature space needs to be updated based on the comprehensive score, and finally output the emotional state category prediction result of the facial expression image.

2. The student sentiment analysis method based on the multimodal dynamic memory large model as claimed in claim 1 is characterized in that: In step S1, the real emotional state category labels include five categories, namely happiness, confusion, anxiety, concentration, and depression.

3. The student sentiment analysis method based on the multimodal dynamic memory large model as claimed in claim 1 is characterized in that: In step S2, the specific process of using the visual cue learning method to perform cue fine-tuning on the qth encoding layer of the image encoder is as follows: AS21: Get the classification token representation learned by the (q-1)th encoding layer of the image encoder, multiply the classification token representation by the linear projection matrix to obtain the projected classification token representation, perform layer normalization on the projected classification token representation, and then apply the multi-head self-attention mechanism to obtain the processed classification token representation, and add the processed classification token representation and the projected classification token representation as the summary token of the qth encoding layer; AS22: Add the classification token representation to the randomly initialized learnable vector as the first intermediate token representation, perform layer normalization on the first intermediate token representation, and then apply the multi-head self-attention mechanism to obtain the processed first intermediate token representation, and add the processed first intermediate token representation to the first intermediate token representation as the local prompt token of the qth encoding layer; AS23: Randomly initialize the global prompt token of the qth encoding layer, obtain the image features output by the (q-1)th encoding layer of the image encoder, concatenate the image features with the local prompt tokens, the global prompt tokens and the summary tokens to form the original visual input representation, normalize the original visual input representation by layer and then apply the pre-trained self-attention mechanism to obtain the processed visual input representation, add the processed visual input representation to the original visual input representation to obtain the original visual output representation, delete the local prompt tokens, the global prompt tokens and the summary tokens from the original visual output representation to obtain the processed visual output representation, normalize the processed visual output representation by layer and process it by a feedforward neural network to obtain the initial image features, and add the initial image features to the processed visual output representation as the image features output by the qth encoding layer of the image encoder.

4. The student sentiment analysis method based on the multimodal dynamic memory large model as claimed in claim 3 is characterized in that: In step S2, the specific process of obtaining the visual aggregate representation is as follows: multiplying the image features output by the last encoding layer of the image encoder with a linear projection matrix to obtain the projected image features, concatenating the projected image features corresponding to each facial expression image, and then performing an average pooling operation to output the visual aggregate representation.

5. The student sentiment analysis method based on the multimodal dynamic memory large model as claimed in claim 1 is characterized in that: In step S2, the specific process of generating the visual classification representation by the output module is as follows: taking the visual aggregation representation corresponding to a facial expression image as the query representation, searching for the feature corresponding to the real emotional state category label y of the facial expression image in the dynamic feature space and using it as the key representation and the value representation, processing the query representation by the first projection function to obtain the first projection feature, processing the key representation by the second projection function and transposing it to obtain the second projection feature, processing the value representation by the third projection function to obtain the third projection feature, performing cosine similarity calculation on the first projection feature and the second projection feature to obtain the first similarity feature, taking the first similarity feature as the input variable of the sharpness function, after processing by the sharpness function, obtaining the sharpness feature weight, weighting the third projection feature and the sharpness feature weight in the form of multiplication to obtain the second similarity feature, processing the second similarity feature by the fourth projection function to obtain the element of the yth row in the visual classification representation, processing all facial expression images, and finally forming the visual classification representation.

6. The student sentiment analysis method based on the multimodal dynamic memory large model as claimed in claim 5 is characterized in that: The specific processing process in the projection function is as follows: the features input into the projection function are passed through a fully connected layer with all parameters randomly initialized to obtain the first intermediate features, the first intermediate features are added to the features input into the projection function to form a residual connection to obtain the second intermediate features, the second intermediate features are L2 normalized along the feature dimension to obtain the features output by the projection function.

7. The student sentiment analysis method based on the multimodal dynamic memory large model as claimed in claim 5 is characterized in that: The function form of the sharpness function Rd(·) is: Rd(x)=exp(-w r (1-x)) Where x represents the variable input to the sharpness function; w r Represents a hyperparameter that adjusts sharpness.

8. The student sentiment analysis method based on the multimodal dynamic memory large model as claimed in claim 1, characterized in that: In step S3, the specific process of determining whether the dynamic feature space needs to be updated based on the comprehensive score is as follows: S31: calculating an entropy value based on the first predicted probability corresponding to the visual aggregate representation of the facial expression image to be classified as the current predicted entropy, calculating an entropy value based on the first predicted probability corresponding to each feature in the dynamic feature space as the historical predicted entropy, averaging all historical predicted entropies as the average historical entropy, adding the current predicted entropy, the average historical entropy and a preset constant as the total confidence score, and taking the ratio of the preset confidence score to the total confidence score as the comprehensive score of the facial expression image; S32: When the dynamic feature space reaches a preset capacity threshold, if the comprehensive score of the facial expression image to be classified is greater than or equal to the lowest comprehensive score in the dynamic feature space, the visual aggregate representation of the facial expression image to be classified replaces the features corresponding to the lowest comprehensive score in the dynamic feature space to update the dynamic feature space; if the comprehensive score of the facial expression image to be classified is less than the lowest comprehensive score in the dynamic feature space, the dynamic feature space remains unchanged.

9. A student sentiment analysis system based on a multimodal dynamic memory large model, characterized in that: include: A data acquisition module is used to obtain facial expression images of different students when watching online education courses and their corresponding real emotional state category labels; The model acquisition module is used to input facial expression images into the multimodal dynamic memory large model for training. During the training process of the multimodal dynamic memory large model, each facial expression image is first projected by a linear projection layer, and each facial expression image corresponds to a token embedding sequence. The token embedding sequence is added to the spatial position encoding result of the facial expression image and the initialized classification token representation as the input token sequence, and the input token sequence is input into the image encoder. The visual cue learning method is used to perform cue fine-tuning on each encoding layer of the image encoder, and finally a visual aggregate representation is obtained. The description related to the facial behavior is processed by a text segmentation tool to form a segmented text sequence, and the segmented text sequence is added to the image encoder. The columns are input into the text encoder together with the learnable text representation to obtain the text aggregation representation, the visual aggregation representation is multiplied by the transposed text aggregation representation and then output the first prediction probability through Softmax, the visual aggregation representation is input into the output module together with the historical features stored in the dynamic feature space to obtain the visual classification representation, the visual aggregation representation is stored in the dynamic feature space as a new feature, the visual aggregation representation is multiplied by the transposed visual classification representation and then output the second prediction probability through Softmax, the weighted sum of the first prediction probability and the second prediction probability is taken as the final prediction probability, and the cross entropy loss is calculated based on the final prediction probability and the true emotional state category label to update the multimodal dynamic memory large model parameters; The result acquisition module is used to input the facial expression image to be classified into the trained multimodal dynamic memory large model. When the dynamic feature space reaches the preset capacity threshold, the comprehensive score corresponding to the facial expression image to be classified is calculated, and based on the comprehensive score, it is determined whether the dynamic feature space needs to be updated, and finally the emotional state category prediction result of the facial expression image is output.

10. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the student sentiment analysis method based on the multimodal dynamic memory large model as described in any one of claims 1 to 8 when executing the computer program.

Images