Text main driving-based learner multi-modal sentiment analysis method and device

Abstract

The application discloses a learner multi-modal sentiment analysis method and device based on text main driving, extracts multi-modal data with student related sentiment information embedded in an online classroom, uses a language representation pre-training model BERT to perform feature extraction on text modal data, uses an LSTM to perform feature extraction on audio and visual modal data, uses a cross-modal attention mechanism to fuse multi-modal information, and outputs a final result of multi-modal feature fusion, and performs sentiment analysis according to the final fusion result. The contrast learning technology is used to promote single-modal feature coding quality, maintain task related modal data uniqueness, ensure that multi-modal fusion results sufficiently learn unique sentiment information of various modal data generated in a classroom, improve learner participation in an online learning environment, and thus promote teaching quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of sentiment analysis, in particular a learner multi-modal sentiment analysis method and device based on text main driving. BACKGROUND

[0002] Multi-modal data fusion for learners mainly includes feature layer fusion, model layer fusion and decision layer fusion. Among them, model layer fusion is the commonly used fusion method by researchers, which models the intra-modal and inter-modal relationship to mine the inter-modal dynamic relationship. The commonly used model layer fusion models at present include convolutional neural network, long short-term memory network, Transformer and attention mechanism, etc. For example, MulT model designs a cross-modal Transformer based on attention mechanism. The input of traditional Transformer is the same modal data, while the input of Transformer in MulT is two different modal data, and through this cross-modal attention mechanism, the non-alignment problem in multi-modal data can be effectively solved. However, MulT model assumes that the contribution of each modal in multi-modal data to the final learner sentiment analysis is the same, which is unreasonable. Studies by some scholars show that for learner sentiment analysis of text and audio-video data generated in the online learning process, the contribution of text data is more significant than other modalities.

[0003] In view of this characteristic of text data, Interaction Canonical Correlation Network uses deep canonical correlation analysis method to study the implicit feature correlation between multi-modal data, and obtains multi-modal feature representation based on text. However, Interaction Canonical Correlation Network does not distinguish between each modal in the multi-modal fusion process, but only uses a canonical correlation analysis loss to train the model, without designing different fusion weights for different modalities, which reduces the performance of the multi-modal fusion network. In addition, some researchers use other modalities as auxiliary to correct the feature representation of the text modality under the premise of taking the text modality as the leading one, and design a gating mechanism to judge the effective information amount of the auxiliary modality, so as to filter out the modalities with noise and improve the accuracy of sentiment analysis. However, the above method does not take into account the uniqueness of each modality data in the online learning process, and directly removes the modalities with noise, which is equivalent to losing the unique emotional information that the modality should have. In addition, the quality of the feature representation of the multi-modal data fusion result of the learner depends to a great extent on the degree of feature representation in the sentiment aspect in the early single-modal feature coding process. In view of this problem, most of the solutions are limited to optimizing the feature representation from the original input to the fusion result, without focusing on mining the emotional characteristics of the single modality itself.

[0004] Therefore, the above technical problems existing in the related art are urgent to be solved. SUMMARY

[0005] The present application aims to solve one of the technical problems in the related art. To this end, the embodiments of the present application provide a learner multi-modal sentiment analysis method and device based on text main driving, which can mine sentiment information in multi-modal data and obtain good multi-modal joint representation.

[0006] According to an aspect of the embodiments of the present application, a learner multi-modal sentiment analysis method based on text main driving is provided, the method comprising:

[0007] extracting multi-modal data in which student-related sentiment information is embedded in an online classroom;

[0008] For data of the text modality, a language representation pre-training model BERT is used for feature extraction;

[0009] For data of the audio and visual modalities, an LSTM is used for feature extraction;

[0010] Multi-modal information is fused by using a cross-modal attention mechanism to output a final result of multi-modal feature fusion;

[0011] Learner sentiment analysis is performed according to the final output of multi-modal feature fusion.

[0012] In one of the embodiments, the multi-modal data in which student-related sentiment information is embedded in an online classroom is extracted, comprising:

[0013] Text information is transcribed from the original video;

[0014] Acoustic features are extracted by using OVAREP software;

[0015] Facial action units are extracted by using a facial action coding system;

[0016] Face embedding features are extracted by using an expression recognition model.

[0017] In one of the embodiments, the formula for feature extraction by using the language representation pre-training model BERT is:

[0018]

[0019] wherein X t represents a feature vector of the text modality, BERT(·) represents a language representation pre-training model BERT based on Transformer, and is a training parameter of the model, l t represents a modality sequence length, d t represents a modality feature dimension.

[0020] In one of the embodiments, the formula for feature extraction using LSTM is as follows:

[0021]

[0022] where BiLSTM(·) represents feature extraction using BiLSTM model, I m represents pre-processed model input, l m represents sequence length of modality m, d m represents feature dimension of modality m, when m=a, X m represents feature vector of audio modality, represents model parameters for training audio modality feature extraction; when m=v, X m represents feature vector of visual modality, represents model parameters for training visual modality feature extraction.

[0023] In one of the embodiments, multi-modal information is fused using cross-modal attention mechanism, and the cross-modal attention layer is CA(·) function, including:

[0024]

[0025] wherein, and are inputs of the function, L(·) represents layer normalization operation, Q, K, and V represent query, key, and value respectively, T is modality fusion information, d k represents feature dimension of modality k.

[0026] In one of the embodiments, the final result of multi-modal feature fusion is output, including:

[0027] The text-centered feature fusion module is defined, and the mathematical formula is as follows:

[0028]

[0029]

[0030] wherein N is the number of layers of TAF sub-module, and M is the number of layers of TVF sub-module. and are the i-th layer input of TVF sub-module, and are the j-th layer input of TAF sub-module, l t represents modality sequence length, d t represents modality feature dimension;

[0031] The final output of multi-modal feature fusion is obtained, and the mathematical formula includes:

[0032]

[0033] wherein is the final output of the TVF sub-module, is the final output of the TAF sub-module, Z s is the final output of the multi-modal feature fusion module.

[0034] In one of the embodiments, the method further comprises: performing sentiment prediction according to the final output of the multi-modal feature fusion, comprising:

[0035] performing sentiment polarity prediction by a multi-layer perception, and the mathematical formula comprises:

[0036]

[0037] wherein σ(·) represents a ReLU activation function, W y is a training parameter of the activation function, is the final predicted sentiment polarity, Z s is the final output of the multi-modal feature fusion module.

[0038] In one of the embodiments, the method further comprises:

[0039] adopting mean absolute error to calculate sentiment prediction loss, and the formula comprises:

[0040]

[0041] wherein n is the number of all samples in the data set, y i and respectively represent the real sentiment polarity and the predicted sentiment polarity of the i-th sample, is the sentiment prediction loss.

[0042] According to an aspect of the embodiment of the present application, a learner multi-modal sentiment analysis device based on text main driving is provided, and the device comprises:

[0043] A first module is configured to extract multi-modal data with student-related sentiment information embedded in an online classroom.

[0044] A second module is configured to perform feature extraction on data of a text mode by using a language representation pre-training model BERT.

[0045] A third module is configured to perform feature extraction on data of audio and visual modes by using an LSTM.

[0046] A fourth module is configured to fuse multi-modal information by using a cross-modal attention mechanism and output a final result of multi-modal feature fusion.

[0047] A fifth module is configured to perform sentiment analysis based on the final output of the multi-modal feature fusion.

[0048] According to an aspect of the embodiments of the present application, a text-main-driven learner multi-modal sentiment analysis device is provided, which comprises:

[0049] at least one processor;

[0050] at least one memory configured to store at least one program;

[0051] The at least one processor is configured to implement the text-main-driven learner multi-modal sentiment analysis method according to the embodiments of the present application when executing the at least one program.

[0052] The text-main-driven learner multi-modal sentiment analysis method and device provided by the embodiments of the present application have the following beneficial effects: The embodiments of the present application extract multi-modal data in which student-related sentiment information is embedded in an online classroom; for text modal data, a language representation pre-training model BERT is used to extract features; for audio and visual modal data, an LSTM is used to extract features; a cross-modal attention mechanism is used to fuse multi-modal information, and output the final result of multi-modal feature fusion; sentiment analysis is performed based on the final output of multi-modal feature fusion. The use of contrast learning technology promotes the quality of single-modal feature encoding, maintains the uniqueness of task-related modal data, ensures that the multi-modal fusion result sufficiently learns the unique sentiment information of various modal data generated in the classroom, improves the participation of learners in the online learning environment, and thus promotes the teaching quality.

[0053] Additional aspects and advantages of the present application will be given in part in the following description, become apparent from the following description, or be learned by practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0055] Figure 1 A flowchart of the text-main-driven learner multi-modal sentiment analysis method provided by the embodiments of the present application is shown in the following figure:

[0056] Figure 2 A schematic diagram of the text-main-driven learner multi-modal sentiment analysis device provided by the embodiments of the present application is shown in the following figure:

[0057] Figure 3Another text main driving-based learner multi-modal sentiment analysis device provided by the embodiment of the application. DETAILED DESCRIPTION

[0058] In order for those skilled in the art to better understand the scheme of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0059] The terms "first", "second", "third", and "fourth" and the like in the specification of the present application and claims and drawings are used to distinguish different objects, not to describe a particular order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or device.

[0060] In the present application, the phrase "embodiment" means that the specific features, structures or characteristics described in conjunction with the embodiment can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment to other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in the present application can be combined with other embodiments.

[0061] In the post-epidemic era, online learning has become a pillar of high-quality education development in the new era with the support of related technologies. However, the "emotional deficiency" problem caused by the inability of students and teachers to communicate face-to-face seriously hinders the in-depth development of the deep integration of artificial intelligence and education in the new era. The difficulty of perceiving the emotions of learners in the online education environment makes it difficult for teachers to accurately grasp the learning state of students, affecting the effect. Therefore, using student facial expressions, voice tone, and text expression and other multi-modal data to analyze learner emotions has become a hot topic of research in recent years, and the key to research is to fully mine emotional information from multi-modal data and obtain good multi-modal joint representation.

[0062] Multimodal data fusion for learners mainly includes three types: feature level fusion, model level fusion and decision level fusion. Among them, model level fusion is the common fusion method used by researchers, which models the relationship between intra-modal and inter-modal to mine the dynamic relationship between modalities. The commonly used model level fusion models include convolutional neural network, long short-term memory network, Transformer and attention mechanism, etc. For example, MulT model designs a cross-modal Transformer based on attention mechanism. The input of traditional Transformer is the same modal data, while the input of Transformer in MulT is two different modal data. Through this cross-modal attention mechanism, the non-alignment problem in multimodal data can be effectively solved. However, MulT model assumes that the contribution of each modality to the final learner sentiment analysis is the same, which is unreasonable. Studies have shown that for learner sentiment analysis of text and audio-video data generated in the online learning process, the contribution of text data is more significant than other modalities.

[0063] In view of this characteristic of text data, Interaction Canonical Correlation Network uses deep canonical correlation analysis method to study the implicit feature correlation between multimodal data, and obtains multimodal feature representation based on text. However, Interaction Canonical Correlation Network does not distinguish between modalities in the process of multimodal fusion, but only uses a canonical correlation analysis loss to train the model, without designing different fusion weights for different modalities, which reduces the performance of the multimodal fusion network. In addition, some researchers use other modalities as auxiliary to correct the feature representation of the text modality under the premise of text modality as the leading role, and design a gating mechanism to judge the effective information amount of auxiliary modalities, so as to filter out modalities with noise and improve the accuracy of sentiment analysis. However, the above methods do not consider the uniqueness of each modality data in online learning, and directly remove the modalities with noise, which is equivalent to losing the unique emotional information that the modality should have. In addition, the quality of the feature representation of the multimodal data fusion result of the learner depends to a great extent on the degree of feature representation in the early single-modal feature coding process. In view of this problem, most of the solutions are limited to optimizing the feature representation from the original input to the fusion result, without focusing on mining the emotional characteristics of single modalities.

[0064] Therefore, in view of the above problems, the application utilizes a cross-modal attention mechanism, designs a text-driven modal-specific fusion method, and integrates contrast learning into a multi-modal sentiment analysis framework, designs a single-modal sentiment feature generation model, uses noise contrast estimation loss function to optimize the generated feature vector, maintains the uniqueness of single-modal data, and better fuses multiple modal data for learner sentiment analysis.

[0065] Figure 1 A flowchart of a text-driven learner multi-modal sentiment analysis method provided by an embodiment of the application is shown in Figure 1 The text-driven learner multi-modal sentiment analysis method provided by the embodiment of the application includes:

[0066] S101, extracting multi-modal data with student-related sentiment information embedded in an online classroom.

[0067] S102, for data of the text modality, using a language representation pre-training model BERT for feature extraction.

[0068] In social media, users express their ideas through videos, and video data contains spoken text information, audio information with speech intonation, and visual information with facial expressions and gestures. We manually transcribe the text information from the original video, extract word vectors using GloVe, and obtain Acoustic features are extracted using OVAREP software The extracted features are related to emotion and tone; frames are extracted from the video using the MTCNN face detection algorithm to identify faces, facial action units are extracted through the facial action coding system, including facial shape, HoG features, head posture, head direction, and eye gaze direction related emotional information, and finally facial embedding features are extracted using an expression recognition model to obtain

[0069] S103, for data of the audio and visual modalities, using LSTM for feature extraction.

[0070] S104, using a cross-modal attention mechanism to fuse multi-modal information and output the final result of multi-modal feature fusion.

[0071] S105, performing sentiment analysis according to the final output of multi-modal feature fusion.

[0072] Optionally, the multi-modal data with student-related sentiment information embedded in the online classroom includes: transcribing text information from the original video; extracting acoustic features using OVAREP software; extracting facial action units through the facial action coding system; and extracting facial embedding features through the expression recognition model.

[0073] The performance of learner multimodal sentiment analysis models hinges on the quality of fused features, which largely depends on the quality of upstream single-modal feature encoding. Common solutions include studying the model's backpropagation loss function or using techniques to adjust the geometric properties of the feature space, bringing data of the same category closer together and distancing data of different categories. However, these methods neglect the preservation of task-related information. Therefore, this application, after obtaining fused features using a modality-specific fusion method, designs a single-modal feature generation model. This model generates single-modal feature vectors from the fusion results, and then calculates the similarity between the generated single-modal feature vectors and the original vectors extracted from single-modal features. A noise-contrast estimation loss function based on contrastive learning is used to maximize this similarity, ultimately improving the quality of single-modal feature encoding and maintaining the uniqueness of task-related modality data.

[0074] In this embodiment, the formula for feature extraction using the language representation pre-trained model BERT is as follows:

[0075]

[0076] Among them, X t The feature vector represents the text modality, and BERT(·) represents the Transformer-based language representation pre-trained model BERT. These are the training parameters of the model, l t d represents the length of the modal sequence. t This represents the modal feature dimension.

[0077] The single-modal feature extraction process in this application is as follows: We extract the preprocessed multimodal data I m Input the data into the model. For different modal data, select a feature extraction method that suits its modal characteristics.

[0078] For the text modality, we use the language representation pre-trained model BERT for feature extraction:

[0079]

[0080] Among them, X t The feature vector represents the text modality, and BERT(·) represents the Transformer-based language representation pre-trained model BERT. These are the training parameters of the model. t d represents the length of the modal sequence. t This represents the modal feature dimension.

[0081] For audio and visual modalities, we utilize LSTM for feature extraction. Considering the non-aligned problem of multi-modal sentiment data, i.e., the future sentiment state affects the current emotional expression, we adopt a bidirectional long short-term memory artificial neural network (BiLSTM) to process the features of audio and visual modalities:

[0082]

[0083] where BiLSTM(·) denotes feature extraction using the BiLSTM model. When m=a, X m denotes the feature vector of the audio modality, denotes the model parameter for training the audio modality feature extraction; conversely, when m=v, X m denotes the feature vector of the visual modality, denotes the model parameter for training the visual modality feature extraction.

[0084] It should be noted that the formula for feature extraction using LSTM is:

[0085]

[0086] where BiLSTM(·) denotes feature extraction using the BiLSTM model, I m denotes the pre-processed model input, l m denotes the sequence length of the modality m, d m denotes the feature dimension of the modality m, when m=a, X m denotes the feature vector of the audio modality, denotes the model parameter for training the audio modality feature extraction; when m=v, X m denotes the feature vector of the visual modality, denotes the model parameter for training the visual modality feature extraction.

[0087] After extracting the modality features, multi-modal feature fusion based on modality specificity is needed. The specific process is as follows: first, in order to ensure that each element of the input sequence has sufficient perception ability to its neighborhood elements, we input the feature vectors of each modality to a one-dimensional convolution layer:

[0088]

[0089] where g(·) denotes one-dimensional convolution operation, k m denotes the size of the convolution kernel. After this convolution operation, has the ability to perceive the temporal context.

[0090] Then, we fuse multi-modal information by using cross-modal attention mechanism. Note that in this step, we only consider fusing text-audio modal and text-visual modal, that is, we fully consider the importance of text modal in the fusion process, and only align audio and visual modal through text modal. We first define the fusion method of cross-modal attention mechanism (take text-audio modal fusion as an example), define the cross-modal attention layer as CA(·) function,

[0091]

[0092]

[0093]

[0094]

[0095] wherein, and are the inputs of the function, L(·) represents the layer normalization operation. Q, K and V represent Query, Key and Value respectively, and W Q ,W K ,W V are the normalized function parameter matrices of Q, K and V respectively. In the process of fusing text and audio modal, Query is text data, and Key and Value are both video data, that is, the interaction information of text and video data is obtained through the mapping relationship of Query to a series of key-value pairs. The specific process is: and First, Q, K and V are obtained by linear transformation through L(·); then the inner product of Q and K is obtained to obtain the attention weight, and the degree of inner product is adjusted by in the process, and finally multiplied by V to obtain the attention value based on . Note that the application uses multi-head attention, so multiple CA(·) operations are performed at a time, and finally the results are spliced to obtain the final fusion result:

[0096]

[0097] wherein, h represents the number of attention levels (heads), and W o is the parameter matrix. Finally, we obtain the text-audio modal fusion information and the text-visual modal fusion information

[0098] Next, we define the text-centered feature fusion module. In this module, we consider two aspects of fusion, and design two sub-modules TAF and TVF. Among them, TAF is used to fuse audio and text modalities, and TVF is used to fuse visual and text modalities. Specifically, the input information of the sub-module is initialized first:

[0099]

[0100]

[0101] where N is the number of layers of the TAF sub-module, and M is the number of layers of the TVF sub-module. and is the input of the i-th layer of the TVF sub-module, and is the input of the j-th layer of the TAF sub-module. The output of the i-th layer of the TVF sub-module is The output of the j-th layer of the TAF sub-module is The two sub-modules TVF and TAF are in parallel relationship, so when N=M, the input of the i-th layer of TVF is the output of the i-1-th layer of TVF and TAF, i.e. i=j=k=q. But when N≠M, then at some time, TAF and TVF have one module that has completed the loop. If N>M and N>i>M, it means that the TAF module has completed the loop, so k=M; if N<M and N<j<M, it means that the TVF module has completed the loop, so q=N. The output of the i-th layer of TVF and the output of the j-th layer of TAF are:

[0102]

[0103]

[0104] where is the feature vector of the text modality. From the formula, we can see that we use text information to repeatedly fuse with other modalities, and finally get:

[0105]

[0106] where is the final output of the TVF sub-module, is the final output of the TAF sub-module. Z s is the final output of the multi-modal feature fusion module.

[0107] The application extracts multi-modal data embedded with student-related emotional information in an online classroom; for text modal data, a language representation pre-training model BERT is used for feature extraction; for audio and visual modal data, LSTM is used for feature extraction; a cross-modal attention mechanism is used to fuse multi-modal information, and output the final result of multi-modal feature fusion; emotional analysis is performed according to the final output of multi-modal feature fusion. The use of contrast learning technology promotes the quality of single-modal feature encoding, maintains the uniqueness of task-related modal data, ensures that the multi-modal fusion result fully learns the unique emotional information of various modal data generated in the classroom, and improves the participation of learners in the online learning environment, thereby promoting the quality of teaching.

[0108] In addition, the application can also be based on the single-modal feature optimization of generative contrast learning, specifically: in the previous fusion step, our goal is to maximize the fusion of similar information in each modal data. However, each modal data has its unique data characteristics, so we use contrast learning prediction encoding to maintain task-related modal unique information. The specific method is to use the fused feature vector Z s Predict single-modal data through backpropagation Use generated single-modal data and real single-modal data X m Calculate the correlation, where b represents the number of samples in a training batch. Finally, use the noise contrast estimation method to maximize the emotional relevance information of each single-modal data.

[0109] First, design a neural network for generating single-modal data in reverse:

[0110]

[0111]

[0112] where F θ (·) is a neural network composed of parameters θ, is the single-modal data generated by the generation network. ‖·‖2 is the Euclidean norm, and we get the unit vector by dividing the Euclidean norm. The norm of a matrix is defined as the ability of its maximum stretching vector, so we maximize the norm to get and X m The maximum correlation is defined as the r(·) function.

[0113] Since our single-modal data is not annotated with emotional polarity, we need to maximize r(·) in an unsupervised manner when training single-modal feature representation. Therefore, we construct a noise contrast loss function suitable for unsupervised learning to optimize r(·)

[0114]

[0115] Finally, in this contrastive learning, we take the current sample as the positive sample and other samples in the same training batch as the negative samples:

[0116]

[0117] where b represents the number of samples in a batch. represents the current sample, i.e., the positive sample. represents the negative sample.

[0118] Finally, we obtain three noise contrastive estimates of the single modality, and finally add the three values to obtain the final contrastive prediction encoding score:

[0119]

[0120] where and represent the noise contrastive estimates of the text, visual, and audio modalities, respectively.

[0121] Optionally, the present application can also perform sentiment prediction according to the final output of multi-modal feature fusion, including:

[0122] Perform sentiment polarity prediction through a multi-layer perceptron, and the mathematical formula includes:

[0123]

[0124] where σ(·) represents the ReLU activation function, W y is the training parameter of the activation function, is the final predicted sentiment polarity, and Z s is the final output of the multi-modal feature fusion module.

[0125] Since our model is a regression task, we use the mean absolute error (MAE) to calculate the task-related sentiment prediction loss when training the model:

[0126]

[0127] where n is the number of all samples in the data set, y i and represent the real sentiment polarity and the predicted sentiment polarity of the i-th sample, respectively. Finally, we calculate the weighted sum of the two loss functions to obtain the loss function of the entire model:

[0128]

[0129] Here, α is a hyperparameter that adjusts the weights of the contrastive learner loss function.

[0130] Furthermore, this application also provides a text-driven learner multimodal sentiment analysis device, such as... Figure 2 As shown, the device includes:

[0131] The first module is used to extract multimodal data containing embedded emotional information of students in online classrooms;

[0132] The second module is used to extract features from text modal data using the language representation pre-trained model BERT.

[0133] The third module is used to extract features from audio and visual modal data using LSTM.

[0134] The fourth module is used to fuse multimodal information using a cross-modal attention mechanism and output the final result of multimodal feature fusion.

[0135] The fifth module is used for sentiment analysis based on the final output of multimodal feature fusion.

[0136] Furthermore, this application also provides a text-driven learner multimodal sentiment analysis device, such as... Figure 3 As shown, the device includes:

[0137] At least one processor;

[0138] At least one memory for storing at least one program;

[0139] When at least one of the programs is executed by at least one of the processors, the text-driven learner multimodal sentiment analysis method described in the preceding embodiments is implemented.

[0140] Similarly, the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0141] In alternative embodiments, the functions / operations depicted in the flowcharts can occur out of the order depicted. For example, two operations shown in succession can in fact be executed substantially concurrently or the operations sometimes can be executed in the reverse order depending upon the functionality / operations involved. Also, embodiments presented and described in the flowcharts are provided by way of example only. The steps presented and / or described in the flowcharts are not exhaustive and additional steps can be performed or the described steps can be modified. Alternative embodiments are within the scope of the disclosure.

[0142] Furthermore, although the present application is described in the context of functional modules, it is to be understood that one or more of the functions and / or features can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It is also to be understood that detailed discussion of the actual implementation of each module is unnecessary to an understanding of the present application. Rather, the actual implementation is within the routine of an engineer's knowledge given the nature of the devices disclosed in the present application, the properties of the various functional modules, and the interrelationships of the modules. Therefore, the disclosure of the present application is not to be limited to particular implementations described, since the generic principles of the application have been used to facilitate a disclosure of the application. It will be readily apparent to those skilled in the art that certain aspects of the disclosed application can be put into effect by software comprising instructions for a computer to execute. However, it will be appreciated by the skilled person that the application is not limited to software and can certainly be implemented by hardware or firmware or a combination thereof.

[0143] The logic and / or steps represented in the flowcharts and / or otherwise described herein, for example, can be embodied in non-transitory computer-readable media, which can be executed by an instruction execution system, apparatus, or device, such as a computer-based system, processor, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this specification, a "computer-readable medium" can be any means that can contain, store, communicate, propagate or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

[0144] It should be understood that portions of the application can be realized with hardware, software, firmware or a combination thereof. In the foregoing embodiments, a plurality of steps or methods can be realized with software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if realized with hardware, and as in another embodiment, any one or a combination of the following technologies known in the art can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.

[0145] In the above description of the present specification, the description of the terms "one embodiment", "another embodiment" or "certain embodiments" or the like means that the specific features, structures, materials or characteristics described in connection with the embodiments or examples are included in at least one embodiment or example of the present application. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0146] Although the embodiments of the present application have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the claims and their equivalents.

[0147] The above, the above embodiments are only to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A text-main-driven learner multi-modal sentiment analysis method, characterized in that, The method includes: Extracting multimodal data containing embedded emotional information about students in online classrooms; For text modal data, feature extraction is performed using the language representation pre-trained model BERT. For audio and visual modal data, LSTM is used for feature extraction; Multimodal information is fused using a cross-modal attention mechanism, and the final result of multimodal feature fusion is output. Sentiment analysis is performed based on the final output of multimodal feature fusion; The method further includes: Based on the final result of multimodal feature fusion, backpropagation is performed to obtain the predicted single-modal data; The correlation is calculated based on the predicted single-modal data and the actual single-modal data; The correlation is optimized using a noise contrast estimation method to maximize the sentiment-related information of each single modality.

2. The text-main-driven learner multi-modal sentiment analysis method according to claim 1, wherein, Extract multimodal data containing embedded student-related emotional information from online classrooms, including: Text information is transcribed from the original video; Acoustic features were extracted using OVAREP software. Facial motion units are extracted using a facial motion coding system; Facial embedding features are extracted using an expression recognition model. 3.The text-based learner multi-modal sentiment analysis method of claim 1, wherein, The formula for feature extraction using the language representation pre-trained model BERT is as follows: wherein X t represents the feature vector of the text modality, BERT(·) represents a Transformer-based language representation pre-training model BERT, and is the training parameter of the model, l t represents the modality sequence length, d t represents the modality feature dimension.

4. The text-main-driven learner multi-modal sentiment analysis method according to claim 1, wherein, The formula for feature extraction using LSTM is as follows: where BiLSTM(·) denotes feature extraction using a BiLSTM model, I m denotes the pre-processed model input, l m denotes the sequence length of modality m, d m denotes the feature dimension of modality m, when m = a, X m denotes the feature vector of the audio modality, denotes the model parameters for training the audio modality feature extraction; when m = v, X m denotes the feature vector of the visual modality, denotes the model parameters for training the visual modality feature extraction.

5. The text-main-driven learner multi-modal sentiment analysis method according to claim 1, wherein, Multimodal information is fused using a cross-modal attention mechanism. The cross-modal attention layer is a CA(·) function, including: wherein, and are inputs of the function, L(·) represents a layer normalization operation, Q, K, V represent query, key and value respectively, T is the modal fusion information, d k represents the feature dimension of the modal k.

6. The text-driven learner multimodal sentiment analysis method according to claim 1, characterized in that, The final result of multimodal feature fusion is output, including: Define a text-centric feature fusion module, with the following mathematical formula: where N is the number of layers of the TAF sub-module, and M is the number of layers of the TVF sub-module. and is the i-th layer input of the TVF sub-module, and is the j-th layer input of the TAF sub-module, l t denotes the length of the modality sequence, d t denotes the dimension of the modality feature; The final result of multimodal feature fusion is obtained through mathematical formulas including: in This is the final output of the TVF submodule. Z is the final output of the TAF submodule. s It is the final output of the multimodal feature fusion module.

7. The text-main-driven learner multi-modal sentiment analysis method according to claim 1, wherein, The method further includes: performing sentiment prediction based on the final output of multimodal feature fusion, including: Emotional polarity prediction using a multilayer perceptron includes the following mathematical formula: where σ(·) denotes the ReLU activation function, W y is the trained parameter of the activation function, is the final predicted sentiment polarity, Z s is the final output of the multi-modal feature fusion module.

8. The text-main-driven learner multi-modal sentiment analysis method according to claim 7, wherein, The method further includes: The sentiment prediction loss is calculated using the mean absolute error formula, which includes: where n is the number of all samples in the dataset, y i and represents the true sentiment polarity and the predicted sentiment polarity of the i-th sample, respectively, is the sentiment prediction loss.

9. A text-main-driven learner multi-modal sentiment analysis apparatus, characterized in that, The device includes: The first module is used to extract multimodal data containing embedded emotional information of students in online classrooms; The second module is used to extract features from text modal data using the language representation pre-trained model BERT. The third module is used to extract features from audio and visual modal data using LSTM. The fourth module is used to fuse multimodal information using a cross-modal attention mechanism and output the final result of multimodal feature fusion. The fifth module is used for sentiment analysis based on the final output of multimodal feature fusion; The device is also used to perform the following steps: Based on the final result of multimodal feature fusion, backpropagation is performed to obtain the predicted single-modal data; The correlation is calculated based on the predicted single-modal data and the actual single-modal data; The correlation is optimized using a noise contrast estimation method to maximize the sentiment-related information of each single modality.

10. A text-main-driven learner multi-modal sentiment analysis apparatus, characterized in that, The device includes: At least one processor; At least one memory for storing at least one program; The text-driven learner multimodal sentiment analysis method as described in any one of claims 1-7 is implemented when at least one of the programs is executed by at least one of the processors.

Images