Online collaborative session emotion recognition method and system for uncertain mode missing

Abstract

The invention relates to an online collaborative session emotion recognition method and system oriented to uncertain mode missing. The method comprises the following steps: segmenting collaborative session data into a plurality of sub collaborative session data according to different stages; and converting each piece of sub-collaboration session data into session text data, and performing emotion labeling on each piece of sub-collaboration session data. And extracting single-mode features in the preprocessed sub-collaboration session data, and capturing context information in the sub-collaboration session data through an LSTM layer to obtain each mode representation. And carrying out splicing processing on the plurality of single-mode features, and exploring cross-mode emotion promotion information between the multi-mode fusion features and the single-mode features. And constructing a multi-view emotion reasoning graph, and updating node representation through a graph attention mechanism to obtain fused utterance representation. And training the session emotion prediction layer by fusing utterance representation to obtain a collaborative session emotion recognition model, and calling the collaborative session emotion recognition model to recognize the preprocessed current collaborative session data.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence and collaborative conversation processing, and in particular to an online collaborative conversation emotion recognition method and system for uncertain modality loss. Background Art

[0002] Online collaborative learning has great advantages in cultivating and developing learners' collaborative communication skills, problem-solving skills, and high-order thinking. At present, online collaborative learning has become a very important learning paradigm in higher education. It mainly uses video conferencing and other methods to conduct collaborative conversations to achieve problem solutions or specific learning goals. Due to factors such as time and space barriers, learners are prone to confusion, negative learning emotions, etc. during online collaborative learning conversations. If teachers fail to identify and perceive students' bad learning emotions in a timely manner, and then fail to provide appropriate emotional support and learning intervention in a timely manner, it will make it difficult to effectively promote the collaborative learning conversation process, and ultimately affect the collaborative learning experience and learning outcomes. Therefore, it is very critical to timely and accurately identify the emotional state of learners during collaborative learning conversations, which can provide important decision-making basis for teachers to formulate personalized emotional support services in a timely manner.

[0003] At present, most of the studies on emotion analysis of learners in collaborative learning sessions are based on post-event manual analysis, that is, to understand the learners' emotional state through questionnaire surveys or manually coded videos after the collaborative learning. Obviously, they cannot identify the learners' emotional state and its changes in real time during the collaborative session, and it is difficult to provide corresponding collaborative learning interventions. One of the advantages of online methods is that the interactive session data in the collaborative learning process can be recorded, providing a data basis for teachers and managers to deeply understand the collaborative learning process and collaborative learning effects. Therefore, in recent years, some studies have begun to use the multimodal interactive session data generated in the online collaborative learning process, and proposed methods and models for automatic recognition of learners' emotions based on multimodal data fusion. However, most of these studies have not considered the impact of the contextual context in the collaborative learning process on the learners' emotional state, nor the impact of the learners themselves and other learners on the current learning emotional state, resulting in poor recognition results. In addition, such studies are almost all based on the assumption that each data modality is complete in the collaborative learning process, and do not take into account the fact that due to network connection, background noise and other reasons, uncertain modal data is often missing in the actual collaborative learning session (for example, student A forgets to turn on the camera at time 1, resulting in the loss of visual modality, and student B loses voice data at time 5 due to network reasons, etc.). Although discarding or simply interpolating the missing modal data can solve the problem of missing uncertain modalities, it will bring about problems such as loss of collaborative session context and scarce session data, and further make the emotion recognition effect difficult to meet the actual teaching application.

[0004] In summary, existing multimodal emotion recognition models are difficult to meet the learners' emotion recognition needs in actual complex collaborative conversation scenarios. Summary of the invention

[0005] Based on this, it is necessary to provide an online collaborative conversation emotion recognition method and system for uncertain modality loss that can meet the learners' emotion recognition needs in actual complex collaborative conversation scenarios in response to the above technical problems.

[0006] The present invention provides an online collaborative conversation emotion recognition method for uncertain modality loss, the method comprising:

[0007] Acquire collaborative session data through an audio and video acquisition device, and divide the collaborative session data into a plurality of sub-collaborative session data having a time sequence relationship according to different stages of the collaborative learning process;

[0008] Performing emotion annotation on each sub-collaboration session data to determine the learner's emotional state in each sub-collaboration session data;

[0009] Calling a unimodal feature extractor to extract unimodal features from the preprocessed sub-collaboration session data, and using the unimodal features as input of an LSTM layer to capture context information in the sub-collaboration session data, so as to obtain representations of each modality corresponding to the sub-collaboration session data;

[0010] The plurality of single-modal features are concatenated to obtain a multi-modal fusion feature, and the cross-modal emotion promotion information between the multi-modal fusion feature and the single-modal feature is explored based on the Transformer architecture;

[0011] Based on the enhanced unimodal features, a multi-view sentiment reasoning graph with a dynamic context learning window is constructed, and node representations in the multi-view sentiment reasoning graph are updated through a graph attention mechanism to obtain a fused discourse representation of the sub-collaborative session data;

[0012] The fused speech representation is used as the input of the conversation emotion prediction layer to train a collaborative conversation emotion recognition model, and the collaborative conversation emotion recognition model is called to recognize the preprocessed current collaborative conversation data to obtain a corresponding emotion recognition result;

[0013] Among them, the collaborative session data is obtained by the audio and video acquisition device collecting the conversation interaction content in the online collaborative learning process, including video data and audio data. The preprocessing is used to perform modality filling on the sub-collaborative session data after emotion annotation. The single-modal extractor includes pre-trained fast-RCNN, Wav2vec2.0 and BERT models. The multi-view emotion reasoning graph includes a text modality emotion reasoning graph, an audio modality emotion reasoning graph and a video modality emotion reasoning graph. The conversation emotion prediction layer consists of a fully connected layer and a Softmax layer, which is used to output predicted emotion labels for the fused discourse representation.

[0014] In one embodiment, the step of acquiring the collaborative session data through an audio and video acquisition device and dividing the collaborative session data into a plurality of sub-collaborative session data having a time sequence relationship according to different stages of the collaborative learning process, further comprises:

[0015] Further dividing each of the sub-collaborative conversation data into a plurality of collaborative discourse data in a chronological order according to the conversation turns of different learners in the collaborative learning process;

[0016] Preprocessing the plurality of collaborative discourse data so that when any modal data is missing in the collaborative discourse data, UNK in the vocabulary of the BERT model is called to fill in the missing corresponding modal data;

[0017] Among them, each collaborative discourse data includes at least multiple rounds of collaborative interaction discourse of two learners, and each round of the collaborative interaction discourse has modal representations of audio modality, video modality and text modality, and the multiple collaborative discourse data are used to construct a collaborative session dataset in combination with sub-collaborative session data with emotion annotations. When the audio modality or video modality data is missing in the collaborative discourse data, the audio modality or video modality data is filled with blank frames.

[0018] In one of the embodiments, the calling of a unimodal feature extractor to extract unimodal features from the preprocessed sub-collaboration session data, and using the unimodal features as input of an LSTM layer to capture contextual information in the sub-collaboration session data to obtain representations of each modality corresponding to the sub-collaboration session data, includes:

[0019] Calling the pre-trained fast-RCNN, Wav2vec2.0 and BERT models as the unimodal feature extractors to respectively extract the original unimodal features of the collaborative discourse data, and using the original unimodal features as inputs of the three LSTM layers to capture the contextual features of the original collaborative session;

[0020] Each modal data is input into the fully connected layer for feature dimension conversion, so as to map the modal features corresponding to each modal data into the same dimensional space, and obtain the aligned representations of each modality.

[0021] In one embodiment, the concatenation of the plurality of single-modal features to obtain a multi-modal fusion feature, and exploring cross-modal emotion promotion information between the multi-modal fusion feature and the single-modal feature based on a Transformer architecture, includes:

[0022] When the flow direction of the emotional information is from the multimodal feature to the unimodal feature, the unimodal feature is set as a query vector, and the multimodal feature is set as a key-value pair, and complementary emotional information associated with the unimodal feature is extracted from the multimodal feature; and

[0023] When the emotional information flows from the unimodal feature to the multimodal feature, the multimodal feature is set as the query vector, and the unimodal feature is set as the key-value pair, so as to extract key emotional information from the unimodal feature to enhance the multimodal feature.

[0024] In one of the embodiments, the step of constructing a multi-view sentiment reasoning graph with a dynamic context learning window based on enhanced unimodal features, and updating node representations in the multi-view sentiment reasoning graph through a graph attention mechanism to obtain a fused discourse representation of the sub-collaborative session data includes:

[0025] Constructing the text modality emotion reasoning graph, the audio modality emotion reasoning graph and the video modality emotion reasoning graph for each of the collaborative discourse data, wherein the nodes in each reasoning graph are used to represent the collaborative interaction discourse of each round;

[0026] The multimodal features and unimodal features are fused through a graph attention mechanism to obtain enhanced unimodal features to initialize the collaborative interaction discourse represented by each node.

[0027] In one of the embodiments, the method of constructing a multi-view sentiment reasoning graph with a dynamic context learning window based on enhanced unimodal features, and updating the node representations in the multi-view sentiment reasoning graph through a graph attention mechanism to obtain a fused discourse representation of the sub-collaborative session data, further includes:

[0028] Acquire the speech density and the number of speeches of each learner in the collaborative learning process in the collaborative speech data, and calculate the context connection window of the collaborative speech data based on the speech density and the number of speeches to set the dynamic context learning window;

[0029] Initializing the edge relationship and edge weight of each modal sentiment reasoning graph respectively, wherein the edge weight is calculated and determined based on the sentiment semantic similarity between the current node and a plurality of neighboring nodes in the dynamic context learning window;

[0030] The graph attention mechanism is used to update the information of the current node according to the types of different edges between the neighboring nodes and the dependencies between the nodes.

[0031] In one embodiment, the fused speech representation is used as the input of the conversation emotion prediction layer to train a collaborative conversation emotion recognition model, and the collaborative conversation emotion recognition model is called to recognize the preprocessed current collaborative conversation data to obtain a corresponding emotion recognition result, including:

[0032] Sending the fused speech representation to multiple decoders respectively to reconstruct the initial representation of each modality, and calculating the loss between the reconstructed value and the true value; and

[0033] Based on the multimodal representation and the annotated emotion labels of the collaborative speech data, and in combination with the multimodal representation and emotion labels of the positive example and the multimodal representation and emotion labels of the negative example, the contrast loss between different collaborative speech data is calculated;

[0034] Based on the loss between the reconstructed value and the true value and the contrast loss, the loss function of the collaborative conversation emotion recognition model is determined, and the model parameters are learned through back propagation and gradient descent algorithms to obtain the collaborative conversation emotion recognition model under optimal model parameters.

[0035] The present invention also provides an online collaborative conversation emotion recognition system for uncertain modality loss, the system comprising:

[0036] A session segmentation module, used to obtain collaborative session data through an audio and video acquisition device, and segment the collaborative session data into a plurality of sub-collaborative session data having a time sequence relationship according to different stages of the collaborative learning process;

[0037] The emotion annotation module is used to perform emotion annotation on each sub-collaboration session data to determine the emotional state of the learner in each sub-collaboration session data;

[0038] A modal feature acquisition module, used for calling a unimodal feature extractor to extract unimodal features from the preprocessed sub-collaboration session data, and using the unimodal features as input of an LSTM layer to capture contextual information in the sub-collaboration session data, so as to obtain representations of each modality corresponding to the sub-collaboration session data;

[0039] A feature fusion module, used to concatenate multiple single-modal features to obtain multi-modal fusion features, and explore cross-modal emotion promotion information between the multi-modal fusion features and the single-modal features based on the Transformer architecture;

[0040] A fused discourse representation module, used to construct a multi-view sentiment reasoning graph with a dynamic context learning window based on enhanced unimodal features, and update the node representations in the multi-view sentiment reasoning graph through a graph attention mechanism to obtain a fused discourse representation of the sub-collaborative session data;

[0041] An emotion recognition module is used to use the fused speech representation as the input of the conversation emotion prediction layer, train a collaborative conversation emotion recognition model, and call the collaborative conversation emotion recognition model to recognize the preprocessed current collaborative conversation data to obtain a corresponding emotion recognition result;

[0042] Among them, the collaborative session data is obtained by the audio and video acquisition device collecting the conversation interaction content in the online collaborative learning process, including video data and audio data. The preprocessing is used to perform modality filling on the sub-collaborative session data after emotion annotation. The single-modal extractor includes pre-trained fast-RCNN, Wav2vec2.0 and BERT models. The multi-view emotion reasoning graph includes a text modality emotion reasoning graph, an audio modality emotion reasoning graph and a video modality emotion reasoning graph. The conversation emotion prediction layer consists of a fully connected layer and a Softmax layer, which is used to output predicted emotion labels for the fused discourse representation.

[0043] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method for online collaborative conversation emotion recognition for uncertain modality loss as described in any one of the above is implemented.

[0044] The present invention also provides a computer storage medium storing a computer program, wherein when the computer program is executed by a processor, the method for online collaborative conversation emotion recognition for uncertain modality loss as described in any one of the above is implemented.

[0045] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for online collaborative conversation emotion recognition oriented to uncertain modality loss.

[0046] The above-mentioned online collaborative conversation emotion recognition method and system for the lack of uncertain modalities can meet the automatic and accurate recognition of learners' emotional states during actual online collaborative conversations by extracting and fusing multi-dimensional collaborative conversation emotion information in the absence of uncertain modalities, thereby supporting collaborative learning teachers to implement some targeted emotion intervention strategies to specific learners or collaborative learning groups in a timely manner, in order to improve learners' online collaborative learning experience, enhance learners' online collaborative learning motivation, and improve the learning outcomes of collaborative groups. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A schematic diagram of the process of the online collaborative conversation emotion recognition method for uncertain modality loss provided by the present invention;

[0049] Figure 2 A schematic diagram of the overall process of an online collaborative conversation emotion recognition method for uncertain modality loss in a specific embodiment provided by the present invention;

[0050] Figure 3 A schematic diagram of an online collaborative session emotion recognition system architecture of an online collaborative session emotion recognition method for uncertain modality loss in a specific embodiment provided by the present invention;

[0051] Figure 4 A schematic diagram of the structure of the online collaborative conversation emotion recognition system for uncertain modality loss provided by the present invention;

[0052] Figure 5 This is a diagram of the internal structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] Combine the following Figures 1 to 5 The present invention describes an online collaborative conversation emotion recognition method and system for uncertain modality loss.

[0055] like Figure 1 As shown, in one embodiment, a method for emotion recognition in online collaborative conversations with uncertain modality loss includes the following steps:

[0056] Step S110, acquiring collaborative session data through an audio and video acquisition device, and dividing the collaborative session data into a plurality of sub-collaborative session data having a time sequence relationship according to different stages of the collaborative learning process.

[0057] The collaborative conversation data is obtained by collecting conversation interaction content during the online collaborative learning process using audio and video acquisition equipment, and includes video data and audio data.

[0058] In some embodiments, the online collaborative conversation emotion recognition method for uncertain modality loss provided by the present invention obtains collaborative conversation data through an audio and video acquisition device, and divides the collaborative conversation data into multiple sub-collaborative conversation data with a time sequence relationship according to different stages of the collaborative learning process, and then includes the following steps:

[0059] Step S210: further divide each sub-collaborative conversation data into a plurality of collaborative discourse data in a chronological order according to the conversation turns of different learners in the collaborative learning process.

[0060] Step S220, pre-processing the plurality of collaborative discourse data, so that when any modal data is missing in the collaborative discourse data, UNK in the vocabulary of the BERT model is called to fill in the missing corresponding modal data.

[0061] Among them, each collaborative discourse data includes at least multiple rounds of collaborative interaction discourse of two learners, and each round of collaborative interaction discourse has modal representations of audio modality, video modality and text modality, and multiple collaborative discourse data are used to construct a collaborative conversation dataset in combination with sub-collaborative conversation data with emotion annotations. When audio modality or video modality data is missing in the collaborative discourse data, the audio modality or video modality data is filled with blank frames.

[0062] Step S120 , emotion annotation is performed on each sub-collaboration session data to determine the emotional state of the learner in each sub-collaboration session data.

[0063] Step S130, calling a unimodal feature extractor to extract unimodal features from the preprocessed sub-collaboration session data, and using the unimodal features as input to the LSTM layer to capture contextual information in the sub-collaboration session data, so as to obtain representations of each modality corresponding to the sub-collaboration session data.

[0064] Among them, preprocessing is used to fill the modality of the sub-collaboration session data after emotion annotation, and the single-modal extractor includes pre-trained fast-RCNN, Wav2vec2.0 and BERT models.

[0065] In some embodiments, the online collaborative session emotion recognition method for uncertain modality loss provided by the present invention calls a unimodal feature extractor to extract unimodal features in the preprocessed sub-collaboration session data, and uses the unimodal features as the input of the LSTM layer to capture the contextual information in the sub-collaboration session data, so as to obtain the representations of each modality corresponding to the sub-collaboration session data, specifically including the following steps:

[0066] Step S131, calling the pre-trained fast-RCNN, Wav2vec2.0 and BERT models as unimodal feature extractors to extract the original unimodal features of the collaborative discourse data respectively, and using the original unimodal features as inputs of the three LSTM layers respectively to capture the contextual features of the original collaborative session.

[0067] In step S132, each modal data is input into a fully connected layer for feature dimension conversion, so as to map the modal features corresponding to each modal data into the same dimensional space, and obtain aligned representations of each modality.

[0068] In step S140, multiple unimodal features are concatenated to obtain multimodal fusion features, and cross-modal emotion promotion information between the multimodal fusion features and the unimodal features is explored based on the Transformer architecture.

[0069] In some embodiments, the method for online collaborative conversation emotion recognition for uncertain modality loss provided by the present invention concatenates multiple single-modal features to obtain multi-modal fusion features, and explores cross-modal emotion promotion information between multi-modal fusion features and single-modal features based on the Transformer architecture, specifically including the following steps:

[0070] Step S141, when the flow direction of emotional information is from multimodal features to unimodal features, the unimodal features are set as query vectors, and the multimodal features are set as key-value pairs, and complementary emotional information associated with the unimodal features is extracted from the multimodal features.

[0071] Step S142, when the flow direction of emotional information is from unimodal features to multimodal features, the multimodal features are set as query vectors, and the unimodal features are set as key-value pairs, so as to extract key emotional information from the unimodal features to enhance the multimodal features.

[0072] Step S150, constructing a multi-view sentiment reasoning graph with a dynamic context learning window based on the enhanced unimodal features, and updating the node representations in the multi-view sentiment reasoning graph through a graph attention mechanism to obtain a fused discourse representation of the sub-collaborative session data.

[0073] Among them, the multi-view emotion reasoning graph includes text modality emotion reasoning graph, audio modality emotion reasoning graph and video modality emotion reasoning graph.

[0074] In some embodiments, the online collaborative conversation emotion recognition method for uncertain modality loss provided by the present invention constructs a multi-view emotion reasoning graph with a dynamic context learning window based on enhanced unimodal features, and updates the node representations in the multi-view emotion reasoning graph through a graph attention mechanism to obtain a fused discourse representation of the sub-collaborative conversation data, specifically comprising the following steps:

[0075] Step S151, constructing a text modality emotion reasoning graph, an audio modality emotion reasoning graph, and a video modality emotion reasoning graph for each collaborative discourse data, and the nodes in each reasoning graph are used to represent each round of collaborative interaction discourse.

[0076] Step S152, multimodal features and unimodal features are fused through a graph attention mechanism to obtain enhanced unimodal features to initialize the collaborative interaction discourse represented by each node.

[0077] In some embodiments, the online collaborative session emotion recognition method for uncertain modality loss provided by the present invention constructs a multi-view emotion reasoning graph with a dynamic context learning window based on enhanced unimodal features, and updates the node representations in the multi-view emotion reasoning graph through a graph attention mechanism to obtain a fused discourse representation of the sub-collaborative session data, and specifically includes the following steps:

[0078] Step S153, obtaining the discourse density and the number of discourses of each learner in the collaborative learning process in the collaborative discourse data, and calculating the context connection window of the collaborative discourse data based on the discourse density and the number of discourses to set a dynamic context learning window.

[0079] Step S154, initializing the edge relationship and edge weight of each modal sentiment reasoning graph respectively, where the edge weight is calculated and determined based on the sentiment semantic similarity between the current node and multiple neighboring nodes in the dynamic context learning window.

[0080] Step S155, using the graph attention mechanism, updates the information of the current node according to the types of different edges between neighboring nodes and the dependencies between nodes.

[0081] Step S160, using the fused speech representation as the input of the conversation emotion prediction layer, training a collaborative conversation emotion recognition model, and calling the collaborative conversation emotion recognition model to recognize the preprocessed current collaborative conversation data to obtain a corresponding emotion recognition result.

[0082] Among them, the conversation sentiment prediction layer consists of a fully connected layer and a Softmax layer, which is used to output the predicted sentiment label for the fused discourse representation.

[0083] In some embodiments, the online collaborative conversation emotion recognition method for uncertain modality loss provided by the present invention uses the fused speech representation as the input of the conversation emotion prediction layer, trains to obtain a collaborative conversation emotion recognition model, and calls the collaborative conversation emotion recognition model to recognize the preprocessed current collaborative conversation data to obtain the corresponding emotion recognition result, which specifically includes the following steps:

[0084] Step S161, sending the fused speech representation to multiple decoders respectively to reconstruct the initial representation of each modality and calculate the loss between the reconstructed value and the true value.

[0085] Step S162, based on the multimodal representation and annotated emotion labels of the collaborative discourse data, and in combination with the multimodal representation and emotion labels of the positive example and the multimodal representation and emotion labels of the negative example, the contrast loss between different collaborative discourse data is calculated.

[0086] Step S163, based on the loss and contrast loss between the reconstructed value and the true value, determine the loss function of the collaborative conversation emotion recognition model, and learn the model parameters through back propagation and gradient descent algorithms to obtain the collaborative conversation emotion recognition model under the optimal model parameters.

[0087] The above-mentioned online collaborative conversation emotion recognition method for uncertain modality missing can meet the automatic and accurate recognition of learners' emotional states in actual online collaborative conversations by extracting and fusing multi-dimensional collaborative conversation emotion information in the case of uncertain modality missing, thereby supporting collaborative learning teachers to implement some targeted emotion intervention strategies to specific learners or collaborative learning groups in a timely manner, in order to improve learners' online collaborative learning experience, enhance learners' online collaborative learning motivation, and improve the learning outcomes of collaborative groups.

[0088] See also Figure 2 and Figure 3As shown, in a specific embodiment, the online collaborative conversation emotion recognition method for uncertain modality loss provided by the present invention uses multi-channel conversation behavior data collected in the actual online collaborative learning process to construct a collaborative conversation data set containing three modalities of text, audio and image, and performs preliminary data cleaning on each modality data. Then, the collaborative conversation emotion recognition model for uncertain modality loss is trained, including extracting single-modal features respectively using multiple pre-trained models and capturing initial single-modal context information using the LSTM model; constructing an emotion promotion module between multi-modality and single-modality based on the Transformer architecture to capture the emotional semantic context information in the collaborative conversation process from multiple perspectives and dimensions; then, constructing a multi-perspective emotion reasoning graph with a dynamic context learning window, including initializing graph nodes based on the aforementioned emotion promotion module, defining the type of edges in the graph based on the interaction of learners in the collaborative conversation process, initializing edge weights based on the semantic similarity of nodes, and dynamically setting the context learning window based on the speech density of learners to update node features using the node information propagation mechanism in the graph.

[0089] Finally, the emotional information learned from the emotional promotion module and the emotional reasoning graph module is spliced ​​and fused to obtain the final multimodal representation of each collaborative utterance. This multimodal fusion representation is used for collaborative learning of emotion recognition, and is simultaneously sent to three single-modal decoders for reconstructing the original input data information. It is also used to conduct comparative learning of multimodal representations between different samples under the constraint of emotional labels, and then the loss function of the final model training is obtained based on three types of loss functions to guide the training process of the emotion recognition model. This collaborative conversation emotion recognition model can be used for learner emotion recognition in uncertain modality-missing collaborative conversation situations.

[0090] In this embodiment, the following steps are specifically included:

[0091] Step 1: Collaborative conversation data collection: Use audio and video acquisition equipment to collect conversation interaction content during the online collaborative learning process, including video data and audio data of conversation interaction during the online collaborative learning process collected by audio and video acquisition equipment.

[0092] Step 2: Construction of collaborative conversation dataset. First, the original audio and video data are coarsely segmented into collaborative conversations of different groups at different stages according to the different stages of the collaborative learning process; then, for each collaborative conversation, the audio and video data are further segmented into collaborative discourses with a time sequence according to the conversation turns of different learners in the collaborative interaction process, and the corresponding conversation text data is generated using an audio transcription tool; finally, two experts in the field of educational technology perform sentiment annotation on the collaborative conversations to determine the learner's emotional state contained in each collaborative discourse. At this point, the online collaborative learning process can be regarded as an online collaborative conversation dataset composed of multiple online collaborative conversations, each of which contains multiple rounds of collaborative interaction discourses of at least two learners, and each discourse is expressed in three modalities: audio, video, and text. Therefore, a collaborative conversation data can be represented as The set of learners involved in the dialogue is S = {s1, s2, s3, ..., s M}, words Depend on Say, among them A mapping function that represents an utterance to its corresponding speaker.

[0093] Step 3: Data preprocessing. In the actual collaborative learning process, due to network connection, background noise and other reasons, some speech samples u i Missing data from one or two modalities (assuming that the utterance sample u i If a speech sample has complete data of at least one modality), it needs to be preprocessed, including: if the text modality of a speech sample is missing, it is filled with UNK in the BERT vocabulary; if the audio and / or video modalities of a speech sample are missing, it is filled with blank frames.

[0094] Step 4: Training of emotion recognition model for collaborative conversation with uncertain modality loss, including:

[0095] (1) Extracting the initial unimodal context features of the utterance:

[0096] Specifically, the pre-trained fast-RCNN, Wav2vec2.0 and BERT models are used as unimodal feature extractors to extract the original unimodal features of each utterance. The original unimodal features are then fed into three LSTM layers to capture the initial collaborative session context information. Finally, the information of each modality is sent to the fully connected layer for feature dimension change, so that the three modal features are mapped to the same dimensional space to obtain the aligned representation of each modality.

[0097] (2) Construct an emotion promotion module based on the Transformer architecture.

[0098] Specifically, a Transformer-based emotion promotion unit is constructed for each modality to achieve mutual promotion between multimodal information and unimodal information. m Perform simple splicing to obtain the initial multimodal fusion features, that is, Furthermore, based on the Transformer architecture, we explore the cross-modal sentiment promotion information between multimodality and each unimodality, that is, when the direction of sentiment information flow is from multimodality to unimodality, the features of modality m are set as query vector Q, and the multimodal features are set as key-value pairs (K and V) to extract complementary sentiment information related to the current modality m from multimodality. Similarly, when the direction of sentiment information flow is from unimodality to multimodality, the multimodal features are set as query vector Q of the current sentiment promotion direction, and the unimodal features are set as key-value pairs (K and V) to extract important sentiment information from unimodality to enhance multimodal features. The process is expressed as:

[0099]

[0100] In the formula, MHCA refers to the multi-head cross-modal interaction calculation based on the Transformer architecture. After the first emotion promotion layer, the single modal feature is updated to The multimodal features are updated as After l emotion promotion layers, the unimodal feature is updated to The multimodal features are updated to F l Both unimodal and multimodal features contain emotional clues that complement each other’s contexts.

[0101] (3) Constructing a multi-view sentiment reasoning graph with dynamic context learning window:

[0102] It is planned to construct three modal interaction graphs for each dialogue (including text modal sentiment reasoning graph Audio modality sentiment inference graph Video modality sentiment inference graph ), each node in the graph represents a discourse.

[0103] First, the representation of each node in the graph is initialized. The unimodal features and multimodal features learned by the above-mentioned emotion promotion module are fused based on the attention mechanism to obtain enhanced unimodal representations, which are used to initialize the corresponding node features in each modal interaction graph. The process is described as:

[0104]

[0105] In the formula, and b1 is a learnable parameter, Initialize features for nodes of the interaction graph of modality m.

[0106] Secondly, the context learning window in the sentiment reasoning graph is dynamically set. Specifically, the speech density of each learner in the dialogue during the collaboration process is first calculated. Assuming that the current speech u i By S A Sent, calculate the weight α of the number of learner A's utterances in the entire conversation. Secondly, multiply the weight α by the sequence number i of the current utterance in the conversation to calculate the number of utterances that the learner corresponding to the current utterance has issued. Finally, calculate the context connection window ω of the current utterance, that is, ω = i*α + S-1 (where S represents the number of learners participating in the current collaborative session, and S-1 represents considering other learners except the current learner), so as to dynamically set the context learning window of the current utterance.

[0107] Finally, the edge relationship and edge weight are initialized for each modal interaction graph. Assume that for the sentiment inference graph of modality m, the edge relationship is determined by the speaker type. If the two utterance nodes of an edge are both issued by the same learner, the edge type is "same learner"; otherwise, the edge type is "different learner". The edge weight is calculated based on the sentiment semantic similarity between the current node and the ω neighboring nodes in the dynamic context window. Specifically, the similarity between each node and all nodes in the context window is calculated as:

[0108]

[0109] In the formula, represents the initialization features of node i in the sentiment inference graph of modality m, represents the initialization features of node j within the context window ω, Represents the edge weight between the current node i and node j in the sentiment reasoning graph of modality m. So far, the construction of the sentiment reasoning graphs of the three modalities can be completed respectively.

[0110] (4) Update node representation in the sentiment reasoning graph based on graph attention mechanism:

[0111] The graph attention mechanism is used to update the aggregated neighbor node information, and the current node information is updated according to the type of different edges and the dependencies between nodes. After one layer of graph convolution, the edge weight is updated as follows:

[0112]

[0113] In the formula, and are the initial features of nodes i and j in the sentiment reasoning graph of modality m, a T and W αrepresents the learnable parameters, N(u i ) The set of neighbor nodes of the current node within the context learning window ω, is the updated edge weight between node i and node j in the sentiment reasoning graph of modality m. Then, according to the updated weight, the current discourse node after a layer of graph convolution information propagation is represented as:

[0114]

[0115] For the sentiment reasoning graph of modality m, the sentiment reasoning information learned after r layers of graph convolution information propagation can be represented as Based on the attention mechanism, the emotional reasoning information under this modality or perspective can be calculated as:

[0116]

[0117] In the formula, and b2 is a learnable parameter, G r It is the emotional reasoning feature representation of discourse node i obtained by aggregation and updating after graph information propagation based on speaker information and emotional semantic information.

[0118] (5) Training of emotion recognition model for collaborative conversation with uncertain modality loss:

[0119] Based on the above steps, the emotional information learned by the emotional promotion module and the emotional reasoning graph module are concatenated and fused to obtain the final fused discourse representation, which is used as the input of the conversation emotion prediction layer. The conversation emotion prediction layer consists of a fully connected layer and a Softmax layer, which outputs a predicted emotion label for each fused discourse representation.

[0120] Assume that the original input of utterance i is Its true sentiment label is y i The emotional interaction information learned by the emotion promotion module for utterance i is F l , the emotional reasoning information learned through the emotional reasoning graph is G r , concatenate and fuse them to obtain the final multimodal representation Z = Concat (F l ,G r ). After the multimodal representation Z passes through the conversation sentiment prediction layer, the predicted sentiment label is Therefore, the objective function of the conversation emotion recognition task is:

[0121]

[0122] Among them, N represents the number of utterances in the conversation, and C represents the number of emotion categories.

[0123] At the same time, it is considered that in the process of real collaborative conversations, some modal data of some discourses are often missing due to various reasons. This method fills the missing data in the preprocessing stage, but the filled data may be quite different from the real data. Therefore, this method also considers the reconstruction loss of the data, that is, it encourages the emotion recognition model to reconstruct the complete collaborative conversation data from the potential multimodal representation space as much as possible to guide the model to learn the real semantic information of the missing part. Specifically, the final multimodal fusion representation Z is sent to the three decoders respectively to reconstruct the initial representation of each modality, and the loss between the reconstructed value and the true value is calculated:

[0124]

[0125] Among them, Encoder indicates that the reconstruction encoder can be composed of a fully connected layer, θ m represents the encoder parameters, λ m represents the missing rate of mode m, X m and They represent the original unimodal features and reconstructed features of modality m respectively.

[0126] In addition, in order to make the multimodal features learned by the model present the sample points of the same emotion category as close as possible in the feature space (i.e., positive examples), and the sample points of different emotion categories as far away as possible (i.e., negative examples), this method also considers calculating the contrast loss between sample points. Specifically, assuming that the speech sample u i The final multimodal representation of i And the sentiment label is y i , its positive example u j The multimodal representation of i , its sentiment label y j =y i ; Its negative example u k The multimodal representation of k , its sentiment label y k ≠y i , then the contrast loss is:

[0127]

[0128] Among them, sim represents the cosine similarity function, τ is the temperature coefficient, L cl (z i ) represents the calculation of the speech sample u in each training batch i The contrast loss, L cl represents the sum of the contrast losses of all samples in the entire training set. Finally, the training loss function of the collaborative conversation emotion recognition model for uncertain modality loss is:

[0129]

[0130] in, and is the hyperparameter coefficient to balance the relationship between the three types of losses. The parameters of the collaborative conversation emotion recognition model are learned through back propagation and stochastic gradient descent algorithms. When the objective function reaches the optimal solution, the conversation emotion recognition model with the optimal model parameters is obtained.

[0131] Step 5: Emotion recognition of new collaborative conversation data.

[0132] Specifically, the new samples are preprocessed in the same way as the training data, so that they become data types that can be recognized and efficiently processed by the collaborative conversation emotion recognition model; then the preprocessed data of each channel are sent to the trained collaborative conversation emotion recognition model to finally obtain the emotion prediction result for the sample.

[0133] In this embodiment, the focus is on how to solve the problem of missing discourse modalities that often occur in real collaborative conversation scenarios. It is proposed to fill in the preprocessing stage first, and finally make the learned emotional features as close to the real information as possible through reconstruction loss and contrast loss in the model training stage, so that it has a high practical application value. Secondly, this method constructs an emotional promotion module based on the Transformer architecture to achieve the two-way flow and mutual promotion of complementary emotional information between single modality and multimodality, that is, it retains the unique emotional feature information of single modality and enhances the mining of multimodal complementary emotional information. In addition, this method also dynamically sets the context learning window based on the frequency of speech and the number of speakers in the conversation, and initializes the edge weights in the emotional graph by calculating the similarity between nodes, avoiding the interference of irrelevant noise information, and retaining and enhancing the emotional information contained in the node features themselves.

[0134] In addition, this method can solve the problem that even if some data is partially missing due to network reasons in real online collaborative conversation scenarios, it can also automatically and accurately perceive the emotional state of learners in the collaborative learning process, provide decision-making basis for teachers' personalized learning emotional intervention and support, and help improve learners' collaborative learning experience, improve learners' collaborative learning motivation, and improve the collaborative learning results of collaborative groups. Compared with traditional manual recognition or multimodal emotion recognition models, this method and system take into account the current situation of actual online collaborative learning scenarios (such as the existence of multi-channel recording data) and problems (such as the partial loss of some modes of some discourses due to network, equipment, etc.), and consider how to better capture conversation context information, interaction information between modalities, and interaction information between different learners from the perspective of method design, so as to learn emotional information more comprehensively from incomplete data. Therefore, this method is more scientific and more robust, and has better emotional recognition effects.

[0135] The following is a description of the online collaborative session emotion recognition system for uncertain modality loss provided by the present invention. The online collaborative session emotion recognition system for uncertain modality loss described below and the online collaborative session emotion recognition method for uncertain modality loss described above can refer to each other.

[0136] like Figure 4 As shown, in one embodiment, an online collaborative conversation emotion recognition system for uncertain modality loss includes a conversation segmentation module 410, an emotion annotation module 420, a modal feature acquisition module 430, a feature fusion module 440, a fused discourse representation module 450 and an emotion recognition module 460.

[0137] The session segmentation module 410 is used to obtain collaborative session data through an audio and video acquisition device, and segment the collaborative session data into a plurality of sub-collaborative session data having a time sequence relationship according to different stages of the collaborative learning process.

[0138] The emotion annotation module 420 is used to perform emotion annotation on each sub-collaboration session data to determine the emotion state of the learner in each sub-collaboration session data.

[0139] The modal feature acquisition module 430 is used to call the unimodal feature extractor to extract the unimodal features in the preprocessed sub-collaboration session data, and use the unimodal features as the input of the LSTM layer to capture the contextual information in the sub-collaboration session data to obtain the representations of each modality corresponding to the sub-collaboration session data.

[0140] The feature fusion module 440 is used to concatenate multiple unimodal features to obtain multimodal fusion features, and explore the cross-modal emotion promotion information between the multimodal fusion features and the unimodal features based on the Transformer architecture.

[0141] The fused discourse representation module 450 is used to construct a multi-view sentiment reasoning graph with a dynamic context learning window based on enhanced unimodal features, and to update the node representations in the multi-view sentiment reasoning graph through a graph attention mechanism to obtain a fused discourse representation of the sub-collaborative session data.

[0142] The emotion recognition module 460 is used to use the fused speech representation as the input of the conversation emotion prediction layer, train a collaborative conversation emotion recognition model, and call the collaborative conversation emotion recognition model to recognize the preprocessed current collaborative conversation data to obtain corresponding emotion recognition results.

[0143] Among them, the collaborative session data is obtained by collecting the conversation interaction content in the online collaborative learning process through audio and video acquisition equipment, including video data and audio data. The preprocessing is used to fill the modality of the sub-collaborative session data after emotion annotation. The single-modal extractor includes pre-trained fast-RCNN, Wav2vec2.0 and BERT models. The multi-view emotion reasoning graph includes text modality emotion reasoning graph, audio modality emotion reasoning graph and video modality emotion reasoning graph. The conversation emotion prediction layer consists of a fully connected layer and a Softmax layer, which is used to output the predicted emotion label for the fused discourse representation.

[0144] In this embodiment, the online collaborative conversation emotion recognition system for uncertain modality loss provided by the present invention further includes a data preprocessing module for:

[0145] According to the conversation turns of different learners in the collaborative learning process, each sub-collaborative conversation data is further divided into multiple collaborative discourse data with a chronological order.

[0146] Preprocess multiple collaborative discourse data so that when any modality data is missing in the collaborative discourse data, UNK in the vocabulary of the BERT model is called to fill in the missing corresponding modality data.

[0147] Among them, each collaborative discourse data includes at least multiple rounds of collaborative interaction discourse of two learners, and each round of collaborative interaction discourse has modal representations of audio modality, video modality and text modality, and multiple collaborative discourse data are used to construct a collaborative conversation dataset in combination with sub-collaborative conversation data with emotion annotations. When audio modality or video modality data is missing in the collaborative discourse data, the audio modality or video modality data is filled with blank frames.

[0148] In this embodiment, in the online collaborative conversation emotion recognition system for uncertain modality loss provided by the present invention, the modality feature acquisition module 430 is specifically used for:

[0149] The pre-trained fast-RCNN, Wav2vec2.0, and BERT models are called as unimodal feature extractors to extract the original unimodal features of the collaborative discourse data respectively, and the original unimodal features are used as the input of the three LSTM layers respectively to capture the contextual features of the original collaborative conversation.

[0150] Each modal data is input into the fully connected layer for feature dimension transformation, so as to map the modal features corresponding to each modal data into the same dimensional space and obtain the aligned representations of each modality.

[0151] In this embodiment, in the online collaborative conversation emotion recognition system for uncertain modality loss provided by the present invention, the feature fusion module 440 is specifically used for:

[0152] When the flow direction of emotional information is from multimodal features to unimodal features, the unimodal features are set as query vectors, and the multimodal features are set as key-value pairs, and complementary emotional information associated with the unimodal features is extracted from the multimodal features.

[0153] When the direction of emotional information flow is from unimodal features to multimodal features, the multimodal features are set as query vectors, and the unimodal features are set as key-value pairs, so as to extract key emotional information from the unimodal features to enhance the multimodal features.

[0154] In this embodiment, in the online collaborative conversation emotion recognition system for uncertain modality loss provided by the present invention, the fusion discourse representation module 450 is specifically used for:

[0155] For each collaborative discourse data, a text modality sentiment reasoning graph, an audio modality sentiment reasoning graph, and a video modality sentiment reasoning graph are constructed. The nodes in each reasoning graph are used to represent the collaborative interaction discourse of each round.

[0156] Multimodal features and unimodal features are fused through the graph attention mechanism to obtain enhanced unimodal features to initialize the collaborative interaction discourse represented by each node.

[0157] In this embodiment, in the online collaborative conversation emotion recognition system for uncertain modality loss provided by the present invention, the fusion discourse representation module 450 is further used for:

[0158] The discourse density and the number of discourses of each learner in the collaborative learning process are obtained, and the context connection window of the collaborative discourse data is calculated based on the discourse density and the number of discourses to set a dynamic context learning window.

[0159] The edge relations and edge weights of each modal sentiment reasoning graph are initialized respectively. The edge weights are calculated and determined based on the sentiment semantic similarity between the current node and multiple neighboring nodes in the dynamic context learning window.

[0160] The graph attention mechanism is used to update the information of the current node according to the types of different edges between neighboring nodes and the dependencies between nodes.

[0161] In this embodiment, in the online collaborative conversation emotion recognition system for uncertain modality loss provided by the present invention, the emotion recognition module 460 is specifically used for:

[0162] The fused speech representation is sent to multiple decoders to reconstruct the initial representation of each modality and calculate the loss between the reconstructed value and the true value.

[0163] Based on the multimodal representation and annotated sentiment labels of collaborative discourse data, and combined with the multimodal representation and sentiment labels of positive examples and the multimodal representation and sentiment labels of negative examples, the contrast loss between different collaborative discourse data is calculated.

[0164] Based on the loss and contrast loss between the reconstructed value and the true value, the loss function of the collaborative conversation emotion recognition model is determined, and the model parameters are learned through back propagation and gradient descent algorithms to obtain the collaborative conversation emotion recognition model under the optimal model parameters.

[0165] Figure 5 An example of a physical structure diagram of an electronic device is shown. The electronic device may be a smart terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The electronic device includes a processor, an internal memory and a network interface connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, an online collaborative session emotion recognition method for uncertain modality loss is implemented, and the method includes:

[0166] Acquire collaborative session data through audio and video acquisition equipment, and divide the collaborative session data into multiple sub-collaborative session data with a time sequence relationship according to different stages of the collaborative learning process;

[0167] Performing emotion annotation on each sub-collaboration session data to determine the learner's emotional state in each sub-collaboration session data;

[0168] The unimodal feature extractor is called to extract the unimodal features from the preprocessed sub-collaboration session data, and the unimodal features are used as the input of the LSTM layer to capture the context information in the sub-collaboration session data, so as to obtain the representations of each modality corresponding to the sub-collaboration session data;

[0169] Multiple single-modal features are concatenated to obtain multi-modal fusion features, and the cross-modal sentiment promotion information between multi-modal fusion features and single-modal features is explored based on the Transformer architecture;

[0170] Based on the enhanced unimodal features, a multi-view sentiment reasoning graph with a dynamic context learning window is constructed, and the node representations in the multi-view sentiment reasoning graph are updated through the graph attention mechanism to obtain the fused discourse representation of the sub-collaborative session data;

[0171] The fused speech representation is used as the input of the conversation emotion prediction layer to train the collaborative conversation emotion recognition model, and the collaborative conversation emotion recognition model is called to recognize the preprocessed current collaborative conversation data to obtain the corresponding emotion recognition result;

[0172] Among them, the collaborative session data is obtained by collecting the conversation interaction content in the online collaborative learning process through audio and video acquisition equipment, including video data and audio data. The preprocessing is used to fill the modality of the sub-collaborative session data after emotion annotation. The single-modal extractor includes pre-trained fast-RCNN, Wav2vec2.0 and BERT models. The multi-view emotion reasoning graph includes text modality emotion reasoning graph, audio modality emotion reasoning graph and video modality emotion reasoning graph. The conversation emotion prediction layer consists of a fully connected layer and a Softmax layer, which is used to output the predicted emotion label for the fused discourse representation.

[0173] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present invention, and does not constitute a limitation on the electronic device to which the scheme of the present invention is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0174] On the other hand, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, implements an online collaborative conversation emotion recognition method for uncertain modality loss, the method comprising:

[0175] Acquire collaborative session data through audio and video acquisition equipment, and divide the collaborative session data into multiple sub-collaborative session data with a time sequence relationship according to different stages of the collaborative learning process;

[0176] Performing emotion annotation on each sub-collaboration session data to determine the learner's emotional state in each sub-collaboration session data;

[0177] The unimodal feature extractor is called to extract the unimodal features from the preprocessed sub-collaboration session data, and the unimodal features are used as the input of the LSTM layer to capture the context information in the sub-collaboration session data, so as to obtain the representations of each modality corresponding to the sub-collaboration session data;

[0178] Multiple single-modal features are concatenated to obtain multi-modal fusion features, and the cross-modal sentiment promotion information between multi-modal fusion features and single-modal features is explored based on the Transformer architecture;

[0179] Based on the enhanced unimodal features, a multi-view sentiment reasoning graph with a dynamic context learning window is constructed, and the node representations in the multi-view sentiment reasoning graph are updated through the graph attention mechanism to obtain the fused discourse representation of the sub-collaborative session data;

[0180] The fused speech representation is used as the input of the conversation emotion prediction layer to train the collaborative conversation emotion recognition model, and the collaborative conversation emotion recognition model is called to recognize the preprocessed current collaborative conversation data to obtain the corresponding emotion recognition result;

[0181] Among them, the collaborative session data is obtained by collecting the conversation interaction content in the online collaborative learning process through audio and video acquisition equipment, including video data and audio data. The preprocessing is used to fill the modality of the sub-collaborative session data after emotion annotation. The single-modal extractor includes pre-trained fast-RCNN, Wav2vec2.0 and BERT models. The multi-view emotion reasoning graph includes text modality emotion reasoning graph, audio modality emotion reasoning graph and video modality emotion reasoning graph. The conversation emotion prediction layer consists of a fully connected layer and a Softmax layer, which is used to output the predicted emotion label for the fused discourse representation.

[0182] In another aspect, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and when the processor executes the computer instructions, a method for online collaborative session emotion recognition for uncertain modality loss is implemented, the method comprising:

[0183] Acquire collaborative session data through audio and video acquisition equipment, and divide the collaborative session data into multiple sub-collaborative session data with a time sequence relationship according to different stages of the collaborative learning process;

[0184] Performing emotion annotation on each sub-collaboration session data to determine the learner's emotional state in each sub-collaboration session data;

[0185] The unimodal feature extractor is called to extract the unimodal features from the preprocessed sub-collaboration session data, and the unimodal features are used as the input of the LSTM layer to capture the context information in the sub-collaboration session data, so as to obtain the representations of each modality corresponding to the sub-collaboration session data;

[0186] Multiple single-modal features are concatenated to obtain multi-modal fusion features, and the cross-modal sentiment promotion information between multi-modal fusion features and single-modal features is explored based on the Transformer architecture;

[0187] Based on the enhanced unimodal features, a multi-view sentiment reasoning graph with a dynamic context learning window is constructed, and the node representations in the multi-view sentiment reasoning graph are updated through the graph attention mechanism to obtain the fused discourse representation of the sub-collaborative session data;

[0188] The fused speech representation is used as the input of the conversation emotion prediction layer to train the collaborative conversation emotion recognition model, and the collaborative conversation emotion recognition model is called to recognize the preprocessed current collaborative conversation data to obtain the corresponding emotion recognition result;

[0189] Among them, the collaborative session data is obtained by collecting the conversation interaction content in the online collaborative learning process through audio and video acquisition equipment, including video data and audio data. The preprocessing is used to fill the modality of the sub-collaborative session data after emotion annotation. The single-modal extractor includes pre-trained fast-RCNN, Wav2vec2.0 and BERT models. The multi-view emotion reasoning graph includes text modality emotion reasoning graph, audio modality emotion reasoning graph and video modality emotion reasoning graph. The conversation emotion prediction layer consists of a fully connected layer and a Softmax layer, which is used to output the predicted emotion label for the fused discourse representation.

[0190] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory.

[0191] By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0192] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0193] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A method for emotion recognition in online collaborative conversations with uncertain modality loss, characterized in that: The method comprises: Acquire collaborative session data through an audio and video acquisition device, and divide the collaborative session data into a plurality of sub-collaborative session data having a time sequence relationship according to different stages of the collaborative learning process; Performing emotion annotation on each sub-collaboration session data to determine the learner's emotional state in each sub-collaboration session data; Calling a unimodal feature extractor to extract unimodal features from the preprocessed sub-collaboration session data, and using the unimodal features as input of an LSTM layer to capture context information in the sub-collaboration session data, so as to obtain representations of each modality corresponding to the sub-collaboration session data; The plurality of single-modal features are concatenated to obtain a multi-modal fusion feature, and the cross-modal emotion promotion information between the multi-modal fusion feature and the single-modal feature is explored based on the Transformer architecture; Based on the enhanced unimodal features, a multi-view sentiment reasoning graph with a dynamic context learning window is constructed, and node representations in the multi-view sentiment reasoning graph are updated through a graph attention mechanism to obtain a fused discourse representation of the sub-collaborative session data; The fused speech representation is used as the input of the conversation emotion prediction layer to train a collaborative conversation emotion recognition model, and the collaborative conversation emotion recognition model is called to recognize the preprocessed current collaborative conversation data to obtain a corresponding emotion recognition result; Among them, the collaborative session data is obtained by the audio and video acquisition device collecting the conversation interaction content in the online collaborative learning process, including video data and audio data. The preprocessing is used to perform modality filling on the sub-collaborative session data after emotion annotation. The single-modal extractor includes pre-trained fast-RCNN, Wav2vec2.0 and BERT models. The multi-view emotion reasoning graph includes a text modality emotion reasoning graph, an audio modality emotion reasoning graph and a video modality emotion reasoning graph. The conversation emotion prediction layer consists of a fully connected layer and a Softmax layer, which is used to output predicted emotion labels for the fused discourse representation.

2. The method for online collaborative conversation emotion recognition for uncertain modality loss according to claim 1 is characterized in that: The step of acquiring the collaborative session data through the audio and video acquisition device and dividing the collaborative session data into a plurality of sub-collaborative session data having a time sequence relationship according to different stages of the collaborative learning process, further comprises: Further dividing each of the sub-collaborative conversation data into a plurality of collaborative discourse data in a chronological order according to the conversation turns of different learners in the collaborative learning process; Preprocessing the plurality of collaborative discourse data so that when any modal data is missing in the collaborative discourse data, UNK in the vocabulary of the BERT model is called to fill in the missing corresponding modal data; Among them, each collaborative discourse data includes at least multiple rounds of collaborative interaction discourse of two learners, and each round of the collaborative interaction discourse has modal representations of audio modality, video modality and text modality, and the multiple collaborative discourse data are used to construct a collaborative session dataset in combination with sub-collaborative session data with emotion annotations. When the audio modality or video modality data is missing in the collaborative discourse data, the audio modality or video modality data is filled with blank frames.

3. The method for online collaborative conversation emotion recognition for uncertain modality loss according to claim 2 is characterized in that: The calling of the unimodal feature extractor to extract the unimodal features in the preprocessed sub-collaboration session data, and using the unimodal features as the input of the LSTM layer to capture the context information in the sub-collaboration session data, so as to obtain the representations of each modality corresponding to the sub-collaboration session data, includes: Calling the pre-trained fast-RCNN, Wav2vec2.0 and BERT models as the unimodal feature extractors to respectively extract the original unimodal features of the collaborative discourse data, and using the original unimodal features as inputs of the three LSTM layers to capture the contextual features of the original collaborative session; Each modal data is input into the fully connected layer for feature dimension conversion, so as to map the modal features corresponding to each modal data into the same dimensional space, and obtain the aligned representations of each modality.

4. The method for online collaborative conversation emotion recognition for uncertain modality loss according to claim 3 is characterized in that: The step of concatenating the plurality of single-modal features to obtain a multi-modal fusion feature, and exploring cross-modal emotion promotion information between the multi-modal fusion feature and the single-modal feature based on a Transformer architecture includes: When the flow direction of the emotional information is from the multimodal feature to the unimodal feature, the unimodal feature is set as a query vector, and the multimodal feature is set as a key-value pair, and complementary emotional information associated with the unimodal feature is extracted from the multimodal feature; and When the emotional information flows from the unimodal feature to the multimodal feature, the multimodal feature is set as the query vector, and the unimodal feature is set as the key-value pair, so as to extract key emotional information from the unimodal feature to enhance the multimodal feature.

5. The method for online collaborative conversation emotion recognition for uncertain modality loss according to claim 4 is characterized in that: The method of constructing a multi-view sentiment reasoning graph with a dynamic context learning window based on the enhanced unimodal features, and updating the node representations in the multi-view sentiment reasoning graph through a graph attention mechanism to obtain a fused discourse representation of the sub-collaborative session data, includes: Constructing the text modality emotion reasoning graph, the audio modality emotion reasoning graph and the video modality emotion reasoning graph for each of the collaborative discourse data, wherein the nodes in each reasoning graph are used to represent the collaborative interaction discourse of each round; The multimodal features and unimodal features are fused through a graph attention mechanism to obtain the enhanced unimodal features to initialize the collaborative interaction discourse represented by each node.

6. The method for online collaborative conversation emotion recognition for uncertain modality loss according to claim 5 is characterized in that: The method of constructing a multi-view sentiment reasoning graph with a dynamic context learning window based on the enhanced unimodal features, and updating the node representations in the multi-view sentiment reasoning graph through a graph attention mechanism to obtain a fused discourse representation of the sub-collaborative session data, further includes: Acquire the speech density and the number of speeches of each learner in the collaborative learning process in the collaborative speech data, and calculate the context connection window of the collaborative speech data based on the speech density and the number of speeches to set the dynamic context learning window; Initializing the edge relationship and edge weight of each modal sentiment reasoning graph respectively, wherein the edge weight is calculated and determined based on the sentiment semantic similarity between the current node and a plurality of neighboring nodes in the dynamic context learning window; The graph attention mechanism is used to update the information of the current node according to the types of different edges between the neighboring nodes and the dependencies between the nodes.

7. The method for online collaborative conversation emotion recognition for uncertain modality loss according to claim 6, characterized in that: The method uses the fused speech representation as the input of the conversation emotion prediction layer, trains a collaborative conversation emotion recognition model, and calls the collaborative conversation emotion recognition model to recognize the preprocessed current collaborative conversation data to obtain a corresponding emotion recognition result, including: Sending the fused speech representation to multiple decoders respectively to reconstruct the initial representation of each modality, and calculating the loss between the reconstructed value and the true value; and Based on the multimodal representation and the annotated emotion labels of the collaborative speech data, and in combination with the multimodal representation and emotion labels of the positive example and the multimodal representation and emotion labels of the negative example, the contrast loss between different collaborative speech data is calculated; Based on the loss between the reconstructed value and the true value and the contrast loss, the loss function of the collaborative conversation emotion recognition model is determined, and the model parameters are learned through back propagation and gradient descent algorithms to obtain the collaborative conversation emotion recognition model under optimal model parameters.

8. An online collaborative conversation emotion recognition system for uncertain modality loss, characterized in that: The system comprises: A session segmentation module, used to obtain collaborative session data through an audio and video acquisition device, and segment the collaborative session data into a plurality of sub-collaborative session data having a time sequence relationship according to different stages of the collaborative learning process; The emotion annotation module is used to perform emotion annotation on each sub-collaboration session data to determine the emotional state of the learner in each sub-collaboration session data; A modal feature acquisition module, used for calling a unimodal feature extractor to extract unimodal features from the preprocessed sub-collaboration session data, and using the unimodal features as input of an LSTM layer to capture contextual information in the sub-collaboration session data, so as to obtain representations of each modality corresponding to the sub-collaboration session data; A feature fusion module, used to concatenate multiple single-modal features to obtain multi-modal fusion features, and explore cross-modal emotion promotion information between the multi-modal fusion features and the single-modal features based on the Transformer architecture; A fused discourse representation module, used to construct a multi-view sentiment reasoning graph with a dynamic context learning window based on enhanced unimodal features, and update the node representations in the multi-view sentiment reasoning graph through a graph attention mechanism to obtain a fused discourse representation of the sub-collaborative session data; An emotion recognition module is used to use the fused speech representation as the input of the conversation emotion prediction layer, train a collaborative conversation emotion recognition model, and call the collaborative conversation emotion recognition model to recognize the preprocessed current collaborative conversation data to obtain a corresponding emotion recognition result; Among them, the collaborative session data is obtained by the audio and video acquisition device collecting the conversation interaction content in the online collaborative learning process, including video data and audio data. The preprocessing is used to perform modality filling on the sub-collaborative session data after emotion annotation. The single-modal extractor includes pre-trained fast-RCNN, Wav2vec2.0 and BERT models. The multi-view emotion reasoning graph includes a text modality emotion reasoning graph, an audio modality emotion reasoning graph and a video modality emotion reasoning graph. The conversation emotion prediction layer consists of a fully connected layer and a Softmax layer, which is used to output predicted emotion labels for the fused discourse representation.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images