Emotion recognition method and system for online collaborative conversations with uncertain modality loss

Abstract

The present invention relates to an online collaborative conversation emotion recognition method and system for uncertain modality loss, the method comprising: dividing collaborative conversation data into multiple sub-collaborative conversation data according to different stages. Converting each sub-collaborative conversation data into conversation text data, and performing emotion annotation on each sub-collaborative conversation data. Extracting unimodal features from the preprocessed sub-collaborative conversation data, and capturing contextual information in the sub-collaborative conversation data through an LSTM layer to obtain representations of each modality. Splicing multiple unimodal features to explore cross-modal emotion promotion information between multimodal fusion features and unimodal features. Constructing a multi-perspective emotion reasoning graph, and updating node representations through a graph attention mechanism to obtain a fused discourse representation. Training the conversation emotion prediction layer through the fused discourse representation to obtain a collaborative conversation emotion recognition model, and calling the collaborative conversation emotion recognition model to recognize the preprocessed current collaborative conversation data.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and collaborative conversation processing technology, and in particular to an online collaborative conversation emotion recognition method and system for uncertain modality loss. Background Art

[0002] Online collaborative learning offers significant advantages in cultivating and developing learners' collaborative communication, problem-solving, and higher-order thinking skills. Currently, online collaborative learning has become a crucial learning paradigm in higher education, primarily leveraging video conferencing and other methods to conduct collaborative discussions and reach solutions to problems or achieve specific learning goals. Due to factors such as time and space barriers, learners are prone to experiencing learning emotions such as confusion and negativity during online collaborative learning sessions. If teachers fail to promptly identify and perceive students' negative learning emotions, and subsequently fail to provide appropriate emotional support and learning interventions, the collaborative learning process will be difficult to effectively advance, ultimately impacting the collaborative learning experience and outcomes. Therefore, timely and accurate identification of learners' emotional states during collaborative learning sessions is crucial, providing important decision-making support for teachers to develop personalized emotional support services.

[0003] Currently, most research on learner emotion analysis during collaborative learning sessions relies on post-process manual analysis, i.e., assessing learners' emotional states through questionnaires or manually coded videos after the collaborative learning session. Clearly, these methods are unable to identify learners' emotional states and their changes in real time during the collaborative learning session, nor can they provide appropriate collaborative learning interventions. One advantage of online methods is that they can record interactive conversation data during the collaborative learning process, providing a data foundation for teachers and administrators to gain a deeper understanding of the collaborative learning process and its effectiveness. Therefore, in recent years, some studies have begun to utilize the multimodal interactive conversation data generated during online collaborative learning, proposing methods and models for automatic learner emotion recognition based on multimodal data fusion. However, most of these studies fail to consider the impact of the contextual context of the collaborative learning process on learners' emotional states, nor the influence of the learner's own and other learners on their current learning emotional state, resulting in poor recognition results. Furthermore, these studies are almost all based on the assumption that all data modalities are complete during collaborative learning. They fail to consider the fact that missing data in uncertain modalities often occurs during actual collaborative learning sessions due to issues like network connectivity and background noise (for example, Student A forgets to turn on their camera at time 1, resulting in missing visual modal data; Student B loses voice data at time 5 due to network issues). While discarding or simply interpolating missing modal data can address this issue, it also leads to problems like loss of collaborative session context and scarcity of session data, further hindering the effectiveness of emotion recognition in practical teaching applications.

[0004] In summary, existing multimodal emotion recognition models are difficult to meet the learners' emotion recognition needs in actual complex collaborative conversation scenarios. Summary of the Invention

[0005] Based on this, it is necessary to provide an online collaborative conversation emotion recognition method and system for uncertain modality loss that can meet the learners' emotion recognition needs in actual complex collaborative conversation scenarios to address the above technical problems.

[0006] The present invention provides an online collaborative conversation emotion recognition method for uncertain modality loss, the method comprising:

[0007] Acquire collaborative session data through an audio and video acquisition device, and divide the collaborative session data into a plurality of sub-collaborative session data having a time sequence relationship according to different stages of the collaborative learning process;

[0008] Performing emotion annotation on each sub-collaboration session data to determine the learner's emotional state in each sub-collaboration session data;

[0009] Invoking a unimodal feature extractor to extract unimodal features from the preprocessed sub-collaboration session data, and using the unimodal features as input to an LSTM layer to capture contextual information in the sub-collaboration session data, thereby obtaining representations of each modality corresponding to the sub-collaboration session data;

[0010] The plurality of single-modal features are spliced ​​together to obtain a multimodal fusion feature, and the cross-modal emotion promotion information between the multimodal fusion feature and the single-modal feature is explored based on the Transformer architecture;

[0011] Constructing a multi-view sentiment reasoning graph with a dynamic context learning window based on the enhanced unimodal features, and updating the node representations in the multi-view sentiment reasoning graph through a graph attention mechanism to obtain a fused discourse representation of the sub-collaborative session data;

[0012] The fused speech representation is used as the input of the conversation emotion prediction layer to train a collaborative conversation emotion recognition model, and the collaborative conversation emotion recognition model is called to recognize the preprocessed current collaborative conversation data to obtain a corresponding emotion recognition result;

[0013] Among them, the collaborative session data is obtained by the audio and video acquisition device collecting the conversation interaction content in the online collaborative learning process, including video data and audio data. The preprocessing is used to perform modality filling on the sub-collaborative session data after emotion annotation. The single-modal extractor includes pre-trained fast-RCNN, Wav2vec2.0 and BERT models. The multi-view emotion reasoning graph includes a text modality emotion reasoning graph, an audio modality emotion reasoning graph and a video modality emotion reasoning graph. The conversation emotion prediction layer consists of a fully connected layer and a Softmax layer, which is used to output the predicted emotion label for the fused discourse representation.

[0014] In one embodiment, the step of acquiring collaborative session data through an audio and video acquisition device and dividing the collaborative session data into a plurality of sub-collaborative session data having a time sequence relationship according to different stages of the collaborative learning process may further include:

[0015] further dividing each of the sub-collaborative conversation data into a plurality of collaborative discourse data in a chronological order according to conversation turns of different learners in the collaborative learning process;

[0016] Preprocessing the plurality of collaborative discourse data so that when any modal data is missing in the collaborative discourse data, UNK in the vocabulary of the BERT model is called to fill in the missing corresponding modal data;

[0017] Among them, each collaborative discourse data includes at least multiple rounds of collaborative interaction discourse between two learners, and each round of the collaborative interaction discourse has modal representations of audio modality, video modality and text modality, and the multiple collaborative discourse data are used to construct a collaborative session dataset in combination with sub-collaborative session data with emotion annotations. When the collaborative discourse data lacks audio modality or video modality data, the audio modality or video modality data is filled with blank frames.

[0018] In one embodiment, calling a unimodal feature extractor to extract unimodal features from the preprocessed sub-collaboration session data, and using the unimodal features as input to an LSTM layer to capture contextual information in the sub-collaboration session data to obtain representations of each modality corresponding to the sub-collaboration session data includes:

[0019] Calling the pre-trained fast-RCNN, Wav2vec2.0, and BERT models as the unimodal feature extractors to respectively extract original unimodal features of the collaborative discourse data, and using the original unimodal features as inputs of the three LSTM layers to capture contextual features of the original collaborative session;

[0020] Each modal data is input into the fully connected layer for feature dimension conversion, so as to map the modal features corresponding to each modal data to the same dimensional space, and obtain the aligned representations of each modality.

[0021] In one embodiment, the concatenation of the plurality of unimodal features to obtain a multimodal fusion feature, and exploring cross-modal sentiment promotion information between the multimodal fusion feature and the unimodal features based on a Transformer architecture, includes:

[0022] When the direction of flow of emotional information is from multimodal features to unimodal features, the unimodal features are set as query vectors, and the multimodal features are set as key-value pairs, and complementary emotional information associated with the unimodal features is extracted from the multimodal features; and

[0023] When the direction of emotional information flow is from the unimodal feature to the multimodal feature, the multimodal feature is set as the query vector, and the unimodal feature is set as the key-value pair, so as to extract key emotional information from the unimodal feature to enhance the multimodal feature.

[0024] In one embodiment, the method of constructing a multi-view sentiment reasoning graph with a dynamic context learning window based on enhanced unimodal features, and updating node representations in the multi-view sentiment reasoning graph through a graph attention mechanism to obtain a fused discourse representation of the sub-collaborative session data, includes:

[0025] Constructing the text modality emotion reasoning graph, the audio modality emotion reasoning graph, and the video modality emotion reasoning graph for each of the collaborative discourse data, wherein the nodes in each reasoning graph are used to represent the collaborative interaction discourse of each round;

[0026] The multimodal features and unimodal features are fused through a graph attention mechanism to obtain enhanced unimodal features to initialize the collaborative interaction discourse represented by each node.

[0027] In one embodiment, the method of constructing a multi-view sentiment reasoning graph with a dynamic context learning window based on enhanced unimodal features, and updating node representations in the multi-view sentiment reasoning graph through a graph attention mechanism to obtain a fused discourse representation of the sub-collaborative session data, further includes:

[0028] Acquiring the speech density and the number of speeches of each learner in the collaborative learning process in the collaborative speech data, and calculating the context connection window of the collaborative speech data based on the speech density and the number of speeches to set the dynamic context learning window;

[0029] Initializing edge relationships and edge weights for each modal sentiment reasoning graph, respectively, wherein the edge weights are calculated and determined based on sentiment semantic similarities between a current node and a plurality of neighboring nodes within the dynamic context learning window;

[0030] The graph attention mechanism is used to update the information of the current node according to the types of different edges between the neighboring nodes and the dependencies between the nodes.

[0031] In one embodiment, the fused speech representation is used as the input of the conversation emotion prediction layer to train a collaborative conversation emotion recognition model, and the collaborative conversation emotion recognition model is called to recognize the preprocessed current collaborative conversation data to obtain a corresponding emotion recognition result, including:

[0032] Sending the fused speech representation to multiple decoders respectively to reconstruct the initial representation of each modality and calculating the loss between the reconstructed value and the true value; and

[0033] Based on the multimodal representation and the annotated emotion labels of the collaborative speech data, and in combination with the multimodal representation and emotion labels of the positive example and the multimodal representation and emotion labels of the negative example, calculating the contrast loss between different collaborative speech data;

[0034] Based on the loss between the reconstructed value and the true value and the contrast loss, the loss function of the collaborative conversation emotion recognition model is determined, and the model parameters are learned through back propagation and gradient descent algorithms to obtain the collaborative conversation emotion recognition model under the optimal model parameters.

[0035] The present invention also provides an online collaborative conversation emotion recognition system for uncertain modality loss, the system comprising:

[0036] A session segmentation module is used to obtain collaborative session data through an audio and video acquisition device, and to segment the collaborative session data into a plurality of sub-collaborative session data with a time sequence relationship according to different stages of the collaborative learning process;

[0037] The emotion annotation module is used to perform emotion annotation on each sub-collaboration session data to determine the learner's emotional state in each sub-collaboration session data;

[0038] a modal feature acquisition module, configured to call a unimodal feature extractor to extract unimodal features from the preprocessed sub-collaboration session data, and use the unimodal features as input to an LSTM layer to capture contextual information in the sub-collaboration session data, thereby obtaining representations of each modality corresponding to the sub-collaboration session data;

[0039] A feature fusion module is used to splice multiple single-modal features to obtain multimodal fusion features, and explore the cross-modal emotion promotion information between the multimodal fusion features and the single-modal features based on the Transformer architecture;

[0040] A fusion discourse representation module is used to construct a multi-view sentiment reasoning graph with a dynamic context learning window based on the enhanced unimodal features, and to update the node representations in the multi-view sentiment reasoning graph through a graph attention mechanism to obtain a fusion discourse representation of the sub-collaboration session data;

[0041] An emotion recognition module is configured to use the fused speech representation as input to a conversation emotion prediction layer, train a collaborative conversation emotion recognition model, and call the collaborative conversation emotion recognition model to recognize the preprocessed current collaborative conversation data to obtain a corresponding emotion recognition result;

[0042] Among them, the collaborative session data is obtained by the audio and video acquisition device collecting the conversation interaction content in the online collaborative learning process, including video data and audio data. The preprocessing is used to perform modality filling on the sub-collaborative session data after emotion annotation. The single-modal extractor includes pre-trained fast-RCNN, Wav2vec2.0 and BERT models. The multi-view emotion reasoning graph includes a text modality emotion reasoning graph, an audio modality emotion reasoning graph and a video modality emotion reasoning graph. The conversation emotion prediction layer consists of a fully connected layer and a Softmax layer, which is used to output the predicted emotion label for the fused discourse representation.

[0043] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements any of the above-mentioned methods for online collaborative conversation emotion recognition for uncertain modality loss.

[0044] The present invention also provides a computer storage medium storing a computer program, wherein when the computer program is executed by a processor, the method for online collaborative conversation emotion recognition oriented to uncertain modality loss as described above is implemented.

[0045] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-described methods for online collaborative conversation emotion recognition oriented to uncertain modality loss.

[0046] The above-mentioned online collaborative conversation emotion recognition method and system for uncertain modality loss can meet the automatic and accurate recognition of learners' emotional states during actual online collaborative conversations by extracting and fusing multi-dimensional collaborative conversation emotion information in the case of uncertain modality loss, thereby supporting collaborative learning teachers to implement some targeted emotional intervention strategies to specific learners or collaborative learning groups in a timely manner, in order to improve learners' online collaborative learning experience, enhance learners' online collaborative learning motivation, and improve the learning outcomes of collaborative groups. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 A flow chart of the method for emotion recognition in online collaborative conversations with uncertain modality loss provided by the present invention;

[0049] Figure 2 This is a schematic diagram of the overall process of the online collaborative conversation emotion recognition method for uncertain modality loss in a specific embodiment provided by the present invention;

[0050] Figure 3 A schematic diagram of an online collaborative session emotion recognition system architecture for an online collaborative session emotion recognition method for uncertain modality loss in a specific embodiment provided by the present invention;

[0051] Figure 4 A schematic diagram of the structure of the online collaborative conversation emotion recognition system for uncertain modality loss provided by the present invention;

[0052] Figure 5 This is a diagram of the internal structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0054] The following combination Figures 1 to 5 The present invention describes an online collaborative conversation emotion recognition method and system for uncertain modality loss.

[0055] like Figure 1 As shown, in one embodiment, a method for emotion recognition in online collaborative conversations with uncertain modality loss includes the following steps:

[0056] Step S110 : Acquire collaborative session data through an audio and video acquisition device, and divide the collaborative session data into a plurality of sub-collaborative session data having a time sequence relationship according to different stages of the collaborative learning process.

[0057] The collaborative session data is obtained by collecting the interactive content of the online collaborative learning session using audio and video acquisition equipment, and includes video data and audio data.

[0058] In some embodiments, the method for online collaborative conversation emotion recognition for uncertain modality loss provided by the present invention acquires collaborative conversation data through an audio and video acquisition device, and divides the collaborative conversation data into multiple sub-collaborative conversation data with a time sequence relationship according to different stages of the collaborative learning process, and then includes the following steps:

[0059] Step S210 : further dividing each sub-collaborative conversation data into a plurality of collaborative discourse data in a chronological order according to the conversation turns of different learners in the collaborative learning process.

[0060] Step S220 , pre-processing the plurality of collaborative discourse data, so that when any modal data is missing in the collaborative discourse data, UNK in the vocabulary of the BERT model is called to fill in the missing corresponding modal data.

[0061] Among them, each collaborative discourse data includes at least multiple rounds of collaborative interaction discourse between two learners, and each round of collaborative interaction discourse has modal representations of audio modality, video modality and text modality. Multiple collaborative discourse data are used to construct a collaborative conversation dataset in combination with sub-collaborative conversation data with emotion annotations. When audio modality or video modality data is missing in the collaborative discourse data, the audio modality or video modality data is filled with blank frames.

[0062] Step S120 : performing emotion annotation on each sub-collaboration session data to determine the learner's emotional state in each sub-collaboration session data.

[0063] In step S130 , a unimodal feature extractor is called to extract unimodal features from the preprocessed sub-collaboration session data, and the unimodal features are used as input to the LSTM layer to capture contextual information in the sub-collaboration session data, so as to obtain representations of each modality corresponding to the sub-collaboration session data.

[0064] Among them, preprocessing is used to fill the modality of the sub-collaborative conversation data after emotion annotation, and the single-modal extractor includes pre-trained fast-RCNN, Wav2vec2.0 and BERT models.

[0065] In some embodiments, the method for online collaborative session emotion recognition for uncertain modality loss provided by the present invention calls a unimodal feature extractor to extract unimodal features from preprocessed sub-collaborative session data, and uses the unimodal features as input to the LSTM layer to capture contextual information in the sub-collaborative session data to obtain representations of each modality corresponding to the sub-collaborative session data. Specifically, the method includes the following steps:

[0066] In step S131, the pre-trained fast-RCNN, Wav2vec2.0, and BERT models are called as unimodal feature extractors to extract the original unimodal features of the collaborative discourse data respectively, and the original unimodal features are used as inputs of the three LSTM layers respectively to capture the contextual features of the original collaborative session.

[0067] In step S132 , each modal data is input into a fully connected layer for feature dimension conversion, so as to map the modal features corresponding to each modal data into the same dimensional space, thereby obtaining aligned representations of each modality.

[0068] In step S140 , multiple unimodal features are concatenated to obtain multimodal fusion features, and cross-modal emotion promotion information between the multimodal fusion features and the unimodal features is explored based on the Transformer architecture.

[0069] In some embodiments, the method for emotion recognition in online collaborative conversations with uncertain modality loss provided by the present invention concatenates multiple unimodal features to obtain multimodal fusion features, and explores cross-modal emotion promotion information between the multimodal fusion features and unimodal features based on the Transformer architecture, specifically including the following steps:

[0070] Step S141: When the flow direction of emotional information is from multimodal features to unimodal features, the unimodal features are set as query vectors, and the multimodal features are set as key-value pairs, and complementary emotional information associated with the unimodal features is extracted from the multimodal features.

[0071] Step S142: When the direction of emotional information flow is from unimodal features to multimodal features, the multimodal features are set as query vectors and the unimodal features are set as key-value pairs to extract key emotional information from the unimodal features to enhance the multimodal features.

[0072] In step S150 , a multi-view sentiment reasoning graph with a dynamic context learning window is constructed based on the enhanced unimodal features, and the node representations in the multi-view sentiment reasoning graph are updated through the graph attention mechanism to obtain the fused discourse representation of the sub-collaborative session data.

[0073] Among them, the multi-perspective emotion reasoning graph includes text modality emotion reasoning graph, audio modality emotion reasoning graph and video modality emotion reasoning graph.

[0074] In some embodiments, the present invention provides an online collaborative conversation emotion recognition method for uncertain modality loss, which constructs a multi-view emotion reasoning graph with a dynamic context learning window based on enhanced unimodal features, and updates the node representations in the multi-view emotion reasoning graph through a graph attention mechanism to obtain a fused discourse representation of the sub-collaborative conversation data. Specifically, the method includes the following steps:

[0075] Step S151: construct a text modality emotion reasoning graph, an audio modality emotion reasoning graph, and a video modality emotion reasoning graph for each collaborative discourse data. The nodes in each reasoning graph are used to represent the collaborative interaction discourse of each round.

[0076] In step S152, the multimodal features and the unimodal features are fused through the graph attention mechanism to obtain enhanced unimodal features to initialize the collaborative interaction discourse represented by each node.

[0077] In some embodiments, the method for online collaborative conversation emotion recognition for uncertain modality loss provided by the present invention constructs a multi-view emotion reasoning graph with a dynamic context learning window based on enhanced unimodal features, and updates the node representations in the multi-view emotion reasoning graph through a graph attention mechanism to obtain a fused discourse representation of the sub-collaborative conversation data, specifically including the following steps:

[0078] Step S153: Obtain the discourse density and the number of discourses of each learner in the collaborative learning process in the collaborative discourse data, and calculate the context connection window of the collaborative discourse data based on the discourse density and the number of discourses to set a dynamic context learning window.

[0079] In step S154, the edge relationships and edge weights of each modal emotion reasoning graph are initialized respectively. The edge weights are calculated and determined based on the emotion semantic similarity between the current node and multiple neighboring nodes in the dynamic context learning window.

[0080] In step S155, the graph attention mechanism is used to update the information of the current node according to the types of different edges between neighboring nodes and the dependencies between nodes.

[0081] In step S160 , the fused discourse representation is used as the input of the conversation emotion prediction layer to train a collaborative conversation emotion recognition model, and the collaborative conversation emotion recognition model is called to recognize the preprocessed current collaborative conversation data to obtain a corresponding emotion recognition result.

[0082] Among them, the conversation emotion prediction layer consists of a fully connected layer and a Softmax layer, which is used to output the predicted emotion label for the fused discourse representation.

[0083] In some embodiments, the method for online collaborative conversation emotion recognition for uncertain modality loss provided by the present invention uses the fused discourse representation as the input of the conversation emotion prediction layer, trains a collaborative conversation emotion recognition model, and calls the collaborative conversation emotion recognition model to recognize the preprocessed current collaborative conversation data to obtain the corresponding emotion recognition result, specifically including the following steps:

[0084] In step S161 , the fused speech representation is sent to multiple decoders respectively to reconstruct the initial representation of each modality and calculate the loss between the reconstructed value and the true value.

[0085] Step S162 , based on the multimodal representation and the annotated emotion labels of the collaborative discourse data, and in combination with the multimodal representation and emotion labels of the positive example and the multimodal representation and emotion labels of the negative example, calculate the contrast loss between different collaborative discourse data.

[0086] In step S163 , based on the loss and contrast loss between the reconstructed value and the true value, the loss function of the collaborative conversation emotion recognition model is determined, and the model parameters are learned through back propagation and gradient descent algorithms to obtain the collaborative conversation emotion recognition model under the optimal model parameters.

[0087] The above-mentioned online collaborative conversation emotion recognition method for uncertain modality loss can automatically and accurately identify the learner's emotional state during the actual online collaborative conversation by extracting and fusing the multi-dimensional collaborative conversation emotion information in the case of uncertain modality loss, thereby supporting collaborative learning teachers to implement some targeted emotional intervention strategies to specific learners or collaborative learning groups in a timely manner, in order to improve learners' online collaborative learning experience, enhance learners' online collaborative learning motivation, and improve the learning outcomes of collaborative groups.

[0088] See also Figure 2 and Figure 3As shown in a specific embodiment, the present invention provides an online collaborative conversation emotion recognition method for uncertain modality loss, which uses multi-channel conversation behavior data collected during actual online collaborative learning to construct a collaborative conversation dataset containing three modalities: text, audio, and image, and performs preliminary data cleaning on each modal data. Furthermore, a collaborative conversation emotion recognition model for uncertain modality loss is trained, including extracting unimodal features using multiple pre-trained models and capturing initial unimodal context information using an LSTM model; constructing an emotion promotion module between multimodality and unimodality based on the Transformer architecture to capture the emotional semantic context information in the collaborative conversation process from multiple perspectives and dimensions; further, constructing a multi-perspective emotion reasoning graph with a dynamic context learning window, including initializing graph nodes based on the aforementioned emotion promotion module, defining the edge type in the graph based on the learner's interaction during the collaborative conversation, initializing the edge weight based on the semantic similarity of the node, and dynamically setting the context learning window based on the learner's speech density to update the node features using the node information propagation mechanism in the graph.

[0089] Finally, the emotional information learned from the emotion facilitation module and the emotion reasoning graph module is concatenated and fused to obtain a final multimodal representation of each collaborative utterance. This multimodal fusion representation is used for collaborative learning of emotion recognition and simultaneously fed into three unimodal decoders to reconstruct the original input data. It is also used to conduct comparative learning of multimodal representations between different samples under the constraint of emotion labels. The final model training loss function is then derived based on three loss functions to guide the training process of the emotion recognition model. This collaborative conversation emotion recognition model can be used to identify learner emotions in collaborative conversations with uncertain modalities.

[0090] In this embodiment, the following steps are specifically included:

[0091] Step 1: Collaborative session data collection. Use audio and video acquisition equipment to collect the conversation interaction content during the online collaborative learning process, including the video data and audio data of the conversation interaction during the online collaborative learning process collected by the audio and video acquisition equipment.

[0092] Step 2: Construction of collaborative conversation dataset. First, the original audio and video data are coarsely divided into collaborative conversations of different groups at different stages according to the different stages of the collaborative learning process; then, for each collaborative conversation, the audio and video data are further divided into collaborative discourses with a time sequence relationship according to the conversation turns of different learners in the collaborative interaction process, and the corresponding conversation text data is generated using an audio transcription tool; finally, two experts in the field of educational technology perform sentiment annotation on the collaborative conversations to determine the learner's emotional state contained in each collaborative discourse. At this point, the online collaborative learning process can be regarded as an online collaborative conversation dataset composed of multiple online collaborative conversations, each of which contains multiple rounds of collaborative interaction discourses of at least two learners, and each discourse is expressed in three modalities: audio, video, and text. Therefore, a collaborative conversation data can be represented as The set of learners involved in the dialogue is S = {s1,s2,s3,...,s M}, discourse Depend on Say, among them A mapping function representing an utterance to its corresponding speaker.

[0093] Step 3: Data preprocessing. In the actual collaborative learning process, due to network connection, background noise and other reasons, some speech samples u i Missing data from one or two modalities (assuming that the speech sample u i At least one modality of complete data), then it needs to be preprocessed, including: if the text modality of a speech sample is missing, it is filled with UNK in the BERT vocabulary; if the audio and / or video modality of a speech sample is missing, it is filled with blank frames.

[0094] Step 4: Training a collaborative conversation emotion recognition model for uncertain modality loss, specifically including:

[0095] (1) Extracting the initial unimodal context features of the utterance:

[0096] Specifically, the pre-trained fast-RCNN, Wav2vec2.0 and BERT models are used as unimodal feature extractors to extract the original unimodal features of each utterance. The original unimodal features are then fed into three LSTM layers to capture the initial collaborative session context information. Finally, the information of each modality is sent to the fully connected layer for feature dimension change, so that the three modal features are mapped to the same dimensional space, and the aligned representation of each modality is obtained.

[0097] (2) Construct an emotion promotion module based on the Transformer architecture.

[0098] Specifically, a Transformer-based emotion promotion unit is constructed for each modality to achieve mutual promotion between multimodal information and unimodal information. First, the above unimodal feature H m Perform simple splicing to obtain the initial multimodal fusion features, that is, Furthermore, based on the Transformer architecture, we explore the cross-modal emotional promotion information between multimodality and each unimodality. That is, when the direction of emotional information flow is from multimodality to unimodality, the features of modality m are set as query vector Q, and the multimodal features are set as key-value pairs (K and V) to extract complementary emotional information related to the current modality m from the multimodality. Similarly, when the direction of emotional information flow is from unimodality to multimodality, the multimodal features are set as query vector Q of the current emotional promotion direction, and the unimodal features are set as key-value pairs (K and V) to extract important emotional information from the unimodality to enhance the multimodal features. The process is expressed as:

[0099]

[0100] In the formula, MHCA refers to the multi-head cross-modal interaction calculation based on the Transformer architecture. After the first emotion promotion layer, the single modal feature is updated to The multimodal features are updated as After l emotion promotion layers, the unimodal feature is updated to Multimodal features are updated to F l Both unimodal and multimodal features contain emotional clues that complement each other's contexts.

[0101] (3) Constructing a multi-view sentiment reasoning graph with dynamic context learning window:

[0102] It is planned to construct three modal interaction diagrams for each dialogue (including text modal sentiment reasoning diagram Audio modality sentiment inference graph Video modality sentiment inference graph ), each node in the graph represents a discourse.

[0103] First, the representation of each node in the graph is initialized. The unimodal features and multimodal features learned by the above-mentioned emotion promotion module are fused based on the attention mechanism to obtain enhanced unimodal representations, which are used to initialize the corresponding node features in each modal interaction graph. The process is described as follows:

[0104]

[0105] Where, and b1 is a learnable parameter, Initialize features for the nodes of the interaction graph of modality m.

[0106] Secondly, the context learning window in the emotional reasoning graph is dynamically set. Specifically, the speech density of each learner in the dialogue during the collaboration process is first calculated. Assume that the current speech u i By S A , calculate the weight α of learner A's utterances in the entire conversation. Secondly, multiply the weight α by the sequence number i of the current utterance in the conversation to calculate the number of utterances issued by the corresponding learner. Finally, calculate the context connection window ω of the current utterance, that is, ω = i*α + S-1 (where S represents the number of learners participating in the current collaborative conversation, and S-1 represents the consideration of learners other than the current learner), to dynamically set the context learning window of the current utterance.

[0107] Finally, edge relationships and edge weights are initialized for each modal interaction graph. Assume that for the sentiment inference graph of modality m, edge relationships are determined by speaker type. If both utterance nodes of an edge are emitted by the same learner, the edge type is "same learner"; otherwise, the edge type is "different learners." Edge weights are calculated based on the sentiment semantic similarity between the current node and its ω neighboring nodes within the dynamic context window. Specifically, the similarity between each node and all nodes in the context window is calculated as:

[0108]

[0109] Where, represents the initialization features of node i in the sentiment reasoning graph of modality m, represents the initialization features of node j within the context window ω, Represents the edge weight between the current node i and node j in the sentiment reasoning graph of modality m. At this point, the construction of the three modal sentiment reasoning graphs can be completed respectively.

[0110] (4) Update node representation in the sentiment reasoning graph based on the graph attention mechanism:

[0111] The graph attention mechanism is used to update the aggregated neighbor node information, and the current node information is updated according to the type of different edges and the dependencies between nodes. After one layer of graph convolution, the edge weight is updated as follows:

[0112]

[0113] Where, and They are the initial features of node i and node j in the sentiment reasoning graph of modality m, a T and W αrepresents the learnable parameters, N(u i ) The set of neighbor nodes of the current node within the context learning window ω, is the updated edge weight between node i and node j in the sentiment reasoning graph of modality m. Then, according to the updated weight, the current discourse node representation after a layer of graph convolution information propagation is updated as follows:

[0114]

[0115] For the emotional reasoning graph of modality m, the emotional reasoning information learned after r-layer graph convolution information propagation can be represented as Based on the attention mechanism, the emotional reasoning information under this modality or perspective can be calculated as follows:

[0116]

[0117] Where, and b2 are learnable parameters, G r It is the emotional reasoning feature representation of discourse node i obtained by aggregation and updating after graph information propagation based on speaker information and emotional semantic information.

[0118] (5) Training of emotion recognition models for collaborative conversations with uncertain modality loss:

[0119] Based on the above steps, the emotional information learned by the emotion promotion module and the emotion reasoning graph module is concatenated and fused to obtain the final fused utterance representation, which serves as the input to the conversation emotion prediction layer. The conversation emotion prediction layer consists of a fully connected layer and a softmax layer, and outputs a predicted emotion label for each fused utterance representation.

[0120] Assume that the original input of utterance i is Its true emotional label is y i The emotional interaction information learned by the emotion promotion module for discourse i is F l , the emotional reasoning information learned through the emotional reasoning graph is G r , and concatenate and fuse them to obtain the final multimodal representation Z = Concat(F l ,G r ). The predicted emotion label obtained after the multimodal representation Z passes through the conversation emotion prediction layer is Therefore, the objective function of the conversation emotion recognition task is:

[0121]

[0122] Where N represents the number of utterances in a conversation, and C represents the number of emotion categories.

[0123] At the same time, considering that in real collaborative conversations, due to various reasons, some modal data of certain utterances is often missing, this method fills in the missing data during the preprocessing stage, but the filled data may differ significantly from the real data. Therefore, this method also considers the reconstruction loss of the data, that is, it encourages the emotion recognition model to reconstruct the complete collaborative conversation data from the potential multimodal representation space as much as possible, so as to guide the model to learn the real semantic information of the missing parts. Specifically, the final multimodal fusion representation Z is sent to three decoders respectively to reconstruct the initial representation of each modality, and the loss between the reconstructed value and the true value is calculated:

[0124]

[0125] Among them, Encoder represents the reconstruction encoder which can be composed of a fully connected layer, θ m represents the encoder parameters, λ m represents the missing rate of mode m, X m and represent the original unimodal features and reconstructed features of modality m respectively.

[0126] In addition, in order to make the multimodal features learned by the model show that the sample points of the same emotion category are as close as possible (i.e., positive examples) in the feature space, and the sample points of different emotion categories are as far away as possible (i.e., negative examples), this method also considers calculating the contrast loss between sample points. Specifically, assuming that the speech sample u i The final multimodal representation of z i And the sentiment label is y i , its positive example u j The multimodal representation of z i , its sentiment label y j =y i ; Its negative example u k The multimodal representation of z k , its sentiment label y k ≠y i , then the contrast loss is:

[0127]

[0128] Among them, sim represents the cosine similarity function, τ is the temperature coefficient, L cl (z i ) represents the calculation of the speech sample u in each training batch i Contrastive loss, L cl represents the sum of the contrast losses of all samples in the entire training set. Finally, the training loss function of the collaborative conversation emotion recognition model for uncertain modality loss is:

[0129]

[0130] in, and is the hyperparameter coefficient to balance the relationship between the three types of losses. The parameters of the collaborative conversational emotion recognition model are learned through backpropagation and stochastic gradient descent algorithms. When the objective function reaches the optimal solution, the conversational emotion recognition model with the optimal model parameters is obtained.

[0131] Step 5: Emotion recognition of new collaborative conversation data.

[0132] Specifically, the new samples are preprocessed in the same way as the training data, so that they become data types that can be recognized and efficiently processed by the collaborative conversation emotion recognition model; then the preprocessed channel data are sent to the trained collaborative conversation emotion recognition model to finally obtain the emotion prediction result for the sample.

[0133] In this embodiment, the focus is on how to solve the problem of missing discourse modalities that often occur in real collaborative conversation scenarios. It is proposed to first fill in the gaps in the preprocessing stage, and finally make the learned emotional features as close to the real information as possible through reconstruction loss and contrast loss in the model training stage, so that it has high practical application value. Secondly, the method constructs an emotional promotion module based on the Transformer architecture to achieve a two-way flow and mutual promotion of complementary emotional information between single modality and multimodality, which retains the unique emotional feature information of single modality and enhances the mining of multimodal complementary emotional information. In addition, the method also dynamically sets the context learning window based on the frequency of speech and the number of speakers in the conversation, and initializes the edge weights in the emotional graph by calculating the similarity between nodes, avoiding the interference of irrelevant noise information, and retaining and enhancing the emotional information contained in the node features themselves.

[0134] Furthermore, this method can address the problem of missing some modalities of data in real online collaborative conversations due to network issues or other factors. It can also automatically and accurately perceive learners' emotional states during collaborative learning, providing a decision-making basis for teachers' personalized emotional intervention and support. This helps enhance learners' collaborative learning experience, improve their collaborative learning motivation, and improve the collaborative learning outcomes of collaborative groups. Compared to traditional manual recognition or multimodal emotion recognition models, this method and system take into account the current status of actual online collaborative learning scenarios (such as the presence of multi-channel recorded data) and problems (such as the missing modalities of some utterances due to network and equipment issues). From a method design perspective, it considers how to better capture conversation context information, interaction information between modalities, and interaction information between different learners, so as to more comprehensively learn emotional information from incomplete data. Therefore, this method is more scientific and robust, and has superior emotion recognition results.

[0135] The following describes the online collaborative session emotion recognition system for uncertain modality loss provided by the present invention. The online collaborative session emotion recognition system for uncertain modality loss described below and the online collaborative session emotion recognition method for uncertain modality loss described above can refer to each other.

[0136] like Figure 4 As shown, in one embodiment, an online collaborative conversation emotion recognition system for uncertain modality loss includes a conversation segmentation module 410, an emotion labeling module 420, a modal feature acquisition module 430, a feature fusion module 440, a fused discourse representation module 450 and an emotion recognition module 460.

[0137] The session segmentation module 410 is configured to acquire collaborative session data through an audio and video acquisition device, and segment the collaborative session data into a plurality of sub-collaborative session data having a time sequence relationship according to different stages of the collaborative learning process.

[0138] The emotion annotation module 420 is used to perform emotion annotation on each sub-collaboration session data to determine the learner's emotion state in each sub-collaboration session data.

[0139] The modal feature acquisition module 430 is used to call the unimodal feature extractor to extract the unimodal features in the preprocessed sub-collaboration session data, and use the unimodal features as the input of the LSTM layer to capture the contextual information in the sub-collaboration session data to obtain the modal representations corresponding to the sub-collaboration session data.

[0140] The feature fusion module 440 is used to splice multiple unimodal features to obtain multimodal fusion features, and explore the cross-modal emotion promotion information between the multimodal fusion features and unimodal features based on the Transformer architecture.

[0141] The fused discourse representation module 450 is used to construct a multi-view sentiment reasoning graph with a dynamic context learning window based on enhanced unimodal features, and update the node representations in the multi-view sentiment reasoning graph through a graph attention mechanism to obtain a fused discourse representation of the sub-collaborative session data.

[0142] The emotion recognition module 460 is used to use the fused speech representation as the input of the conversation emotion prediction layer, train a collaborative conversation emotion recognition model, and call the collaborative conversation emotion recognition model to recognize the preprocessed current collaborative conversation data to obtain corresponding emotion recognition results.

[0143] Among them, the collaborative session data is obtained by audio and video acquisition equipment from the conversation interaction content in the online collaborative learning process, including video data and audio data. The preprocessing is used to fill the modality of the sub-collaborative session data after emotion annotation. The single-modal extractor includes pre-trained fast-RCNN, Wav2vec2.0 and BERT models. The multi-view emotion reasoning graph includes text modality emotion reasoning graph, audio modality emotion reasoning graph and video modality emotion reasoning graph. The conversation emotion prediction layer consists of a fully connected layer and a Softmax layer, which is used to output the predicted emotion label for the fused discourse representation.

[0144] In this embodiment, the online collaborative conversation emotion recognition system for uncertain modality loss provided by the present invention further includes a data preprocessing module for:

[0145] According to the conversation turns of different learners in the collaborative learning process, each sub-collaborative conversation data is further divided into multiple collaborative discourse data with a time sequence.

[0146] Preprocess multiple collaborative discourse data so that when any modal data is missing in the collaborative discourse data, UNK in the vocabulary of the BERT model is called to fill in the missing corresponding modal data.

[0147] Among them, each collaborative discourse data includes at least multiple rounds of collaborative interaction discourse between two learners, and each round of collaborative interaction discourse has modal representations of audio modality, video modality and text modality. Multiple collaborative discourse data are used to construct a collaborative conversation dataset in combination with sub-collaborative conversation data with emotion annotations. When audio modality or video modality data is missing in the collaborative discourse data, the audio modality or video modality data is filled with blank frames.

[0148] In this embodiment, in the online collaborative conversation emotion recognition system for uncertain modality loss provided by the present invention, the modality feature acquisition module 430 is specifically used to:

[0149] The pre-trained fast-RCNN, Wav2vec2.0, and BERT models are called as unimodal feature extractors to extract the original unimodal features of the collaborative discourse data respectively, and the original unimodal features are used as the input of three LSTM layers to capture the contextual features of the original collaborative session.

[0150] Each modal data is input into the fully connected layer for feature dimension conversion, so as to map the modal features corresponding to each modal data to the same dimensional space and obtain the aligned modal representations.

[0151] In this embodiment, the feature fusion module 440 of the online collaborative conversation emotion recognition system for uncertain modality loss provided by the present invention is specifically configured to:

[0152] When the direction of emotional information flow is from multimodal features to unimodal features, the unimodal features are set as query vectors, and the multimodal features are set as key-value pairs, and complementary emotional information associated with the unimodal features is extracted from the multimodal features.

[0153] When the direction of emotional information flow is from unimodal features to multimodal features, the multimodal features are set as query vectors and the unimodal features are set as key-value pairs to extract key emotional information from the unimodal features to enhance the multimodal features.

[0154] In this embodiment, the online collaborative conversation emotion recognition system for uncertain modality loss provided by the present invention, the fusion discourse representation module 450 is specifically used to:

[0155] For each collaborative discourse data, a text modality sentiment reasoning graph, an audio modality sentiment reasoning graph, and a video modality sentiment reasoning graph are constructed. The nodes in each reasoning graph are used to represent the collaborative interaction discourse of each round.

[0156] Multimodal features and unimodal features are fused through the graph attention mechanism to obtain enhanced unimodal features to initialize the collaborative interaction discourse represented by each node.

[0157] In this embodiment, the online collaborative conversation emotion recognition system for uncertain modality loss provided by the present invention, the fusion discourse representation module 450 is further configured to:

[0158] The discourse density and the number of discourses of each learner in the collaborative learning process are obtained, and the context connection window of the collaborative discourse data is calculated based on the discourse density and the number of discourses to set a dynamic context learning window.

[0159] The edge relationships and edge weights of each modality sentiment reasoning graph are initialized respectively. The edge weights are calculated based on the sentiment semantic similarity between the current node and multiple neighboring nodes in the dynamic context learning window.

[0160] The graph attention mechanism is used to update the information of the current node according to the types of different edges between neighboring nodes and the dependencies between nodes.

[0161] In this embodiment, the emotion recognition system for online collaborative conversations with uncertain modality loss provided by the present invention, the emotion recognition module 460 is specifically configured to:

[0162] The fused speech representation is sent to multiple decoders to reconstruct the initial representation of each modality and calculate the loss between the reconstructed value and the true value.

[0163] Based on the multimodal representation and annotated sentiment labels of collaborative discourse data, and combined with the multimodal representation and sentiment labels of positive examples and the multimodal representation and sentiment labels of negative examples, the contrast loss between different collaborative discourse data is calculated.

[0164] Based on the loss and contrast loss between the reconstructed value and the true value, the loss function of the collaborative conversation emotion recognition model is determined, and the model parameters are learned through back propagation and gradient descent algorithms to obtain the collaborative conversation emotion recognition model with the optimal model parameters.

[0165] Figure 5 The following is a schematic diagram of the physical structure of an electronic device. The electronic device may be a smart terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The electronic device includes a processor, an internal memory and a network interface connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, an online collaborative session emotion recognition method for uncertain modality loss is implemented, and the method includes:

[0166] Acquire collaborative session data through audio and video acquisition equipment, and divide the collaborative session data into multiple sub-collaborative session data with a time sequence relationship according to different stages of the collaborative learning process;

[0167] Performing emotion annotation on each sub-collaboration session data to determine the learner's emotional state in each sub-collaboration session data;

[0168] The unimodal feature extractor is called to extract unimodal features from the preprocessed sub-collaboration session data. The unimodal features are used as input to the LSTM layer to capture the contextual information in the sub-collaboration session data, thereby obtaining the corresponding modal representations of the sub-collaboration session data.

[0169] Multiple single-modal features are spliced ​​together to obtain multimodal fusion features, and the cross-modal sentiment promotion information between multimodal fusion features and single-modal features is explored based on the Transformer architecture;

[0170] Based on the enhanced unimodal features, a multi-view sentiment reasoning graph with a dynamic context learning window is constructed. The node representations in the multi-view sentiment reasoning graph are updated through the graph attention mechanism to obtain the fused discourse representation of the sub-collaborative session data.

[0171] The fused discourse representation is used as the input of the conversation emotion prediction layer to train a collaborative conversation emotion recognition model. The collaborative conversation emotion recognition model is then called to recognize the preprocessed current collaborative conversation data to obtain the corresponding emotion recognition results.

[0172] Among them, the collaborative session data is obtained by audio and video acquisition equipment from the conversation interaction content in the online collaborative learning process, including video data and audio data. The preprocessing is used to fill the modality of the sub-collaborative session data after emotion annotation. The single-modal extractor includes pre-trained fast-RCNN, Wav2vec2.0 and BERT models. The multi-view emotion reasoning graph includes text modality emotion reasoning graph, audio modality emotion reasoning graph and video modality emotion reasoning graph. The conversation emotion prediction layer consists of a fully connected layer and a Softmax layer, which is used to output the predicted emotion label for the fused discourse representation.

[0173] Those skilled in the art will understand that Figure 5 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the electronic device to which the solution of the present invention is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0174] On the other hand, the present invention also provides a computer storage medium storing a computer program. When the computer program is executed by a processor, a method for emotion recognition in online collaborative conversations for uncertain modality loss is implemented. The method comprises:

[0175] Acquire collaborative session data through audio and video acquisition equipment, and divide the collaborative session data into multiple sub-collaborative session data with a time sequence relationship according to different stages of the collaborative learning process;

[0176] Performing emotion annotation on each sub-collaboration session data to determine the learner's emotional state in each sub-collaboration session data;

[0177] The unimodal feature extractor is called to extract unimodal features from the preprocessed sub-collaboration session data. The unimodal features are used as input to the LSTM layer to capture the contextual information in the sub-collaboration session data, thereby obtaining the corresponding modal representations of the sub-collaboration session data.

[0178] Multiple single-modal features are spliced ​​together to obtain multimodal fusion features, and the cross-modal sentiment promotion information between multimodal fusion features and single-modal features is explored based on the Transformer architecture;

[0179] Based on the enhanced unimodal features, a multi-view sentiment reasoning graph with a dynamic context learning window is constructed. The node representations in the multi-view sentiment reasoning graph are updated through the graph attention mechanism to obtain the fused discourse representation of the sub-collaborative session data.

[0180] The fused discourse representation is used as the input of the conversation emotion prediction layer to train a collaborative conversation emotion recognition model. The collaborative conversation emotion recognition model is then called to recognize the preprocessed current collaborative conversation data to obtain the corresponding emotion recognition results.

[0181] Among them, the collaborative session data is obtained by audio and video acquisition equipment from the conversation interaction content in the online collaborative learning process, including video data and audio data. The preprocessing is used to fill the modality of the sub-collaborative session data after emotion annotation. The single-modal extractor includes pre-trained fast-RCNN, Wav2vec2.0 and BERT models. The multi-view emotion reasoning graph includes text modality emotion reasoning graph, audio modality emotion reasoning graph and video modality emotion reasoning graph. The conversation emotion prediction layer consists of a fully connected layer and a Softmax layer, which is used to output the predicted emotion label for the fused discourse representation.

[0182] In another aspect, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and when the processor executes the computer instructions, implements a method for emotion recognition in an online collaborative session for uncertain modality loss, the method comprising:

[0183] Acquire collaborative session data through audio and video acquisition equipment, and divide the collaborative session data into multiple sub-collaborative session data with a time sequence relationship according to different stages of the collaborative learning process;

[0184] Performing emotion annotation on each sub-collaboration session data to determine the learner's emotional state in each sub-collaboration session data;

[0185] The unimodal feature extractor is called to extract unimodal features from the preprocessed sub-collaboration session data. The unimodal features are used as input to the LSTM layer to capture the contextual information in the sub-collaboration session data, thereby obtaining the corresponding modal representations of the sub-collaboration session data.

[0186] Multiple single-modal features are spliced ​​together to obtain multimodal fusion features, and the cross-modal sentiment promotion information between multimodal fusion features and single-modal features is explored based on the Transformer architecture;

[0187] Based on the enhanced unimodal features, a multi-view sentiment reasoning graph with a dynamic context learning window is constructed. The node representations in the multi-view sentiment reasoning graph are updated through the graph attention mechanism to obtain the fused discourse representation of the sub-collaborative session data.

[0188] The fused discourse representation is used as the input of the conversation emotion prediction layer to train a collaborative conversation emotion recognition model. The collaborative conversation emotion recognition model is then called to recognize the preprocessed current collaborative conversation data to obtain the corresponding emotion recognition results.

[0189] Among them, the collaborative session data is obtained by audio and video acquisition equipment from the conversation interaction content in the online collaborative learning process, including video data and audio data. The preprocessing is used to fill the modality of the sub-collaborative session data after emotion annotation. The single-modal extractor includes pre-trained fast-RCNN, Wav2vec2.0 and BERT models. The multi-view emotion reasoning graph includes text modality emotion reasoning graph, audio modality emotion reasoning graph and video modality emotion reasoning graph. The conversation emotion prediction layer consists of a fully connected layer and a Softmax layer, which is used to output the predicted emotion label for the fused discourse representation.

[0190] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory.

[0191] By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0192] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0193] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A method for emotion recognition in online collaborative conversations with uncertain modality loss, characterized by: The method comprises: Acquire collaborative session data through an audio and video acquisition device, and divide the collaborative session data into a plurality of sub-collaborative session data having a time sequence relationship according to different stages of the collaborative learning process; further dividing each of the sub-collaborative conversation data into a plurality of collaborative discourse data in a chronological order according to conversation turns of different learners in the collaborative learning process; Preprocessing the plurality of collaborative discourse data so that when any modal data is missing in the collaborative discourse data, UNK in the vocabulary of the BERT model is called to fill in the missing corresponding modal data; Each collaborative discourse data set includes at least multiple rounds of collaborative interaction discourse between two learners, and each round of collaborative interaction discourse has modal representations of audio modality, video modality, and text modality. The multiple collaborative discourse data sets are used to construct a collaborative session dataset in combination with sub-collaborative session data with emotion annotations. When the collaborative discourse data lacks audio modality or video modality data, the audio modality or video modality data is filled with blank frames. Performing emotion annotation on each sub-collaboration session data to determine the learner's emotional state in each sub-collaboration session data; Invoking a unimodal feature extractor to extract unimodal features from the preprocessed sub-collaboration session data, and using the unimodal features as input to an LSTM layer to capture contextual information in the sub-collaboration session data, thereby obtaining representations of each modality corresponding to the sub-collaboration session data; The plurality of single-modal features are spliced ​​together to obtain a multimodal fusion feature, and the cross-modal emotion promotion information between the multimodal fusion feature and the single-modal feature is explored based on the Transformer architecture; Constructing a multi-view sentiment reasoning graph with a dynamic context learning window based on the enhanced unimodal features, and updating the node representations in the multi-view sentiment reasoning graph through a graph attention mechanism to obtain a fused discourse representation of the sub-collaborative session data; The fused speech representation is used as the input of the conversation emotion prediction layer to train a collaborative conversation emotion recognition model, and the collaborative conversation emotion recognition model is called to recognize the preprocessed current collaborative conversation data to obtain a corresponding emotion recognition result; Among them, the collaborative session data is obtained by the audio and video acquisition device collecting the conversation interaction content in the online collaborative learning process, including video data and audio data. The preprocessing is used to perform modality filling on the sub-collaborative session data after emotion annotation. The single-modal extractor includes pre-trained fast-RCNN, Wav2vec2.0 and BERT models. The multi-view emotion reasoning graph includes a text modality emotion reasoning graph, an audio modality emotion reasoning graph and a video modality emotion reasoning graph. The conversation emotion prediction layer consists of a fully connected layer and a Softmax layer, which is used to output the predicted emotion label for the fused discourse representation.

2. The method for online collaborative conversation emotion recognition based on uncertain modality loss according to claim 1 is characterized in that: The calling of the unimodal feature extractor to extract unimodal features from the preprocessed sub-collaboration session data, and using the unimodal features as input to the LSTM layer to capture contextual information in the sub-collaboration session data to obtain representations of each modality corresponding to the sub-collaboration session data, includes: Calling the pre-trained fast-RCNN, Wav2vec2.0, and BERT models as the unimodal feature extractors to respectively extract original unimodal features of the collaborative discourse data, and using the original unimodal features as inputs of the three LSTM layers to capture contextual features of the original collaborative session; Each modal data is input into the fully connected layer for feature dimension conversion, so as to map the modal features corresponding to each modal data to the same dimensional space, and obtain the aligned representations of each modality.

3. The method for online collaborative conversation emotion recognition based on uncertain modality loss according to claim 2 is characterized in that: The step of concatenating the plurality of single-modal features to obtain a multi-modal fusion feature, and exploring cross-modal emotion promotion information between the multi-modal fusion feature and the single-modal feature based on a Transformer architecture includes: When the direction of flow of emotional information is from multimodal features to unimodal features, the unimodal features are set as query vectors, and the multimodal features are set as key-value pairs, and complementary emotional information associated with the unimodal features is extracted from the multimodal features; and When the direction of emotional information flow is from the unimodal feature to the multimodal feature, the multimodal feature is set as the query vector, and the unimodal feature is set as the key-value pair, so as to extract key emotional information from the unimodal feature to enhance the multimodal feature.

4. The method for online collaborative conversation emotion recognition based on uncertain modality loss according to claim 3 is characterized in that: The method of constructing a multi-view sentiment reasoning graph with a dynamic context learning window based on the enhanced unimodal features and updating the node representations in the multi-view sentiment reasoning graph through a graph attention mechanism to obtain a fused discourse representation of the sub-collaborative session data includes: Constructing the text modality emotion reasoning graph, the audio modality emotion reasoning graph, and the video modality emotion reasoning graph for each of the collaborative discourse data, wherein the nodes in each reasoning graph are used to represent the collaborative interaction discourse of each round; The multimodal features and unimodal features are fused through a graph attention mechanism to obtain the enhanced unimodal features to initialize the collaborative interaction discourse represented by each node.

5. The method for online collaborative conversation emotion recognition based on uncertain modality loss according to claim 4 is characterized in that: The method further includes: constructing a multi-view sentiment reasoning graph with a dynamic context learning window based on the enhanced unimodal features, and updating the node representations in the multi-view sentiment reasoning graph through a graph attention mechanism to obtain a fused discourse representation of the sub-collaborative session data. Acquiring the speech density and the number of speeches of each learner in the collaborative learning process in the collaborative speech data, and calculating the context connection window of the collaborative speech data based on the speech density and the number of speeches to set the dynamic context learning window; Initializing edge relationships and edge weights for each modal sentiment reasoning graph, respectively, wherein the edge weights are calculated and determined based on sentiment semantic similarities between a current node and a plurality of neighboring nodes within the dynamic context learning window; The graph attention mechanism is used to update the information of the current node according to the types of different edges between the neighboring nodes and the dependencies between the nodes.

6. The method for online collaborative conversation emotion recognition based on uncertain modality loss according to claim 5 is characterized in that: The fused speech representation is used as the input of the conversation emotion prediction layer to train a collaborative conversation emotion recognition model, and the collaborative conversation emotion recognition model is called to recognize the preprocessed current collaborative conversation data to obtain a corresponding emotion recognition result, including: Sending the fused speech representation to multiple decoders respectively to reconstruct the initial representation of each modality and calculating the loss between the reconstructed value and the true value; and Based on the multimodal representation and the annotated emotion labels of the collaborative speech data, and in combination with the multimodal representation and emotion labels of the positive example and the multimodal representation and emotion labels of the negative example, calculating the contrast loss between different collaborative speech data; Based on the loss between the reconstructed value and the true value and the contrast loss, the loss function of the collaborative conversation emotion recognition model is determined, and the model parameters are learned through back propagation and gradient descent algorithms to obtain the collaborative conversation emotion recognition model under the optimal model parameters.

7. An online collaborative conversation emotion recognition system for uncertain modality loss, characterized by: The system comprises: A session segmentation module is used to obtain collaborative session data through an audio and video acquisition device, and to segment the collaborative session data into a plurality of sub-collaborative session data with a time sequence relationship according to different stages of the collaborative learning process; Data preprocessing module, used to: According to the conversation turns of different learners in the collaborative learning process, each sub-collaborative conversation data is further divided into multiple collaborative discourse data with a time sequence; Preprocess multiple collaborative discourse data to fill in the missing corresponding modal data by calling UNK in the vocabulary of the BERT model when any modal data is missing in the collaborative discourse data; Each collaborative discourse data set includes at least multiple rounds of collaborative interaction discourse between two learners, and each round of collaborative interaction discourse has modal representations of audio modality, video modality, and text modality. Multiple collaborative discourse data sets are used to construct a collaborative conversation dataset in combination with sub-collaborative conversation data with emotion annotations. When audio modality or video modality data is missing in the collaborative discourse data, the audio modality or video modality data is filled with blank frames. The emotion annotation module is used to perform emotion annotation on each sub-collaboration session data to determine the learner's emotional state in each sub-collaboration session data; a modal feature acquisition module, configured to call a unimodal feature extractor to extract unimodal features from the preprocessed sub-collaboration session data, and use the unimodal features as input to an LSTM layer to capture contextual information in the sub-collaboration session data, thereby obtaining representations of each modality corresponding to the sub-collaboration session data; A feature fusion module is used to splice multiple single-modal features to obtain multimodal fusion features, and explore the cross-modal emotion promotion information between the multimodal fusion features and the single-modal features based on the Transformer architecture; A fusion discourse representation module is used to construct a multi-view sentiment reasoning graph with a dynamic context learning window based on the enhanced unimodal features, and to update the node representations in the multi-view sentiment reasoning graph through a graph attention mechanism to obtain a fusion discourse representation of the sub-collaboration session data; An emotion recognition module is configured to use the fused speech representation as input to a conversation emotion prediction layer, train a collaborative conversation emotion recognition model, and call the collaborative conversation emotion recognition model to recognize the preprocessed current collaborative conversation data to obtain a corresponding emotion recognition result; Among them, the collaborative session data is obtained by the audio and video acquisition device collecting the conversation interaction content in the online collaborative learning process, including video data and audio data. The preprocessing is used to perform modality filling on the sub-collaborative session data after emotion annotation. The single-modal extractor includes pre-trained fast-RCNN, Wav2vec2.0 and BERT models. The multi-view emotion reasoning graph includes a text modality emotion reasoning graph, an audio modality emotion reasoning graph and a video modality emotion reasoning graph. The conversation emotion prediction layer consists of a fully connected layer and a Softmax layer, which is used to output the predicted emotion label for the fused discourse representation.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images