A multimodal dialogue sentiment recognition method and system

Abstract

The disclosure provides a multi-modal dialogue emotion recognition method and system, including obtaining a dialogue data set and a true emotion label of a dialogue, the dialogue data set including multiple dialogues and modal features contained in each dialogue; training a multi-modal dialogue emotion recognition model, inputting the dialogue data set into the multi-modal dialogue emotion recognition model to obtain a predicted emotion label of each dialogue, calculating a cross-entropy loss of the model according to the predicted emotion label and the true emotion label of each dialogue, and obtaining a trained multi-modal dialogue emotion recognition model when the cross-entropy loss of the model is the smallest; inputting dialogue data to be recognized into the trained multi-modal dialogue emotion recognition model to obtain an emotion recognition result. In the model training stage, a modal parameter decoupling technology and a fine-grained self-adaptive gradient adjustment mechanism are fused, effectively solving the problem of multi-modal marginal effect decrease caused by gradient distribution imbalance between different modalities.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of emotion recognition, and particularly relates to a multi-modal dialogue emotion recognition method and system. BACKGROUND

[0002] Emotion recognition tasks have strong application value, and span multiple application fields, including healthcare, education and services. The dialogue emotion recognition task (Emotion Recognition in Conversation, ERC) has attracted common interest from multiple disciplines, from artificial intelligence and cognitive science to social science, due to its universality. The main goal of the dialogue emotion recognition task is to detect the emotional color accompanying each utterance in the dialogue. However, the way humans perceive emotions is diverse, and includes not only the content of the dialogue itself, but also non-verbal elements such as changes in tone of voice and changes in facial expressions. Therefore, to accurately capture emotions, it is also necessary to consider multiple perception methods, including linguistic and non-linguistic indicators.

[0003] With the development of computer vision, many studies have begun to use multi-modal information (or multiple perception methods) to provide the model with diversified perspectives to enhance the model's ability to recognize dialogue emotions. However, through analysis of existing research, it can be found that simply increasing the number of perception methods is not always proportional to the improvement of the performance of dialogue emotion recognition models. For example, when introducing audio, visual or audio and visual dual-modal information to a dialogue emotion recognition model that only relies on pure text information, the model performance does not necessarily continue to improve with the increase in the number of perception methods. Therefore, there is a marginal effect problem in the field of multi-modal emotion recognition, that is, increasing additional information sources (such as text, audio, vision, etc.) may reach a limit in performance at a certain point, after which there is no obvious improvement, and may even cause the performance of the model to decrease. Therefore, one of the focuses of current research is to explore the reasons for the marginal effect diminishing phenomenon in the field of multi-modal emotion recognition and to solve the performance bottleneck in multi-modal emotion recognition.

[0004] The reasons for the multimodal marginal diminishing effect can be summarized as follows in different research fields: 1) the problem of aggravated model overfitting caused by the increase in the number of modalities; 2) the problem of modality imbalance caused by the difference in cross-modality feature specifications; and 3) the unbalanced optimization of the model to different modalities. However, in the unique context of the dialogue sentiment recognition task, there is no problem of aggravated model overfitting caused by the increase in the number of modalities, and there is no problem of modality imbalance caused by the difference in cross-modality feature specifications. By decoupling the learning process of the dialogue sentiment recognition model on different modalities, it can be found that there is a difference in the optimization speed of different modalities, and the modality with a slower optimization speed is more inhibited. In order to more deeply analyze the reasons for the unbalanced optimization of the model to different modalities, the average gradient allocated to each modality in the joint training process is observed. The inventor finds that, in the joint training process, the inhibited modality is allocated a smaller average gradient, and some modalities cannot converge in the joint training because of being inhibited, which leads to the decline in the performance of the multimodal model. SUMMARY

[0005] In order to effectively solve the multimodal marginal diminishing effect caused by the unbalanced gradient allocation between different modalities, the present application proposes a multimodal dialogue sentiment recognition method and system. In the model training stage, attention is focused on the neurons of the sentiment recognition model, the dominant modality and the non-dominant modality of the neurons are identified by using the modality parameter decoupling technology, and then the modulation coefficients of each modality are calculated based on the loss function, so as to re-allocate the gradient for the dominant modality and the non-dominant modality, balance the proportion of the inhibited modality and the dominant modality, and thus relieve the gradient imbalance problem between each modality.

[0006] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0007] In the first aspect, the present application provides a multimodal dialogue sentiment recognition method, comprising:

[0008] obtaining a dialogue data set and a true sentiment label of a dialogue, wherein the dialogue data set comprises a plurality of dialogues and modality features contained in each dialogue;

[0009] training a multimodal dialogue sentiment recognition model, inputting the dialogue data set into the multimodal dialogue sentiment recognition model to obtain a predicted sentiment label of each dialogue, calculating a cross-entropy loss of the model according to the predicted sentiment label and the true sentiment label of each dialogue, and obtaining a trained multimodal dialogue sentiment recognition model when the cross-entropy loss of the model is the smallest;

[0010] inputting dialogue data to be recognized into the trained multimodal dialogue sentiment recognition model to obtain a sentiment recognition result.

[0011] Preferably, the training of the multimodal dialogue sentiment recognition model comprises:

[0012] obtain a dialogue dataset and a true sentiment label of a dialogue, the dialogue dataset comprising a plurality of dialogues and a plurality of multi-modal features contained in each dialogue; extract the multi-modal features contained in each dialogue to obtain full-modal features;

[0013] erase one type of modal feature in the full-modal features to obtain erased modal features; and obtain erased modal features according to the full-modal features and the erased modal features;

[0014] calculate a response degree of each neuron in the multi-modal dialogue sentiment recognition model to the erased modal features;

[0015] divide the neurons into dominant modalities and non-dominant modalities according to the response degree;

[0016] input the dialogue dataset into the multi-modal dialogue sentiment recognition model to obtain a predicted sentiment label of each dialogue, calculate a cross-entropy loss of the model and a cross-entropy loss of the erased modal features according to the predicted sentiment label of each dialogue and the true sentiment label, and calculate an optimization speed of each modality according to the cross-entropy loss of the model and the cross-entropy loss of the erased modal features;

[0017] calculate a modality modulation coefficient according to the optimization speed of each modality;

[0018] redistribute gradients for the dominant modalities and the non-dominant modalities according to the modality modulation coefficient.

[0019] Preferably, the calculation of the response degree of each neuron in the multi-modal dialogue sentiment recognition model to the erased modal features is specifically:

[0020]

[0021] wherein a i is the i-th neuron in the multi-modal dialogue sentiment recognition model; u m，k represents the k-th value in the m-th modality information in the dialogue in the dialogue dataset, M represents the number of modalities contained in the current dialogue, and D represents the feature dimension of the dialogue; and the dialogue with the m-th modality information erased is denoted as represents the k-th value in the m-th modality information in the dialogue .

[0022] Preferably, the division of the neurons into dominant modalities and non-dominant modalities according to the response degree is specifically defined as an indicator function expression:

[0023] and k≠m

[0024] where | | is an absolute value operation, is the value of the indicator function II(*), is the response degree of the neuron based on the mth modality, is the response degree of the neuron based on the kth modality, and k≠m; the parameter ∈ is a threshold value;

[0025] When the response degree of the neuron to the mth modality is greater than that to all other modalities, the indicator function value is 1, and the mth modality is the dominant modality of the neuron; otherwise, it is 0, and the mth modality is a non-dominant modality of the neuron.

[0026] Preferably, the dialogue data set is input into the multi-modal dialogue sentiment recognition model to obtain a predicted sentiment label of each dialogue, and a cross-entropy loss of the model and a cross-entropy loss of the erased modality feature are calculated according to the predicted sentiment label and the true sentiment label of each dialogue; the optimization speed of each modality is calculated according to the cross-entropy loss of the model and the cross-entropy loss of the erased modality feature, specifically:

[0027]

[0028] where, denotes a set of parameters of the multi-modal dialogue sentiment recognition model, denotes a set of dialogue features of the mini-batch at the tth iteration step; N t and N c denote the total number of dialogues and the total number of sentiment categories in the set of dialogue features, respectively; y i，j denotes a sentiment label value of the jth category of the ith dialogue, y i，j = 1 when the ith utterance belongs to the jth category, otherwise y i，j = 0; e i，j denotes a predicted probability value of the jth category of the ith dialogue.

[0029] The optimization speed measurement formula of the mth modality at the tth iteration step is as follows:

[0030]

[0031] where, is a set of utterance features of the mini-batch in which the mth modality information of all utterances is erased.

[0032] Preferably, the modulation coefficients of each modality are calculated according to the optimization speed of each modality, specifically:

[0033]

[0034] where, is the optimization speed of the m-th modality, is the optimization speed of the k-th modality, denotes the modulation coefficient of the m-th modality, and the modulation coefficient of the m-th modality decreases with the increase of the optimization proportion of the m-th modality , and when the value of the modulation coefficient of the m-th modality is close to 1, the optimization proportion of the m-th modality reaches a minimum value, and the parameter γ is a hyperparameter for controlling the modulation degree.

[0035] Preferably, the gradient is redistributed for the dominant modalities and the non-dominant modalities according to the modality modulation coefficients, specifically, the modality-shared parameters are updated for the dominant modalities and the non-dominant modalities:

[0036]

[0037] wherein, denotes the modality-shared parameter connecting the i-th neuron of the (l-1)-th layer and the j-th neuron of the l-th layer in the model; is the indication coefficient of the i-th neuron of the (l-1)-th layer to the m-th modality calculated by the modality indicator; z m is the modality modulation coefficient; η denotes the learning rate.

[0038] In a second aspect, the present application provides a multi-modal dialogue sentiment recognition system, comprising:

[0039] an extraction module configured to obtain a dialogue data set and a real sentiment label of a dialogue, wherein the dialogue data set comprises a plurality of dialogues and modality features contained in each dialogue;

[0040] a training module configured to train a multi-modal dialogue sentiment recognition model, input the dialogue data set into the multi-modal dialogue sentiment recognition model to obtain a predicted sentiment label of each dialogue, calculate a cross-entropy loss of the model according to the predicted sentiment label and the real sentiment label of each dialogue, and obtain a trained multi-modal dialogue sentiment recognition model when the cross-entropy loss of the model is the smallest;

[0041] a recognition module configured to input dialogue data to be recognized into the trained multi-modal dialogue sentiment recognition model to obtain a sentiment recognition result.

[0042] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the steps of the multi-modal dialogue sentiment recognition method described above.

[0043] In a fourth aspect, the present application provides a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the multi-modal dialogue sentiment recognition method described above when executing the program.

[0044] Compared with the prior art, the present application has the following beneficial effects:

[0045] (1) The present application introduces a plug-in method, designs a modal parameter decoupling technology and a fine-grained adaptive gradient adjustment mechanism, which can be embedded into existing multi-modal dialogue emotion recognition methods. Through the synergistic effect of the two, the model can re-allocate the gradient to each modality according to the optimization speed ratio of different modalities at the parameter level, thereby achieving balance between modalities. This method does not require substantial modification of existing models, achieving seamless integration of the model, and the present application is more robust and has high adaptability compared with existing methods. BRIEF DESCRIPTION OF DRAWINGS

[0046] The accompanying drawings, which form a part of the present application, are used to provide further understanding of the present application, and the illustrative embodiments of the present application and their description serve the purpose of explaining the present application. The present application is not limited in scope by the illustrative embodiments.

[0047] Figure 1 A flowchart of a multi-modal dialogue sentiment recognition model training method provided for the first embodiment of the present application;

[0048] Figure 2 A model training process schematic diagram of a multi-modal dialogue sentiment recognition model training method provided for the first embodiment of the present application.

[0049] Wherein, 1 - modal parameter decoupling part, 2 - fine-grained adaptive gradient adjustment part. DETAILED DESCRIPTION

[0050] The present application will be further described below in conjunction with the drawings and embodiments.

[0051] Explanation of technical terms:

[0052] (1) Model parameters: Parameters are learnable variables in neural networks, used to control the connections and information transmission between neurons.

[0053] (2) Parameter decoupling: In neural networks, decoupling usually refers to separating or decomposing parameters that are originally tightly coupled into different components in order to better understand and control the behavior of the network. In the present application, parameter decoupling refers to decoupling parameters shared by each modality (modality-shared parameters).

[0054] (3) Neuron: Neuron in neural network (sentiment recognition model).

[0055] (4) Multi-modal erasing feature: For example, the current multi-modal feature is text feature, visual feature, and audio feature. The erasing modal feature of the visual feature is to set the input visual modal feature to zero.

[0056] As described in the background, there is a problem of gradient imbalance between different modalities in the task of dialogue sentiment recognition. Therefore, the following embodiments give a training method and device of a multi-modal dialogue sentiment recognition model, wherein the training method is divided into two parts, 1) modal parameter decoupling technology (MDP). The modal parameter decoupling technology realizes the identification of the unique parameters of a specific modality. Specifically, it first evaluates the sensitivity / response degree of each neuron to different modalities, and then classifies the parameters according to the sensitivity / response degree. 2) Fine-grained adaptive gradient modulation mechanism (FAGM). The fine-grained adaptive gradient modulation mechanism can effectively improve the problem of gradient imbalance in the task of dialogue sentiment recognition. Specifically, according to the proportion of the optimization speed of different modalities, the fine-grained adaptive gradient modulation mechanism emphasizes the suppressed modality by readjusting the gradient of each modality, balances the proportion of the suppressed modality and the dominant modality, and thus relieves the gradient imbalance problem between each modality.

[0057] Embodiment one

[0058] As shown in Figure 1 , the embodiment discloses a training method of a multi-modal dialogue sentiment recognition model, comprising the following steps:

[0059] S1: obtaining a dialogue data set and a true sentiment label of a dialogue, wherein the dialogue data set comprises a plurality of dialogues and modal features contained in each dialogue;

[0060] S2: training a multi-modal dialogue sentiment recognition model, inputting the dialogue data set into the multi-modal dialogue sentiment recognition model to obtain a predicted sentiment label of each dialogue, calculating a cross-entropy loss of the model according to the predicted sentiment label and the true sentiment label of each dialogue, and obtaining a trained multi-modal dialogue sentiment recognition model when the cross-entropy loss of the model is the smallest;

[0061] S3: inputting dialogue data to be recognized into the trained multi-modal dialogue sentiment recognition model to obtain a sentiment recognition result.

[0062] Specifically, compared to existing technologies, this application focuses on optimizing the model training stage. The diminishing marginal effect of multimodal effects in dialogue emotion recognition tasks stems from the problem of unbalanced gradient allocation. Through the synergistic effect of modal parameter decoupling technology and fine-grained adaptive gradient adjustment mechanism, the model can redistribute gradients to each modality at the parameter level according to the optimization speed ratio of different modalities, thereby achieving a balance between the modalities and making the recognition results obtained by the trained emotion recognition model more accurate.

[0063] 1. Problem Definition

[0064] The dialogue sentiment recognition task can be described as: predicting the dialogue sequence U = {U1, U2, ..., U...} N Each dialogue U in} k The tag e k , where N represents the number of dialogues in the dialogue sequence. In the setup of a multimodal dialogue sentiment recognition task, each utterance should contain multiple modal features, i.e. Where M represents the number of modal features contained in the current discourse, and D represents the feature dimension of the discourse.

[0065] Specifically, multimodal features include text features, audio features, and visual features.

[0066] 2. Modal parameter decoupling technology

[0067] Modal parameter decoupling techniques play a crucial role in rebalancing gradient allocation between modes. They can effectively address the complexity challenges brought about by decoupling gradients between different modes, i.e., modal parameter sharing in multimodal models.

[0068] Specifically, firstly, model parameters are divided into two types: modality-unique parameters and modality-shared parameters, depending on whether they are shared across modalities. These parameters exist independently in each modality or are shared among modalities, respectively.

[0069] Next, in order to achieve decoupling of modal shared parameters, the modal parameter decoupling technique focuses on neurons and assumes that each neuron is sensitive to different modalities. This allows each neuron to classify each modality into a dominant modality and a non-dominant modality relative to the neuron based on its response to different modalities.

[0070] like Figure 2 As shown, since each neuron in the model learns from the input modal information, the neuron is represented as:

[0071] a = f(wx + b)

[0072] Here, f is the transfer function, x is the input feature vector, w is the modality-shared parameter, and b is the bias value. Therefore, the neuron and the modality-shared parameter are inseparable, thus enabling the identification of the dominant and non-dominant modalities of the neuron using modality parameter decoupling techniques.

[0073] The specific steps are as follows:

[0074] (1) Neuron sensitivity / response measurement

[0075] Let a i For the i-th neuron, in order to measure neuron a i Dialogue The m-th modal information The degree of responsiveness, therefore, firstly, design a new discourse. The m-th mode This is transformed into a noise-free vector, i.e., the information of the m-th modality is erased. Next, neuron a is calculated. i Dialogue The m-th modal information The higher the responsiveness of a neuron, the greater its focus on the current modality, meaning it will allocate a larger gradient to the current modality. Therefore, the change in the responsiveness of each neuron can approximately reflect its relative importance to the various modalities.

[0076]

[0077] (2) Adaptability calculation between neurons and modalities

[0078] The neuron-modality fit calculation aims to classify neurons based on their responsiveness to different modalities. In simpler terms, it identifies a neuron's tendency to process specific modalities. Specifically, the indicator function is defined as follows:

[0079]

[0080] Where |*| represents the absolute value operation. The value of the index function II(*) This refers to the neuron's response based on the m-th modality. This represents the neuron's responsiveness to the k-th mode. Its value is 1 when the neuron's responsiveness to the m-th mode is greater than that to all other modes, indicating the m-th mode is the neuron's dominant mode; otherwise, it is 0, indicating the m-th mode is the neuron's non-dominant mode. The parameter ∈ is a threshold used to determine the exclusivity of the selected neuron for the chosen mode; that is, the larger the value of ∈, the greater the probability that the selected neuron belongs only to the selected mode.

[0081] The indicator coefficient obtained from the indicator function It illustrates the relationship between the i-th neuron and the M modalities. In short, when... When , it indicates that the i-th neuron is adapted to the m-th modality. It is important to note that for s... i In this case, it can only have one modality with a value of 1 at a time, meaning that each neuron can only be assigned to one modality at a time to avoid repeated modulation of the gradient.

[0082] like Figure 2 As shown, a dialogue dataset and the real sentiment labels of the dialogues are obtained. The dialogue dataset includes multiple dialogues and the multimodal features contained in each dialogue. The multimodal features contained in each dialogue are extracted to obtain the full modal features.

[0083] Erasing one type of modal feature from the full modal features yields the erased modal features;

[0084] The method for obtaining the erased modal features is as follows: erase one type of modal features from the full modal features to obtain the erased modal features; and obtain the erased modal features based on the full modal features and the erased modal features.

[0085] Specifically, Figure 2 In section 1, the modal parameter decoupling assumes the erased modal feature is the second modal feature, and calculates the response degree of each neuron to the second modal feature. It can be seen that there are highlighted neurons in the neural network (highlighting indicates...). Based on this, it can be determined that the second modality feature is the dominant modality of the highlighted neuron.

[0086] The modality parameter decoupling technique essentially aims to identify which mode a current neuron is more sensitive to. Based on this, during the loss optimization phase, it aims to suppress the influence of that mode on the direction of neuron parameter optimization, so that the direction of gradient descent is no longer solely controlled by the dominant mode. In this way, non-dominant modes can also be fully learned.

[0087] 3. Fine-grained adaptive gradient adjustment mechanism

[0088] Because modal loss can reflect the imbalanced optimization between different modalities, the optimization speed of each modality can be rebalanced based on its own loss. In short, the optimization objective of dialogue sentiment recognition is to minimize the cross-entropy loss at each iteration step. The cross-entropy loss for dialogue sentiment recognition is defined as follows:

[0089]

[0090] in, The parameter set representing the multimodal model. denotes the set of utterance features for the mini-batch at the t-th iteration step. N t and N c denote the total number of utterances and the total number of sentiment classes in the set of utterance features, respectively. y i，j denotes the label value of the j-th class for the i-th utterance, y i，j = 1 when the i-th utterance belongs to the j-th class, otherwise y i，j = 0. e i，j denotes the predicted probability value of the j-th class for the i-th utterance.

[0091] (1) Measure the optimization speed ratio

[0092] The optimization speed of the m-th modality at the t-th iteration step is measured as follows:

[0093]

[0094] where, is the set of utterance features for the mini-batch with the m-th modality information erased from all utterances. By using the softmax operation for normalization, the diversity of the optimization speed can reflect the dominance of the current modality in model training, because erasing the dominant modality can lead to more significant changes in loss. Then, we calculate the modulation coefficient of the m-th modality using the following formula:

[0095] (2) Calculate the modulation coefficient

[0096] The modulation coefficient of the m-th modality is calculated as follows:

[0097]

[0098] where, is the optimization speed of the m-th modality, is the optimization speed of the k-th modality, denotes the modulation coefficient of the m-th modality, and the modulation coefficient of the m-th modality decreases as the optimization ratio of the m-th modality increases, and when the value of the modulation coefficient of the m-th modality is close to 1, the optimization ratio of the m-th modality reaches a minimum value. Therefore, the modulation coefficient of the dominant modality is the smallest. In addition, the parameter γ is a hyperparameter that controls the degree of modulation.

[0099] The adjustment coefficient is essentially equivalent to changing the direction of gradient descent. If there is no adjustment coefficient, the direction of gradient descent will be more biased to the dominant mode. Therefore, the decoupling process is equivalent to finding which mode has the greatest impact on the neuron, and the gradient adjustment mechanism is to weaken the impact of the dominant mode on the gradient descent direction, or in other words, to make the gradient descent direction slightly biased towards the non-dominant mode. In this way, the problem of non-dominant mode not being able to learn better can be solved.

[0100] (3) Update the mode parameters

[0101] At the t-th iteration step, the model parameter is updated as follows during the modulation of the stochastic gradient descent (SGD) optimization method:

[0102]

[0103] wherein, represents the mode sharing parameter connecting the i-th neuron of the l-1-th layer and the j-th neuron of the l-th layer in the model. is the mode indicator calculated by the i-th neuron of the l-1-th layer to the m-th mode (calculated by formula 2). η represents the learning rate.

[0104] By updating the model parameters, the adjustment of the gradient descent direction is realized. As shown in Figure 2 , the gradient descent part in the fine-grained adaptive gradient adjustment part 2 is adjusted. The black arrow represents the unbalanced gradient descent direction dominated by the dominant mode before the modulation coefficient is added, and the gray arrow represents the gradient descent direction after the modulation coefficient is adjusted and the optimization speed of each mode is balanced.

[0105] Note that in order to improve the efficiency of the modulation, a modulation interval τ is introduced to adjust the modulation frequency. Specifically, at the end of every t iteration step, the mode indication coefficient and the modulation coefficient are updated (calculated by formula 2 and formula 5, respectively), and the rest of the time, these coefficients remain unchanged to reduce the amount of calculation.

[0106] In the model training stage, the present application combines the mode parameter decoupling technology and the fine-grained adaptive gradient adjustment mechanism. First, the dominant mode and the non-dominant mode of the neuron are identified using the mode parameter decoupling technology, and then the modulation coefficient of each mode is calculated based on the loss function to redistribute the gradient for the dominant mode and the non-dominant mode, so that the proportion of the suppressed mode and the dominant mode is balanced, thereby effectively alleviating the gradient imbalance problem between each mode. Inputting the dialog data to be identified into the trained multi-modal dialog emotion recognition model can improve the accuracy of the output emotion recognition result.

[0107] Embodiment two

[0108] The embodiment provides a multi-modal dialogue emotion recognition system, comprising:

[0109] An extraction module is configured to acquire dialogue data sets and real emotion labels of dialogues, wherein the dialogue data sets comprise multiple dialogues and modal features contained in each dialogue;

[0110] A training module is configured to train a multi-modal dialogue emotion recognition model, input the dialogue data sets into the multi-modal dialogue emotion recognition model, obtain predicted emotion labels of each dialogue, calculate cross-entropy loss of the model according to the predicted emotion labels and the real emotion labels of each dialogue, and obtain the trained multi-modal dialogue emotion recognition model when the cross-entropy loss of the model is the minimum.

[0111] An identification module is configured to input dialogue data to be identified into the trained multi-modal dialogue emotion recognition model to obtain an emotion recognition result.

[0112] Embodiment three

[0113] The embodiment provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement steps in the multi-modal dialogue emotion recognition method in the above embodiment one.

[0114] First, the sensitivity / response degree of each neuron to different modalities is evaluated, then the parameters are classified according to the sensitivity / response degree, and finally, the fine-grained adaptive gradient adjustment mechanism is used to effectively improve the gradient distribution imbalance problem in the dialogue emotion recognition task. Specifically, according to the proportion of the optimization speed of different modalities, the fine-grained adaptive gradient adjustment mechanism emphasizes the suppressed modality by readjusting the gradient of each modality, balances the proportion of the suppressed modality and the dominant modality, and thus relieves the gradient imbalance problem between each modality.

[0115] Embodiment four

[0116] The embodiment provides a computer device, which comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, and the processor implements steps in the multi-modal dialogue emotion recognition method in the above embodiment one when executing the program.

[0117] First, the sensitivity / response degree of each neuron to different modalities is evaluated, then the parameters are classified according to the sensitivity / response degree, and finally, the fine-grained adaptive gradient adjustment mechanism is used to effectively improve the gradient distribution imbalance problem in the dialogue emotion recognition task. Specifically, according to the proportion of the optimization speed of different modalities, the fine-grained adaptive gradient adjustment mechanism emphasizes the suppressed modality by readjusting the gradient of each modality, balances the proportion of the suppressed modality and the dominant modality, and thus relieves the gradient imbalance problem between each modality.

[0118] The steps or modules involved in the above embodiments two to four correspond to embodiment one, and the detailed description can refer to the relevant description part of embodiment one. The term "computer readable storage medium" should be understood to include a single medium or multiple media of one or more instruction sets; it should also be understood to include any medium capable of storing, encoding, or carrying instruction sets for execution by a processor and causing the processor to perform any of the methods in the present application.

[0119] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for multi-modal dialogue sentiment recognition, characterized in that, The method comprises the following steps: obtaining a dialogue data set and a real emotion label of a dialogue, wherein the dialogue data set comprises a plurality of dialogues and modal features contained in each dialogue; training a multi-modal dialogue emotion recognition model, inputting the dialogue data set into the multi-modal dialogue emotion recognition model to obtain a predicted emotion label of each dialogue, and calculating a cross-entropy loss of the model according to the predicted emotion label and the real emotion label of each dialogue, wherein the trained multi-modal dialogue emotion recognition model is obtained when the cross-entropy loss of the model is the smallest; the training of the multi-modal dialogue emotion recognition model specifically comprises the following steps: obtaining a dialogue data set and a real emotion label of a dialogue, wherein the dialogue data set comprises a plurality of dialogues and multi-modal features contained in each dialogue; extracting the multi-modal features contained in each dialogue to obtain full-modal features; erasing a type of modal features in the full-modal features to obtain erased modal features; obtaining erased modal features according to the full-modal features and the erased modal features; calculating the response degree of each neuron in the multi-modal dialogue emotion recognition model to the erased modal features; dividing the dominant modal and the non-dominant modal of the neuron according to the response degree; inputting the dialogue data set into the multi-modal dialogue emotion recognition model to obtain a predicted emotion label of each dialogue, and calculating a cross-entropy loss of the model and a cross-entropy loss of the erased modal features according to the predicted emotion label and the real emotion label of each dialogue; calculating the optimization speed of each modal according to the cross-entropy loss of the model and the cross-entropy loss of the erased modal features; calculating the modulation coefficient of each modal according to the optimization speed of each modal; and reassigning the gradient to the dominant modal and the non-dominant modal according to the modulation coefficient of each modal; inputting dialogue data to be recognized into the trained multi-modal dialogue emotion recognition model to obtain an emotion recognition result. 2.The multi-modal dialogue sentiment recognition method of claim 1, wherein, The calculation of the response degree of each neuron in the multi-modal dialogue emotion recognition model to the erased modal features specifically comprises the following steps: in, In the multimodal dialogue emotion recognition model, the first One neuron; This represents the dialogue in the dialogue dataset. The Middle The first modal information One value, This indicates the number of modalities contained in the current dialogue. Represents the feature dimensions of the dialogue; will erase the first A dialogue with modal information is denoted as express The Middle The first modal information Values. 3.The multi-modal dialogue sentiment recognition method of claim 1, wherein, The division of the dominant modal and the non-dominant modal of the neuron according to the response degree specifically comprises the following steps: defining an indicator function expression: wherein, is an absolute value operation, is an indicator function of value, is a response of the neuron based on the first modality, is a response of the neuron based on the second modality, and ; parameter is a threshold value; When the neuron is on the first When the response of a mode is greater than that of all other modes, the indicator function value is 1. The first mode is the dominant mode of the neuron; otherwise, it is 0. Each modality is a non-dominant modality of the neuron.

4. The multi-modal dialogue sentiment recognition method of claim 1, wherein, The inputting of the dialogue data set into the multi-modal dialogue emotion recognition model to obtain a predicted emotion label of each dialogue, the calculation of a cross-entropy loss of the model and a cross-entropy loss of the erased modal features according to the predicted emotion label and the real emotion label of each dialogue, and the calculation of the optimization speed of each modal according to the cross-entropy loss of the model and the cross-entropy loss of the erased modal features specifically comprise the following steps: in, This represents the parameter set of a multimodal model. Indicates the first The set of utterance features of mini-batch in each iteration step; and These represent the total number of discourses and the total number of sentiment categories in the set of discourse features, respectively. Indicates the first The first of the words The label value of the nth category, when the nth category is... The word belongs to the first When there are multiple categories, Otherwise ; Indicates the first The first of the words The predicted probability values ​​for each category; In the first iteration step, the optimization speed measurement formula of the first modality is as follows: In the second iteration step, the optimization speed measurement formula of the first modality is as follows: In the second iteration step, the optimization speed measurement formula of the first modality is as wherein, is the set of utterance features for the mini-batch with the mth modality information erased from all utterances. 5.The multi-modal dialogue sentiment recognition method of claim 1, wherein, The calculation of the modulation coefficient of each modal according to the optimization speed of each modal specifically comprises the following steps: in, This refers to the optimization speed of the m-th mode. This refers to the optimization speed of the k-th mode. Let m represent the modulation coefficient of the m-th mode, and let m be the modulation coefficient of the m-th mode. Modulation coefficients of each mode With the optimization ratio of the m-th mode As it increases, it decreases, when the first Modulation coefficients of each mode When the value is close to 1, the optimization ratio of the m-th mode is... To reach the minimum value, the parameter It is a hyperparameter that controls the degree of modulation. 6.The multi-modal dialogue sentiment recognition method of claim 1, wherein, The reassignment of the gradient to the dominant modal and the non-dominant modal according to the modulation coefficient of each modal specifically comprises the following steps: updating the modal shared parameters of the dominant modal and the non-dominant modal: in, The model represents the connection of the first The first layer The first neuron and the second The first layer Modal shared parameters among neurons; The modal indicator calculates the first... The first layer The first neuron pairs with the first The indicator coefficients for each mode; This represents the learning rate.

7. A multi-modal conversational sentiment recognition system, characterized in that, The method comprises the following steps: an extraction module is configured to obtain a dialogue data set and a real emotion label of a dialogue, wherein the dialogue data set comprises a plurality of dialogues and modal features contained in each dialogue; The training module is configured to train the multi-modal dialogue sentiment recognition model, input the dialogue data set into the multi-modal dialogue sentiment recognition model, obtain a predicted sentiment label of each dialogue, calculate a cross-entropy loss of the model according to the predicted sentiment label and a real sentiment label of each dialogue, and obtain the trained multi-modal dialogue sentiment recognition model when the cross-entropy loss of the model is the smallest. The training of the multi-modal dialogue sentiment recognition model specifically includes: obtaining a dialogue data set and a real sentiment label of a dialogue, wherein the dialogue data set includes a plurality of dialogues and multi-modal features contained in each dialogue; extracting the multi-modal features contained in each dialogue to obtain full-modal features; erasing a type of modal features in the full-modal features to obtain erased modal features; obtaining erased modal features according to the full-modal features and the erased modal features; calculating a response degree of each neuron in the multi-modal dialogue sentiment recognition model to the erased modal features; dividing dominant modal and non-dominant modal of the neuron according to the response degree; inputting the dialogue data set into the multi-modal dialogue sentiment recognition model to obtain a predicted sentiment label of each dialogue, calculating a cross-entropy loss of the model and a cross-entropy loss of the erased modal features according to the predicted sentiment label and the real sentiment label of each dialogue; calculating an optimization speed of each modal according to the cross-entropy loss of the model and the cross-entropy loss of the erased modal features; calculating a modulation coefficient of each modal according to the optimization speed of each modal; and reassigning gradients for the dominant modal and the non-dominant modal according to the modulation coefficient of each modal. The recognition module is configured to input dialogue data to be recognized into the trained multi-modal dialogue sentiment recognition model to obtain a sentiment recognition result.

8. A computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the multi-modal dialogue sentiment recognition method according to any one of claims 1-6.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the multi-modal dialogue sentiment recognition method according to any one of claims 1-6 when executing the program.

Images