Emotion recognition method, device and equipment based on multi-modal interaction and medium

Abstract

The invention relates to the field, in particular to an emotion recognition method and device based on multi-modal interaction, equipment and a medium. According to the method, each single-modal feature is subjected to embedded representation, and then interactive branches of every two modal pairs are constructed, so that fusion features are generated, and potential complementary relationships are mined. Specifically, through a task performance evaluation mechanism, each fusion branch is dynamically graded, and a leading branch and an auxiliary branch are divided. Under the guidance of the main branch, the auxiliary branch executes structure optimization, so that the overall representation capability is improved. According to the invention, a relation filtering module is introduced, and the auxiliary branches are guided to pay attention to more discriminative structural features by comparing similarity relation matrixes between samples between the dominant branches and the auxiliary branches. According to the guiding mechanism, the similarity relation matrix is used as a medium, and redundancy expression can be effectively removed while complementarity is reserved. A weight adaptive mechanism is also introduced, and the guiding strength is dynamically adjusted according to sample characteristics, so that the guiding strategy is more flexible and generalization.

Description

Technical Field

[0001] This invention relates to the field of emotion recognition, specifically to a method, apparatus, device, and medium based on multimodal interaction. Background Technology

[0002] Emotion recognition is achieved by analyzing multimodal data, including text, audio, and video data, to identify whether a user is depressed or anxious. Current technologies ignore the differences in how each modality represents emotion across various application scenarios. In other words, when training emotion recognition models using data from all modalities, they treat all modalities equally, thus weakening the contribution of more discriminative modalities and ultimately reducing the accuracy of emotion recognition.

[0003] In summary, existing technologies have reduced the accuracy of emotion recognition.

[0004] Therefore, existing technologies still need to be improved and enhanced. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides an emotion recognition method, apparatus, device, and medium based on multimodal interaction, which solves the problem of reduced emotion recognition accuracy in existing technologies.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides an emotion recognition method based on multimodal interaction, which uses a trained recognition model to achieve emotion recognition, wherein the training method of the recognition model includes:

[0008] Acquire multimodal sample data generated by users in application scenarios and users' real emotional tags in the application scenarios, and combine the multimodal sample data in pairs to obtain various modality pairs;

[0009] Based on the combination of each modality pair, virtual sentiment tags are obtained;

[0010] Determine the error between the real emotion tag and the virtual emotion tag corresponding to each modality pair combination, and classify each modality pair combination into a dominant branch and an auxiliary branch based on the error corresponding to each modality pair combination;

[0011] Based on the dominant branch, data unrelated to emotion contained in the auxiliary branch is filtered to obtain filtered data. The total loss function required to train the recognition model is constructed based on the filtered data so that the trained recognition model is suitable for the application scenario.

[0012] In one implementation, the recognition model includes a modal encoder, a cross-modal interaction network, and a sentiment prediction module; based on each modal pair combination, a virtual sentiment label is obtained, including:

[0013] The modal encoder is applied to the two modal sample data within each modal pair combination to obtain the features of the two modal sample data.

[0014] The cross-modal interaction network is applied to the two features of each modality pair combination to fuse the two features, resulting in a fused feature for each modality pair combination.

[0015] A multilayer perceptron network with the same network structure as the sentiment prediction module is obtained, and the multilayer perceptron network is applied to the fusion features of each modality pair combination to obtain a virtual sentiment label corresponding to each modality pair combination.

[0016] In one implementation, based on the error corresponding to each mode pair combination, the various mode pair combinations are classified into dominant branches and auxiliary branches, including:

[0017] Determine the first error during the first iteration of training the recognition model and the previous error during the last iteration of training the recognition model from the errors.

[0018] Based on the initial error and the previous error, the current error is obtained;

[0019] Based on the current error corresponding to each mode pair combination, the mode pair combination corresponding to the minimum current error is taken as the dominant branch, and the mode pair combinations other than the dominant branch are taken as the auxiliary branches.

[0020] In one implementation, based on the dominant branch, data unrelated to sentiment contained in the auxiliary branches is filtered to obtain the filtered data, including:

[0021] When the error corresponding to the dominant branch is less than a set multiple of the error corresponding to the auxiliary branch, a sample relationship matrix of the dominant branch is constructed based on the fusion characteristics of the mode pair combination corresponding to the dominant branch, denoted as the dominant relationship matrix; and a sample relationship matrix of the auxiliary branch is constructed based on the fusion characteristics of the mode pair combination corresponding to the auxiliary branch, denoted as the auxiliary relationship matrix.

[0022] Tensor separation is performed on the dominant relation matrix. Based on the tensor-separated dominant relation matrix, data unrelated to sentiment contained in the auxiliary relation matrix of the auxiliary branches is filtered to obtain the filtered data.

[0023] In one implementation, a total loss function is constructed based on the filtered data to train the recognition model, so that the trained recognition model is suitable for the application scenario, including:

[0024] Determine the weights of the filtered data, and obtain the filtering loss function based on the weights and the filtered data;

[0025] Determine the multimodal contrast loss function of the modal encoder;

[0026] The features of all the modal sample data are fused to obtain the first fused feature;

[0027] The fusion features of all the modal pairs are fused to obtain the second fusion feature;

[0028] The sentiment prediction module is applied to the first fusion feature and the second fusion feature to obtain the predicted sentiment label;

[0029] Based on the predicted sentiment label and the real sentiment label, a prediction loss function is obtained;

[0030] Based on the filtering loss function, the multimodal contrast loss function, and the prediction loss function, the total loss function is obtained.

[0031] In one implementation, the multimodal sample data includes text data, audio data, and video data, and the modal encoder includes BERT and Transformer, wherein BERT is used to encode the text data, and Transformer is used to encode the audio data and the video data.

[0032] In one implementation, the recognition model includes a modal encoder, a cross-modal interaction network, and a sentiment prediction module. Sentiment recognition is achieved using a trained recognition model, including:

[0033] Acquire real-time multimodal data generated by the user in the application scenario;

[0034] The modality encoder is applied to the multimodal real-time data to obtain multimodal real-time features;

[0035] The multimodal real-time features are fused to obtain the first real-time fused feature;

[0036] The multimodal real-time features are combined in pairs to obtain various real-time feature pairs;

[0037] The cross-modal interaction network is applied to each of the aforementioned real-time feature pairs to obtain the second real-time fused feature;

[0038] The emotion prediction module predicts the user's real-time emotion based on the first real-time fusion feature and the second real-time fusion feature.

[0039] Secondly, embodiments of the present invention also provide an emotion recognition device based on multimodal interaction, wherein the device comprises the following components:

[0040] The modality combination module is used to acquire multimodal sample data generated by users in application scenarios and users' real emotional tags in the application scenarios, and to combine the multimodal sample data in pairs to obtain various modality pairs.

[0041] A virtual label prediction module is used to obtain virtual sentiment labels based on each of the modality pairs.

[0042] An error calculation module is used to determine the error between the real emotion tag and the virtual emotion tag corresponding to each modality pair combination, and based on the error corresponding to each modality pair combination, classify each modality pair combination into a dominant branch and an auxiliary branch.

[0043] The loss function construction module is used to filter the emotion-irrelevant data contained in the auxiliary branch based on the dominant branch to obtain the filtered data, and to construct the total loss function required to train the recognition model based on the filtered data, so that the trained recognition model is suitable for the application scenario.

[0044] Thirdly, embodiments of the present invention also provide a terminal device, wherein the terminal device includes a memory, a processor, and an emotion recognition program based on multimodal interaction stored in the memory and executable on the processor, wherein when the processor executes the emotion recognition program based on multimodal interaction, it implements the steps of the emotion recognition method based on multimodal interaction described above.

[0045] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing an emotion recognition program based on multimodal interaction, wherein when the emotion recognition program based on multimodal interaction is executed by a processor, the steps of the emotion recognition method based on multimodal interaction described above are implemented.

[0046] Beneficial Effects: This invention categorizes modality pairs into dominant and auxiliary branches based on the error between the virtual and real sentiment labels corresponding to each modality pair generated by the user in an application scenario. Then, using the dominant branch as a benchmark, it filters out sentiment-irrelevant data from the auxiliary branches to obtain the filtered data. Based on this filtered data, it constructs the total loss function required to train the recognition model. Finally, it uses the trained recognition model to identify the user's sentiment in the application scenario. From the above analysis, it can be seen that this invention uses virtual sentiment labels to evaluate the performance differences of different modality pairs in application scenarios, and then categorizes the modality pairs according to these performance differences. During training, high-performing modality pairs can guide the recognition model to filter out sentiment-irrelevant data contained in poorly performing modalities. This allows the trained recognition model to filter out sentiment-irrelevant data and focus on sentiment-related data, thereby improving the accuracy of the recognition model. Attached Figure Description

[0047] Figure 1 This is an overall flowchart of the present invention;

[0048] Figure 2 This is a flowchart of a multimodal emotion recognition based on hierarchical guidance and complementary enhancement in an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of the relationship filtering module and the sample filtering weight self-adjustment mechanism in an embodiment of the present invention;

[0050] Figure 4 A structural diagram of the emotion recognition device based on multimodal interaction provided by the present invention;

[0051] Figure 5 This is a block diagram illustrating the internal structure of a terminal device provided in an embodiment of the present invention. Detailed Implementation

[0052] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments and accompanying drawings. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0053] Research has found that identifying user emotions—specifically, whether a user is depressed or anxious—through multimodal data such as text, audio, and video data is crucial. Current technologies neglect the differences in how each modality represents emotion across various application scenarios. In other words, when training emotion recognition models using data from all modalities, they treat all modalities equally, thus weakening the contribution of stronger modalities and ultimately reducing the accuracy of emotion recognition.

[0054] To address the aforementioned technical problems, this invention provides an emotion recognition method, apparatus, device, and medium based on multimodal interaction, which solves the problem of reduced emotion recognition accuracy in existing technologies.

[0055] The emotion recognition method based on multimodal interaction in this embodiment can be applied to terminal devices, which can be terminal products with data processing capabilities, such as computers, etc., where modal interaction refers to modal fusion. In this embodiment, as... Figure 1 As shown, the emotion recognition method based on multimodal interaction specifically includes the following steps:

[0056] S100: Obtain multimodal sample data generated by the user in the application scenario and the user's real emotional tags in the application scenario; combine the multimodal sample data in pairs to obtain various modality pairs.

[0057] S200, based on the combination of each modality pair, a virtual sentiment label is obtained;

[0058] S300, determine the error between the real emotion tag and the virtual emotion tag corresponding to each modality pair combination, and classify each modality pair combination into a dominant branch and an auxiliary branch based on the error corresponding to each modality pair combination;

[0059] S400, using the dominant branch as a benchmark, filter out the emotion-irrelevant data contained in the auxiliary branch to obtain the filtered data, and construct the total loss function required to train the recognition model based on the filtered data, so that the trained recognition model is suitable for the application scenario.

[0060] When the emotion recognition method based on steps S100 to S400 is applied to the social media analysis application scenario, since text modal data can provide stable and reliable language information, and although facial expressions in video modal data contain a lot of redundant information due to interference from changes in lighting, the micro-expression changes contained therein are crucial for emotion recognition. Therefore, the branch containing text modal data is the dominant branch, and the branch containing video modal data is the auxiliary branch. The recognition model is trained to learn to filter out the redundant information in video modal data from text modal data and retain micro-expression data, so as to learn to supplement text modal data with micro-expression data in video modal data. When the recognition model is trained using the above training method and applied to the social media analysis application scenario, it can identify micro-expression data in video modal data to supplement text modal data, thereby improving the emotion recognition accuracy of the recognition model.

[0061] Since different application scenarios have different modal data that can serve as the dominant branch, steps S100, S200, S300, and S400 are executed for each application scenario to train a recognition model that is suitable for that application scenario.

[0062] The multimodal sample data in step S100 consists of multiple modal data of the user, such as text data, audio data, and video data (these three modalities are as follows). Figure 2 (As shown). The pairwise combinations in step S100 are the pairwise combinations of text data, audio data, and video data.

[0063] like Figure 2 As shown, the recognition model includes a modal encoder, a cross-modal interaction network (CMI, which is prior art), and a sentiment prediction module (which is prior art). Step S200, which combines each modal pair to obtain a virtual sentiment label, includes the following specific steps S201, S202, and S203:

[0064] S201, apply the modal encoder to the two modal sample data within each modal pair combination to obtain the features of the two modal sample data.

[0065] Use X i H′ represents the data of the i-th modality sample. i Let N represent the feature of the i-th modal sample data, and let N represent the total number of modal sample data. The text modal data is encoded using a BERT encoder to obtain the features of the text modal data. The audio modal data and video modal data are encoded using a Transformer-based encoder to obtain the features of these two modalities respectively.

[0066] S202, apply the cross-modal interaction network to the two features of each modality pair combination to fuse the two features and obtain the fused feature of each modality pair combination.

[0067] Use H k The fusion feature of the combination of the k-th mode pair is represented as follows:

[0068]

[0069] In the formula, H m H represents the feature of one modality sample data in the combination of the k-th modality pairs. n CMI represents the feature of another modality sample data of the combination of the k-th modality pair. kThe cross-modal interaction network is set up for the k-th modal pair combination. Each modal pair combination has its own independent cross-modal interaction network. That is, there is as many cross-modal interaction networks as there are modal pair combinations. Each cross-modal interaction network processes each modal pair combination in parallel to improve the processing speed. The cross-modal interaction network consists of Proj(·), tanh, and Conv. Proj(·) is a projection network, tanh represents nonlinear activation, and Conv is a convolution-based fusion network.

[0070] S203, Obtain a multilayer perceptron network with the same network structure as the sentiment prediction module, apply the multilayer perceptron network to the fusion features of each modality pair combination to obtain a virtual sentiment tag corresponding to each modality pair combination.

[0071] In essence, the fused features of a modality pair are input into a multilayer perceptron network (MLB). This MLB then uses these features, based on the user's two modal sample data, to deduce the user's possible sentiment label, which is the virtual sentiment label. For example, if one modality pair consists of the user's historical text and audio data, the fused features are the combined features of the historical text and audio data. The MLB then uses these combined features to predict the user's possible sentiment label, which is the virtual sentiment label. If the virtual sentiment label matches the user's actual sentiment label, it indicates that the historical text and audio data provide more useful information for sentiment prediction. These two modality data can then guide other modality pair combinations to learn and leverage their advantages. Conversely, if the virtual sentiment label differs significantly from the user's actual sentiment label, it indicates that the fusion learning of the historical text and audio data at this stage is insufficient, and their potential effective information has not yet been effectively extracted. In this case, the historical text and audio data need to be guided by other modality pair combinations for more efficient data mining.

[0072] The aforementioned multilayer perceptron network has the same structure as the sentiment prediction module in the recognition model. This multilayer perceptron network is a trained network, which is used to assist in training the recognition model.

[0073] This embodiment uses This represents the virtual sentiment tag corresponding to the k-th modality pair combination:

[0074]

[0075] In the formula, MLP represents a multilayer perceptron network, and detach represents tensor separation.

[0076] In this embodiment, step S300, which classifies each mode pair combination into a dominant branch and an auxiliary branch based on the error corresponding to each mode pair combination, includes the following specific steps S301, S302, and S303:

[0077] S301, determine the first error during the first iteration of training the recognition model and the previous error during the previous iteration of training the recognition model in the error.

[0078] use This represents the initial error of the k-th mode pair combination during the first training iteration:

[0079]

[0080] In the formula, MAE represents the mean absolute error calculation result, and gt represents the true sentiment label.

[0081] use t represents the previous error of the k-th mode pair combination during the previous training iteration, and t represents the current iteration number.

[0082] S302, Based on the initial error and the previous error, obtain the current error.

[0083]

[0084] S303, based on the current error corresponding to each mode pair combination, the mode pair combination corresponding to the smallest current error is taken as the dominant branch, and the mode pair combinations other than the dominant branch are taken as the auxiliary branches.

[0085] Use A best Indicates the dominant branch during the current training iteration: arg represents the smallest The corresponding mode pair combination.

[0086] This embodiment, through a task-aware hierarchical mechanism (which divides the data into dominant and auxiliary branches), can effectively identify which modalities excel in providing discriminative representations and which modalities contain complementary cues to be explored. This provides precise guidance for subsequent complementary enhancement and redundancy elimination, laying the foundation for more efficient multimodal fusion.

[0087] In this embodiment, when the error corresponding to the dominant branch is less than a set multiple of the error corresponding to the auxiliary branch, step S400 is executed; otherwise, all branches participate in the training of the recognition model with equal status.

[0088] Let s represent the dominant branch A. bestThat is, A best Let s be the variable, t represent the auxiliary branch, and S represent the auxiliary branch. s S represents the error of the dominant branch in the current iteration. t This represents the error of the auxiliary branch in the current iteration. The error corresponding to the dominant branch is less than a set multiple of the error corresponding to the auxiliary branch, i.e., S. s <τ·S t τ is a hyperparameter, which means τ is a parameter less than 1.

[0089] Step S400, which uses the dominant branch as a reference to filter out data unrelated to sentiment contained in the auxiliary branches to obtain the filtered data, includes the following specific steps S401, S402, and S403:

[0090] S401, based on the fusion features of the mode pair combination corresponding to the dominant branch, construct the sample relationship matrix of the dominant branch, denoted as the dominant relationship matrix G. s :

[0091]

[0092] T represents the transpose of the matrix, H s The fusion feature representing the mode pair combination corresponding to the dominant branch s, i.e., H s For H k The fusion features of modal pair combinations that act as the dominant branch.

[0093] S402, based on the fusion features of the modality pair combination corresponding to the auxiliary branch, construct the sample relationship matrix of the auxiliary branch, denoted as the auxiliary relationship matrix G. t :

[0094]

[0095] H t The fusion feature representing the combination of mode pairs corresponding to the auxiliary branch t, ​​i.e., H t For H k The fusion features of modal pair combinations that serve as auxiliary branches.

[0096] S403, perform tensor separation on the dominant relationship matrix. Using the tensor-separated dominant relationship matrix as a benchmark, filter out sentiment-irrelevant data contained in the auxiliary relationship matrices of the auxiliary branches to obtain the filtered data D. s→t :

[0097]

[0098] In the formula, MSE represents mean squared error, and detach represents tensor separation operation. Tensor separation is a technical implementation method, which essentially filters auxiliary branches based on the dominant relation matrix.

[0099] Step S403 is in Figure 3 In the relational filtering module, the dominant branch guides the auxiliary filtering branches to obtain the filtered data D. s→t In other words, the relation filtering module aims to use the distribution of inter-sample relations in the dominant branch as a semantic filter to guide auxiliary branches to focus on task-related complementary signals, while eliminating redundant and noisy interactions. This module is implemented by referencing the cross-branch guidance characteristics in the distillation mechanism, introducing the idea into multimodal learning, and improving the guidance method by using soft alignment of sample distribution rather than hard alignment of features, thereby achieving the elimination of redundancy and the mining of complementary information in the auxiliary branches.

[0100] In this embodiment, step S400, which involves constructing the total loss function required to train the recognition model based on the filtered data, includes the following specific steps S404 to S4010:

[0101] S404, determine the weights of the filtered data, and obtain the filtering loss function based on the weights and the filtered data.

[0102] Use W st The weights of the filtered data are:

[0103] softmax represents normalization, which ensures matrix scaling consistency. s and f t H respectively s and H t The characteristic projection function, Indicates feature concatenation, W st The weights correspond to the samples. These weights incorporate the local feature context of the modality pair and self-regulate the guidance strength, thereby automatically achieving the optimal solution at the guidance parameter optimization level.

[0104] Use L rf Representative filtering loss function:

[0105] N represents the total number of samples, which is also the total number of users. Each user's multimodal sample data can generate a D. s→t , This refers to the filtering loss function generated from the multimodal sample data of the l-th user. In the same round of training iterations, the multimodal sample data of all users are successively input into the recognition model, ensuring that each user has a filtering loss function. The total number of samples can also be the total number of multimodal sample data generated by the same user in different historical periods, where multimodal sample data from the same historical period constitute one sample.

[0106] This loss function, trained end-to-end, drives the complementary branch to retain unique complementary cues while eliminating redundant information that differs too much from task cues, enhancing cross-modal complementarity. This allows the model to more flexibly adapt to the characteristics of different modalities and the complex interactions between them. The relationship filtering module further introduces a dynamic weight adaptation mechanism to ensure that the dominant modality's enhancement of the complementary modality is efficient and targeted, avoiding unnecessary noise introduction and maximizing the cross-modal complementarity potential. This provides cleaner and more discriminative fusion features for classification or regression tasks.

[0107] S405, Determine the multimodal contrast loss function L of the modal encoder. con :

[0108] This embodiment uses the similarity between sample data from different modalities to characterize the multimodal contrast loss function L. con .

[0109] Taking text modal data (t represents text), audio modal data (a represents audio), and video modal data (v represents video) as examples, the multimodal contrastive loss function L is given. con The calculation formula is as follows:

[0110]

[0111] in, h represents the modal sample data, which is the X mentioned above. i .

[0112] Among them, f u (·)(u∈{m,n}) is a linear projection, H′ u (u∈{m,n}) are the features projected onto the contrastive learning space. It is the definition of the similarity between mode m of the i-th sample and mode n of the j-th sample, that is, the (i,j) element of the similarity matrix.

[0113] The multimodal contrast loss function L in this embodiment con The aim is to maximize the similarity between different modalities of the same sample. Simultaneously minimize the similarity between different samples This improves the model's discriminative and generalization abilities in multimodal semantic representation.

[0114] S406, the features of all the modal sample data are fused to obtain the first fused feature.

[0115] The feature H′ of the modal sample data in step S201 i The fusion process yields the first fusion feature: g1 represents a converged network.

[0116] S407, fuse the fusion features of all the said modal pair combinations to obtain the second fusion feature.

[0117] The fusion feature H of all mode pairs combined in S202 k By performing fusion, a second fusion feature is obtained: g2 is also a converged network.

[0118] S408, Apply the sentiment prediction module to the first fusion feature and the second fusion feature to obtain the predicted sentiment label.

[0119]

[0120] In the formula, Fusion represents a unified fusion network, and MLP represents a unified fusion network. Pred The multilayer perceptron network represents the sentiment prediction module (represented by Pred for prediction).

[0121] S409, Based on the predicted sentiment label and the real sentiment label, obtain the prediction loss function L. emo :

[0122]

[0123] In the formula, N represents the total number of samples mentioned above. The representative recognition model outputs the predicted sentiment label based on the i-th sample in N, gt i The true sentiment label represents the i-th sample.

[0124] S4010, based on the filtering loss function, the multimodal contrast loss function, and the prediction loss function, the total loss function L is obtained. total :

[0125] L total =L emo +w1·L con +w2·L rf

[0126] Here, w1 and w2 represent the weight factors of the contrastive learning loss and the relation filtering loss, respectively. As hyperparameters, they can be flexibly adjusted according to specific application scenarios to achieve the optimal balance between the loss terms.

[0127] The training of the recognition model can be completed through the above steps S100, S200, S300 and S400. Since the single-modal features and intermodal correlation information are effectively integrated during the training process, the recognition model’s ability to comprehensively express and discriminate multimodal emotional information is significantly improved.

[0128] Another embodiment, based on the above embodiments, provides a sentiment prediction method, including S501 to S506:

[0129] S501, acquire the multimodal real-time data generated by the user in the application scenario.

[0130] Multimodal real-time data includes real-time text data, audio data, and video data.

[0131] S502, apply the modal encoder to the multimodal real-time data respectively to obtain multimodal real-time features.

[0132] S503, the multimodal real-time features are fused to obtain the first real-time fused feature.

[0133] The same fusion method as in step S406 is used to fuse the real-time features of the multimodal systems.

[0134] S504, the multimodal real-time features are combined in pairs to obtain each real-time feature pair.

[0135] S505, apply the cross-modal interaction network to each of the real-time feature pairs to obtain the second real-time fused features.

[0136] That is, each real-time feature pair is input into its respective cross-modal interaction network (CMI) to obtain several real-time fused feature pairs. Then, the same fusion method as in step S407 is used to perform secondary fusion on the several real-time fused feature pairs to obtain the second real-time fused feature.

[0137] S506, the emotion prediction module predicts the user's real-time emotion based on the first real-time fusion feature and the second real-time fusion feature.

[0138] In summary, this invention draws upon the concept of knowledge distillation and applies it to multimodal learning scenarios. It designs a task-aware modality hierarchical mechanism to dynamically identify the dominant modality branch that performs best in the task, guiding the feature learning process of auxiliary branches. Building upon this, the invention further proposes a relation filtering module to remove redundant information and extract complementary information. Combined with a weight adaptive mechanism, it dynamically adjusts the guidance intensity and gradually integrates the original features and enhanced information through a progressive fusion strategy. This significantly improves the model's robustness, generalization ability, and modeling accuracy in emotion recognition tasks.

[0139] This embodiment also provides an emotion recognition device based on multimodal interaction, such as... Figure 4 As shown, the device comprises the following components:

[0140] Modality combination module 01 is used to acquire multimodal sample data generated by users in application scenarios and users' real emotional tags in the application scenarios, and to combine the multimodal sample data in pairs to obtain various modality pairs.

[0141] Virtual label prediction module 02 is used to obtain virtual sentiment labels based on each of the modality pairs;

[0142] Error calculation module 03 is used to determine the error between the real emotion tag and the virtual emotion tag corresponding to each modality pair combination, and based on the error corresponding to each modality pair combination, classify each modality pair combination into a dominant branch and an auxiliary branch.

[0143] The loss function construction module 04 is used to filter the emotion-irrelevant data contained in the auxiliary branch based on the dominant branch to obtain the filtered data, and to construct the total loss function required to train the recognition model based on the filtered data, so that the trained recognition model is suitable for the application scenario.

[0144] Based on the above embodiments, the present invention also provides a terminal device, the principle block diagram of which can be as follows: Figure 5 As shown, the terminal device includes a processor, memory, network interface, and display screen connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an emotion recognition method based on multimodal interaction. The display screen of the terminal device can be a liquid crystal display (LCD) or an e-ink display.

[0145] Those skilled in the art will understand that Figure 5 The schematic diagram shown is only a partial structural diagram related to the present invention and does not constitute a limitation on the terminal device to which the present invention is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0146] In one embodiment, a terminal device is provided, comprising a memory, a processor, and an emotion recognition program based on multimodal interaction stored in the memory and executable on the processor. When the processor executes the emotion recognition program based on multimodal interaction, it implements the following operation instructions:

[0147] Acquire multimodal sample data generated by users in application scenarios and users' real emotional tags in the application scenarios, and combine the multimodal sample data in pairs to obtain various modality pairs;

[0148] Based on the combination of each modality pair, virtual sentiment tags are obtained;

[0149] Determine the error between the real emotion tag and the virtual emotion tag corresponding to each modality pair combination, and classify each modality pair combination into a dominant branch and an auxiliary branch based on the error corresponding to each modality pair combination;

[0150] Based on the dominant branch, data unrelated to emotion contained in the auxiliary branch is filtered to obtain filtered data. The total loss function required to train the recognition model is constructed based on the filtered data so that the trained recognition model is suitable for the application scenario.

[0151] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A sentiment recognition method based on multimodal interaction, employing a trained recognition model to achieve sentiment recognition, characterized in that, The training methods for the recognition model include: Acquire multimodal sample data generated by users in application scenarios and users' real emotional tags in the application scenarios, and combine the multimodal sample data in pairs to obtain various modality pairs; Based on the combination of each modality pair, virtual sentiment tags are obtained; Determine the error between the real emotion tag and the virtual emotion tag corresponding to each modality pair combination, and classify each modality pair combination into a dominant branch and an auxiliary branch based on the error corresponding to each modality pair combination; Based on the dominant branch, data unrelated to emotion contained in the auxiliary branch is filtered to obtain filtered data. The total loss function required to train the recognition model is constructed based on the filtered data so that the trained recognition model is suitable for the application scenario.

2. The emotion recognition method based on multimodal interaction as described in claim 1, characterized in that, The recognition model includes a modal encoder, a cross-modal interaction network, and a sentiment prediction module; Based on the combination of each modality pair, virtual sentiment tags are obtained, including: The modal encoder is applied to the two modal sample data within each modal pair combination to obtain the features of the two modal sample data. The cross-modal interaction network is applied to the two features of each modality pair combination to fuse the two features, resulting in a fused feature for each modality pair combination. A multilayer perceptron network with the same network structure as the sentiment prediction module is obtained, and the multilayer perceptron network is applied to the fusion features of each modality pair combination to obtain a virtual sentiment label corresponding to each modality pair combination.

3. The emotion recognition method based on multimodal interaction as described in claim 1, characterized in that, Based on the error corresponding to each mode pair combination, each mode pair combination is classified into a dominant branch and an auxiliary branch, including: Determine the first error during the first iteration of training the recognition model and the previous error during the last iteration of training the recognition model from the errors. Based on the initial error and the previous error, the current error is obtained; Based on the current error corresponding to each mode pair combination, the mode pair combination corresponding to the minimum current error is taken as the dominant branch, and the mode pair combinations other than the dominant branch are taken as the auxiliary branches.

4. The emotion recognition method based on multimodal interaction as described in claim 2, characterized in that, Based on the dominant branch, data unrelated to sentiment contained in the auxiliary branches is filtered to obtain the filtered data, including: When the error corresponding to the dominant branch is less than a set multiple of the error corresponding to the auxiliary branch, a sample relationship matrix of the dominant branch is constructed based on the fusion characteristics of the mode pair combination corresponding to the dominant branch, denoted as the dominant relationship matrix; and a sample relationship matrix of the auxiliary branch is constructed based on the fusion characteristics of the mode pair combination corresponding to the auxiliary branch, denoted as the auxiliary relationship matrix. Tensor separation is performed on the dominant relation matrix. Based on the tensor-separated dominant relation matrix, data unrelated to sentiment contained in the auxiliary relation matrix of the auxiliary branches is filtered out to obtain the filtered data.

5. The emotion recognition method based on multimodal interaction as described in claim 2, characterized in that, The total loss function required to train the recognition model is constructed based on the filtered data, including: Determine the weights of the filtered data, and obtain the filtering loss function based on the weights and the filtered data; Determine the multimodal contrast loss function of the modal encoder; The features of all the modal sample data are fused to obtain the first fused feature; The fusion features of all the modal pairs are fused to obtain the second fusion feature; The sentiment prediction module is applied to the first fusion feature and the second fusion feature to obtain the predicted sentiment label; Based on the predicted sentiment label and the real sentiment label, a prediction loss function is obtained; Based on the filtering loss function, the multimodal contrast loss function, and the prediction loss function, the total loss function is obtained.

6. The emotion recognition method based on multimodal interaction as described in any one of claims 2-5, characterized in that, The multimodal sample data includes text data, audio data, and video data. The modal encoder includes BERT and Transformer. BERT is used to encode the text data, and Transformer is used to encode the audio data and the video data.

7. The emotion recognition method based on multimodal interaction as described in claim 1, characterized in that, The recognition model includes a modal encoder, a cross-modal interaction network, and a sentiment prediction module. It employs a trained recognition model to achieve sentiment recognition, including: Acquire real-time multimodal data generated by the user in the application scenario; The modal encoder is applied to the multimodal real-time data to obtain multimodal real-time features; The multimodal real-time features are fused to obtain the first real-time fused feature; The multimodal real-time features are combined in pairs to obtain various real-time feature pairs; The cross-modal interaction network is applied to each of the aforementioned real-time feature pairs to obtain the second real-time fused feature; The sentiment prediction module predicts the user's real-time sentiment based on the first real-time fusion feature and the second real-time fusion feature.

8. An emotion recognition device based on multimodal interaction, characterized in that, The device comprises the following components: The modality combination module is used to acquire multimodal sample data generated by users in application scenarios and users' real emotional tags in the application scenarios, and to combine the multimodal sample data in pairs to obtain various modality pairs. A virtual label prediction module is used to obtain virtual sentiment labels based on each of the modality pairs. An error calculation module is used to determine the error between the real emotion tag and the virtual emotion tag corresponding to each modality pair combination, and based on the error corresponding to each modality pair combination, classify each modality pair combination into a dominant branch and an auxiliary branch. The loss function construction module is used to filter the emotion-irrelevant data contained in the auxiliary branch based on the dominant branch to obtain the filtered data, and to construct the total loss function required to train the recognition model based on the filtered data, so that the trained recognition model is suitable for the application scenario.

9. A terminal device, characterized in that, The terminal device includes a memory, a processor, and an emotion recognition program based on multimodal interaction stored in the memory and executable on the processor. When the processor executes the emotion recognition program based on multimodal interaction, it implements the steps of the emotion recognition method based on multimodal interaction as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an emotion recognition program based on multimodal interaction, which, when executed by a processor, implements the steps of the emotion recognition method based on multimodal interaction as described in any one of claims 1-7.

Images