A multi-modal emotion recognition method and device, and a storage medium

Abstract

The application discloses a kind of based on multi-granularity window mutual attention multi-modal sentiment recognition method, device and storage medium, wherein method includes: obtaining training sample and test sample;According to the sample obtained, training multi-modal model is carried out sentiment recognition task, and model is tested;Multi-modal sample in actual environment is input into the model after training, and the predicted emotional state is output;Wherein multi-modal model is carried out following processing to input sample data: feature extractor generates the initial feature vector of each mode according to sample data;Interaction module generates multi-granularity features according to initial feature vector and multi-granularity window mutual attention mechanism;Fusion module generates deep fusion features according to multi-granularity features;Classifier generates predicted emotional state according to deep fusion features.The application provides a flexible mutual attention window through multi-granularity window mutual attention mechanism, so that the performance of multi-modal sentiment recognition is improved, and can be widely used in emotional computing technology field.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of affective computing, and particularly relates to a multi-modal emotion recognition method and device based on multi-granularity window mutual attention and a storage medium. BACKGROUND

[0002] Affective computing is an emerging research field aiming to develop systems that can recognize, interpret, process and simulate human emotions. People study affective computing largely to simulate empathy so that machines can interpret the emotional state of humans, behave accordingly and respond appropriately to emotions. Since the 21st century, with the rapid development of computer technology, there has been an increasing demand for research on affective computing, and relevant research has accelerated year by year. As an essential part of intelligent human-computer interaction systems, emotion recognition has attracted great attention from academia and industry.

[0003] In general, according to the data modalities, emotion recognition systems can be divided into single-modal emotion recognition systems and multi-modal emotion recognition systems. Single-modal emotion recognition systems include one of the modalities such as text, speech, video, and electroencephalogram. Multi-modal emotion recognition systems contain more information, such as semantic relationships in text, prosody and intonation in sound, and human expressions in video. Multiple modalities can complement each other to help machines better understand emotions and then provide feedback.

[0004] In recent years, the Transformer method has become a new trend in multi-modal emotion recognition systems. The Transformer method can be divided into self-attention mechanism and mutual-attention mechanism in the multi-modal scene: the Query, Key and Value of the self-attention mechanism come from the same modality, emphasizing intra-modal interaction; the Query and Key, Value of the mutual-attention mechanism come from different modalities, emphasizing inter-modal interaction. In addition, according to the attention range, the Transformer method can be divided into global full attention and local window attention: full attention can achieve long-distance interaction, but has the problems of large computational complexity and noise introduction; window attention only performs attention calculation within the window, reducing computational redundancy.

[0005] The window mutual attention mechanism can better solve the inter-modal interaction problem in multi-modal sentiment recognition, but it still has the following shortcomings: (i) multi-modal data is time series misaligned, even if collected in the same time period, the text in the basic unit of words and the signals such as speech, video and electroencephalogram in the basic unit of frames are different, and the number of frames corresponding to a word is not fixed, which does not match the window mutual attention which requires equidistant windows. (ii) The sentiment is asynchronous between different modalities. The sentiment information of the text modality is more likely to be concentrated in some emotionally inclined words, while the sentiment information of other modalities may occur at other time points, which makes it difficult for the sentiment information between different modality windows to interact. SUMMARY

[0006] To at least partially solve one of the technical problems existing in the prior art, the purpose of the present application is to provide a multi-modal sentiment recognition method based on multi-granularity window mutual attention, a device and a storage medium.

[0007] The technical solution adopted by the present application is:

[0008] A multi-modal sentiment recognition method based on multi-granularity window mutual attention, comprising the following steps:

[0009] Obtain a multi-modal database, obtain training samples and test samples from the multi-modal database, and obtain the true sentiment state of each sample;

[0010] Train a multi-modal model Transformer according to the training samples to perform a sentiment recognition task;

[0011] Measure the performance of the multi-modal model according to the test samples, and obtain the best model;

[0012] Input the multi-modal samples in the actual environment into the obtained best model, and output the predicted sentiment state;

[0013] The multi-modal model comprises a feature extractor, an interaction module, a fusion module and a classifier; the multi-modal model processes the input samples as follows:

[0014] The feature extractor generates an initial feature vector for each modality according to the sample data;

[0015] The interaction module generates multi-granularity features according to the initial feature vectors and the multi-granularity window mutual attention mechanism;

[0016] The fusion module generates deep fusion features according to the multi-granularity features;

[0017] The classifier generates a predicted sentiment state according to the deep fusion features;

[0018] In the multi-modal model training process, a preset loss function and an optimizer are used to fit the predicted emotional state and the real emotional state to realize the training of the model.

[0019] Further, the multi-modal database contains two modal data in text modal, speech modal, video modal and electroencephalogram modal, and various modal data have time sequence and are collected by different devices in the same period of time.

[0020] The training sample and the test sample are obtained according to the multi-modal database, comprising:

[0021] The sentence-level data is cut from the multi-modal database, and the cut data is processed to obtain sample data.

[0022] Taking the text and speech modal as an example, each sample data is composed of the following parts:

[0023] (1) The speech waveform of a sentence U = {U1,... U m}, wherein m is the number of data points of the speech waveform;

[0024] (2) The text content of a sentence T = {T1,... T n}, wherein n is the number of words of the text content;

[0025] (3) The real emotional label L.

[0026] Further, the feature extractor generates an initial feature vector of each modal according to the sample data, comprising:

[0027] Obtaining a large-scale pre-training model of each modal; extracting pre-training features P U / T of each modal according to the sample data and the pre-training model;

[0028] The pre-training features of each modal are input into the respective single-modal model to calculate the corresponding initial feature vector , wherein F i U / T is a high-dimensional feature vector at frame level or word level.

[0029] Further, the single-modal model generates the initial feature vector through a multi-head self-attention mechanism:

[0030] Each head of the multi-head attention mechanism network contains Query, Key, Value corresponding W q , W k , W vThree matrices, pre-training features are multiplied by the three matrices one by one to obtain Q, K, and V matrices, then the Q matrix and the transposed K matrix are multiplied, and the self-attention relationship is obtained after the Softmax layer, and then multiplied by the V matrix to generate an attention vector, and finally through a linear layer to generate an initial feature vector:

[0031] Q=W q P, K=W k P, V=W v P

[0032]

[0033] F=f(Attention)

[0034] Wherein, d is the dimension of the hidden layer vector after linear transformation of W q , W k , W v .

[0035] Further, the interaction module generates multi-granularity features according to the initial feature vector and the multi-granularity window interaction mechanism, comprising:

[0036] Windowing the feature vectors of the two modalities, and ensuring that the number of windows of each modality is the same;

[0037] The two modalities are sequentially taken as the main modality X U / T and the secondary modality X T / U , the secondary modality X T / U is multi-granularized to provide interaction information for the main modality X U / T ;

[0038] Using the windowed main modality window to generate a matrix Q w , using the windowed and multi-granularized secondary modality to generate a matrix K w and a matrix V w , and performing window attention interaction between modalities:

[0039]

[0040]

[0041] In the formula, W q , W k , W v are respectively the matrices corresponding to Query, Key, and Value contained in each head of the multi-head attention mechanism network, and d is the dimension of the hidden layer vector after linear transformation of W q , W k , W v .

[0042] Further, the sub-modal X T / U After multi-granularity, the main modal X U / T Provide interaction information, including:

[0043] By non-overlapping windowing of the sub-modal, a plurality of sub-modal window data with n frames are generated, each window segment as the starting K w ={K w1 ,..., K wn} and V w ={V w1 ,..., V wn}; Each window is extended from both ends to both sides, and the length of the extension is determined by different granularities. N new windows of different lengths are generated from the original window. The original window is the shortest window, which is the most important, and the longer the window, the less important it is, and the larger the kernel used. Splice the windows after different granularity pooling to generate new Key and Value, and perform attention calculation with the main modal Query to generate multi-granularity features:

[0044]

[0045]

[0046]

[0047]

[0048] Further, the fusion module generates deep fusion features according to the multi-granularity features, including:

[0049] Splice the multi-granularity features U and T of the two modalities as a whole on the time axis; use a multi-layer multi-head self-attention mechanism network to integrate the whole feature, and perform intra-modal and inter-modal information fusion at the token level to generate deep fusion features:

[0050] Q=W q [U, T], K=W k [U, T], V=W v [U, T]

[0051] The deep features H U , H T of the two modalities are respectively pooled to generate feature vectors after the time axis, and the two feature vectors are spliced to obtain the final multi-modal feature vector.

[0052] Further, the classifier generates a predicted emotional state according to the deep fusion features, including:

[0053] The multi-modal feature vector is transformed by using a fully connected layer with nonlinear activation, and then is subjected to Softmax, and the predicted emotional state is obtained by taking the subscript of the maximum prediction probability:

[0054] H Concat =[TP(H U ), TP(H T )]

[0055] P=Softmax(W c H Concat +b c )

[0056]

[0057] Wherein, TP represents time axis pooling, H U and H T are deep features of two modalities, W c and b c represent parameters of an output linear layer, is a set of emotion categories corresponding to the data set, is a predicted emotion category obtained by taking the subscript of the maximum probability.

[0058] Further, the preset loss function is a cross-entropy function, and the expression is:

[0059]

[0060] In the formula, N is the number of training samples, is a set of emotion categories corresponding to the data set, P i and y i are a predicted emotional state and a real emotional state respectively.

[0061] Another technical solution adopted by the application is:

[0062] A multi-modal emotion recognition device based on a multi-granularity window attention, comprising:

[0063] At least one processor;

[0064] At least one memory for storing at least one program;

[0065] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0066] Another technical solution adopted by the application is:

[0067] A computer readable storage medium, wherein a processor executable program is stored, the processor executable program is used to execute the above method when executed by a processor.

[0068] The beneficial effects of the present application are: the present application provides a flexible mutual attention window through the multi-granularity window mutual attention mechanism, and uses the emotional information in the modalities to help its own emotional mining, thereby improving the performance of multi-modal emotion recognition. BRIEF DESCRIPTION OF DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the present application or the prior art. It should be understood that the drawings in the following introduction are only for the convenience of clearly describing part of the embodiments of the technical solutions of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the premise of the drawings.

[0070] Figure 1 is a step flow chart of a multi-modal emotion recognition method based on multi-granularity window mutual attention in an embodiment of the present application;

[0071] Figure 2 is a training flow chart of a multi-modal emotion recognition method based on multi-granularity window mutual attention in an embodiment of the present application;

[0072] Figure 3 is a model framework diagram of a multi-modal emotion recognition method based on multi-granularity window mutual attention in an embodiment of the present application. DETAILED DESCRIPTION

[0073] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar reference signs represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application. For the step numbers in the following embodiments, they are only set for the convenience of description and explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0074] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by up, down, front, back, left, right, etc. is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0075] In the description of the present application, the meaning of one or more is one or more, the meaning of multiple is two or more, greater than, less than, more than, etc. are understood as not including the number, above, below, within, etc. are understood as including the number. If it is described to the first, second, only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or implicitly indicating the order of the indicated technical features.

[0076] In the description of the present application, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be broadly understood, and those skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0077] Referring to Figure 1 , Figure 2 and Figure 3 , the embodiment provides a multi-modal emotion recognition method based on multi-granularity window mutual attention, which reasonably captures dynamic changing emotion information through multi-granularity window mutual attention, and enhances the stability of the traditional emotion recognition model. The method specifically includes the following steps:

[0078] S1, obtaining a multi-modal database, obtaining training samples and test samples according to the database, and obtaining the real emotion state of each sample.

[0079] Among them, the training sample and the test sample are obtained by cutting and parsing the multi-modal database, and specifically include:

[0080] The multi-modal database contains any two of modalities such as text, speech, video, and electroencephalogram, and each modality data has time sequence and is collected by different devices in the same period of time; The sentence-level data is cut from the database. Taking the text and speech modalities as an example, each sample data is composed of the following parts:

[0081] (1) The speech waveform of a sentence U={U1,...U m}, where m is the number of data points of the speech waveform;

[0082] (2) The text content of a sentence T={T1,...T n}, where n is the number of words of the text content;

[0083] (3) Real emotion label L.

[0084] In this example, the emotion dataset uses the IEMOCAP database, which records approximately 12 hours of conversations between 10 actors across five sessions, encompassing both text and speech modalities. Consistent with international practice, a 50-fold cross-validation evaluation method was employed, with data from each of the five sessions used as test samples, while data from the remaining four sessions served as training samples. The labeling system includes expressions such as "happy," "sad," "angry," and "neutral." After data preprocessing, the cleaned text, speech, and labels for the four commonly used labels are retained.

[0085] S2. Train the multimodal model Transformer for emotion recognition tasks based on the training samples; the feature extractor generates initial features for each modality based on the pre-trained model and the multi-head self-attention model; the interaction module generates multi-granularity features based on the multi-granularity window mutual attention model; the fusion module generates deep fusion features based on the self-attention model; the classifier generates predicted emotional states based on the nonlinear layer; the loss function and optimizer train the model by fitting the predicted emotional states and the real emotional states.

[0086] The following is an explanation using text and voice modes as examples.

[0087] First, we obtain large-scale pre-trained models WavLM and RoBERTa for speech and text modalities respectively, and extract pre-trained features P from training samples in turn. U / T ; The pre-trained features of each modality are then calculated through their respective single-modal models to obtain the corresponding initial feature vector Among them, F i U / T is a high-dimensional feature vector at the frame / word level. The unimodal model here is implemented based on a multi-head self-attention mechanism, where each head contains W q 、W k 、W v The three matrices, pre-trained features and these three matrices are multiplied one by one to obtain the Q, K, and V matrices. Then the Q matrix is ​​multiplied by the transposed K matrix, and the self-attention relationship is obtained through the Softmax layer. It is then multiplied by the V matrix to generate the attention vector, and finally the initial feature vector is generated through the linear layer:

[0088] Q=W q PK=W k PV=W v P

[0089]

[0090] F=f(Attention)

[0091] Where d is the time after W q 、Wk , W v The dimension of the hidden layer vector after linear transformation of three matrices.

[0092] The initial feature vector is obtained, and the interaction module generates multi-granularity features based on a multi-granularity window interaction attention mechanism. Specifically, the speech and text are taken as the main modality X U / T and the secondary modality X T / U respectively in turn, after windowing, each window of the secondary modality extends from both ends to both sides, and the length of the extension is determined by different granularities, generating N new windows of different lengths from the original window. The original window is the shortest window, which has the greatest importance, and the longer the window, the less important it is, and the larger the pooling kernel used; the windows after different granularity pooling are spliced to generate new Key and Value, and attention calculation is performed with the Query of the main modality to generate multi-granularity features, providing flexible and rich interaction information for the main modality.

[0093]

[0094]

[0095]

[0096]

[0097]

[0098] The multi-granularity features of the two modalities are spliced along the time axis, and the fusion module generates deep fusion features based on a multi-head self-attention network:

[0099] Q = W q [U, T] K = W k [U, T] V = W v [U, T]

[0100] The deep features H U and H T of the two modalities are respectively pooled to remove the time axis to generate feature vectors, and the two feature vectors are spliced to obtain the final multi-modal feature vector; the classifier uses a fully connected layer with nonlinear activation, and the multi-modal feature vector is transformed and passed through Softmax to obtain the predicted sentiment state by taking the subscript of the maximum prediction probability:

[0101] H Concat = [TP(H U ), TP(H T )]

[0102] P = Softmax(W c H Concat + b c )

[0103]

[0104] where TP denotes temporal-spatial pooling, W c , b c denote parameters of the output linear layer, is a set of emotion categories corresponding to the dataset, is a predicted emotion category obtained by taking the index of the maximum probability;

[0105] The loss function of the training stage is a cross-entropy function, where N is the number of training samples, P i , y i are the predicted emotion state and the true emotion state, respectively:

[0106]

[0107] S3, measure the performance of the model according to the test samples, and save the model with the best performance.

[0108] According to the test samples, the performance of the model is measured, and the model with the best performance is saved. Optionally, the measurement index is the average of the weighted accuracy and the unweighted accuracy, so as to prevent the prediction from being biased in the case of data imbalance, and the parameters and structure of the entire model with the best performance are saved.

[0109] S4, input the multi-modal samples in the actual environment into the obtained model with the best performance, and output the predicted emotion state.

[0110] The multi-modal samples in the open environment generate the predicted emotion state through the model with the best performance. In the actual open environment, the voice recorded by various microphone devices and the corresponding transcription text constitute new input samples, and the saved end-to-end model is used to predict the emotion state result and feedback.

[0111] The embodiment also provides a multi-modal emotion recognition device based on a multi-granularity window attention, comprising:

[0112] at least one processor;

[0113] at least one memory for storing at least one program;

[0114] When the at least one program is executed by the at least one processor, the at least one processor implements Figure 1 the method.

[0115] The multi-modal sentiment recognition device based on the multi-granularity window mutual attention according to the embodiment can execute the multi-modal sentiment recognition method based on the multi-granularity window mutual attention provided by the method embodiment of the application, can execute the implementation steps of any combination of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0116] The application further discloses a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to execute the method shown in the method embodiment of the application. Figure 1

[0117] The embodiment further provides a storage medium storing instructions or programs that can execute the multi-modal sentiment recognition method based on the multi-granularity window mutual attention provided by the method embodiment of the application. When the instructions or programs are executed, the implementation steps of any combination of the method embodiment can be executed, and the corresponding functions and beneficial effects of the method are possessed.

[0118] In some alternative embodiments, the functions / operations mentioned in the block diagram can not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, two blocks shown in succession can actually be executed substantially simultaneously or the blocks can sometimes be executed in reverse order. In addition, the embodiments presented and described in the flowcharts of the application are provided by way of example, with the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operations and logical flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of larger operations are independently executed.

[0119] In addition, although the application is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the described functions and / or features can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It can also be understood that a detailed discussion of the actual implementation of each module is unnecessary for an understanding of the application. Rather, given the properties, functions and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be within the routine skill of the engineer, given the present disclosure. Therefore, those skilled in the art can implement the application as set forth in the claims using ordinary skill and without undue experimentation. It can also be understood that the disclosed specific concepts are merely illustrative and are not intended to limit the scope of the application, which is determined by the full scope of the appended claims and their equivalents.​

[0120] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions of the present application can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0121] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logic functions, which can be specifically embodied in any computer readable medium for use by or in conjunction with an instruction execution system, device or apparatus, such as a computer-based system, a system including a processor or other system that can fetch and execute instructions from the instruction execution system, device or apparatus. For the purpose of the present specification, the "computer readable medium" can be any device that can contain, store, communicate, propagate or transport programs for use by or in conjunction with the instruction execution system, device or apparatus, or in conjunction with these instruction execution systems, devices or apparatus.

[0122] More specific examples (a non-exhaustive list) of the computer readable medium include the following: an electrical connection having one or more wires (electrical devices), a portable computer diskette (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer readable medium can even be paper or other suitable medium on which the program can be printed, because the program can be electronically obtained, for example, by optical scanning of the paper or other medium, followed by editing, interpreting or otherwise processing, if necessary, in other suitable ways, to obtain the program electronically, and then storing it in the computer memory.

[0123] It should be understood that various parts of the present application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following technologies, known in the art, or their combinations can be used: a discrete logic circuit having logic gates for implementing logic functions on data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array (PGA), a field programmable gate array (FPGA), or the like.

[0124] In the above description of the present specification, the description of the terms "one embodiment / one example", "another embodiment / another example", or "certain embodiments / certain examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiments or examples are included in at least one embodiment or example of the present application. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0125] Although the embodiments of the present application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made thereto without departing from the principles and spirit of the present application, the scope of which is defined by the claims and their equivalents.

[0126] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the above-described embodiments, and those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present application, and these equivalent modifications or substitutions are included in the scope defined by the claims of the present application.

Claims

1. A multimodal emotion recognition method based on multi-granularity window mutual attention, characterized in that: The following steps are involved: Obtain a multimodal database, obtain training samples and test samples based on the multimodal database, and obtain the true emotional state of each sample; Train a multimodal model for emotion recognition tasks based on training samples; Measure the performance of the multimodal model based on the test samples and obtain the best performing model; Input multimodal samples from the actual environment into the best performing model and output the predicted emotional state; The multimodal model includes a feature extractor, an interaction module, a fusion module, and a classifier; the multimodal model performs the following processing on the input sample data: The feature extractor generates the initial feature vector of each modality based on the sample data; The interaction module generates multi-granularity features based on the initial feature vector and the multi-granularity window mutual attention mechanism; The fusion module generates deep fusion features based on multi-granularity features; The classifier generates predicted emotional states based on the deep fusion features; The interaction module generates multi-granularity features based on the initial feature vector and the multi-granularity window mutual attention mechanism, including: Divide the feature vectors of the two modes into windows and ensure that the number of windows of each mode is the same; The two modes are respectively regarded as the main mode X U / T and submodal X T / U , submodal X T / U After multi-granularity, the main mode X U / T Provide interactive information; Use the main modal window after windowing to generate the matrix Q w , using the windowed and multi-granular submodal generation matrix K w and matrix V w , perform window attention interaction between modalities: Where W q 、W k 、W v Each head of the multi-head attention mechanism network contains the matrix corresponding to Query, Key, and Value, and d is the matrix of W q 、W k 、W v The dimensions of the hidden layer vector after the three matrices are linearly transformed.

2. A multimodal emotion recognition method based on multi-granularity window mutual attention according to claim 1, characterized in that: The multimodal database includes two modal data among text modality, voice modality, video modality, and EEG modality, and each modal data has time series characteristics; The obtaining of training samples and test samples according to the multimodal database includes: Sentence-level data is segmented from the multimodal database, and the segmented data is processed to obtain sample data.

3. The multimodal emotion recognition method based on multi-granularity window mutual attention according to claim 1 is characterized in that: The feature extractor generates an initial feature vector for each modality based on the sample data, including: Obtain the pre-trained model for each modality; extract the pre-trained features p of each modality based on the sample data and the pre-trained model U / T ; The pre-trained features of each modality are passed through their respective single-modal models to calculate the corresponding initial feature vector Among them, F i U / T It is a high-dimensional feature vector at the frame level or word level.

4. The multimodal emotion recognition method based on multi-granularity window mutual attention according to claim 3 is characterized in that: The unimodal model generates the initial feature vector through a multi-head self-attention mechanism: Each head of the multi-head attention mechanism network contains W corresponding to Query, Key, and Value q 、W k 、W v The three matrices, pre-trained features and these three matrices are multiplied one by one to obtain the Q, K, and V matrices. Then the Q matrix is ​​multiplied by the transposed K matrix, and the self-attention relationship is obtained through the Softmax layer. It is then multiplied by the V matrix to generate the attention vector, and finally the initial feature vector is generated through the linear layer: Q=W q P,K=W k P,V=W v P F=f(Attention) Where d is the time after W q 、W k 、W v The dimensions of the hidden layer vector after the linear transformation of the three matrices.

5. The multimodal emotion recognition method based on multi-granularity window mutual attention according to claim 1, characterized in that: The submodal X T / U After multi-granularity, the main mode X U / T Provides interaction information, including: By performing non-overlapping window division on the submodal data, multiple submodal window data with n frames are generated, and each window is used as the starting K w ={K w1 ,...,K wn } and V w ={V w1 ,...,V wn Each window is extended from both ends to both sides. The extension length is determined by different granularities. N new windows of different lengths are generated for the original window. The original window is the shortest window and has the greatest importance. The longer the window, the less important it is. The larger the pooling kernel is. The windows after pooling of different granularities are spliced ​​together to generate new keys and values. The attention calculation is performed with the query of the main mode to generate multi-granularity features:

6. The multimodal emotion recognition method based on multi-granularity window mutual attention according to claim 1, characterized in that: The fusion module generates deep fusion features based on multi-granularity features, including: The multi-granularity features U and T of the two modalities are spliced ​​together on the time axis as a whole. A multi-layer multi-head self-attention mechanism network is used to integrate the overall features, and information fusion within and between modalities is performed at the token level to generate deep fusion features: Q=W q [U,T],K=W k [U,T],V=W v [U,T] Deep features H of two modalities U 、H T After pooling the time axis separately, feature vectors are generated, and the two feature vectors are concatenated to obtain the final multimodal feature vector.

7. The multimodal emotion recognition method based on multi-granularity window mutual attention according to claim 1, characterized in that: The classifier generates a predicted emotional state based on the deep fusion features, including: Using a fully connected layer with nonlinear activation, the multimodal feature vector is transformed and passed through Softmax, and the predicted emotional state is obtained by taking the subscript of the maximum predicted probability: H Concat =[TP(H U ),TP(H T )] P=Softmax(W c H Concat +b c ) Among them, TP represents time axis pooling, H U and H T is the deep feature of the two modalities, W c 、b c represents the parameters of the output linear layer, is the set of emotion categories corresponding to the dataset, is the predicted emotion category obtained by taking the subscript with the maximum probability.

8. A multimodal emotion recognition device based on multi-granularity window mutual attention, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

9. A computer-readable storage medium storing a program executable by a processor, characterized in that: The processor-executable program is used to perform the method according to any one of claims 1 to 7 when executed by the processor.

Images