Emotion recognition method and device, computer readable storage medium and terminal device

Abstract

The application belongs to the technical field of artificial intelligence, and particularly relates to a sentiment recognition method and device, a computer readable storage medium and a terminal device. The method comprises the following steps: acquiring input information of each modality; extracting feature sequences of each modality from the input information of each modality; extracting context information of each modality from the feature sequences of each modality; performing fusion processing on the context information of each modality to obtain fused context information; calculating representation vectors of each modality according to the fused context information and the feature sequences of each modality; performing fusion processing on the representation vectors of each modality to obtain a fused representation vector; and processing the fused representation vector using a classifier to obtain a sentiment recognition result. In the application, the fused context information is obtained by fusing the context information of each modality, so that the information in other modalities can be fully utilized, and the accuracy of the sentiment recognition result is effectively improved.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to an emotion recognition method, device, computer-readable storage medium and terminal equipment. Background Technology

[0002] Human emotions in a natural state can elicit multimodal responses. Compared to methods that use only single-modal information for emotion recognition, methods that fuse multimodal information for emotion recognition have better performance. However, in existing multimodal emotion recognition methods, the calculation of the representation vectors of each modality is still independent, failing to fully utilize information from other modalities, resulting in lower accuracy of emotion recognition results. Summary of the Invention

[0003] In view of this, embodiments of this application provide an emotion recognition method, apparatus, computer-readable storage medium, and terminal device to solve the problem of low accuracy of recognition results in existing emotion recognition methods.

[0004] A first aspect of this application provides an emotion recognition method, which may include:

[0005] Obtain input information for each modality;

[0006] Extract the feature sequences of each modality from the input information of each modality;

[0007] Extract the context information of each modality from the feature sequences of each modality;

[0008] The context information of each modality is fused to obtain fused context information;

[0009] Using the fusion context information as a reference, the attention distribution in time and feature dimension is calculated for the feature sequences of each modality;

[0010] Calculate the representation vector for each modality based on the attention distribution;

[0011] The representation vectors of each modality are fused to obtain a fused representation vector;

[0012] The fused representation vector is processed using a preset classifier to obtain the emotion recognition result.

[0013] In one specific implementation of the first aspect, the fusion processing of the context information of each modality to obtain fused context information may include:

[0014] The context information of each modality is fused using a preset linear weighting model to obtain the fused context information.

[0015] In one specific implementation of the first aspect, the fusion processing of the context information of each modality to obtain fused context information may include:

[0016] The context information of each modality is fused using a preset attention model to obtain the fused context information.

[0017] In one specific implementation of the first aspect, the fusion processing of the representation vectors of each modality to obtain a fused representation vector may include:

[0018] The representation vectors of each modality are concatenated sequentially to obtain the fused representation vector;

[0019] or

[0020] The fused representation vector is obtained by sequentially performing inner or outer product calculations on the representation vectors of each modality.

[0021] In one specific implementation of the first aspect, processing the fused representation vector using a preset classifier to obtain the emotion recognition result may include:

[0022] The fused representation vector is input into the classifier, and the probability values ​​of each sentiment category output by the classifier are obtained.

[0023] The emotion category with the highest probability value is taken as the emotion recognition result.

[0024] In one specific implementation of the first aspect, the modalities may include a visual modal, a speech modal, and a text modal.

[0025] A second aspect of this application provides an emotion recognition device, which may include:

[0026] The input information acquisition module is used to acquire input information for each modality;

[0027] The feature sequence extraction module is used to extract the feature sequences of each modality from the input information of each modality.

[0028] The context information extraction module is used to extract the context information of each modality from the feature sequences of each modality.

[0029] The context information fusion module is used to fuse the context information of each modality to obtain fused context information;

[0030] The representation vector calculation module is used to calculate the attention distribution in time and feature dimension for the feature sequences of each modality, with reference to the fusion context information; and to calculate the representation vector of each modality according to the attention distribution.

[0031] The representation vector fusion module is used to fuse the representation vectors of various modalities to obtain a fused representation vector.

[0032] The emotion recognition module is used to process the fused representation vector using a preset classifier to obtain the emotion recognition result.

[0033] In one specific implementation of the second aspect, the context information fusion module may include:

[0034] The first context information fusion unit is used to fuse the context information of each modality using a preset linear weighting model to obtain the fused context information.

[0035] In one specific implementation of the second aspect, the context information fusion module may include:

[0036] The second context information fusion unit is used to fuse the context information of each modality using a preset attention model to obtain the fused context information.

[0037] In one specific implementation of the second aspect, the representation vector fusion module may include:

[0038] The first representation vector fusion unit is used to concatenate the representation vectors of each modality in sequence to obtain the fused representation vector;

[0039] or

[0040] The second representation vector fusion unit is used to calculate the inner or outer product of the representation vectors of each modality in sequence to obtain the fused representation vector.

[0041] In one specific implementation of the second aspect, the emotion recognition module may include:

[0042] The probability value calculation unit is used to input the fused representation vector into the classifier and obtain the probability values ​​of each sentiment category output by the classifier.

[0043] The emotion recognition result determination unit is used to select the emotion category with the highest probability value as the emotion recognition result.

[0044] In one specific implementation of the second aspect, the modalities may include a visual modal, a speech modal, and a text modal.

[0045] A third aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described emotion recognition methods.

[0046] A fourth aspect of this application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above-described emotion recognition methods.

[0047] The fifth aspect of this application provides a computer program product that, when run on a terminal device, causes the terminal device to execute the steps of any of the above-described emotion recognition methods.

[0048] The beneficial effects of this application embodiment compared with the prior art are as follows: This application embodiment acquires input information of each modality; extracts feature sequences of each modality from the input information of each modality; extracts context information of each modality from the feature sequences of each modality; fuses the context information of each modality to obtain fused context information; calculates the representation vector of each modality based on the fused context information and the feature sequences of each modality; fuses the representation vectors of each modality to obtain a fused representation vector; and processes the fused representation vector using a preset classifier to obtain the emotion recognition result. Through this application embodiment, in the process of calculating the representation vector of each modality, the fused context information obtained by fusing the context information of each modality is used, thereby making full use of information from other modalities and effectively improving the accuracy of emotion recognition results. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart of one embodiment of an emotion recognition method in this application.

[0051] Figure 2 This is an example diagram that uses multimodal fused context information instead of unimodal context information as a reference.

[0052] Figure 3 This is a schematic diagram of a traditional multimodal emotion recognition method;

[0053] Figure 4 This is a schematic diagram of a multimodal emotion recognition method based on cross-modal context in an embodiment of this application;

[0054] Figure 5 This is a structural diagram of one embodiment of an emotion recognition device according to the present application.

[0055] Figure 6 This is a schematic block diagram of a terminal device in an embodiment of this application. Detailed Implementation

[0056] To make the inventive objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0057] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0058] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0059] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0060] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0061] Furthermore, in the description of this application, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0062] In this embodiment, the context information of each modality can be fused to obtain fused context information. Then, the representation vector of each modality is calculated based on the fused context information, thereby making full use of the information in other modalities and effectively improving the accuracy of emotion recognition results.

[0063] The executing entity of this application embodiment can be any type of terminal device, including but not limited to mobile phones, tablet computers, desktop computers, laptops, handheld computers, and robots.

[0064] Please see Figure 1 One embodiment of an emotion recognition method in this application may include:

[0065] Step S101: Obtain the input information for each modality.

[0066] Each modality includes a visual modality, a speech modality, a text modality, and at least two other modalities.

[0067] For ease of description, this application embodiment uses three modal modules, including visual modality, speech modality and text modality, as examples for illustration. In practical applications, more or fewer modalities can be selected according to specific circumstances, and the specific modalities selected can also be flexibly set. This application embodiment does not make specific limitations on this.

[0068] The input information for the visual modality is video frames, the input information for the speech modality is speech frames, and the input information for the text modality is text data. The input information for each modality can be provided by the user, retrieved from a database, or generated from live recordings. The input information for each modality corresponds to the same user and the same time period. For example, to perform emotion recognition on user A during a specific speech, it is necessary to simultaneously acquire video frames of user A's face, speech frames of the speech, and text data of the speech content.

[0069] Step S102: Extract the feature sequences of each modality from the input information of each modality.

[0070] In one specific implementation of this application, the input information of the visual modality can be input into a pre-trained video feature extraction neural network to obtain a video feature sequence; the input information of the audio modality can be input into a pre-trained audio feature extraction neural network to obtain an audio feature sequence; and the input information of the text modality can be input into a pre-trained text feature extraction neural network to obtain a text feature sequence.

[0071] Since audio and video data both have a time dimension, while text data does not, audio and video feature sequences both possess a time dimension, while text feature sequences do not. Depending on the specific situation, text feature sequences can be aligned to the time dimension of audio data, or vice versa. Taking the alignment of text feature sequences to the time dimension of audio data as an example, we can find the text semantic feature data corresponding to the phonemes in the text feature sequence and align each text semantic feature data point in the text feature sequence with the time dimension of the phonemes.

[0072] Step S103: Extract the context information of each modality from the feature sequences of each modality.

[0073] In the embodiments of this application, any existing context information extraction method can be adopted according to the actual situation, including but not limited to models such as LSTM and Transformer, as well as techniques such as transfer learning. Specifically, these techniques can be used to extract video context information from video feature sequences, audio context information from audio feature sequences, and text context information from text feature sequences.

[0074] Step S104: Perform fusion processing on the context information of each modality to obtain fused context information.

[0075] In one specific implementation of this application, a preset linear weighted model can be used to fuse the context information of each modality. That is, each modality's context information is assigned a corresponding weight, and the context information of each modality is weighted and summed according to the weight to obtain the fused context information.

[0076] In another specific implementation of the embodiments of this application, a preset attention model or any other information fusion processing model in the prior art can be used to fuse the context information of each modality to obtain fused context information. The embodiments of this application do not specifically limit this.

[0077] Step S105: Calculate the representation vector of each modality based on the fusion context information and the feature sequences of each modality.

[0078] In traditional multimodal emotion recognition methods, a crucial step is to compute the attention of features distributed over an indeterminate time dimension, and then transform the time-dimension feature sequence into a fixed-length unimodal representation vector based on the attention distribution. This process utilizes the contextual information of the corresponding unimodal mode as a reference for attention computation. For example, the contextual information of the corresponding unimodal mode is used as a query to perform an inner product calculation with the feature sequence, and then the result is converted into attention weights using a softmax function.

[0079] A limitation of traditional multimodal emotion recognition methods is that, when calculating the representation of a single modality, they assume that its contextual information is extracted only from the input information of that modality. However, contextual information actually has cross-modal mapping characteristics. For example, when speech information is unclear, its speech modality performance can be estimated from text information. Based on cross-modal information, more comprehensive contextual features can be obtained. Figure 2 As shown, information from the speech modality alone can predict the current phoneme as "fulan." However, using an Automatic Speech Recognition (ASR) language model, the actual text is predicted to be "Hunan," and the phoneme mapped from the text modality to the speech modality is "hunan." This demonstrates that using both text and speech modalities allows for the acquisition of more contextual information than using only the speech modality.

[0080] Based on this principle, in this embodiment, more comprehensive and accurate contextual information can be extracted using multimodal information to guide the representation calculation of each unimodal modality. Specifically, the fused multimodal contextual information is used instead of the unimodal contextual information as a reference. Attention distributions in time and feature dimensions are calculated for the feature sequences of each modality, and representation vectors for each modality are calculated based on these attention distributions. Specifically, video representation vectors can be calculated based on fused contextual information and video feature sequences, audio representation vectors can be calculated based on fused contextual information and audio feature sequences, and text representation vectors can be calculated based on fused contextual information and text feature sequences.

[0081] Step S106: Perform fusion processing on the representation vectors of each modality to obtain the fused representation vector.

[0082] In one specific implementation of this application, the representation vectors of each modality can be concatenated sequentially to obtain a fused representation vector. In another specific implementation of this application, the representation vectors of each modality can be sequentially calculated using an inner product or an outer product to obtain a fused representation vector.

[0083] Step S107: Process the fused representation vector using a preset classifier to obtain the emotion recognition result.

[0084] In this embodiment, any classifier from the prior art can be used for emotion recognition, depending on the actual situation. The specific model structure of the classifier is not specifically limited here. After obtaining the fused standard vector, the fused representation vector can be input into the classifier, and the probability values ​​of each emotion category output by the classifier can be obtained. These emotion categories may include, but are not limited to, happiness, sadness, and anger. Finally, the emotion category with the highest probability value can be taken as the emotion recognition result.

[0085] Figure 3 The diagram shown illustrates a traditional multimodal emotion recognition method. Figure 4 The diagram shown is a schematic of the multimodal emotion recognition method based on cross-modal context in an embodiment of this application. By comparing the two, it can be seen that in the process of calculating the representation vector of each modality, the embodiment of this application uses the fused context information obtained by fusing the context information of each modality, thereby making full use of the information in other modalities and effectively improving the accuracy of emotion recognition results.

[0086] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0087] Corresponding to the emotion recognition method described in the above embodiments, Figure 5 This diagram illustrates a structural diagram of an embodiment of an emotion recognition device provided in this application.

[0088] In this embodiment, an emotion recognition device may include:

[0089] The input information acquisition module 501 is used to acquire input information for each modality;

[0090] The feature sequence extraction module 502 is used to extract the feature sequences of each modality from the input information of each modality.

[0091] The context information extraction module 503 is used to extract the context information of each modality from the feature sequences of each modality.

[0092] The context information fusion module 504 is used to fuse the context information of each modality to obtain fused context information;

[0093] The representation vector calculation module 505 is used to calculate the representation vector of each modality based on the fusion context information and the feature sequence of each modality.

[0094] The representation vector fusion module 506 is used to fuse the representation vectors of each modality to obtain a fused representation vector.

[0095] The emotion recognition module 507 is used to process the fused representation vector using a preset classifier to obtain the emotion recognition result.

[0096] In one specific implementation of this application embodiment, the context information fusion module may include:

[0097] The first context information fusion unit is used to fuse the context information of each modality using a preset linear weighting model to obtain the fused context information.

[0098] In one specific implementation of this application embodiment, the context information fusion module may include:

[0099] The second context information fusion unit is used to fuse the context information of each modality using a preset attention model to obtain the fused context information.

[0100] In one specific implementation of this application embodiment, the representation vector calculation module may include:

[0101] The attention distribution calculation unit is used to calculate the attention distribution in time and feature dimension for the feature sequences of each modality, with reference to the fusion context information.

[0102] The representation vector calculation unit is used to calculate the representation vector of each modality according to the attention distribution.

[0103] In one specific implementation of this application embodiment, the representation vector fusion module may include:

[0104] The first representation vector fusion unit is used to concatenate the representation vectors of each modality in sequence to obtain the fused representation vector;

[0105] or

[0106] The second representation vector fusion unit is used to calculate the inner or outer product of the representation vectors of each modality in sequence to obtain the fused representation vector.

[0107] In one specific implementation of this application embodiment, the emotion recognition module may include:

[0108] The probability value calculation unit is used to input the fused representation vector into the classifier and obtain the probability values ​​of each sentiment category output by the classifier.

[0109] The emotion recognition result determination unit is used to select the emotion category with the highest probability value as the emotion recognition result.

[0110] In one specific implementation of this application, the modalities may include a visual modal, a speech modal, and a text modal.

[0111] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0112] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0113] Figure 6 A schematic block diagram of a terminal device provided in an embodiment of this application is shown. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0114] like Figure 6 As shown, the terminal device 6 in this embodiment includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. When the processor 60 executes the computer program 62, it implements the steps in the various emotion recognition method embodiments described above, for example... Figure 1 Steps S101 to S107 are shown. Alternatively, when the processor 60 executes the computer program 62, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 5 The functions of modules 501 to 507 are shown.

[0115] For example, the computer program 62 may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 62 in the terminal device 6.

[0116] The terminal device 6 can be a computing device such as a mobile phone, tablet computer, desktop computer, laptop, handheld computer, or robot. Those skilled in the art will understand that... Figure 6 This is merely an example of terminal device 6 and does not constitute a limitation on terminal device 6. It may include more or fewer components than shown, or combine certain components, or different components. For example, terminal device 6 may also include input / output devices, network access devices, buses, etc.

[0117] The processor 60 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0118] The memory 61 can be an internal storage unit of the terminal device 6, such as a hard drive or memory of the terminal device 6. The memory 61 can also be an external storage device of the terminal device 6, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the terminal device 6. Furthermore, the memory 61 can include both internal and external storage units of the terminal device 6. The memory 61 is used to store the computer program and other programs and data required by the terminal device 6. The memory 61 can also be used to temporarily store data that has been output or will be output.

[0119] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0120] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0121] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0122] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0123] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0124] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0125] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0126] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An emotion recognition method, characterized by, The method comprises the following steps: obtaining input information of each modality; extracting feature sequences of each modality from the input information of each modality respectively; extracting context information of each modality from the feature sequences of each modality respectively; performing fusion processing on the context information of each modality to obtain fused context information; calculating attention distribution on time and feature dimension of the feature sequences of each modality with reference to the fused context information respectively; calculating representation vectors of each modality according to the attention distribution respectively; performing fusion processing on the representation vectors of each modality to obtain a fused representation vector; processing the fused representation vector using a preset classifier to obtain a sentiment recognition result.

2. The emotion recognition method of claim 1, wherein, The fusion processing on the context information of each modality to obtain the fused context information comprises: performing fusion processing on the context information of each modality using a preset linear weighting model to obtain the fused context information.

3. The emotion recognition method of claim 1, wherein, The fusion processing on the context information of each modality to obtain the fused context information comprises: performing fusion processing on the context information of each modality using a preset attention model to obtain the fused context information.

4. The emotion recognition method of claim 1, wherein, The fusion processing on the representation vectors of each modality to obtain the fused representation vector comprises: sequentially splicing the representation vectors of each modality to obtain the fused representation vector; or sequentially performing inner product or outer product calculation on the representation vectors of each modality to obtain the fused representation vector. The processing of the fused representation vector using the preset classifier to obtain the sentiment recognition result comprises:

5. The emotion recognition method of claim 1, wherein, inputting the fused representation vector into the classifier and obtaining probability values of each sentiment category output by the classifier; taking the sentiment category with the maximum probability value as the sentiment recognition result. The modalities comprise a visual modality, a speech modality and a text modality.

6. The emotion recognition method of any one of claims 1 to 5, characterized in that, The method comprises the following steps:

7. An emotion recognition apparatus, characterized by, an input information obtaining module is configured to obtain input information of each modality; a feature sequence extracting module is configured to extract feature sequences of each modality from the input information of each modality respectively; a context information extracting module is configured to extract context information of each modality from the feature sequences of each modality respectively; a context information fusion module is configured to perform fusion processing on the context information of each modality to obtain fused context information; a representation vector calculating module is configured to calculate attention distribution on time and feature dimension of the feature sequences of each modality with reference to the fused context information respectively, and calculate representation vectors of each modality according to the attention distribution respectively; a representation vector fusion module is configured to perform fusion processing on the representation vectors of each modality to obtain a fused representation vector; a sentiment recognition module is configured to process the fused representation vector using a preset classifier to obtain a sentiment recognition result. The computer program is executed by the processor to implement the steps of the sentiment recognition method in any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The processor executes the computer program to implement the steps of the sentiment recognition method in any one of claims 1 to 6.

9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, ​

Images