An emotion recognition method, device, equipment and storage medium

Abstract

The application provides an emotion recognition method, device and equipment and a storage medium, and belongs to the technical field of computers. The method comprises the following steps: acquiring multiple types of modal data; extracting feature information of various types of modal data respectively, and determining multiple types of to-be-fused types; performing feature fusion on the feature information of the modal data belonging to the to-be-fused types to obtain initial fused features; performing classification and recognition processing on the initial fused features based on a target classifier obtained through pre-training to obtain an initial recognition result; performing feature fusion and classification and recognition processing on the initial recognition result and the feature information of at least one type of modal data to obtain emotion recognition results of the multiple types of modal data. The application can improve the accuracy of the recognition result.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to an emotion recognition method, apparatus, device, and storage medium. Background Technology

[0002] Currently, numerous technological products such as sales robots and customer service robots are being rapidly adopted in various fields. However, intelligent interaction technology between robots and humans requires recognizing the user's current emotional state based on their external language, facial expressions, and gestures, in order to intelligently provide emotional feedback. Therefore, accurately identifying user emotions is a pressing issue that needs to be addressed.

[0003] In existing technologies, most emotion recognition methods employ feature-level fusion or decision-level fusion. Feature-level fusion refers to fusing features from multiple modalities before encoding, which can effectively extract interaction information between modalities. Decision-level fusion is fusion at the decision-making level after the classifier encoding process. Its main goal is to determine the credibility of each modality and then coordinate and jointly make decisions.

[0004] However, feature-level fusion tends to overlook intramodal interaction information during the recognition process, resulting in poor accuracy. Decision-level fusion does not consider the mutual influence between modalities, thus leading to poor reliability of the recognition results. Summary of the Invention

[0005] The purpose of this application is to provide an emotion recognition method, apparatus, device, and storage medium that can improve the accuracy of recognition results.

[0006] The embodiments of this application are implemented as follows:

[0007] One aspect of this application provides an emotion recognition method, the method comprising:

[0008] Acquire various types of modal data, including text, speech, and image data;

[0009] Feature information of various types of modal data is extracted, and multiple types to be fused are identified among these types.

[0010] Feature fusion is performed on the feature information of the modal data belonging to the type to be fused to obtain the initial fused features;

[0011] The initial fused features are classified and identified based on the pre-trained target classifier to obtain the initial identification result, where the target classifier is the classifier that matches the initial fused features.

[0012] The initial recognition results and the feature information of at least one type of modality data are subjected to feature fusion and classification recognition processing to obtain emotion recognition results of multiple types of modality data, wherein at least one type is a type among multiple types.

[0013] Optionally, feature information of various types of modal data can be extracted, including:

[0014] Various types of modal data are input into a pre-trained bidirectional long short-term memory network model to obtain contextual features of various types of modal data;

[0015] By inputting various types of modal data into a pre-trained self-attention mechanism network model, contextual semantic features of various types of modal data can be obtained.

[0016] Based on contextual features and contextual semantic features, feature information of various types of modal data is obtained.

[0017] Optionally, before inputting various types of modal data into a pre-trained bidirectional long short-term memory network model to obtain the contextual features of various types of modal data, the process further includes:

[0018] If the current type belongs to the speech type, then the Mel-frequency cepstral coefficient feature information is extracted from the modal data of the current type as the modal data of the current type;

[0019] If the current type is an image type, then extract the text data from the modal data of the current type as the modal data of the current type.

[0020] Optionally, the types to be merged include: text type and speech type;

[0021] Feature fusion is performed on the feature information of the modal data belonging to the type to be fused to obtain the initial fused features, including:

[0022] The feature information of text-type modal data and the feature information of speech-type modal data are fused to obtain the initial fused features.

[0023] Optionally, feature fusion and classification processing are performed on the initial recognition results and feature information of at least one type of modality data to obtain emotion recognition results for multiple types of modality data, including:

[0024] The initial recognition results are fused into the feature information of the first type of modality data to obtain the current fused features;

[0025] Based on the pre-trained current classifier, the current fused features are classified and identified to obtain the current identification result, where the current classifier is the classifier that matches the current fused features;

[0026] The current recognition result is used as the new initial recognition result, and the feature information of the second type of modality data is used as the feature information of the new first type of modality data. The step of fusing the initial recognition result into the feature information of the first type of modality data is executed again. This process is repeated until the current recognition result meets the preset conditions or the number of iterations reaches the preset number of iterations. Here, the second type is the type after the first type.

[0027] Based on the feature information of various types of modal data and the current recognition results, emotion recognition results for multiple types of modal data are obtained.

[0028] Optionally, based on the feature information of various types of modal data and the current recognition results, emotion recognition results for multiple types of modal data can be obtained, including:

[0029] Based on classifiers corresponding to various types, the feature information of various types of modal data is classified and identified to obtain the single-modal emotion recognition results of various types of modal data;

[0030] Based on the current recognition results and the single-modal emotion recognition results of various types of modal data, emotion recognition results for multiple types of modal data are obtained.

[0031] Optionally, based on the current recognition results and the unimodal emotion recognition results of various types of modal data, emotion recognition results for multiple types of modal data are obtained, including:

[0032] Based on the current recognition result, the weight of the current recognition result, the single-modal emotion recognition results of various types of modal data, and the weight of the single-modal emotion recognition results of various types of modal data, emotion recognition results of multiple types of modal data are obtained.

[0033] In another aspect of this application, an emotion recognition device is provided, which includes: an acquisition module, an extraction module, a fusion module, a training module, and a result module;

[0034] The acquisition module is used to acquire various types of modal data, including text, speech, and image data.

[0035] The extraction module is used to extract feature information from various types of modal data and determine the various types to be fused.

[0036] The fusion module is used to perform feature fusion on the feature information of modal data belonging to the type to be fused, and obtain the initial fused features;

[0037] The training module is used to classify and recognize the initial fused features based on the pre-trained target classifier to obtain the initial recognition result, wherein the target classifier is a classifier that matches the initial fused features;

[0038] The results module is used to perform feature fusion and classification recognition processing on the initial recognition results and feature information of at least one type of modality data to obtain emotion recognition results of multiple types of modality data, wherein at least one type is a type among multiple types.

[0039] Optionally, the extraction module is specifically used to input various types of modal data into a pre-trained bidirectional long short-term memory network model to obtain contextual features of various types of modal data; input various types of modal data into a pre-trained self-attention mechanism network model to obtain contextual semantic features of various types of modal data; and obtain feature information of various types of modal data based on contextual features and contextual semantic features.

[0040] Optionally, the extraction module is further configured to extract Mel-frequency cepstral coefficient feature information from the modal data of the current type as the modal data of the current type if the current type belongs to the speech type; and to extract text data from the modal data of the current type as the modal data of the current type if the current type belongs to the image type.

[0041] Optionally, the types to be fused include: text type and speech type; the fusion module is specifically used to perform feature fusion processing on the feature information of the text type modal data and the feature information of the speech type modal data to obtain the initial fused features.

[0042] Optionally, the result module is used to fuse the initial recognition result into the feature information of the first type of modality data to obtain the current fused feature; perform classification and recognition processing on the current fused feature based on the pre-trained current classifier to obtain the current recognition result, wherein the current classifier is a classifier that matches the current fused feature; take the current recognition result as the new initial recognition result, take the feature information of the second type of modality data as the new feature information of the first type of modality data, and re-execute the step of fusing the initial recognition result into the feature information of the first type of modality data, repeat the execution until the current recognition result meets the preset conditions or the number of iterations reaches the preset number of iterations, wherein the second type is the type after the first type; and obtain the emotion recognition results of multiple types of modality data based on the feature information of various types of modality data and the current recognition result.

[0043] Optionally, the results module is specifically used to classify and recognize the feature information of various types of modal data based on classifiers corresponding to various types, and obtain the single-modal emotion recognition results of various types of modal data; based on the current recognition results and the single-modal emotion recognition results of various types of modal data, the emotion recognition results of multiple types of modal data are obtained.

[0044] Optionally, the results module is specifically used to obtain the emotion recognition results of multiple types of modal data based on the current recognition result, the weight of the current recognition result, the single-modal emotion recognition results of various types of modal data, and the weight of the single-modal emotion recognition results of various types of modal data.

[0045] In another aspect of this application, a computer device is provided, including: a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the steps of an emotion recognition method.

[0046] In another aspect of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of an emotion recognition method.

[0047] The beneficial effects of the embodiments of this application include:

[0048] This application provides an emotion recognition method, apparatus, device, and storage medium that can acquire multiple types of modal data, extract feature information from each type of modal data, determine multiple types to be fused, and perform feature fusion on the feature information of the modal data belonging to the types to be fused to obtain initial fused features. Then, based on a pre-trained target classifier, the initial fused features are classified and recognized to obtain an initial recognition result. Further, feature fusion and classification recognition processing can be performed on the initial recognition result and the feature information of at least one type of modal data. Specifically, feature fusion of the feature information of the modal data to be fused enables early-stage fusion in the data extraction phase, allowing for more effective extraction of interaction information between modalities. Feature fusion of the initial recognition result and the feature information of at least one type of modal data enables late-stage model fusion in the decision-making phase, allowing for the identification of the credibility of each modality and subsequent coordination and joint decision-making. In other words, hybrid fusion can be achieved through the above methods, combining the advantages of feature-level fusion and decision-level fusion, resulting in more accurate and effective emotion recognition results. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 A flowchart illustrating the emotion recognition method provided in this application embodiment;

[0051] Figure 2 This is another schematic diagram of the emotion recognition method provided in the embodiments of this application;

[0052] Figure 3 This is another schematic diagram of the emotion recognition method provided in the embodiments of this application;

[0053] Figure 4 This is another schematic diagram of the emotion recognition method provided in the embodiments of this application;

[0054] Figure 5 This is another schematic diagram of the emotion recognition method provided in the embodiments of this application;

[0055] Figure 6 This is a schematic diagram of the structure of the emotion recognition device provided in the embodiments of this application;

[0056] Figure 7 A schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0058] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0059] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0060] In the description of this application, it should be noted that the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0061] It should be noted that the emotion recognition method provided in this application embodiment can be applied to robots in various industries, such as sales robots and customer service robots in the insurance industry. Specifically, it can be a humanoid robot configured with the above-mentioned program, or it can be a virtual robot set in a computer equipment system. No specific restrictions are made here. Any robot that can communicate with users is acceptable.

[0062] The following section will explain in detail the specific implementation process of the emotion recognition method provided in the embodiments of this application.

[0063] Figure 1 Please refer to the flowchart of the emotion recognition method provided in the embodiments of this application. Figure 1 The method includes:

[0064] S110: Acquire multiple types of modal data.

[0065] These include various types such as text, voice, and image.

[0066] Optionally, the execution subject of this method can be a computer device, where modal data can be user-input data. For example, in an application scenario, when the computer device is the corresponding customer service robot, it can be the user's words to the customer service robot, or text, images, and other information sent.

[0067] Among them, text type can be a type containing text information, voice type can be a type containing voice messages, and image type can be a type containing related image files; no specific restrictions are imposed here.

[0068] These data can be collected in real time or pre-collected and then processed before input, and can be configured according to the needs of the actual application.

[0069] It should be noted that, in order to process these modal data in a more uniform and efficient manner, preprocessing can also be performed on these modal data, such as data acquisition, data cleaning, data format normalization, and data denoising. One or more of the above preprocessing methods can be selected according to actual needs, and no specific restrictions are imposed here.

[0070] S120: Extract feature information from various types of modal data and determine the various types to be fused.

[0071] Optionally, after obtaining the above modal data, feature extraction can be performed on these modal data. Different extraction methods can be used for different types of modal data to obtain corresponding feature information. Furthermore, after the extraction is completed, multiple types to be fused can be determined among multiple types.

[0072] Among them, multiple modal data types to be fused can refer to modal data that need to be fused in terms of features, such as speech and text types.

[0073] The feature information can specifically be information used to represent the modality data. For example, for text types, it can be keywords, contextual logical relationships, etc., and for speech types, it can be the corresponding key paragraphs, etc. There are no specific restrictions here, and the specific type of feature information can be determined based on actual needs.

[0074] S130: Perform feature fusion on the feature information of the modal data belonging to the type to be fused to obtain the initial fused features.

[0075] Optionally, after determining the modal data of the type to be fused in the above manner, feature fusion can be performed on the feature information of the modal data belonging to the type to be fused to obtain the initial fused features. For example, feature fusion can be performed on the feature information of text type modal data and the feature information of speech type modal data to obtain the initial fused features.

[0076] Specifically, this feature fusion can be feature-level fusion, which is the fusion of two features directly.

[0077] S140: Based on the pre-trained target classifier, the initial fused features are classified and identified to obtain the initial identification result.

[0078] The target classifier is the classifier that matches the features after initial fusion.

[0079] Optionally, the target classifier can be a classifier that matches the initial fused features, that is, the target classifier can be used to perform classification and recognition processing on the initial fused features.

[0080] It should be noted that the target classifier can be pre-trained. For example, an initial classifier can be set first, and after relevant training based on the initial fused features, a classifier that matches the initial fused features can be obtained, which is the target classifier mentioned above.

[0081] S150: Perform feature fusion and classification recognition processing on the initial recognition results and feature information of at least one type of modality data to obtain emotion recognition results of multiple types of modality data.

[0082] Among them, at least one type is a type among multiple types.

[0083] Optionally, after obtaining the initial recognition result, feature fusion and classification recognition processing can be performed on the initial recognition result and the feature information of at least one type of modality data. The feature fusion can be decision-level fusion, that is, fusion at the decision level after the classifier encoding process.

[0084] Through the above fusion and further classification and recognition processing, emotion recognition results of various types of modal data can be obtained.

[0085] The obtained emotion recognition result can be specific to types such as anger, happiness, surprise, or neutral, and the corresponding result can be set according to actual needs.

[0086] This application provides an emotion recognition method that can acquire multiple types of modal data, extract feature information from each type of modal data, determine multiple types to be fused, and perform feature fusion on the feature information of the modal data belonging to the types to be fused to obtain initial fused features. Then, based on a pre-trained target classifier, the initial fused features are classified and recognized to obtain an initial recognition result. Further, feature fusion and classification and recognition processing can be performed on the initial recognition result and the feature information of at least one type of modal data. Specifically, feature fusion of the feature information of the modal data to be fused enables early-stage fusion in the data extraction stage, allowing for more effective extraction of interaction information between modalities. Feature fusion of the initial recognition result and the feature information of at least one type of modal data enables late-stage model fusion in the decision-making stage, allowing for the identification of the credibility of each modality and subsequent coordination and joint decision-making. In other words, this method achieves hybrid fusion, combining the advantages of feature-level fusion and decision-level fusion, resulting in more accurate and effective emotion recognition.

[0087] The following is a detailed explanation of another specific implementation process of the emotion recognition method provided in the embodiments of this application.

[0088] Figure 2 For another flowchart illustrating the emotion recognition method provided in this application embodiment, please refer to... Figure 2 Feature information of various types of modal data is extracted, including:

[0089] S210: Input various types of modal data into a pre-trained bidirectional long short-term memory network model to obtain contextual features of various types of modal data.

[0090] Optionally, the bidirectional long short-term memory network model (BiLSTM) can be pre-trained. By inputting various types of modal data into the model, the contextual features of various types of modal data can be obtained.

[0091] It should be noted that the input to this model usually requires text data. Text-type modal data can be input directly, while other types of modal data, such as speech and image data, can be converted into text data before input.

[0092] S220: Input various types of modal data into a pre-trained self-attention mechanism network model to obtain contextual semantic features of various types of modal data.

[0093] Optionally, the self-attention mechanism network model can be a model used to determine the corresponding semantics from the input text. By inputting various types of modal data into the self-attention mechanism network model, the contextual semantic features of various types of modal data can be obtained.

[0094] It should be noted that, similar to the bidirectional long short-term memory network model, the self-attention mechanism network model takes text data as input. For other types of modal data that are not text, they can be converted into text before being input into the model.

[0095] S230: Based on contextual features and contextual semantic features, obtain feature information of various types of modal data.

[0096] Optionally, S210-S220 can be performed separately or sequentially, without specific timing restrictions. The specific configuration can be based on actual needs. Figure 2 We will explain this by taking each example separately.

[0097] After obtaining the aforementioned contextual features and contextual semantic features, feature information for various types of modal data can be obtained.

[0098] Specifically, contextual features and contextual semantic features can be represented by vectors, and the concatenation of vectors can be used as feature information for various types of modal data.

[0099] The emotion recognition method provided in this application embodiment can input various types of modal data into a pre-trained bidirectional long short-term memory network model to obtain contextual features of various types of modal data; input various types of modal data into a pre-trained self-attention mechanism network model to obtain contextual semantic features of various types of modal data; and obtain feature information of various types of modal data based on contextual features and contextual semantic features. In this way, by determining contextual features and contextual semantic features through two models respectively, the feature information of modal data can be represented more accurately, that is, more accurate feature information can be obtained.

[0100] Optionally, before inputting various types of modal data into a pre-trained bidirectional long short-term memory network model to obtain the contextual features of various types of modal data, the method further includes: if the current type is speech, then extracting Mel-frequency cepstral coefficient feature information from the modal data of the current type as the modal data of the current type; if the current type is image, then extracting text data from the modal data of the current type as the modal data of the current type.

[0101] Optionally, the type of modal data can be determined before inputting it into the bidirectional long short-term memory network model. For text data, it can be input directly, while for speech and image data, corresponding conversion processing is required.

[0102] In the specific conversion process, for a speech type, the Mel Frequency Cepstrum Coefficient (MFCC) feature information can be extracted from the modal data as the modal data of the current type. After information extraction in this way, the text data corresponding to the speech type can be obtained, and then the obtained text data can be input into the bidirectional long short-term memory network model for feature extraction.

[0103] Correspondingly, for image types, text data can be extracted from modal data as the modal data of the current type. For example, text data can be extracted from an image through image recognition, and this text data can be used as the modal data of the image type. Then, it can be input into a bidirectional long short-term memory network model for feature extraction.

[0104] Optionally, the types to be fused include: text type and speech type; feature fusion is performed on the feature information of the modal data belonging to the types to be fused to obtain the initial fused features, including: performing feature fusion processing on the feature information of the text type modal data and the feature information of the speech type modal data to obtain the initial fused features.

[0105] Optionally, the feature fusion process typically focuses on speech and text types. Specifically, it can involve fusing text-type feature information into speech-type feature information. In other words, it can perform feature fusion processing on the feature information of text-type modal data and the feature information of speech-type modal data to obtain initial fused features. These fused features can be obtained by classifying and encoding the features of text-type modal data and then adding them to the speech-type modal data. This feature-level feature fusion can achieve multimodal fusion of deep semantic information of text and features of speech.

[0106] The following is a detailed explanation of another specific implementation process of the emotion recognition method provided in the embodiments of this application.

[0107] Figure 3 For another flowchart illustrating the emotion recognition method provided in this application embodiment, please refer to... Figure 3 The initial recognition results and feature information from at least one type of modality data are subjected to feature fusion and classification recognition processing to obtain emotion recognition results for multiple types of modality data, including:

[0108] S310: The initial recognition result is fused into the feature information of the first type of modality data to obtain the current fused feature.

[0109] Optionally, after obtaining the initial recognition result, the initial recognition result can be fused into the feature information of the first type of modal data. For example, if the first type is text, the current fused feature is the result obtained by fusing the initial recognition result into the feature information of the text type modal data.

[0110] S320: Based on the pre-trained current classifier, classify and recognize the current fused features to obtain the current recognition result.

[0111] The current classifier is the classifier that matches the current fused features.

[0112] Optionally, the current classifier can include a variety of types, such as CNN (Convolutional Neural Networks), LSTM (Long Short-Term Memory), or RNN (Recurrent Neural Network), which can be selected according to actual needs, that is, the classifier that matches the current fused features.

[0113] The current classifier can be used to perform corresponding classification and recognition processes to obtain the current recognition result.

[0114] S330: Take the current recognition result as the new initial recognition result, take the feature information of the second type of modality data as the feature information of the new first type of modality data, and re-execute the step of fusing the initial recognition result into the feature information of the first type of modality data. Repeat this process until the current recognition result meets the preset conditions or the number of cycles reaches the preset number.

[0115] The second type is the type that follows the first type.

[0116] Optionally, the above steps can be repeated, that is, the current recognition result can be used as the new initial recognition result, the feature information of the second type of modality data can be used as the feature information of the new first type of modality data, and the step of fusing the initial recognition result into the feature information of the first type of modality data can be repeated until the current recognition result meets the preset conditions or the number of cycles reaches the preset number.

[0117] It should be noted that, for example, when it is necessary to perform the feature fusion of the feature information of text type modal data and the feature information of speech type modal data, if the first type is text type, then the second type can be speech type; correspondingly, if the first type is speech type, then the second type is text type, that is, the cyclic fusion process is achieved in this way.

[0118] S340: Based on the feature information of various types of modal data and the current recognition results, obtain the emotion recognition results of various types of modal data.

[0119] Optionally, after obtaining the current recognition result after meeting the preset conditions or after the number of iterations reaches the preset number, emotion recognition results for multiple types of modal data can be obtained based on the feature information of various types of modal data and the current recognition result.

[0120] In other words, emotions can be classified based on the recognition results, and the emotion represented by the final recognition result can be determined, such as anger, happiness, surprise, neutrality, etc.

[0121] The emotion recognition method provided in this application embodiment can fuse the initial recognition result into the feature information of the first type of modality data to obtain the current fused feature; perform classification and recognition processing on the current fused feature based on the pre-trained current classifier to obtain the current recognition result; use the current recognition result as the new initial recognition result, use the feature information of the second type of modality data as the new feature information of the first type of modality data, and repeat the step of fusing the initial recognition result into the feature information of the first type of modality data, repeating the process until the current recognition result meets the preset conditions or the number of iterations reaches the preset number; based on the feature information of various types of modality data and the current recognition result, emotion recognition results for multiple types of modality data are obtained. Through the above-described cyclical or multiple recognition process, more accurate emotion recognition results can be obtained.

[0122] The following is a detailed explanation of another specific implementation process of the emotion recognition method provided in the embodiments of this application.

[0123] Figure 4 For another flowchart illustrating the emotion recognition method provided in this application embodiment, please refer to... Figure 4 Based on the feature information of various types of modal data and the current recognition results, emotion recognition results for multiple types of modal data are obtained, including:

[0124] S410: Based on the classifiers corresponding to various types, the feature information of various types of modal data is classified and recognized to obtain the single-modal emotion recognition results of various types of modal data.

[0125] Among them, the single-modal emotion recognition result refers to the recognition result obtained by classifying and recognizing the feature information of the modal data of one type based on the corresponding classifier of one type.

[0126] For example, the result of classifying and recognizing the feature information of the modal data of the text type using a classifier corresponding to the text type is the single modal emotion recognition result of the text type modal data. Other types are similar, and examples will not be repeated here.

[0127] S420: Based on the current recognition results and the single-modal emotion recognition results of various types of modal data, obtain emotion recognition results for multiple types of modal data.

[0128] Optionally, after obtaining the current recognition result through step S330, emotion recognition results for multiple types of modal data can be obtained based on the current recognition result and the single-modal emotion recognition results of various types of modal data.

[0129] It should be noted that a single modality often cannot provide a complete description of information such as speech and text. By extending from a single modality to a multimodality in the above way, the comprehensiveness and accuracy of emotion recognition results can be improved.

[0130] The emotion recognition method provided in this application embodiment can classify and recognize the feature information of various types of modal data based on classifiers corresponding to various types, thereby obtaining single-modal emotion recognition results for various types of modal data; based on the current recognition results and the single-modal emotion recognition results for various types of modal data, emotion recognition results for multiple types of modal data can be obtained. This approach enables the extension from single-modal emotion recognition results to multi-modal emotion recognition results, improving the comprehensiveness and completeness of the emotion recognition results.

[0131] Optionally, based on the current recognition result and the single-modal emotion recognition results of various types of modal data, emotion recognition results of multiple types of modal data can be obtained, including: based on the current recognition result, the weight of the current recognition result, the single-modal emotion recognition results of various types of modal data, and the weight of the single-modal emotion recognition results of various types of modal data, emotion recognition results of multiple types of modal data can be obtained.

[0132] It should be noted that the weight of the current recognition result can be set according to the actual needs. Correspondingly, the weight of the single-modal emotion recognition result of various types of modal data can be set according to historical data or actual needs.

[0133] After obtaining the above weights, emotion recognition results for multiple types of modal data can be obtained based on the current recognition result, the weight of the current recognition result, the single-modal emotion recognition results of various types of modal data, and the weights of the single-modal emotion recognition results of various types of modal data.

[0134] To more clearly explain the emotion recognition method provided in the embodiments of this application, the following example uses the fusion of text-type modal data and speech-type modal data for a specific explanation.

[0135] Figure 5 For another flowchart illustrating the emotion recognition method provided in this application embodiment, please refer to... Figure 5 ,in, Figure 5The text data shown refers to text-type modal data, and the speech data refers to speech-type modal data. The text data and speech data are respectively processed through a bidirectional long short-term memory network model and a self-attention mechanism network model to obtain text feature information and speech feature information. The text feature information can be classified by a target classifier and then fused with the speech feature information at the feature level to obtain the initial fused features. Then, the text feature information and the initial fused features can be fused at the decision level to obtain the emotion recognition result.

[0136] The following describes the apparatus, device, and storage medium used to implement the emotion recognition method provided in this application. The specific implementation process and technical effects are described above and will not be repeated below.

[0137] Figure 6 Please refer to the structural schematic diagram of the emotion recognition device provided in the embodiments of this application. Figure 6 The device includes: an acquisition module 610, an extraction module 620, a fusion module 630, a training module 640, and a result module 650;

[0138] The acquisition module 610 is used to acquire various types of modal data, including text, speech and image types.

[0139] The extraction module 620 is used to extract feature information of various types of modal data respectively, and to determine various types to be fused among the various types;

[0140] The fusion module 630 is used to perform feature fusion on the feature information of the modal data belonging to the type to be fused, so as to obtain the initial fused features.

[0141] The training module 640 is used to classify and recognize the initial fused features based on the pre-trained target classifier to obtain the initial recognition result, wherein the target classifier is a classifier that matches the initial fused features;

[0142] The result module 650 is used to perform feature fusion and classification recognition processing on the initial recognition result and the feature information of at least one type of modal data to obtain the emotion recognition result of multiple types of modal data, wherein at least one type is a type among multiple types.

[0143] Optionally, the extraction module 620 is specifically used to input various types of modal data into a pre-trained bidirectional long short-term memory network model to obtain contextual features of various types of modal data; input various types of modal data into a pre-trained self-attention mechanism network model to obtain contextual semantic features of various types of modal data; and obtain feature information of various types of modal data based on contextual features and contextual semantic features.

[0144] Optionally, the extraction module 620 is further configured to extract Mel-frequency cepstral coefficient feature information from the modal data of the current type as the modal data of the current type if the current type belongs to the speech type; and to extract text data from the modal data of the current type as the modal data of the current type if the current type belongs to the image type.

[0145] Optionally, the types to be fused include: text type and speech type; the fusion module 630 is specifically used to perform feature fusion processing on the feature information of the text type modal data and the feature information of the speech type modal data to obtain the initial fused features.

[0146] Optionally, the result module 650 is used to fuse the initial recognition result into the feature information of the first type of modality data to obtain the current fused feature; perform classification and recognition processing on the current fused feature based on the pre-trained current classifier to obtain the current recognition result, wherein the current classifier is a classifier that matches the current fused feature; take the current recognition result as the new initial recognition result, take the feature information of the second type of modality data as the new feature information of the first type of modality data, and re-execute the step of fusing the initial recognition result into the feature information of the first type of modality data, repeat the execution until the current recognition result meets the preset conditions or the number of iterations reaches the preset number of iterations, wherein the second type is the type after the first type; and obtain the emotion recognition results of multiple types of modality data based on the feature information of various types of modality data and the current recognition result.

[0147] Optionally, the result module 650 is specifically used to classify and recognize the feature information of various types of modal data based on classifiers corresponding to various types, and obtain the single-modal emotion recognition results of various types of modal data; based on the current recognition results and the single-modal emotion recognition results of various types of modal data, the emotion recognition results of multiple types of modal data are obtained.

[0148] Optionally, the result module 650 is specifically used to obtain the emotion recognition results of multiple types of modal data based on the current recognition result, the weight of the current recognition result, the single-modal emotion recognition results of various types of modal data, and the weight of the single-modal emotion recognition results of various types of modal data.

[0149] The above-described device is used to execute the method provided in the foregoing embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0150] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more microprocessors, or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).

[0151] Figure 7 Please refer to the schematic diagram of the computer device provided in the embodiments of this application. Figure 7 The computer device includes: a memory 710 and a processor 720. The memory 710 stores a computer program that can run on the processor 720. When the processor 720 executes the computer program, it implements the steps of the emotion recognition method.

[0152] Another aspect of this application embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of an emotion recognition method.

[0153] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0154] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0155] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0156] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute certain steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0157] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0158] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An emotion recognition method, characterized in that, The method includes: Acquire multiple types of modal data, including text, speech, and image data; If the current type belongs to the speech type, then the Mel-frequency cepstral coefficient feature information is extracted from the modal data of the current type as the modal data of the current type; If the current type is an image type, then extract text data from the modal data of the current type as the modal data of the current type; Various types of modal data are input into a pre-trained bidirectional long short-term memory network model to obtain contextual features of various types of modal data; By inputting various types of modal data into a pre-trained self-attention mechanism network model, contextual semantic features of various types of modal data can be obtained. Based on the contextual features and the contextual semantic features, feature information of various types of modal data is obtained, and various types to be fused among the multiple types are determined; Feature fusion is performed on the feature information of the modal data belonging to the type to be fused to obtain the initial fused features; The initial fused features are classified and identified based on the pre-trained target classifier to obtain an initial identification result, wherein the target classifier is a classifier that matches the initial fused features; The initial recognition result and the feature information of at least one type of modal data are subjected to feature fusion and classification recognition processing to obtain the emotion recognition result of the multiple types of modal data, wherein the at least one type is a type among the multiple types; The process of feature fusion and classification of the initial recognition result and feature information of at least one type of modality data to obtain the emotion recognition result of the multiple types of modality data includes: The initial recognition result is fused into the feature information of the first type of modality data to obtain the current fused feature; The current fused features are classified and identified based on the pre-trained current classifier to obtain the current identification result, wherein the current classifier is a classifier that matches the current fused features; The current recognition result is used as the new initial recognition result, and the feature information of the second type of modal data is used as the feature information of the new first type of modal data. The step of fusing the initial recognition result into the feature information of the first type of modal data is executed again, and the process is repeated until the current recognition result meets the preset conditions or the number of iterations reaches the preset number of iterations. Here, the second type is the type after the first type. Based on the feature information of various types of modal data and the current recognition results, the emotion recognition results of the various types of modal data are obtained.

2. The emotion recognition method as described in claim 1, characterized in that, The types to be fused include: text type and speech type; The feature fusion of feature information of modal data belonging to the type to be fused, to obtain initial fused features, includes: The feature information of the text-type modal data and the feature information of the speech-type modal data are fused to obtain the initial fused features.

3. The emotion recognition method as described in claim 1, characterized in that, The process of obtaining emotion recognition results for the various types of modal data based on feature information of different types of modal data and the current recognition result includes: Based on classifiers corresponding to various types, the feature information of various types of modal data is classified and identified to obtain the single-modal emotion recognition results of various types of modal data; Based on the current recognition results and the single-modal emotion recognition results of the various types of modal data, the emotion recognition results of the multiple types of modal data are obtained.

4. The emotion recognition method as described in claim 3, characterized in that, The process of obtaining emotion recognition results for the multiple types of modal data based on the current recognition result and the single-modal emotion recognition results of the various types of modal data includes: Based on the current recognition result, the weight of the current recognition result, the single-modal emotion recognition results of various types of modal data, and the weights of the single-modal emotion recognition results of various types of modal data, the emotion recognition results of the multiple types of modal data are obtained.

5. An emotion recognition device, characterized in that, The device includes: an acquisition module, an extraction module, a fusion module, a training module, and a result module; The acquisition module is used to acquire multiple types of modal data, including text, voice, and image data. The extraction module is configured to: if the current type is speech, extract Mel-frequency cepstral coefficient feature information from the modal data of the current type as the modal data of the current type; if the current type is image, extract text data from the modal data of the current type as the modal data of the current type; input the modal data of various types into a pre-trained bidirectional long short-term memory network model to obtain the contextual features of the modal data of various types; input the modal data of various types into a pre-trained self-attention mechanism network model to obtain the contextual semantic features of the modal data of various types; and, based on the contextual features and the contextual semantic features, obtain the feature information of the modal data of various types and determine the various types to be fused among the multiple types. The fusion module is used to perform feature fusion on the feature information of the modal data belonging to the type to be fused, so as to obtain the initial fused features; The training module is used to classify and recognize the initial fused features based on a pre-trained target classifier to obtain an initial recognition result, wherein the target classifier is a classifier that matches the initial fused features; The result module is used to perform feature fusion and classification recognition processing on the initial recognition result and the feature information of at least one type of modal data to obtain the emotion recognition result of the multiple types of modal data, wherein the at least one type is a type among the multiple types; The result module is specifically used for: fusing the initial recognition result into the feature information of the first type of modality data to obtain the current fused feature; performing classification and recognition processing on the current fused feature based on the pre-trained current classifier to obtain the current recognition result, wherein the current classifier is a classifier that matches the current fused feature; using the current recognition result as a new initial recognition result, using the feature information of the second type of modality data as the feature information of the new first type of modality data, and re-executing the step of fusing the initial recognition result into the feature information of the first type of modality data, repeating the process until the current recognition result meets a preset condition or the number of iterations reaches a preset number, wherein the second type is a type after the first type; and obtaining the emotion recognition result of the multiple types of modality data based on the feature information of various types of modality data and the current recognition result.

6. A computer device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program that can run on the processor, and when the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 4.

Images