A multimodal emotion recognition method and system

Abstract

The present disclosure relates to a multimodal emotion recognition method and system, which includes: a multimodal information preprocessing step, collecting user voice data and image data to obtain corresponding voice segment data and facial image data; a multimodal emotion recognition step, performing voice, text and expression visual emotion analysis to obtain corresponding emotional state results; a first HMM processing step, combining the multimodal emotional state results into a set of features, inputting them into the first HMM as observation values, and outputting the corresponding probability of each emotion; a correlation calculation step, using a cross-correlation function (CCF) to calculate the correlation between the results of the multimodal emotional state; and a second HMM processing step, inputting the correlation results and the multimodal emotional state results into the second HMM as observation values, and taking the emotion with the largest output value as the final recognition result.

Description

Technical Field

[0001] The present disclosure relates to the field of biometric intelligent recognition technology, and more specifically to a multimodal emotion recognition method and system. Background Art

[0002] In recent years, with the advancement of computer and communication technologies, new technologies such as artificial intelligence have continued to develop rapidly. Against the backdrop of overall technological progress, how to effectively and accurately dynamically analyze and identify the emotional states of individual participants in various intelligent systems built on artificial intelligence has gradually become a research hotspot.

[0003] Emotion recognition, within the context of artificial intelligence (AI), uses computer engineering to integrate the mechanisms of human emotion to automatically identify an individual's emotional state by acquiring physiological and non-physiological signals. It is a key component of affective computing and a significant interdisciplinary research topic encompassing neuroscience, psychology, cognitive science, computer science, and artificial intelligence. As a specific application area, emotion recognition has found extensive and in-depth applications in various scenarios and fields, including intelligent customer service, personal assistants, education, financial fraud prevention, healthcare, human-computer interaction, and service robotics. For example, in healthcare settings, real-time understanding of a patient's emotional state, particularly those with expressive disorders, allows for optimal care measures tailored to their mood and improves care quality. In product development, identifying a user's emotional state during product use and understanding their user experience can help developers improve product functionality and design products that better meet user needs. In various human-computer interaction systems, real-time recognition of a user's emotional state can make interactions between humans and machines more friendly and natural. In nearly all service industries, including telecommunications, emotion recognition can be applied to customer service to improve customer satisfaction.

[0004] Emotion recognition research focuses on aspects such as facial expressions, voice, heart rate, behavior, text, and physiological signal recognition, which can be used to determine the user's emotional state. However, existing emotion recognition is primarily limited to a single, specific factor, such as voice or biological signals, and relies on large-scale extraction of user features for analysis and detection. Due to computational complexity, it rarely combines multiple factors to utilize information from various aspects for analysis and judgment. Therefore, existing emotion recognition solutions, when using a single data source, often suffer from unstable recognition performance and are prone to errors when the amount of collected data is insufficient or when faced with subjects from different cultural backgrounds or individual differences. Therefore, the industry urgently needs a feasible technical solution that can effectively improve the accuracy and stability of emotion recognition. Summary of the Invention

[0005] To address the inaccuracies and instability of existing emotion recognition solutions, researchers in this field have conducted in-depth research and development using a combination of techniques, including speech recognition, natural language processing, and facial expression recognition. Furthermore, they have conducted multimodal emotion recognition by integrating the correlations between different factors. The present disclosure aims to provide a multimodal emotion recognition method and system that effectively improves the accuracy and stability of emotion recognition.

[0006] The following is a brief overview of the present disclosure to provide a basic understanding of some aspects of the present disclosure. However, it should be understood that this overview is not an exhaustive overview of the present disclosure. It is not intended to define the key or important parts of the present disclosure, nor is it intended to limit the scope of the present disclosure. Its purpose is simply to present certain concepts of the present disclosure in a simplified form as a prelude to the more detailed description that will be given later.

[0007] According to one aspect of the present disclosure, a multimodal emotion recognition method is provided. The method may include: a multimodal information preprocessing step, collecting multimodal original data including at least voice data and image data generated when the emotion recognition object speaks, performing blank segment detection and marking on the voice data in the multimodal original data and storing it as voice segment data, using speech-to-text ASR (Automatic Speech Recognition) to parse the voice audio stream to convert the voice into text segment data, performing face detection and cropping on the image data in the multimodal original data and storing the facial image data in a memory; a multimodal emotion recognition step, performing speech emotion analysis, text emotion analysis and facial expression visual emotion analysis respectively according to the multimodal information including at least voice segment data, text segment data and facial image data obtained in the multimodal information preprocessing step, so as to obtain the results of speech-based emotion state, text-based emotion state and visual-based emotion state; a first Hidden Markov Model (HMM) The present invention also provides a method for the present invention to provide a first HMM processing step, wherein the results of the emotional state obtained from the speech emotion analysis, text emotion analysis and expression visual emotion analysis obtained in the multimodal emotion recognition step are combined into a set of features, which are input into the first HMM as observation values ​​in the first HMM, and the corresponding probability of each emotion among the multiple emotions is output; a correlation calculation step, wherein the correlation between the results of the multimodal emotional state is calculated using a cross-correlation function CCF (Cross-Correlation Function) for the results of the speech-based emotional state, text-based emotional state and visual-based emotional state obtained in the multimodal emotion recognition step; and a second HMM processing step, wherein the correlation result obtained in the correlation calculation step and the probability of each emotion obtained in the first HMM processing step are input into the second HMM as observation values ​​in the second HMM, and the emotion with the largest value among the output state values ​​of each emotion is used as the final emotion recognition result.

[0008] According to another aspect of the present disclosure, a multimodal emotion recognition system is provided. The system may include: a voice acquisition unit for acquiring the voice of the emotion recognition object, obtaining a voice processing rule, detecting and marking the blank segments in the real-time collected voice data based on the voice processing rule to obtain a marking result, and obtaining and saving the initial voice file when the voice acquisition is completed; a voice-to-text unit for converting the input voice information into corresponding text information using a keyword recognition and fuzzy matching method and outputting it; a visual expression acquisition unit for tracking the face of the emotion recognition object and photographing and recording the expression of the emotion recognition object at each moment, photographing a facial expression image containing the face, and cropping the facial region of the facial expression image to obtain facial image data; a voice emotion classifier for taking the voice segment of the voice acquisition unit as input and outputting the result of the emotional state of each emotion among multiple emotions based on a trained built-in model; a text emotion classifier for taking the text segment output by the voice-to-text unit as input and outputting the emotional state of each emotion among multiple emotions. The facial expression emotion classifier takes the facial image data collected by the visual expression acquisition unit as input and outputs the results of the emotional state of each emotion in a plurality of emotions; the first HMM module has 18 states and 6 outputs, the input is the feature vector of the result of the multimodal emotional state, and the output is the corresponding probability of each emotion in a plurality of emotions; the correlation calculation unit uses the cross-correlation function CCF to calculate the correlation between the results of the multimodal emotional state based on speech, the emotional score based on text and the emotional state based on vision obtained by the speech emotion classifier, the text emotion classifier and the facial expression emotion classifier; and the second HMM module inputs the correlation result obtained by the correlation calculation unit and the probability of each emotion output by the first HMM module as the observation value in the second HMM, and takes the emotion with the largest value in the output state value as the final emotion recognition result.

[0009] According to another aspect of the present disclosure, a multimodal emotion recognition device is provided. The device may include: a memory storing instructions; and a processor configured to execute the instructions stored in the memory to perform the multimodal emotion recognition method according to one aspect of the present disclosure.

[0010] According to another aspect of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium may store computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the multimodal emotion recognition method according to one aspect of the present disclosure.

[0011] According to another aspect of the present disclosure, a computer program product is provided, which may include a computer program / instructions that, when executed by a processor, implement the steps of the multimodal emotion recognition method according to one aspect of the present disclosure.

[0012] According to the technical solution disclosed in the present invention, multiple modal information including voice, text, visual expressions, etc. are integrated through multimodal information collection, multimodal information encoding, multimodal data information sequence analysis, and multimodal information correlation analysis, thereby effectively improving the accuracy and stability of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 An exemplary flow chart of a multimodal emotion recognition method 100 according to an embodiment of the present disclosure is shown;

[0014] Figure 2 1 shows an exemplary processing procedure of the text emotion analysis and recognition sub-step S122 in the multimodal emotion recognition method 100 according to an embodiment of the present disclosure;

[0015] Figure 3 An exemplary schematic diagram showing a multimodal emotion recognition system 300 according to an embodiment of the present disclosure;

[0016] Figure 4 An exemplary schematic diagram illustrating a processing procedure performed by the first HMM module 330 according to an embodiment of the present disclosure;

[0017] Figure 5 An exemplary schematic diagram showing the processing process performed by the correlation calculation unit 340 according to an embodiment of the present disclosure;

[0018] Figure 6 An exemplary schematic diagram illustrating a processing procedure performed by the second HMM module 350 according to an embodiment of the present disclosure;

[0019] Figure 7 FIG. 7 shows an exemplary block diagram of a multimodal emotion recognition apparatus 700 according to an embodiment of the present disclosure;

[0020] Figure 8 FIG. 8 shows an exemplary configuration diagram of a computing device 800 that can implement the multimodal emotion recognition method 100 according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions, and numerical values ​​described in these embodiments do not limit the scope of the present disclosure. At the same time, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual proportional relationships. The following description of at least one exemplary embodiment is merely illustrative and is not intended to limit the present disclosure, its application, or use in any way.

[0022] Technologies, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such technologies, methods, and devices should be considered part of the specification. The technology of the present disclosure can be applied to various products.

[0023] For ease of understanding and explanation, as a multimodal emotion recognition method and system according to an embodiment of the present disclosure, multiple modal information including three factors, namely voice, text, and visual expressions, is mainly used as analysis factors for emotion recognition processing, but this is not restrictive. According to the technical purpose of the present disclosure, the analysis factors of the recognition processing can be expanded to also include other suitable arbitrary factors. In addition, as the output classification of the emotion recognition processing, the present disclosure uses the six basic emotions shared by humans proposed by American psychologist Ekman et al. - happiness, anger, fear, sadness, disgust and surprise - as an example for explanation, but this is not restrictive, and the output classification of the emotion recognition processing can also be expanded to include any other suitable range of any other suitable emotion category, or the output classification of the emotion recognition processing can also be narrowed down to include some emotion classifications among the above six basic emotions.

[0024] The following describes the overall process of the multimodal emotion recognition method according to an embodiment of the present disclosure. Figure 1 A schematic diagram of a multimodal emotion recognition method 100 according to an exemplary embodiment of the present disclosure is shown. Here, the method is described using the example of an individual speaking in a human-to-human or human-computer interaction scenario as the subject of emotion recognition. Of course, the method can also be applied to any other appropriate scenario.

[0025] Preferably, the multimodal emotion recognition method 100 according to the embodiment of the present disclosure may generally include the following steps S110 to S150. The details of each step S110 to S150 are as follows:

[0026] Multimodal information preprocessing step S110: collecting multimodal raw data including at least speech data and image data generated by the emotion recognition subject when speaking, detecting and marking blank segments of the speech data in the multimodal raw data, and storing them as speech segment data; parsing the speech audio stream using speech-to-text ASR to convert the speech into text segment data; performing facial detection and cropping on the image data in the multimodal raw data, and storing the facial image data in a memory;

[0027] Multimodal emotion recognition step S120: performing speech emotion analysis, text emotion analysis, and facial expression and visual emotion analysis on the multimodal information obtained in the multimodal information preprocessing step S110, respectively, to obtain a speech-based emotional state, a text-based emotional state, and a visual-based emotional state;

[0028] First HMM processing step S130: combining the results of the emotional state obtained from the speech emotion analysis, text emotion analysis, and expression visual emotion analysis in the multimodal emotion recognition step S120 into a set of features, inputting them into the first HMM as observation values, and outputting the corresponding probabilities P1 to P2 of each of the multiple emotions. n , in the embodiment of the present disclosure, n is set to 6, and of course it can be other appropriate values ​​corresponding to the number of emotion categories;

[0029] Correlation calculation step S140: for the results of the speech-based emotional state, the text-based emotional state, and the visual-based emotional state obtained in the multimodal emotion recognition step S120, using a cross-correlation function (CCF), calculating the correlation between the results of the multimodal emotional state;

[0030] Second HMM processing step S150: The correlation result obtained in the correlation calculation step S140 and the probability of each emotion obtained in the first HMM processing step S130 are input into the second HMM as the observation values ​​in the second HMM, and the emotion with the largest value among the output state values ​​of each emotion is taken as the final emotion recognition result.

[0031] Through the above steps S110 to S150, at least three modalities (voice, text, and visual expressions) are effectively combined for emotion recognition. First, the emotion classification of each modality is performed separately to form a preliminary classification result. Next, the correlation between the modalities is incorporated into the model, and the preliminary classification results are fine-tuned. This can effectively avoid algorithm bias under a single modality and improve the accuracy and stability of the model.

[0032] More specifically, it is preferred that, in the multimodal information preprocessing step S110, upon receiving a voice collection instruction, the voice recording function is started to collect the user's voice, and the blank segments in the real-time collected voice data are detected and marked based on the acquired voice processing rules to obtain a marking result, and an initial voice file is obtained and saved after the voice collection is completed; further, the input voice data can be converted into corresponding text information and output by combining keyword recognition with fuzzy matching.

[0033] Preferably, in the multimodal information preprocessing step S110, a facial expression image including the face of the emotion recognition subject can be captured, the face of the emotion recognition subject can be tracked, and the expressions of the emotion recognition subject at various moments can be captured and recorded. The captured facial expression image can be cropped to obtain facial image data. The captured facial expression image can be uploaded to a server, and the facial expression image can be cropped to obtain facial image data.

[0034] More specifically, preferably, the multimodal emotion recognition step S120 may include a speech emotion analysis and recognition sub-step S121, a text emotion analysis and recognition sub-step S122, and an expression and visual emotion analysis and recognition sub-step S123.

[0035] Among them, in the speech emotion analysis and recognition sub-step S121, it is preferred that, for the speech segment data, acoustic features expressing emotions are extracted from the collected speech signals, and the mapping relationship between the extracted acoustic features and emotions is found, a speech emotion classifier is constructed through feature extraction, and the speech emotion classifier is used to obtain the emotional state of each segment of speech segment data.

[0036] In the text sentiment analysis and identification sub-step S122, preferably, for the text fragment data, an unsupervised text sentiment classifier is constructed based on various lexicons including slang lexicons, sentiment lexicons, negation lexicons, enhancement lexicons, and weakening lexicons, and the text sentiment classifier is used to calculate the score of the emotional state of each text fragment data.

[0037] In the expression visual emotion analysis and recognition sub-step S123, preferably, a convolutional neural network (CNN) framework is used to train an expression emotion classifier for the facial image data of the emotion recognition object, and the expression emotion classifier is used to obtain the result of the emotional state of the emotion recognition object.

[0038] As mentioned above, in the text sentiment analysis and identification sub-step S122, it is preferred to adopt an unsupervised method for processing. Figure 2The following is an exemplary processing procedure of the text emotion analysis and recognition sub-step S122 in the multimodal emotion recognition method 100 according to an embodiment of the present disclosure. First, for the text segment data as input, in step S122-1, the slang in the text segment is replaced based on the slang vocabulary; next, in step S122-2, all emotion-containing words are marked with corresponding emotion scores based on the emotion vocabulary; then, in step S122-3, all intensifiers (such as "absolutely"), weakening words (such as "possible"), and negation words (such as "impossible") and corresponding scores are marked based on the enhancement vocabulary, weakening vocabulary, and negation vocabulary; then, in step S122-4, the score of the emotion word that appears after the intensifier is increased; in step S122-5, the score of the emotion word that appears after the weakening word is decreased; in step S122-6, the score of the emotion word that appears after the negation word is converted into the opposite number; finally, in step S122-7, the average of the scores of all words in the text segment is taken as the emotion score of the text segment. Thus, the unsupervised text emotion analysis and recognition process of the text emotion analysis and recognition sub-step S122 in the multimodal emotion recognition method 100 according to the embodiment of the present disclosure is completed, that is, an unsupervised text emotion classifier is constructed.

[0039] According to the above unsupervised text sentiment analysis and recognition processing, an unsupervised method is adopted for text type sentiment recognition, which does not require a large amount of labeling and has the advantages of high accuracy, short commissioning cycle, high practical value, simplicity and efficiency.

[0040] Preferably, the first HMM used in the first HMM processing step S130 has 18 states and 6 outputs, the input is the feature vector of the result of the multimodal emotional state, and the output is the probability value P1~P n .

[0041] Next, a multimodal emotion recognition system according to an embodiment of the present disclosure is described. Figure 3 An exemplary schematic diagram of a multimodal emotion recognition system 300 according to an embodiment of the present disclosure is shown. The exemplary multimodal emotion recognition system 300 may generally include the following modules 311 to 350, and the details of each module 311 to 350 are as follows:

[0042] The speech collection unit 311 is used to collect the speech of the emotion recognition subject, obtain speech processing rules, detect and mark blank segments in the real-time collected speech data based on the speech processing rules to obtain marking results, and obtain and save initial speech segment data when the speech collection is completed;

[0043] The speech-to-text unit 312 converts the input speech information into corresponding text information by using the method of keyword recognition and fuzzy matching and outputs the corresponding text segment data;

[0044] Image acquisition unit 313: used for tracking the face of the emotion recognition subject and capturing and recording the expression of the emotion recognition subject at various moments, capturing a facial expression image including the face, and performing facial region cropping on the facial expression image to obtain facial image data;

[0045] The speech emotion classifier 321 takes the speech segment data obtained by the speech acquisition unit 311 as input and outputs the emotional state results of each emotion among multiple emotions based on the trained built-in model;

[0046] The text emotion classifier 322 takes the text segment data output by the speech-to-text unit 312 as input and outputs the result of the emotional state of each emotion among multiple emotions;

[0047] The facial expression and emotion classifier 323 takes the facial image data collected by the visual expression collection unit 313 as input and outputs the result of the emotional state of each emotion among multiple emotions;

[0048] The first HMM module 330 has 18 states and 6 outputs. The input is the feature vector of the result of the multimodal emotional state, and the output is the corresponding probability P1~P n ;

[0049] The correlation calculation unit 340 calculates the correlation between each pair of multimodal input information using a cross-correlation function (CCF) for the speech-based emotional state, text-based emotional state, and visual-based emotional state results obtained by the speech emotion classifier 321, the text emotion classifier 322, and the facial expression emotion classifier 323; and

[0050] The second HMM module 350: combines the correlation result obtained by the correlation calculation unit 340 and the probabilities P1 to P1 of each emotion output by the first HMM module 330. n , is input into the second HMM as the observation value in the second HMM, and the emotion with the largest value in the output state value is taken as the final emotion recognition result.

[0051] In the emotion recognition methods of the prior art, single information, such as voice information or visual information, is mainly used. In this case, the recognition results are unstable and easily interfered by external factors, resulting in a decrease in accuracy. As described above, the multimodal emotion recognition method and system according to the embodiment of the present disclosure, compared with the prior art, uses multimodal information as the information source, combines multimodal input information such as voice, text, and vision, and not only understands the user's emotions from the perceptual level such as voice intonation and visual expression, but also performs text-based semantic recognition. Compared with the case of a single information source, the confidence of the recognition result is higher; in addition, in the processing of multimodal information, the correlation between multimodal information is further taken into account, and then the correlation of multimodal information is processed as the input of HMM, which effectively improves the accuracy of emotion recognition.

[0052] Next, an example of a specific implementation process of the multimodal emotion recognition method 100 according to an embodiment of the present disclosure is described in detail.

[0053] First, using the multimodal emotion recognition system 300 of the embodiment of the present disclosure, in the multimodal information preprocessing step S110, the voice acquisition unit 311 is used to collect the voice uttered by the user who is the object of emotion recognition; at the same time, the image acquisition unit 313 is used to collect the user's facial expression information.

[0054] In this case, suppose the text corresponding to the collected user voice is: "I'm not happy about this at all. This part might be exciting, but I really don't like this solution," a total of three sentences. Here, the collected sound is stored in a common structure, such as RIFF (Resources Interchange File Format), as exemplary voice data as shown in Table 1 below. Each row is a RIFF block, with the storage address on the left and the specific voice data on the right. There are three RIFF blocks in total, corresponding to the three segments of the user's voice data.

[0055]

[0056] Table 1

[0057] At this time, for the above-mentioned user voice, the physical features of the voice such as fundamental frequency (pitch) and energy (energy) can be first extracted through the voice processing system; for the above-mentioned many features, methods such as PCA (Principal Component Analysis), LDA (Linear Discriminant Analysis), ResNet (Deep Residual Network) are used to build a feature extractor to obtain new features, that is, the mapping of the original features.

[0058] Based on the new features obtained here, a speech emotion classifier 321 can be constructed by further using methods such as k-NN (k-Nearest Neighbor) algorithm to obtain the respective probabilities of the six emotional states, that is, a six-dimensional vector.

[0059] Assume that for the speech data in step S110, after the speech emotion analysis and recognition sub-step S121, the respective probabilities of the emotional states obtained are, for example:

[0060] (0.01,0.15,0.03,0.54,0.12,0.05)

[0061] (0.52,0.04,0.00,0.00,0.00,0.43)

[0062] (0.04,0.21,0.00,0.11,0.60 0.04)

[0063] In the speech-to-text unit 312, speech-to-text conversion is modeled, integrating technologies such as natural language processing (NLP) and semantic understanding to convert the user's speech into text. The text converted from the user's speech above is: "I'm not happy about this at all. This part might be exciting. I really don't like this solution." Each conversation segment is then divided into smaller text segments based on punctuation. A text segment is defined as a number of words between two punctuation marks. The text segments converted from the above text are: "I'm not happy about this at all," "This part might be exciting," and "I really don't like this solution."

[0064] In the text sentiment analysis and identification sub-step S122, an unsupervised text sentiment classifier 322 is used to perform sentiment scoring on each text segment obtained in the above steps. The specific processing includes, for example:

[0065] In the sentence “I am not happy at all about this”, the score of the emotional word “happy” is 3, the score after adding the negation word “not” is -1×3, and the score after adding the intensifier “not at all” is -3×2;

[0066] In the sentence “This part may be exciting”, the emotional word “excited” scores 4, and after adding the weakening word “maybe”, the score is 4 / 1.5;

[0067] In the sentence “I really don’t like this plan”, the score of the sentiment word “like” is 3.5, and after adding the negation word “no”, the score becomes -1×3.5, and after adding the enhancement word “very”, the score becomes -3.5×2.

[0068] Finally, according to the predefined score-emotion correspondence, the scores are classified into scores corresponding to the emotion categories, and finally form a six-dimensional vector.

[0069] Therefore, the vectors corresponding to each text segment are, for example:

[0070] (0.00,0.00,0.00,-6.00,0.00,0.00)

[0071] (2.67,0.00,0.00,0.00,0.00,0.00)

[0072] (0.00,0.00,0.00,0.00,-7.00,0.00)

[0073] In the expression visual emotion analysis and recognition sub-step S123, the expression-based expression emotion classifier 323 is trained on the collected user facial image data to output the emotional state corresponding to the user's expression. For the user's expression image in each paragraph, the expression emotion classifier 323, which serves as an expression recognition model, obtains the respective probabilities of the six emotional states corresponding to each text segment, i.e., a six-dimensional vector. For example, in the above case, they are:

[0074] (0.02,0.26,0.03,0.34,0.21,0.04)

[0075] (0.57,0.01,0.01,0.00,0.00,0.40)

[0076] (0.10,0.41,0.01,0.06,0.40 0.02)

[0077] Next, the first HMM processing step S130 is entered. Taking the first text segment "I am not happy about this at all" as an example, the processing process is as follows: Figure 4 The emotional state results output in steps S121 to S123 are combined into a set of features, which are input into the first HMM model, i.e., the first HMM module 330, as observation values ​​in the hidden Markov model, to output the corresponding probability of each specific emotion among the multiple emotions.

[0078] The correlation calculation step S140 can be performed in parallel with step S130 or in sequence, such as Figure 5 As shown, for the results of the speech-based emotion probability, text-based emotion score and vision-based emotion probability output in the above steps S121 to S123, in the correlation calculation unit 340, the cross-correlation function CCF is used to calculate the CCF value that can reflect the correlation between the multimodal input information.

[0079] Next, enter the second HMM processing step S150, as shown in Figure 6As shown, according to the probabilities of all emotions obtained by using the first HMM module 330 in the above first HMM processing step S130 and the correlation results obtained by using the correlation calculation unit 340 in the correlation calculation step S140, they are combined into a set of features again and input into the second HMM module 350 as observation values ​​in the hidden Markov model, and the emotion with the largest value in the output state value is taken as the final emotion recognition result, that is, a specific emotion.

[0080] Next, Figure 7 FIG. 7 is an exemplary block diagram of a multimodal emotion recognition apparatus 700 according to an embodiment of the present disclosure.

[0081] like Figure 7 As shown, the multimodal emotion recognition device 700 of an embodiment of the present disclosure includes: a memory 710 and a processor 720 coupled to the memory 710, and the processor 720 is configured to execute the multimodal emotion recognition method 100 in any one embodiment of the present disclosure based on instructions stored in the memory 710.

[0082] The memory 710 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, application programs, a boot loader, a database, and other programs.

[0083] Figure 8 FIG. 8 is a diagram showing an exemplary configuration of a computing device 800 capable of implementing the multimodal emotion recognition method 100 according to an exemplary embodiment of the present disclosure.

[0084] Computing device 800 is an example of a hardware device to which the above-described aspects of the present disclosure can be applied. Computing device 800 can be any machine configured to perform processing and / or computation. Computing device 800 can be, but is not limited to, a workstation, a server, a desktop computer, a laptop computer, a tablet computer, a personal data assistant (PDA), a smartphone, an in-vehicle computer, or a combination thereof.

[0085] like Figure 8As shown, the computing device 800 may include one or more components that can be connected or communicated with a bus 802 via one or more interfaces. The bus 802 may include, but is not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus. The computing device 800 may include, for example, one or more processors 804, one or more input devices 806, and one or more output devices 808. The one or more processors 804 may be any type of processor and may include, but is not limited to, one or more general-purpose processors or special-purpose processors (such as dedicated processing chips). The processor 804 may be configured to implement the multimodal emotion recognition method 100 of the exemplary embodiment of the present disclosure according to the above-mentioned aspects of the present disclosure. The input device 806 may be any type of input device capable of inputting information to the computing device and may include, but is not limited to, a mouse, a keyboard, a touch screen, a microphone, and / or a remote controller. The output device 808 may be any type of device capable of presenting information and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer.

[0086] The computing device 800 may also include or be connected to a non-transitory storage device 814, which may be any non-transitory storage device that can implement data storage and may include, but is not limited to, a disk drive, an optical storage device, a solid-state memory, a floppy disk, a flexible disk, a hard disk, a magnetic tape or any other magnetic medium, a compact disk or any other optical medium, a cache memory and / or any other storage chip or module, and / or any other medium from which a computer can read data, instructions and / or code. The computing device 800 may also include a random access memory (RAM) 810 and a read-only memory (ROM) 812. The ROM 812 may store programs, utilities or processes to be executed in a non-volatile manner. The RAM 810 may provide volatile data storage and store instructions related to the operation of the computing device 800. The computing device 800 may also include a network / bus interface 816 coupled to a data link 818. The network / bus interface 816 can be any type of device or system capable of enabling communication with external devices and / or networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication device and / or a chipset (such as a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, a cellular communication facility, etc.).

[0087] The present disclosure may be implemented as any combination of an apparatus, a system, an integrated circuit, and a computer program on a non-transitory computer-readable medium. One or more processors may be implemented as an integrated circuit (IC), an application-specific integrated circuit (ASIC), or a large-scale integrated circuit (LSI), a system LSI, a super LSI, or a ultra-LSI component that performs some or all of the functions described in the present disclosure.

[0088] The present disclosure includes the use of software, applications, computer programs, or algorithms. The software, applications, computer programs, or algorithms can be stored on a non-transitory computer-readable medium to enable a computer, such as one or more processors, to perform the steps described above and in the accompanying figures. For example, one or more memories can store the software or algorithm in the form of executable instructions, and one or more processors can execute a set of instructions for the software or algorithm to provide various functions according to the embodiments described in the present disclosure.

[0089] Software and computer programs (which may also be referred to as programs, software applications, applications, components, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural language, an object-oriented programming language, a functional programming language, a logic programming language, or assembly language or machine language. The term "computer-readable medium" refers to any computer program product, apparatus, or device for providing machine instructions or data to a programmable data processor, such as magnetic disks, optical disks, solid-state storage devices, memories, and programmable logic devices (PLDs), including computer-readable media that receive machine instructions as computer-readable signals.

[0090] For example, computer-readable media may include dynamic random access memory (DRAM), random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), compact disk read-only memory (CD-ROM) or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to carry or store the desired computer-readable program code in the form of instructions or data structures and can be accessed by a general-purpose or special-purpose computer or a general-purpose or special-purpose processor. As used herein, disk or disc includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks generally reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of the above are also included within the scope of computer-readable media.

[0091] The subject matter of the present disclosure is provided as an example of devices, systems, methods, and programs for performing the features described in the present disclosure. However, in addition to the features described above, other features or variations are also contemplated. It is contemplated that any emerging technology that may replace any of the above-described implementation technologies may be used to implement the components and functions of the present disclosure.

[0092] In addition, the above description provides examples and does not limit the scope, applicability or configuration set forth in the claims. Without departing from the spirit and scope of this disclosure, the function and arrangement of the elements discussed can be changed. Various embodiments can appropriately omit, replace or add various processes or components. For example, features described with respect to certain embodiments can be combined in other embodiments.

[0093] In addition, in the description of the present disclosure, although operations are depicted in a particular order in the drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed to achieve the desired results. In certain circumstances, multitasking and parallel processing may be advantageous.

Claims

1. A multimodal emotion recognition method, comprising: a multimodal information preprocessing step, collecting multimodal raw data including at least speech data and image data generated when the subject of emotion recognition speaks, detecting and marking blank segments of the speech data in the multimodal raw data, and storing them as speech segment data; parsing the speech audio stream using speech-to-text ASR to convert the speech into text segment data; performing facial detection and cropping on the image data in the multimodal raw data, and storing the facial image data in a memory; a multimodal emotion recognition step, performing speech emotion analysis, text emotion analysis, and facial expression and visual emotion analysis, respectively, based on the multimodal information obtained in the multimodal information preprocessing step, which includes at least speech segment data, text segment data, and facial image data, to obtain speech-based emotion state, text-based emotion state, and visual-based emotion state results, wherein the speech-based emotion state, text-based emotion state, and visual-based emotion state results each include a corresponding probability of each of the multiple emotions; a first Hidden Markov Model (HMM) processing step, combining the results of the emotional state obtained according to the speech emotion analysis, text emotion analysis, and facial expression and visual emotion analysis in the multimodal emotion recognition step into a set of features, inputting the features into the first HMM as observation values, and outputting the corresponding probability of each of the multiple emotions; a correlation calculation step, for the results of the speech-based emotional state, the text-based emotional state, and the visual-based emotional state obtained in the multimodal emotion recognition step, using a cross-correlation function (CCF) to calculate the correlation between the results of the multimodal emotional state; as well as In the second HMM processing step, the correlation result obtained in the correlation calculation step and the probability of each emotion obtained in the first HMM processing step are input into the second HMM as the observation values ​​in the second HMM, and the emotion with the largest value among the output state values ​​of each emotion is used as the final emotion recognition result.

2. The multimodal emotion recognition method according to claim 1, wherein: In the multimodal information preprocessing step, when a voice collection instruction is received, the voice recording function is started to collect the user's voice, and the blank segments in the real-time collected voice data are detected and marked based on the acquired voice processing rules to obtain the marking results. After the voice collection is completed, the initial voice file is obtained and saved.

3. The multimodal emotion recognition method according to claim 2, wherein: In the multimodal information preprocessing step, the input voice data is converted into corresponding text information and output by combining keyword recognition with fuzzy matching.

4. The multimodal emotion recognition method according to claim 1, wherein: In the multimodal information preprocessing step, a facial expression image containing the face of the emotion recognition object is captured, the face of the emotion recognition object is tracked and the expressions of the emotion recognition object at various moments are captured and recorded, and the facial region of the captured facial expression image is cropped to obtain facial image data.

5. The multimodal emotion recognition method according to claim 1, wherein: In the multimodal emotion recognition step, for the speech segment data, acoustic features expressing emotions are extracted from the collected speech signals, and the mapping relationship between the extracted acoustic features and emotions is found. A speech emotion classifier is constructed through feature extraction, and the speech emotion classifier is used to obtain the emotional state of each segment of speech segment data.

6. The multimodal emotion recognition method according to claim 1, wherein: In the multimodal emotion recognition step, for the text fragment data, an unsupervised text emotion classifier is constructed based on various lexicons including slang lexicons, emotion lexicons, enhancement lexicons, weakening lexicons, and negation lexicons, and the text emotion classifier is used to calculate the score of the emotional state of each text fragment data.

7. The multimodal emotion recognition method according to claim 1, wherein: In the multimodal emotion recognition step, a convolutional neural network (CNN) framework is used to train an expression emotion classifier for facial image data of the emotion recognition object, and the expression emotion classifier is used to obtain the result of the emotional state of the emotion recognition object.

8. The multimodal emotion recognition method according to claim 1, wherein: The diverse range of emotions includes: happiness, anger, fear, sadness, disgust, and surprise.

9. The multimodal emotion recognition method according to claim 8, wherein: The first HMM has 18 states and 6 outputs, the input is a feature vector of a result of a multimodal emotional state, and the output is a probability value of each emotion.

10. A multimodal emotion recognition system comprising: A speech collection unit is used to collect speech of the emotion recognition subject, obtain speech processing rules, detect and mark blank segments in the real-time collected speech data based on the speech processing rules to obtain marking results, and obtain and save initial speech segment data when speech collection is completed; The speech-to-text unit converts the input speech information into corresponding text information by using the method of keyword recognition and fuzzy matching and outputs the corresponding text segment data; An image acquisition unit is used to track the face of the emotion recognition subject and capture and record the expression of the emotion recognition subject at various moments, capture a facial expression image containing the face, and perform facial region cropping on the facial expression image to obtain facial image data; A speech emotion classifier, which takes the speech segment data obtained by the speech acquisition unit as input and outputs the emotional state results of each emotion among multiple emotions based on the trained built-in model; A text emotion classifier, which takes the text segment data output by the speech-to-text unit as input and outputs the emotional state of each emotion among multiple emotions; an expression emotion classifier, which takes the facial image data collected by the image acquisition unit as input and outputs the result of the emotional state of each emotion among multiple emotions; The first HMM module has 18 states and 6 outputs. The input is a feature vector of a set of features composed of the results of multimodal emotional states, and the output is the corresponding probability of each emotion among multiple emotions; A correlation calculation unit calculates the correlation between the speech-based emotional state, text-based emotional state, and visual-based emotional state results obtained by the speech emotion classifier, the text emotion classifier, and the facial expression emotion classifier using a cross-correlation function (CCF) to calculate the correlation between the multimodal emotional state results. as well as A second HMM module, which inputs the correlation result obtained by the correlation calculation unit and the probability of each emotion output by the first HMM module as the observation value in the second HMM, and takes the emotion with the largest value in the output state value as the final emotion recognition result; Among them, the results of the multimodal emotional state include the results of the emotional state of each emotion output by the speech emotion classifier, the text emotion classifier and the facial expression emotion classifier, and the results of the emotional state of each emotion output by the speech emotion classifier, the text emotion classifier and the facial expression emotion classifier are the corresponding probabilities of each emotion.

11. A multimodal emotion recognition device, comprising: a memory having instructions stored therein; as well as A processor is configured to execute instructions stored in the memory to perform the multimodal emotion recognition method according to any one of claims 1 to 9.

12. A computer-readable storage medium having computer-executable instructions stored thereon, wherein when the computer-executable instructions are executed by one or more processors, the one or more processors are caused to execute the multimodal emotion recognition method according to any one of claims 1 to 9.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the multimodal emotion recognition method according to any one of claims 1 to 9 are implemented.

Images