Multi-modal emotion recognition method and device, equipment, storage medium and program product
By acquiring and processing speech and text data, determining emotional attribute information and filtering low confidence modal representations, the problem of low accuracy of multimodal emotion recognition in the prior art is solved, and more accurate emotion recognition is achieved.
Patent Information
- Application Number
- CN202411169109.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-11-12
AI Technical Summary
The existing multimodal emotion recognition technology cannot effectively utilize single-modal features and lacks interpretability, resulting in low accuracy of emotion recognition and inability to learn the specific contribution of different modes to emotion recognition.
By obtaining speech data and text data, the encoding process is performed separately to obtain intermediate feature representations, the emotional attribute information is determined, and the spatial modal representation is projected to a unified query space to obtain spatial modal representations, and the modal representations with low confidence are filtered according to the confidence score. Fusion processing is performed for emotion recognition.
Effectively learn the contribution of each modal to emotion recognition, improve the accuracy of emotion recognition, and filter misleading noise signals to improve the accuracy of the final result.
Smart Images

Figure CN118942488A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of natural language processing, and in particular to a multimodal emotion recognition method, device, equipment, storage medium and program product. Background Art
[0002] Emotion recognition is an important technology in natural language processing and human-computer interaction. It can help machines accurately understand and identify emotions and respond accordingly, playing a vital role in improving human-computer interaction. Multimodal emotion recognition refers to the use of multiple information sources such as text and voice to provide richer information for in-depth emotion recognition and thus more accurately understand human emotions.
[0003] In the prior art, multimodal emotion recognition usually includes the following steps: unimodal feature extraction, multimodal information interaction and final emotion prediction. However, the multimodal emotion recognition technology in the prior art often cannot effectively utilize unimodal features, lacks interpretability, and cannot learn the specific contribution of different modalities to emotion recognition. Therefore, the relative importance of different modalities is ignored during emotion recognition, resulting in low accuracy of emotion recognition. Summary of the invention
[0004] The present invention provides a multimodal emotion recognition method, device, equipment, storage medium and program product, which are used to solve the problem of low accuracy of emotion recognition in the multimodal emotion recognition technology in the prior art.
[0005] The present invention provides a multimodal emotion recognition method, comprising: acquiring speech data and corresponding text data; encoding the speech data and the text data respectively to obtain an intermediate feature representation of each modal data; determining the emotion attribute information in the intermediate feature representation, and determining a spatial modal representation according to the emotion attribute information; filtering the modal representations with low confidence in the spatial modal representation according to the confidence score of the vector to obtain a target representation vector; fusing the target representation vector, and performing emotion recognition based on the fusion result.
[0006] According to a multimodal emotion recognition method provided by the present invention, the determining of the emotional attribute information in the intermediate feature representation and determining the spatial modal representation based on the emotional attribute information includes: determining the emotional attribute information in the intermediate feature representation of each modal data through a query vector; projecting the speech data, the text data and the emotional attribute information into a unified query space to obtain a spatial modal representation.
[0007] According to a multimodal emotion recognition method provided by the present invention, the low-confidence modal representations in the spatial modal representation are filtered according to the confidence score of the vector to obtain a target representation vector, including: predicting the confidence of the spatial modal representation through a confidence estimation network; and determining the spatial modal representation with a confidence greater than a preset mask threshold as the target representation vector.
[0008] According to a multimodal emotion recognition method provided by the present invention, the voice data and the text data include historical conversations and current conversations, and the target representation vectors are fused and emotion recognition is performed based on the fusion results, including: the target representation vectors are fused and emotion recognition is performed on the current conversation based on the fusion results.
[0009] The present invention also provides a multimodal emotion recognition device, comprising the following modules: an acquisition module and a processing module; the acquisition module is used to acquire voice data and corresponding text data; the processing module is used to encode the voice data and the text data respectively to obtain an intermediate feature representation of each modal data; determine the emotional attribute information in the intermediate feature representation, and determine the spatial modal representation based on the emotional attribute information; filter the modal representations with low confidence in the spatial modal representation according to the confidence score of the vector to obtain a target representation vector; fuse the target representation vector, and perform emotion recognition based on the fusion result.
[0010] According to a multimodal emotion recognition device provided by the present invention, the processing module is used to determine the emotional attribute information in the intermediate feature representation of each modal data through a query vector; the speech data, the text data and the emotional attribute information are projected into a unified query space to obtain a spatial modal representation.
[0011] According to a multimodal emotion recognition device provided by the present invention, the processing module is used to predict the confidence of the spatial modal representation through a confidence estimation network; and determine the spatial modal representation with a confidence greater than a preset mask threshold as the target representation vector.
[0012] According to a multimodal emotion recognition device provided by the present invention, the voice data and the text data include historical conversations and current conversations, and the processing module is used to perform fusion processing on the target representation vector and perform emotion recognition on the current conversation based on the fusion result.
[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the multimodal emotion recognition method as described above is implemented.
[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the multimodal emotion recognition method as described in any one of the above is implemented.
[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the multimodal emotion recognition method as described above is implemented.
[0016] The multimodal emotion recognition method, device, equipment, storage medium and program product provided by the present invention can obtain voice data and corresponding text data; encode the voice data and the text data respectively to obtain the intermediate feature representation of each modal data; determine the emotional attribute information in the intermediate feature representation, and determine the spatial modal representation according to the emotional attribute information; filter the modal representation with low confidence in the spatial modal representation according to the confidence score of the vector to obtain the target representation vector; fuse the target representation vector, and perform emotion recognition based on the fusion result. Through this scheme, since the emotional attribute information can be determined from the intermediate feature representation of each modal data, the emotional clues of different granularities from each modality can be used to effectively learn the contribution of each modality to emotion recognition, thereby improving the accuracy of emotion recognition; since the modal representation with low confidence in the spatial modal representation can be filtered according to the confidence score of the vector, the noise signal that misleads the final emotion recognition result can be filtered, thereby further improving the accuracy of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0018] Figure 1 It is one of the flow charts of the multimodal emotion recognition method provided by the present invention; Figure 2 This is the second flow chart of the multimodal emotion recognition method provided by the present invention; Figure 3 is a schematic diagram of the structure of a multimodal emotion recognition device provided by the present invention; Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0019] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0020] It should be noted that, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.
[0021] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0022] In order to facilitate a clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and order of execution.
[0023] The embodiments of the present application describe some exemplary embodiments for the purpose of explanation. It should be understood that the present application can be implemented in other ways that are not specifically shown in the drawings.
[0024] like Figure 1 As shown, the embodiment of the present application provides a multimodal emotion recognition method, which can be applied to a multimodal emotion recognition device. The multimodal emotion recognition method can include S101-S105: S101. A multimodal emotion recognition device obtains voice data and corresponding text data.
[0025] The above-mentioned voice data and text data include historical conversations and current conversations.
[0026] Optionally, after the multimodal emotion recognition device obtains the voice data and the corresponding text data, it can structure the text data so that it contains historical conversations and current conversations, while maintaining a context-free design for the voice data and only retaining the audio clip of the current conversation.
[0027] S102: The multimodal emotion recognition device encodes the voice data and the text data respectively to obtain an intermediate feature representation of each modal data.
[0028] Specifically, if Figure 2 As shown, the multimodal emotion recognition device can use a dedicated unimodal encoder (Transformer encoder) to encode speech data and text data of different modalities respectively, and retain the intermediate feature representation generated by the Transformer encoder as the unique feature of each input modality.
[0029] Optionally, the above Transformer encoder can be initialized with a pre-trained unimodal model to extract features of a specific modality and build an intra-modal interaction model. For example, the encoder for text data can be initialized with a BERT model, and the encoder for speech data can be initialized with a WavLM model.
[0030] S103: The multimodal emotion recognition device determines the emotion attribute information in the intermediate feature representation, and determines the spatial modal representation according to the emotion attribute information.
[0031] Optionally, the multimodal emotion recognition device can determine the emotional attribute information in the intermediate feature representation of each modal data through a query vector; project the speech data, the text data and the emotional attribute information into a unified query space to obtain a spatial modal representation.
[0032] Specifically, if Figure 2 As shown, the multimodal emotion recognition device can use a set of learnable query vectors to collect and condense the emotional attribute information of each modality in the middle layer, and the query vector will be used as the input of the multimodal fusion encoder. It should be noted that the input of the multimodal fusion encoder also includes the above-mentioned voice data and text data.
[0033] Afterwards, the multimodal emotion recognition device can use self-attention and cross-modal attention to calculate the data and query vectors from different modalities, project them into a unified query space, and obtain a spatial modal representation. Similar to the unimodal encoder, the query vectors can interact through self-attention. Subsequently, cross-modal attention is used to project the information from different modalities into a unified query space, which is similar to a shared multimodal space. In this way, the synergy can be maximized and the heterogeneity gap between different modalities can be minimized.
[0034] Optionally, the vector length of the query vector is smaller than the input length of the unimodal Transformer encoder, so that information can be condensed and computational complexity can be reduced.
[0035] S104. The multimodal emotion recognition device filters the modal representations with low confidence in the spatial modal representation according to the confidence scores of the vectors to obtain a target representation vector.
[0036] Optionally, the multimodal emotion recognition device may predict the confidence of the spatial modal representation through a confidence estimation network; and determine the spatial modal representation with a confidence greater than a preset mask threshold as the target representation vector.
[0037] Specifically, since not all modalities play an equal role in emotion recognition, instance-level dynamic modality-priority learning can be used to process spatial modal representations. The confidence estimation network can take the hidden state of the unimodal encoder of the i-th layer as input and predict a single scalar between 0 and 1. For each instance, if the attribute of a modality in the i-th layer is a key emotional clue, a high confidence score is output, otherwise a low confidence score is output. Considering that when the confidence score is close to 0, the modality can be considered unnecessary or even a noise signal that may mislead the final emotion recognition result, the low-confidence spatial modal representation can be filtered by a preset mask threshold. In this way, unnecessary modalities can be shielded and the degree of fusion of each modality can be dynamically controlled.
[0038] Optionally, the above confidence estimation network is located before the fusion layer.
[0039] S105. The multimodal emotion recognition device performs fusion processing on the target representation vector, and performs emotion recognition based on the fusion result.
[0040] Optionally, the multimodal emotion recognition device may perform a fusion process on the target representation vector through a fusion layer, and classify the fusion result through a classification layer of a multi-layer perceptron, thereby obtaining an emotion recognition classification result of the current conversation.
[0041] In an embodiment of the present application, since the emotional attribute information can be determined from the intermediate feature representation of each modal data, the emotional clues of different granularities from each modality can be utilized to effectively learn the contribution of each modality to emotion recognition, thereby improving the accuracy of emotion recognition; since the low-confidence modal representations in the spatial modal representation can be filtered according to the confidence score of the vector, the noise signals that mislead the final emotion recognition results can be filtered, thereby further improving the accuracy of emotion recognition.
[0042] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiment disclosed herein, the embodiment of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.
[0043] The multimodal emotion recognition method provided in the embodiment of the present application can be executed by a multimodal emotion recognition device or a control module for multimodal emotion recognition in the multimodal emotion recognition device. In the embodiment of the present application, the multimodal emotion recognition device performing the multimodal emotion recognition method is taken as an example to illustrate the multimodal emotion recognition device provided in the embodiment of the present application.
[0044] It should be noted that the embodiment of the present application can divide the functional modules of the multimodal emotion recognition device according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. Optionally, the division of modules in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.
[0045] like Figure 3As shown, an embodiment of the present application provides a multimodal emotion recognition device 300. The multimodal emotion recognition device 300 includes: an acquisition module 301 and a processing module 302. The acquisition module 301 can be used to acquire voice data and corresponding text data; the processing module 302 is used to encode the voice data and the text data respectively to obtain an intermediate feature representation of each modal data; determine the emotional attribute information in the intermediate feature representation, and determine the spatial modal representation according to the emotional attribute information; filter the modal representation with low confidence in the spatial modal representation according to the confidence score of the vector to obtain a target representation vector; perform fusion processing on the target representation vector, and perform emotion recognition based on the fusion result.
[0046] Optionally, the processing module 302 is used to determine the emotional attribute information in the intermediate feature representation of each modal data through the query vector; project the speech data, the text data and the emotional attribute information into a unified query space to obtain a spatial modal representation.
[0047] Optionally, the processing module 302 is used to predict the confidence of the spatial modal representation through a confidence estimation network; and determine the spatial modal representation with a confidence greater than a preset mask threshold as the target representation vector.
[0048] Optionally, the voice data and the text data include historical conversations and current conversations, and the processing module 302 is used to perform fusion processing on the target representation vector and perform emotion recognition on the current conversation based on the fusion result.
[0049] In an embodiment of the present application, since the emotional attribute information can be determined from the intermediate feature representation of each modal data, the emotional clues of different granularities from each modality can be utilized to effectively learn the contribution of each modality to emotion recognition, thereby improving the accuracy of emotion recognition; since the low-confidence modal representations in the spatial modal representation can be filtered according to the confidence score of the vector, the noise signals that mislead the final emotion recognition results can be filtered, thereby further improving the accuracy of emotion recognition.
[0050] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communications interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute a multimodal emotion recognition method, which includes: acquiring speech data and corresponding text data; encoding the speech data and the text data respectively to obtain an intermediate feature representation of each modal data; determining the emotional attribute information in the intermediate feature representation, and determining the spatial modal representation according to the emotional attribute information; filtering the modal representation with low confidence in the spatial modal representation according to the confidence score of the vector to obtain a target representation vector; fusing the target representation vector, and performing emotion recognition based on the fusion result.
[0051] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0052] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multimodal emotion recognition method provided by the above-mentioned methods, which includes: acquiring speech data and corresponding text data; encoding the speech data and the text data respectively to obtain an intermediate feature representation of each modal data; determining the emotional attribute information in the intermediate feature representation, and determining the spatial modal representation based on the emotional attribute information; filtering the low-confidence modal representation in the spatial modal representation according to the confidence score of the vector to obtain a target representation vector; fusing the target representation vector, and performing emotion recognition based on the fusion result.
[0053] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multimodal emotion recognition method provided by the above-mentioned methods, the method comprising: acquiring speech data and corresponding text data; encoding the speech data and the text data respectively to obtain an intermediate feature representation of each modal data; determining the emotion attribute information in the intermediate feature representation, and determining the spatial modal representation based on the emotion attribute information; filtering the modal representations with low confidence in the spatial modal representation according to the confidence score of the vector to obtain a target representation vector; fusing the target representation vector, and performing emotion recognition based on the fusion result.
[0054] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0055] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A multimodal emotion recognition method, characterized in that: include: Obtaining voice data and corresponding text data; Encoding the speech data and the text data respectively to obtain an intermediate feature representation of each modality data; Determining emotional attribute information in the intermediate feature representation, and determining a spatial modal representation based on the emotional attribute information; filtering the modal representations with low confidence in the spatial modal representation according to the confidence scores of the vectors to obtain a target representation vector; The target representation vector is fused, and emotion recognition is performed based on the fusion result.
2. The multimodal emotion recognition method according to claim 1, characterized in that: The determining of the emotional attribute information in the intermediate feature representation and determining the spatial modality representation according to the emotional attribute information includes: Determine the sentiment attribute information in the intermediate feature representation of each modality data through the query vector; The speech data, the text data and the emotional attribute information are projected into a unified query space to obtain a spatial modality representation.
3. The multimodal emotion recognition method according to claim 1 or 2, characterized in that: The filtering of the modal representations with low confidence in the spatial modal representation according to the confidence score of the vector to obtain the target representation vector includes: Predicting the confidence of the spatial modal representation through a confidence estimation network; The spatial modality representation whose confidence is greater than a preset mask threshold is determined as the target representation vector.
4. The multimodal emotion recognition method according to claim 1, characterized in that: The speech data and the text data include historical conversations and current conversations, and the target representation vector is fused and emotion recognition is performed based on the fusion result, including: The target representation vector is fused, and emotion recognition is performed on the current conversation based on the fusion result.
5. A multimodal emotion recognition device, characterized in that: include: Acquisition module and processing module; The acquisition module is used to acquire voice data and corresponding text data; The processing module is used to encode the voice data and the text data respectively to obtain an intermediate feature representation of each modality data; Determine the emotional attribute information in the intermediate feature representation, and determine the spatial modal representation based on the emotional attribute information; filter the modal representations with low confidence in the spatial modal representation according to the confidence score of the vector to obtain a target representation vector; fuse the target representation vector, and perform emotion recognition based on the fusion result.
6. The multimodal emotion recognition device according to claim 5, characterized in that: The processing module is used to determine the emotional attribute information in the intermediate feature representation of each modal data through a query vector; project the speech data, the text data and the emotional attribute information into a unified query space to obtain a spatial modal representation.
7. The multimodal emotion recognition device according to claim 5 or 6, characterized in that: The processing module is used to predict the confidence of the spatial modal representation through a confidence estimation network; and determine the spatial modal representation with a confidence greater than a preset mask threshold as the target representation vector.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the multimodal emotion recognition method as described in any one of claims 1 to 4 is implemented.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multimodal emotion recognition method according to any one of claims 1 to 4 is implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multimodal emotion recognition method according to any one of claims 1 to 4 is implemented.
