Emotion recognition method, emotion recognition model training method, emotion recognition model training device, emotion recognition equipment and medium
By using a comparative learning loss function for cross-modal interaction and feature relationship learning in multimodal emotion recognition, the problem of insufficient accuracy of emotion recognition in traditional technology is solved, and a higher accuracy rate is achieved.
Patent Information
- Application Number
- CN202411764538.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-03
- Publication Date
- 2025-03-25
AI Technical Summary
In traditional multimodal emotion recognition technology, the accuracy of emotion recognition results needs to be improved, mainly because the relationship between the characteristics of emotional data of the same category and the differences between the characteristics of different categories of emotional data.
A emotion recognition model training method is adopted to obtain video, text and audio training data, process and extract feature vectors separately, perform feature dimensionality reduction and joint feature space modeling, and use contrast learning loss functions (including semi-contrast, intra-modal and inter-modal comparison losses) to perform cross-modal interaction and feature relationship learning.
By fully cross-modal interaction and learning the relationship between emotional data of different modalities, the modal gap is reduced and the accuracy of emotional recognition results is improved.
Smart Images
Figure CN119691675A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to an emotion recognition method and its model training method, device, equipment and medium. Background Art
[0002] Multimodal Emotion Recognition (MER) is the automatic recognition of the speaker's emotions. It is widely used in the field of intelligent voice customer service. When a customer has a conversation with an intelligent customer service, the algorithm can automatically identify the customer's emotional state and provide corresponding suggestions and guidance to the intelligent customer service system to improve the customer experience. For example, if a customer shows negative emotions in the conversation, the intelligent customer service can automatically recommend some solutions for that emotion, such as providing better customer support or providing better product services. However, multimodal emotion recognition mostly uses traditional contrastive learning, which forcibly maps the feature representations of emotion data of different modalities into a feature space, and the most commonly used unsupervised contrastive learning completely ignores the connection between features belonging to the same category of emotion data and the differences between features of different categories of emotion data. Summary of the invention
[0003] The main technical problem to be solved by the implementation methods of the present application is that the accuracy of emotion recognition results obtained by traditional multimodal emotion recognition technology needs to be improved.
[0004] To solve the above technical problems, the first technical solution adopted in the implementation mode of the present application is: to provide an emotion recognition model training method, including: obtaining video training data, extracting a video frame sequence of a video modality, as well as text training data of a text modality and audio training data of an audio modality from the video training data; processing the video frame sequence, the text training data and the audio training data respectively through the video processing module, the text processing module and the audio processing module of the emotion recognition model to be trained, and obtaining a first video frame feature, a first text feature vector and a first audio feature vector respectively; performing feature dimensionality reduction processing on the first video frame feature vector, the first text feature vector and the first audio feature vector through the fully connected layer of the emotion recognition model to be trained, and obtaining a second video frame feature vector, a second text feature vector and a second audio feature vector respectively; performing joint feature space modeling through a preset number of contrastive learning loss functions, and processing the second video frame feature vector, the second text feature vector and the second audio feature vector through the joint feature space obtained by modeling, and obtaining an emotion recognition multimodal feature vector; converting the emotion recognition multimodal feature vector into an emotion recognition result corresponding to the emotion recognition model to be trained.
[0005] Optionally, the contrastive learning loss function includes a semi-contrastive loss function, and the step of modeling by the semi-contrastive loss function includes: arbitrarily selecting one of the second video frame feature vector, the second text feature vector and the second audio feature vector as an anchor sample vector, and setting the remaining feature vectors as a positive example set; normalizing the second video frame feature vector, the second text feature vector and the second audio feature vector to obtain a probability distribution of each dimensional feature; the calculation formula of the semi-contrastive loss function is:
[0006]
[0007] Among them, L SCL represents the semi-contrastive loss, E represents the expectation operator, S represents the positive example set, and a m represents the anchor point sample vector, represents the vector in the positive example set, α represents the modal boundary between different modalities, m is represented as a triple, l represents the anchor sample, a represents the positive sample, and v represents the negative sample.
[0008] Optionally, the contrastive learning loss function is an intra-modal contrastive loss function, and the step of modeling by the intra-modal contrastive loss function includes: obtaining two different feature vectors of the same modal data and the same emotion category as a positive example pair; obtaining two different feature vectors of the same modal data and different emotion categories as a negative example pair; the calculation formula of the intra-modal contrastive loss function is:
[0009]
[0010] Among them, L IAMCL represents the intra-modal contrast loss, E represents the expected operator, S represents the positive example pair, and a m represents the anchor vector, represents the vector in the positive example pair, Represents the vector in the positive example pair, N is the number of positive example pairs, and M is the number of negative example pairs.
[0011] Optionally, the contrastive learning loss function is an inter-modal contrastive learning loss, and the step of modeling by the inter-modal contrastive loss function includes: obtaining two different feature vectors of different modal data and the same emotion category as a positive example pair; obtaining two different feature vectors of the same modal data and different emotion categories as a negative example pair; the calculation formula of the inter-modal contrastive loss function is:
[0012]
[0013] Among them, L IEMCL represents the inter-modal contrast loss, E represents the expected operator, S represents the positive example pair, and a m represents the anchor point vector, pi Represents the vector in the positive example pair, n j Represents the vector in the positive example pair, N is the number of positive example pairs, and M is the number of negative example pairs.
[0014] Optionally, the contrastive learning loss function also includes a cross entropy loss function; the calculation formula of the contrastive learning loss function is as follows:
[0015] L=L SCL +L IAMCL +L IEMCL +L CE
[0016] Among them, L represents the contrastive learning loss, L SCL represents the semi-contrastive learning loss, L IAMCL represents the intra-modal contrast loss, L IEMCL represents the inter-modality contrast loss, L CE represents the cross entropy loss.
[0017] Optionally, after the step of converting the emotion recognition multimodal feature vector into the emotion recognition result corresponding to the emotion recognition model to be trained, it also includes: sending the emotion recognition result to manual verification, receiving the emotion recognition verification result returned by the manual verification; and optimizing the contrastive learning loss function and the parameters of the fully connected layer according to the emotion recognition verification result.
[0018] To solve the above technical problems, the second technical solution adopted in the implementation mode of the present application is: to provide an emotion recognition method, including: constructing an initial emotion recognition model, training the initial emotion recognition model through the emotion recognition model training method as described above, to obtain a target emotion recognition model; receiving video data to be recognized, sending the video data to be recognized to the target emotion recognition model, and obtaining corresponding emotion recognition results.
[0019] In order to solve the above technical problems, the third technical solution adopted by the implementation mode of the present application is: to provide an emotion recognition model training device, including: a video training data module, used to obtain video training data, extract the video frame sequence of the video modality in the video training data, as well as the text training data of the text modality and the audio training data of the audio modality; a feature vector extraction module, used to respectively process the video frame sequence, the text training data and the audio training data through the video processing module, the text processing module and the audio processing module of the emotion recognition model to be trained, and respectively obtain the first video frame feature, the first text feature vector and the first audio feature vector; a fully connected layer module, used to process the video frame sequence, the text training data and the audio training data through the video processing module, the text processing module and the audio processing module of the emotion recognition model to be trained The fully connected layer of the emotion recognition model performs feature dimensionality reduction processing on the first video frame feature vector, the first text feature vector and the first audio feature vector, and obtains the second video frame feature vector, the second text feature vector and the second audio feature vector respectively; the multimodal feature vector module is used to perform joint feature space modeling through a preset number of contrastive learning loss functions, and processes the second video frame feature vector, the second text feature vector and the second audio feature vector through the joint feature space obtained by modeling to obtain an emotion recognition multimodal feature vector; the emotion recognition result module is used to convert the emotion recognition multimodal feature vector into an emotion recognition result corresponding to the emotion recognition model to be trained.
[0020] In order to solve the above technical problems, the fourth technical solution adopted by the embodiment of the present application is: to provide an electronic device, including:
[0021] at least one processor; and,
[0022] a memory communicatively connected to the at least one processor; wherein,
[0023] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.
[0024] In order to solve the above technical problems, the fourth technical solution adopted in the implementation mode of the present application is: providing a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by an electronic device, the electronic device executes the method as described above.
[0025] Different from the related art, the present application simultaneously performs contrastive learning and semi-contrastive learning within and / or between modalities, so that the obtained emotion prediction model can fully interact across modalities, learn the relationship between emotion data of different modalities, reduce the gap between emotion data of different modalities, and improve the accuracy of the obtained emotion recognition results. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] One or more embodiments are exemplarily described by corresponding drawings, which do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and the figures in the drawings do not constitute proportional limitations unless otherwise stated.
[0027] Figure 1 It is a schematic diagram of the operating environment of the emotion recognition model training method provided in an embodiment of the present application.
[0028] Figure 2 It is a schematic diagram of the execution flow of the emotion recognition model training method provided in an embodiment of the present application.
[0029] Figure 3 It is a schematic diagram of the execution flow of manual verification in the emotion recognition model training method provided in the embodiment of the present application.
[0030] Figure 4 It is a schematic diagram of the execution flow of the emotion recognition method provided in the embodiment of the present application.
[0031] Figure 5 It is a system structure diagram of the emotion recognition model training device provided in an embodiment of the present application.
[0032] Figure 6 It is a schematic diagram of the hardware structure of an electronic device for executing the emotion recognition model training method and / or emotion recognition method provided in an embodiment of the present application. DETAILED DESCRIPTION
[0033] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0034] It should be noted that, if there is no conflict, the various features in the embodiments of the present application can be combined with each other, all within the scope of protection of the present application. In addition, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a different order than the module division in the device schematic diagram or the order in the flow chart.
[0035] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used in this specification includes any and all combinations of one or more of the related listed items.
[0036] To facilitate understanding of this embodiment, firstly, a method for training an emotion recognition model disclosed in the embodiment of this application is introduced in detail. Figure 1 , Figure 1 Schematic diagram of the operating environment of the emotion recognition model training method provided in the embodiment of the present application. Figure 1 As shown, the executor of the emotion recognition model training method provided in the embodiment of the present application is generally an electronic device with certain computing capabilities, such as a computer device. In some possible implementations, the emotion recognition model training method can be implemented by a processor calling a computer-readable instruction stored in a memory. Figure 1 The computer device in the description can be a server. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. It can be understood that Figure 1 The number of computer devices in the figure is only for reference and can be expanded to any number according to actual needs.
[0037] Please continue reading Figure 2 , Figure 2 is a schematic diagram of the execution flow of the emotion recognition model training method provided in the embodiment of the present application, such as Figure 2 As shown, the following steps are included:
[0038] S11. Obtain video training data, and extract a video frame sequence of a video modality, text training data of a text modality, and audio training data of an audio modality from the video training data.
[0039] S12. Process the video frame sequence, text training data and audio training data respectively through the video processing module, text processing module and audio processing module of the emotion recognition model to be trained, and obtain the first video frame feature, the first text feature vector and the first audio feature vector respectively.
[0040] As an optional implementation, the video processing module in the above step S12 can be built based on the SlowFast model, which is a deep learning model architecture for video understanding and action recognition, proposed by Facebook AI. The model is mainly intended to solve the problem of time scale differences in videos, that is, some actions in the video may occur very quickly (Fast), while others may occur slowly (Slow). The core idea of the SlowFast model is to process video frame sequences through two independent paths: a slower path (Slow Path) is used to capture background information and global structure of motion in the video, while a faster path (Fast Path) focuses on capturing fast-changing motion details. This design enables the model to better understand information at different time scales in the video. The architecture of the SlowFast model involves two main parts: Slow Path is a slower path, which usually samples fewer frames and is used to capture background information and global motion features in the video. Fast Path is a faster path, which usually samples more frames and is used to capture fast motion details in the video. By integrating information from these two paths, the SlowFast model can understand the video content more comprehensively and improve the performance of video understanding and action recognition.
[0041] As an optional implementation, the text processing module in the above step S12 can be constructed based on the BERT (Bidirectional Encoder Representations From Transformers, BERT) model, which is a pre-trained language model developed by Google. The BERT model is based on the Transformer architecture, which includes a Transformer encoder. BERT is a bidirectional model that can simultaneously consider contextual information. Compared with traditional unidirectional language models, it can better understand the context in the text. BERT can understand the contextual information in the sentence, improve the ability to understand the context, can be transferred to various NLP tasks, and greatly reduce the training data requirements for specific tasks.
[0042] As an optional implementation, the audio processing model in the above step S12 can be built based on the Wav2vec model, which is a pre-trained model for speech processing proposed by Facebook AI. The main goal of this model is to learn to extract speech features from raw audio data to improve the performance of speech recognition and speech understanding tasks. The Wav2vec model is based on the ideas of self-supervised learning and semi-supervised learning, combining convolutional neural networks (CNNs) and Transformer models. It uses the following main components: the feature extractor uses a CNN model to extract high-level features from the raw audio waveform, and the pre-trained encoder inputs the extracted feature sequence into the Transformer encoder to learn the representation of the audio data. Wav2vec uses self-supervised learning to learn useful speech representations from raw audio data. It does not require labeled training data, can extract high-quality speech features, and provide better input representations for subsequent tasks.
[0043] S13. Perform feature dimensionality reduction processing on the first video frame feature vector, the first text feature vector, and the first audio feature vector through the fully connected layer of the emotion recognition model to be trained, and obtain a second video frame feature vector, a second text feature vector, and a second audio feature vector respectively.
[0044] S14. Perform joint feature space modeling through a preset number of contrastive learning loss functions, and process the second video frame feature vector, the second text feature vector, and the second audio feature vector through the joint feature space obtained by modeling to obtain an emotion recognition multimodal feature vector.
[0045] As an optional implementation, the contrastive learning loss function in step S14 includes a semi-contrastive loss function, and the step of modeling by the semi-contrastive loss function includes: arbitrarily selecting one of the second video frame feature vector, the second text feature vector, and the second audio feature vector as an anchor sample vector, and setting the remaining feature vectors as a positive example set. Then, the second video frame feature vector, the second text feature vector, and the second audio feature vector are normalized to obtain the probability distribution of each dimensional feature. The calculation formula of the semi-contrastive loss function is:
[0046]
[0047] Among them, L SCL represents the semi-contrastive loss, E represents the expectation operator, S represents the positive example set, and a m represents the anchor point sample vector, p i m Represents the vector in the positive example set, α represents the modal boundary between different modalities, m is represented as a triple, l represents the anchor sample, a represents the positive sample, and v represents the negative sample.
[0048] As an optional implementation, the contrast learning loss function in step S14 is an intra-modal contrast loss function, and the steps of modeling by the intra-modal contrast loss function include: first, obtaining two different feature vectors of the same modal data and the same emotion category as positive example pairs. Then, obtaining two different feature vectors of the same modal data and different emotion categories as negative example pairs. The calculation formula of the intra-modal contrast loss function is:
[0049]
[0050] Among them, L IAMCL represents the intra-modal contrast loss, E represents the expected operator, S represents the positive example pair, and a m represents the anchor vector, represents the vector in the positive pair, Represents the vector in the positive example pair, N is the number of positive example pairs, and M is the number of negative example pairs.
[0051] As an optional implementation, the contrastive learning loss function in the above step S14 is an inter-modal contrastive learning loss, and the steps of modeling by the inter-modal contrastive loss function include: first, obtaining two different feature vectors of different modal data and the same emotion category as positive example pairs. Then, obtaining two different feature vectors of the same modal data and different emotion categories as negative example pairs. Finally, the calculation formula of the inter-modal contrastive loss function is:
[0052]
[0053] Among them, L IEMCL represents the inter-modal contrast loss, E represents the expected operator, S represents the positive example pair, and a m represents the anchor point vector, p i Represents the vector in the positive example pair, n j Represents the vector in the positive example pair, N is the number of positive example pairs, and M is the number of negative example pairs.
[0054] As a preferred implementation, the contrastive learning loss function in step S14 also includes a cross entropy loss function. The calculation formula of the contrastive learning loss function is as follows:
[0055] L=L SCL +L IAMCL +L IEMCL +L CE
[0056] Among them, L represents the contrastive learning loss, L SCL represents the semi-contrastive learning loss, L IAMCL represents the intra-modal contrast loss, L IEMCL represents the inter-modality contrast loss, L CErepresents the cross entropy loss.
[0057] S15. Convert the emotion recognition multimodal feature vector into an emotion recognition result corresponding to the emotion recognition model to be trained.
[0058] Please continue reading Figure 3 , Figure 3 is a schematic diagram of the execution flow of manual verification in the emotion recognition model training method provided in the embodiment of the present application, such as Figure 3 As shown, the manual verification process may include the following steps S16 to S17.
[0059] S16. Send the emotion recognition result to manual verification, and receive the emotion recognition verification result returned by the manual verification.
[0060] S17. Optimize the parameters of the contrastive learning loss function and the fully connected layer according to the emotion recognition verification results.
[0061] Please continue reading Figure 4 , Figure 4 is a schematic diagram of the execution flow of the emotion recognition method provided in the embodiment of the present application, such as Figure 4 As shown, the emotion recognition method includes the following steps S21 to S22.
[0062] S21. Construct an initial emotion recognition model, train the initial emotion recognition model through the emotion recognition model training method, and obtain a target emotion recognition model.
[0063] S22, receiving the video data to be identified, sending the video data to be identified to the target emotion recognition model, and obtaining the corresponding emotion recognition result.
[0064] The embodiment of the present application provides a method for converting an emotion recognition multimodal feature vector into an emotion recognition result corresponding to an emotion recognition model to be trained. By acquiring video training data, a video frame sequence of a video modality, as well as text training data of a text modality and audio training data of an audio modality are extracted from the video training data; the video frame sequence, the text training data and the audio training data are processed by the video processing module, the text processing module and the audio processing module of the emotion recognition model to be trained respectively, and a first video frame feature, a first text feature vector and a first audio feature vector are obtained respectively; feature dimensionality reduction processing is performed on the first video frame feature vector, the first text feature vector and the first audio feature vector by the fully connected layer of the emotion recognition model to be trained, and a second video frame feature vector, a first text feature vector and a first audio feature vector are obtained respectively. feature vector, a second text feature vector and a second audio feature vector; a joint feature space is modeled through a preset number of contrastive learning loss functions, and the second video frame feature vector, the second text feature vector and the second audio feature vector are processed through the joint feature space obtained by modeling to obtain a multimodal feature vector for emotion recognition; the multimodal feature vector for emotion recognition is converted into an emotion recognition result corresponding to the emotion recognition model to be trained, and intra-modal / inter-modal contrastive learning and semi-contrastive learning are performed at the same time. Through these two types of learning, the model can fully explore cross-modal interactions, learn the relationships between samples and classes, and reduce the modal gap. A modal boundary term is introduced to better learn the feature pairs of the same modality, and ultimately the accuracy of the obtained emotion recognition results can be improved through multimodal data.
[0065] Please continue reading Figure 5 , Figure 5 is a schematic diagram of the system structure of the emotion recognition model training device provided in the embodiment of the present application, such as Figure 5 As shown, the emotion recognition model training device 50 includes: a video training data module 51, a feature vector extraction module 52, a fully connected layer module 53, a multimodal feature vector module 54 and an emotion recognition result module 55.
[0066] The video training data module 51 is used to obtain video training data, extract video frame sequences of video modality, text training data of text modality and audio training data of audio modality from the video training data;
[0067] The feature vector extraction module 52 is used to process the video frame sequence, text training data and audio training data through the video processing module, text processing module and audio processing module of the emotion recognition model to be trained, respectively, to obtain the first video frame feature, the first text feature vector and the first audio feature vector respectively;
[0068] The fully connected layer module 53 is used to perform feature dimension reduction processing on the first video frame feature vector, the first text feature vector and the first audio feature vector through the fully connected layer of the emotion recognition model to be trained, and obtain the second video frame feature vector, the second text feature vector and the second audio feature vector respectively;
[0069] The multimodal feature vector module 54 is used to perform joint feature space modeling through a preset number of contrastive learning loss functions, and process the second video frame feature vector, the second text feature vector and the second audio feature vector through the joint feature space obtained by modeling to obtain an emotion recognition multimodal feature vector;
[0070] The emotion recognition result module 55 is used to convert the emotion recognition multimodal feature vector into the emotion recognition result corresponding to the emotion recognition model to be trained as an optional implementation mode, which is specifically used for.
[0071] As an optional implementation, the multimodal feature vector module 54 is further specifically used to arbitrarily select one of the second video frame feature vector, the second text feature vector and the second audio feature vector as an anchor sample vector, and set the remaining feature vectors as a positive example set; normalize the second video frame feature vector, the second text feature vector and the second audio feature vector to obtain a probability distribution of each dimensional feature; the calculation formula of the semi-contrastive loss function is:
[0072]
[0073] Among them, L SCL represents the semi-contrastive loss, E represents the expectation operator, S represents the positive example set, and a m represents the anchor point sample vector, represents the vector in the positive example set, α represents the modal boundary between different modalities, m is represented as a triple, l represents the anchor sample, a represents the positive sample, and v represents the negative sample.
[0074] As an optional implementation, the multimodal feature vector module 54 is further specifically used to obtain two different feature vectors of the same modal data and the same emotion category as a positive example pair; obtain two different feature vectors of the same modal data and different emotion categories as a negative example pair; the calculation formula of the intra-modal contrast loss function is:
[0075]
[0076] Among them, L IAMCL represents the intra-modal contrast loss, E represents the expected operator, S represents the positive example pair, and a m represents the anchor vector, represents the vector in the positive example pair, Represents the vector in the positive example pair, N is the number of positive example pairs, and M is the number of negative example pairs.
[0077] As an optional implementation, the multimodal feature vector module 54 is further specifically used to obtain two different feature vectors of different modal data and the same emotion category as a positive example pair; obtain two different feature vectors of the same modal data and different emotion categories as a negative example pair; the calculation formula of the inter-modal contrast loss function is:
[0078]
[0079] Among them, L IEMCL represents the inter-modal contrast loss, E represents the expected operator, S represents the positive example pair, and a m represents the anchor point vector, p i Represents the vector in the positive example pair, n j Represents the vector in the positive example pair, N is the number of positive example pairs, and M is the number of negative example pairs.
[0080] As an optional implementation, the multimodal feature vector module 54 is also specifically used for calculating the contrastive learning loss function as follows:
[0081] L=L SCL +L IAMCL +L IEMCL +L CE
[0082] Among them, L represents the contrastive learning loss, L SCL represents the semi-contrastive learning loss, L IAMCL represents the intra-modal contrast loss, L IEMCL represents the inter-modality contrast loss, L CE represents the cross entropy loss.
[0083] As an optional implementation, the emotion recognition model training device also includes a manual verification module, which is specifically used to send the emotion recognition result to manual verification, receive the emotion recognition verification result returned by the manual verification; and optimize the contrastive learning loss function and the parameters of the fully connected layer according to the emotion recognition verification result.
[0084] The embodiment of the present application also provides an emotion recognition device, which includes: an emotion recognition model construction module and a video data recognition module. The emotion recognition model construction module is used to construct an initial emotion recognition model, and the initial emotion recognition model is trained by the emotion recognition model training method in the method embodiment to obtain a target emotion recognition model. The video data recognition module is used to receive video data to be recognized, send the video data to be recognized to the target emotion recognition model, and obtain the corresponding emotion recognition result.
[0085] It should be noted that the above-mentioned emotion recognition model training device and / or emotion recognition device can execute the emotion recognition model training method and / or emotion recognition method provided in the embodiments of the present application, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not fully described in the embodiments of the emotion recognition model training device and / or emotion recognition device, please refer to the emotion recognition model training method and / or emotion recognition method provided in the embodiments of the present application.
[0086] Please continue reading Figure 6 , Figure 6 is a schematic diagram of the hardware structure of an electronic device for executing an emotion recognition model training method or an emotion recognition method provided in an embodiment of the present application, such as Figure 6 As shown, the electronic device 600 includes:
[0087] One or more processors 610 and memory 620, Figure 6 A processor 610 is taken as an example.
[0088] The processor 610 and the memory 620 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.
[0089] The memory 620, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the program instructions / modules corresponding to the emotion recognition model training method and / or emotion recognition method in the embodiments of the present application. The processor 610 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 620, that is, implementing the emotion recognition model training method and / or emotion recognition method in the above method embodiments.
[0090] The memory 620 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created according to the use of the emotion recognition model training device and / or the emotion recognition transposition, etc. In addition, the memory 620 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 620 may optionally include a memory remotely arranged relative to the processor 610, and these remote memories may be connected to the emotion recognition model training device and / or the emotion recognition device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0091] The one or more modules are stored in the memory 620, and when executed by the one or more processors 610, the emotion recognition model training method and / or emotion recognition method in any of the above method embodiments are executed, for example, the emotion recognition model training method and / or emotion recognition method described above are executed. Figure 2 Steps S11 to S15 of the method, Figure 3 Steps S16 to S17 of the method, Figure 4 Steps S21 to S22 of the method are implemented Figure 5 The functions of modules 51-55 in.
[0092] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.
[0093] The embodiment of the present application provides a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors, such as Figure 6 A processor 610 in the embodiment may enable the one or more processors to execute the emotion recognition model training method and / or emotion recognition method in any of the above method embodiments, for example, to execute the above described Figure 2 Steps S11 to S15 of the method, Figure 3 Steps S16 to S17 of the method, Figure 4 Steps S21 to S22 of the method are implemented Figure 5 The functions of modules 51-55 in.
[0094] The present application embodiment provides a computer program product, the computer program product includes a computer program stored on a non-volatile computer-readable storage medium, the computer program includes program instructions, when the program instructions are executed by the electronic device, the electronic device is enabled to execute the emotion recognition model training method and / or emotion recognition method in any of the above method embodiments, for example, to execute the above described Figure 2 Steps S11 to S15 of the method, Figure 3 Steps S16 to S17 of the method, Figure 4 Steps S21 to S22 of the method are implemented Figure 5 The functions of modules 51-55 in.
[0095] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0096] Through the description of the above implementation methods, those of ordinary skill in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.
[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Under the concept of the present application, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes in different aspects of the present application as described above, which are not provided in detail for the sake of simplicity. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.
Claims
1. A method for training an emotion recognition model, characterized in that: include: Acquire video training data, and extract a video frame sequence of a video modality, text training data of a text modality, and audio training data of an audio modality from the video training data; The video frame sequence, the text training data and the audio training data are processed respectively by a video processing module, a text processing module and an audio processing module of the emotion recognition model to be trained, and a first video frame feature, a first text feature vector and a first audio feature vector are obtained respectively; Performing feature dimensionality reduction processing on the first video frame feature vector, the first text feature vector, and the first audio feature vector through the fully connected layer of the emotion recognition model to be trained, so as to obtain a second video frame feature vector, a second text feature vector, and a second audio feature vector respectively; Performing joint feature space modeling through a preset number of contrastive learning loss functions, and processing the second video frame feature vector, the second text feature vector, and the second audio feature vector through the joint feature space obtained by modeling to obtain an emotion recognition multimodal feature vector; The emotion recognition multimodal feature vector is converted into an emotion recognition result corresponding to the emotion recognition model to be trained.
2. The emotion recognition model training method according to claim 1, characterized in that: The contrastive learning loss function includes a semi-contrastive loss function, and the step of modeling by using the semi-contrastive loss function includes: arbitrarily selecting one of the second video frame feature vector, the second text feature vector, and the second audio feature vector as an anchor sample vector, and setting the remaining feature vectors as a positive example set; Normalizing the second video frame feature vector, the second text feature vector, and the second audio feature vector to obtain a probability distribution of each dimensional feature; The calculation formula of the semi-contrast loss function is: Among them, L SCL represents the semi-contrastive loss, E represents the expectation operator, S represents the positive example set, and a m represents the anchor point sample vector, p i m represents the vector in the positive example set, α represents the modal boundary between different modalities, m is represented as a triple, l represents the anchor sample, a represents the positive sample, and v represents the negative sample.
3. The emotion recognition model training method according to claim 1, characterized in that: The contrastive learning loss function is an intra-modal contrastive loss function, and the steps of modeling by using the intra-modal contrastive loss function include: Obtain two different feature vectors of the same modality data and the same emotion category as positive example pairs; Obtain two different feature vectors of the same modality data but different emotion categories as negative example pairs; The calculation formula of the intra-modal contrast loss function is: Among them, L IAMCL represents the intra-modal contrast loss, E represents the expected operator, S represents the positive example pair, and a m represents the anchor point vector, p i m Represents the vector in the positive example pair, n j m Represents the vector in the positive example pair, N is the number of positive example pairs, and M is the number of negative example pairs.
4. The emotion recognition model training method according to claim 1, characterized in that: The contrastive learning loss function is an inter-modality contrastive learning loss, and the steps of modeling by using the inter-modality contrastive loss function include: Obtain two different feature vectors of different modal data and the same emotion category as positive example pairs; Obtain two different feature vectors of the same modality data but different emotion categories as negative example pairs; The calculation formula of the inter-modality contrast loss function is: Among them, L IEMCL represents the inter-modal contrast loss, E represents the expected operator, S represents the positive example pair, and a m represents the anchor point vector, p i Represents the vector in the positive example pair, n j Represents the vector in the positive example pair, N is the number of positive example pairs, and M is the number of negative example pairs.
5. The emotion recognition model training method according to claim 1, characterized in that: The contrastive learning loss function also includes a cross entropy loss function; The calculation formula of the contrastive learning loss function is as follows: L=L SCL +L IAMCL +L IEMCL +L CE Among them, L represents the contrastive learning loss, L SCL represents the semi-contrastive learning loss, L IAMCL represents the intra-modal contrast loss, L IEMCL represents the inter-modality contrast loss, L CE represents the cross entropy loss.
6. The emotion recognition model training method according to claim 1, characterized in that: After the step of converting the emotion recognition multimodal feature vector into the emotion recognition result corresponding to the emotion recognition model to be trained, the method further includes: Sending the emotion recognition result to manual verification, and receiving the emotion recognition verification result returned by the manual verification; The contrastive learning loss function and the parameters of the fully connected layer are optimized according to the emotion recognition verification result.
7. An emotion recognition method, characterized in that: include: Constructing an initial emotion recognition model, and training the initial emotion recognition model by the emotion recognition model training method according to any one of claims 1 to 6 to obtain a target emotion recognition model; Receive the video data to be recognized, send the video data to the target emotion recognition model, and obtain the corresponding emotion recognition result.
8. An emotion recognition model training device, characterized in that: include: A video training data module, used to obtain video training data, extract video frame sequences of video modality, text training data of text modality and audio training data of audio modality from the video training data; A feature vector extraction module is used to process the video frame sequence, the text training data and the audio training data through the video processing module, the text processing module and the audio processing module of the emotion recognition model to be trained, respectively, to obtain a first video frame feature, a first text feature vector and a first audio feature vector respectively; A fully connected layer module, used for performing feature dimensionality reduction processing on the first video frame feature vector, the first text feature vector and the first audio feature vector through the fully connected layer of the emotion recognition model to be trained, so as to obtain a second video frame feature vector, a second text feature vector and a second audio feature vector respectively; A multimodal feature vector module, used for performing joint feature space modeling through a preset number of contrastive learning loss functions, and processing the second video frame feature vector, the second text feature vector and the second audio feature vector through the joint feature space obtained by modeling to obtain an emotion recognition multimodal feature vector; The emotion recognition result module is used to convert the emotion recognition multimodal feature vector into the emotion recognition result corresponding to the emotion recognition model to be trained.
9. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 6 or claim 7.
10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by an electronic device, the electronic device executes the method of any one of claims 1 to 6 or claim 7.