Emotion recognition method, pre-training method for emotion recognition model, and electronic device

Abstract

The present disclosure relates to the fields of large model technology and data recognition. Disclosed are an emotion recognition method, a pre-training method for an emotion recognition model, and an electronic device. The emotion recognition method comprises: responding to query data received in a generative dialogue interface, wherein the query data comprises data of at least one of the following modalities: a visual modality, an acoustic mode, a text modality and a context modality; performing emotion recognition on the query data, so as to obtain an initial emotion recognition set, wherein the initial emotion recognition set comprises emotion recognition results corresponding to a plurality of emotion recognition tasks; and on the basis of a target emotion recognition task corresponding to the query data, determining a target emotion recognition result corresponding to the query data. The present disclosure solves the technical problem in the related art of the accuracy of the emotion recognition performed on data being low.

Description

Emotion recognition method, emotion recognition model pre-training method, and electronic device Technical Field

[0001] The present disclosure relates to the fields of large-scale model technology and data recognition, and more specifically, to an emotion recognition method, an emotion recognition model pre-training method, and an electronic device. Background Art

[0002] At present, in some scenarios, how to accurately identify user emotions is the key to improving user experience and satisfaction. By analyzing the user's emotions, we can understand the user's emotional state. However, there are still many challenges in the process of combining multiple data to process emotion recognition tasks. Due to the large gaps in the modality, input and output forms, etc. of user query data, the accuracy of emotion recognition of query data is low.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far.

[0004] Summary of the Invention

[0005] The embodiments of the present disclosure provide an emotion recognition method, an emotion recognition model pre-training method, and an electronic device to at least solve the technical problem of low accuracy in emotion recognition of data in related technologies.

[0006] According to one aspect of an embodiment of the present disclosure, a method for emotion recognition is provided, comprising: responding to query data received in a generative dialogue interface, wherein the query data includes data of at least one of the following modalities: visual modality, acoustic modality, textual modality, and contextual modality; performing emotion recognition on the query data to obtain an initial emotion recognition set, wherein the initial emotion recognition set includes emotion recognition results corresponding to multiple emotion recognition tasks; and determining a target emotion recognition result corresponding to the query data based on a target emotion recognition task corresponding to the query data.

[0007] According to one aspect of an embodiment of the present disclosure, a pre-training method for an emotion recognition model is also provided, including: obtaining a training data set corresponding to multiple emotion recognition tasks, wherein the training data in the training data set includes: multiple training samples, and an emotion label set corresponding to the training samples, and the emotion label set contains emotion labels corresponding to multiple emotion recognition tasks; based on the emotion categories corresponding to the emotion labels, dividing the training data set into multiple data sets, wherein the emotion categories corresponding to multiple training data contained in the same data set are the same; pre-training the initial recognition model based on the same data set to obtain a target recognition model, wherein the target recognition model is used to identify the emotion category corresponding to the input data; pre-training the target recognition model based on the training data set to obtain an emotion recognition model, wherein the emotion recognition model is used to perform emotion recognition on query data, and obtaining an emotion recognition set of the query data, and the emotion recognition set includes emotion recognition results corresponding to multiple emotion recognition tasks.

[0008] According to one aspect of an embodiment of the present disclosure, an emotion recognition method is also provided, including: obtaining data to be recognized, wherein the data to be recognized includes data of at least one of the following modalities: visual modality, acoustic modality, text modality and contextual modality; performing emotion recognition on the data to be recognized to obtain an initial emotion recognition set, wherein the initial emotion set includes emotion recognition results corresponding to multiple different emotion recognition tasks; and determining a target emotion recognition result corresponding to the data to be recognized based on a target emotion recognition task corresponding to the data to be recognized.

[0009] According to one aspect of an embodiment of the present disclosure, a data generation method is also provided, including: responding to query data received in a generative dialogue interface, wherein the query data includes data of at least one of the following modalities: visual modality, acoustic modality, text modality, and contextual modality; performing emotion recognition on the query data to obtain an initial emotion set corresponding to the query data, wherein the initial emotion set includes emotion recognition results corresponding to multiple different emotion recognition tasks; determining a target emotion recognition result corresponding to the query data based on a target emotion recognition task corresponding to the query data; generating a reply result matching the query data based on the target emotion recognition result; and displaying the reply result in the generative dialogue interface.

[0010] According to one aspect of an embodiment of the present disclosure, a data generation method is also provided, including: obtaining query data input in a generative dialogue interface in a dialogue mode, wherein the query data includes data of at least one of the following modalities: visual modality, acoustic modality, text modality, and contextual modality; performing emotion recognition on the query data to obtain an initial emotion set corresponding to the query data, wherein the initial emotion set includes emotion recognition results corresponding to multiple different emotion recognition tasks; based on a target emotion recognition task corresponding to the query data, determining a target emotion recognition result corresponding to the query data; based on the target emotion recognition result, generating a reply result that matches the query data.

[0011] According to one aspect of an embodiment of the present disclosure, an emotion recognition method is also provided, including: obtaining query data by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the query data, and the query data includes data of at least one of the following modalities: visual modality, acoustic modality, text modality and contextual modality; performing emotion recognition on the query data to obtain an initial emotion recognition set, wherein the initial emotion recognition set includes emotion recognition results corresponding to multiple emotion recognition tasks; based on the target emotion recognition task corresponding to the query data, determining the target emotion recognition result corresponding to the query data; outputting the target emotion recognition result by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the target emotion recognition result.

[0012] According to one aspect of an embodiment of the present disclosure, a data generation method is also provided, including: obtaining query data by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the query data, and the query data includes data of at least one of the following modalities: visual modality, acoustic modality, text modality and contextual modality; performing emotion recognition on the query data to obtain an initial emotion recognition set, wherein the initial emotion recognition set includes emotion recognition results corresponding to multiple emotion recognition tasks; based on the target emotion recognition task corresponding to the query data, determining the target emotion recognition result corresponding to the query data; based on the target emotion recognition result, generating a reply result matching the query data; outputting the reply result by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the reply result.

[0013] In an embodiment of the present disclosure, in response to query data received in a generative dialogue interface, wherein the query data includes data of at least one of the following modalities: visual modality, acoustic modality, text modality, and contextual modality; emotion recognition is performed on the query data to obtain an initial emotion recognition set, wherein the initial emotion recognition set includes emotion recognition results corresponding to multiple emotion recognition tasks; based on the target emotion recognition task corresponding to the query data, the target emotion recognition result corresponding to the query data is determined, thereby improving the accuracy of emotion recognition on the data; it is easy to notice that emotion recognition can be performed on the query data in combination with multimodal data to obtain emotion recognition results corresponding to multiple emotion recognition tasks, and by designing and combining multiple emotion recognition tasks, more emotion knowledge of the query data can be obtained, thereby making the target emotion recognition result corresponding to the query data more accurate, thereby achieving the technical effect of accurate emotion recognition on the data, and further solving the technical problem of low accuracy of emotion recognition on the data in related technologies.

[0014] It is easy to note that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present disclosure, and do not constitute a limitation of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation of the present disclosure. In the drawings:

[0016] FIG1 is a schematic diagram of an application scenario of an emotion recognition method according to an embodiment of the present disclosure;

[0017] FIG2 is a flow chart of an emotion recognition method according to Embodiment 1 of the present disclosure;

[0018] FIG3 a is a schematic diagram of a generative dialogue interface according to an embodiment of the present disclosure;

[0019] FIG3 b is a schematic diagram of a web page interface according to an embodiment of the present disclosure;

[0020] FIG3 c is a schematic diagram of a pre-training method based on multimodal emotion joint modeling according to an embodiment of the present disclosure;

[0021] FIG4 is a flow chart of a method for pre-training an emotion recognition model according to Embodiment 2 of the present disclosure;

[0022] FIG5 is a flow chart of an emotion recognition method according to Embodiment 3 of the present disclosure;

[0023] FIG6 is a flow chart of a data generation method according to Embodiment 4 of the present disclosure;

[0024] FIG7 is a flow chart of a data generation method according to Embodiment 5 of the present disclosure;

[0025] FIG8 is a flow chart of an emotion recognition method according to Example 6 of the present disclosure;

[0026] FIG9 is a flow chart of a data generation method according to Embodiment 7 of the present disclosure;

[0027] FIG10 is a schematic diagram of an emotion recognition device according to an embodiment of the present disclosure;

[0028] FIG11 is a schematic diagram of a pre-training device for an emotion recognition model according to an embodiment of the present disclosure;

[0029] FIG12 is a schematic diagram of another emotion recognition device according to an embodiment of the present disclosure;

[0030] FIG13 is a schematic diagram of a data generating device according to an embodiment of the present disclosure;

[0031] FIG14 is a schematic diagram of another data generating device according to an embodiment of the present disclosure;

[0032] FIG15 is a schematic diagram of another emotion recognition device according to an embodiment of the present disclosure;

[0033] FIG16 is a schematic diagram of another emotion recognition device according to an embodiment of the present disclosure;

[0034] FIG17 is a structural block diagram of a computer terminal according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present disclosure.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] The technical solution provided by the present disclosure is mainly implemented using large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters. The large model can also be called a cornerstone model / foundation model (Foundation Model). The large model is pre-trained by large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks and has good generalization capabilities, such as large-scale language models (LLMs) and multi-modal pre-training models.

[0038] It should be noted that when the large model is actually applied, the pre-trained model can be fine-tuned through a small number of samples, so that the large model can be applied to different tasks. For example, the large model can be widely used in natural language processing (NLP), computer vision, speech processing and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, etc. It can also be widely used in natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. In the embodiment of the present disclosure, the data processing through the emotion recognition model in the large model scenario is taken as an example for explanation,

[0039] First, some nouns or terms that appear in the description of the embodiments of the present disclosure are subject to the following explanations:

[0040] Sentiment Analysis (SA): A task that aims to understand people’s emotional states, usually a binary classification (negative / positive) or ternary classification (negative / neutral / positive), or regression (-3 points to 3 points) task.

[0041] Fine-grained sentiment analysis (ABSA): evaluates users' views or opinions on something, which are accumulated over a long period of time.

[0042] Multimodal Sentiment Analysis (MSA): The task of performing sentiment analysis based on data in multiple modalities such as text, audio, and video.

[0043] Emotion Recognition in Conversation (ERC): This task involves identifying the user's emotional feedback, formed by their perception of changes within a short period of time during a conversation. This task is typically a multi-classification task, such as joy, sadness, fear, anger, surprise, and disgust.

[0044] Comment Analysis (CA): Understanding users’ opinions on specific topics, products, or services by analyzing the sentiment, opinions, and attitudes of textual reviews.

[0045] Pre-training: By performing self-supervised or supervised pre-training on a large amount of data, the statistical features of sentiment data are learned, achieving better model prediction results when migrating to downstream tasks.

[0046] There are currently many ways to identify emotions. The first method is conversational emotion recognition, which can automatically detect the emotional features contained in the text of users' daily conversations, but its disadvantage is that it does not take into account additional inputs such as audio and video; the second method is sentiment analysis, which can determine whether the user's comments are positive, negative or neutral. In the automotive field, multimodal voice interaction methods are also deployed, including multimodal sentiment analysis, but the disadvantage is that there is no specific multimodal sentiment analysis method; the third method is multimodal sentiment analysis that combines voice and semantics, but the disadvantage is that it does not take video input into account; the fourth method is a physiological signal analysis subsystem based on a multimodal emotion recognition system, which can recognize emotions based on facial expression images, voice signals, text, human posture, and physiological signals, but the disadvantage is that it relies on a large number of artificial features and does not use pre-trained models, resulting in poor results.

[0047] In order to solve the above problems, the present disclosure provides an emotion recognition method, which can flexibly process unimodal and multimodal data by adopting an emotion joint modeling method, and can introduce task-specific prompting methods to realize joint modeling of emotion recognition tasks. In order to further improve the multimodal emotion perception ability of the model, by designing new pre-training tasks and training strategies, the model can acquire general emotion knowledge across tasks. It has been verified that the method of the present disclosure can not only effectively handle various emotion recognition tasks, but also has excellent cross-task generalization capabilities.

[0048] Example 1

[0049] According to an embodiment of the present disclosure, an emotion recognition method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0050] Taking into account the huge amount of model parameters of the large model and the limited computing resources of the mobile terminal, the above-mentioned emotion recognition method provided by the embodiment of the present disclosure can be applied to the application scenario shown in Figure 1, but is not limited thereto. Figure 1 is a schematic diagram of an application scenario of an emotion recognition method according to an embodiment of the present disclosure. In the application scenario shown in Figure 1, the large model is deployed in a server 10. The server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client devices 20 here can include but are not limited to: smart phones, tablet computers, laptops, PDAs, personal computers, smart home devices, car-mounted devices, etc. The client device 20 can interact with the user through a graphical user interface to call the large model, thereby implementing the method provided by the embodiment of the present disclosure.

[0051] In an embodiment of the present disclosure, a system composed of a client device and a server can perform the following steps: the client device generates query data, wherein the query data includes data of at least one of the following modalities: visual modality, acoustic modality, textual modality, and contextual modality; the server performs emotion recognition on the query data to obtain an initial emotion recognition set, wherein the initial emotion recognition set includes emotion recognition results corresponding to multiple emotion recognition tasks; based on the target emotion recognition task corresponding to the query data, the target emotion recognition result corresponding to the query data is determined. It should be noted that, if the operating resources of the client device can meet the deployment and operation conditions of the large model, the embodiment of the present disclosure can be performed in the client device.

[0052] In the above operating environment, the present disclosure provides an emotion recognition method as shown in Figure 2. Figure 2 is a flow chart of the emotion recognition method according to Example 1 of the present disclosure. As shown in Figure 2, the method may include the following steps:

[0053] Step S202: responding to query data received in the generative dialogue interface.

[0054] The query data includes data of at least one of the following modalities: visual modality, acoustic modality, text modality, and contextual modality.

[0055] The aforementioned generative dialogue interface can be a human-computer interaction interface that uses natural language processing and machine learning methods to enable machines to generate natural language responses. Generative dialogue interfaces can automatically generate responses based on user input. They can also automatically generate responses based on context, where context can be user-to-user interaction input or the user-bot conversation history, making conversations more fluid and flexible. Generative dialogue interfaces are commonly used in applications such as chatbots, intelligent assistants, and virtual customer service.

[0056] The query data mentioned above may be data input by the user in the generative dialogue interface, wherein the query data may be used to query the content that the user needs to know, and the query data may also be used to represent any type of input data, etc., and the query data is not specifically limited here.

[0057] The above-mentioned visual modality may refer to image or video type data; the above-mentioned acoustic modality may refer to sound, voice or music type data; the above-mentioned text modality may refer to text or web page type data; the above-mentioned contextual modality may refer to data associated with previously input query data or the target emotion recognition result corresponding to the previously input query data. For example, the previously input query data may be "Is the weather good today?" The corresponding answer result of the previously input query data may be "The weather is fine today", and the current query data may be "What kind of clothes are suitable to wear today", and the contextual modality data corresponding to the current data may be "Is the weather good today?" The user's emotion can be determined to be happy from the contextual modality data, and based on the contextual modality data, the target emotion recognition result corresponding to the query data can be determined to be happy.

[0058] The application scenarios of the above-mentioned generative dialogue interface include but are not limited to the following scenarios:

[0059] Virtual assistant application scenarios: Generative dialogue interfaces can be used to build intelligent assistants. Users can communicate with the assistants through dialogue to obtain various information, perform tasks, etc. The query data in the virtual assistant application scenario can be any type of data that the user needs to ask the virtual assistant.

[0060] E-commerce customer service application scenario: Generative dialogue interfaces can be used to build online customer service systems for e-commerce platforms. Users can communicate with customer service staff through dialogue to obtain product information, solve problems, etc. The query data in e-commerce customer service application scenarios can include product orders, order issues, and other data.

[0061] Language learning application scenarios: Generative dialogue interfaces can be used to build language learning platforms, where users can practice speaking, listening, grammar, etc. through dialogue and obtain guidance and feedback on language learning. The query data in language learning application scenarios can be user-entered spoken audio, exercises, and other data.

[0062] Tourism consultation application scenario: The generative dialogue interface can be used to build a tourism consultation system, where users can obtain tourism information about the destination, route planning, hotel reservations, etc. through dialogue; the query data in the tourism consultation application scenario can be image information of tourist locations, map information of tourist locations, and other data.

[0063] Medical consultation application scenario: The generative dialogue interface can be used to build a medical consultation platform, where users can communicate with doctors through dialogue to obtain advice and suggestions on health issues. The query data in the medical consultation application scenario can be patient scan images, patient condition information, and other data.

[0064] Application scenarios for robot tour guides: Generative dialogue interfaces can be used to build robot tour guide systems, where users can obtain introductions to scenic spots, navigation instructions, historical knowledge, etc. through dialogue. The query data in robot tour guide application scenarios can include scenic spot images, scenic spot geographic locations, scenic spot information inquiries, and other data.

[0065] Game character application scenario: The generative dialogue interface can be used to build a character dialogue system in the game. Players can promote the development of the game plot by talking to the characters. The query data in the game character application scenario can be text entered by the player, voice input by the player, and other data.

[0066] Media interaction application scenarios: Generative dialogue interfaces can be used to build media interaction platforms, where users can participate in interactive programs, vote, ask questions, etc. through dialogue; the query data in media interaction application scenarios can be users' voting results, questions, and other data.

[0067] Step S204: Perform emotion recognition on the query data to obtain an initial emotion recognition set.

[0068] The initial emotion recognition set includes emotion recognition results corresponding to multiple emotion recognition tasks.

[0069] The above-mentioned emotion recognition tasks include but are not limited to fine-grained emotion analysis tasks, emotion recognition tasks in conversations, multimodal emotion analysis tasks, and comment analysis tasks. There is no limitation on emotion recognition tasks here. Emotion recognition tasks can be set according to actual conditions, and emotion recognition tasks can be added or deleted.

[0070] In an optional embodiment, emotion recognition can be performed on the query data based on multiple emotion recognition tasks to obtain emotion recognition results corresponding to the multiple emotion recognition tasks. During the emotion recognition process on the query data, if the query data contains multiple modalities, feature extraction can be performed on the query data to obtain a unified query vector. Emotion recognition can then be performed on the query data based on the query vector to obtain emotion recognition results corresponding to the multiple emotion recognition tasks.

[0071] Step S206: determining a target emotion recognition result corresponding to the query data based on the target emotion recognition task corresponding to the query data.

[0072] In an optional embodiment, based on the target emotion recognition task corresponding to the query data, the target emotion recognition result corresponding to the query data may be determined from the initial emotion recognition result.

[0073] The above-mentioned target emotion recognition result can be a result obtained corresponding to the target emotion recognition task. For example, the target emotion recognition result can be used to represent the emotional attitude expressed by the query data.

[0074] When the target emotion recognition task is a fine-grained emotion analysis task, the target emotion recognition result can be the emotion label of the query data, that is, the emotional attitude expressed by the query data can be determined through the emotion label; when the target emotion recognition task is an emotion recognition task in a conversation, the target emotion recognition result can be the tone revealed by the query data, that is, whether the query data reveals anger, happiness, etc.; when the target emotion recognition task is a multimodal emotion analysis task, the target emotion recognition result can be the emotional state expressed by the query data; when the target emotion recognition task is a comment analysis task, the target emotion recognition result can be the emotional state expressed by the query data.

[0075] The above method disclosed in the present invention can be applied to the navigation and outbound call platform of intelligent customer service. During the interaction between the voice robot and the customer, the user's emotions can be accurately detected through the above method, so that the platform can automatically use soothing words, transfer to manual or marketing conversion, which can effectively improve the user experience.

[0076] Figure 3a is a schematic diagram of a generative dialogue interface according to an embodiment of the present disclosure. As shown in Figure 3a, a user can input query data in the generative dialogue interface. After emotion recognition is performed on the query data, a reply result can be generated based on the target emotion recognition result corresponding to the query data, and interaction can be performed with the user. One or more interactions can be performed. During the interaction process, by recognizing the emotion of the query data, the user's emotions can be automatically soothed or taken care of, thereby improving the user experience.

[0077] Figure 3b is a schematic diagram of a web page interface according to an embodiment of the present disclosure. As shown in Figure 3b, the user can enter comments on the article content in the web page interface to obtain comment data, which can be the above-mentioned query data. By performing emotion recognition on the comment data, the user's evaluation of the article content can be quickly determined based on the emotion recognition results, so that the articles that the user likes can be quickly screened out.

[0078] It should be noted that the above method of the present disclosure can be applied in many different types of scenarios. The above content is only for illustration and does not impose any limitation. Users can apply the above method according to their needs.

[0079] Through the above steps, the query data received in the generative dialogue interface is responded to, wherein the query data includes data of at least one of the following modalities: visual modality, acoustic modality, text modality and contextual modality; emotion recognition is performed on the query data to obtain an initial emotion recognition set, wherein the initial emotion recognition set includes emotion recognition results corresponding to multiple emotion recognition tasks; based on the target emotion recognition task corresponding to the query data, the target emotion recognition result corresponding to the query data is determined, thereby improving the accuracy of emotion recognition on the data; it is easy to notice that the query data can be emotionally recognized by combining multimodal data to obtain emotion recognition results corresponding to multiple emotion recognition tasks. By combining multiple emotion recognition tasks, more emotional knowledge of the query data can be obtained, so that the target emotion recognition result corresponding to the query data can be made more accurate, thereby achieving the technical effect of accurate emotion recognition on the data, and thus solving the technical problem of low accuracy of emotion recognition on the data in related technologies.

[0080] In the above embodiment of the present disclosure, emotion recognition is performed on the query data to obtain an initial emotion recognition set, which includes: performing emotion recognition on the query data using an emotion recognition model to obtain the initial emotion recognition set.

[0081] The above-mentioned emotion recognition model can be a neural network model or other types of models. There is no limitation on the emotion recognition model here.

[0082] The above-mentioned emotion recognition model can be used to identify query data and obtain emotion recognition results corresponding to the query data across multiple emotion recognition tasks. The above-mentioned emotion recognition model can also be used to identify comment data and obtain emotion recognition results corresponding to the comment data across multiple emotion recognition tasks. The above-mentioned emotion recognition results can also be used to identify any data and obtain emotion recognition results corresponding to the data across multiple emotion recognition tasks.

[0083] In the above embodiment of the present disclosure, the emotion recognition model includes: an encoder, a mask modeling module connected to the encoder, an emotion category prediction module, a contrastive learning module, and an emotion label prediction module.

[0084] The encoder described above can adopt a single-stream architecture based on multi-layer bidirectional feature extraction (Transformer) to jointly process multimodal input. This encoder structure can help emotion recognition models effectively capture interactions and dependencies between modalities, thereby improving the overall performance of emotion recognition models across all tasks.

[0085] The above-mentioned emotion label prediction module can be a decoder, wherein the decoder can be any decoder, and there is no limitation on the architecture and type of the decoder.

[0086] In the above embodiment of the present disclosure, emotion recognition is performed on query data using an emotion recognition model to obtain an initial emotion recognition set, including: using an encoder to extract features from the query data to obtain a query vector corresponding to the query data; and using an emotion label prediction module to perform emotion recognition on the query vector to obtain an initial emotion recognition set.

[0087] For the visual modality, features extracted from a pre-trained mobile network (MobileNet) are used as visual embeddings. For the acoustic modality, filter (fbank) features extracted from the audio analysis library (librosa) are used for acoustic embedding. For the text modality, the Bidirectional and Auto Regressive Transformers (BART) feature extraction model is used.

[0088] In the above embodiment of the present disclosure, an emotion label prediction module is used to perform emotion recognition on a query vector to obtain an initial emotion recognition set, including: obtaining a preset vector set corresponding to multiple emotion recognition tasks, wherein the preset vector set includes at least one preset vector, and the at least one preset vector is used to represent at least one preset emotion corresponding to the emotion recognition task; matching the query vector with at least one preset vector in the preset vector set respectively, and determining a target vector in the preset vector set that successfully matches the query vector; and obtaining an initial emotion recognition result based on the preset emotions corresponding to the target vectors in the multiple preset vector sets.

[0089] The above-mentioned preset vector can be a vector generated by data corresponding to the preset emotion, and the preset emotion can be a pre-set emotion label. The preset emotion here can be a fine-grained label, such as: neutral, angry, depressed, sad, happy, excited, etc., which is not limited here.

[0090] In an optional embodiment, data containing different emotion types can be acquired in advance, and the data can be image data, sound data, video data, or text data. The emotion type corresponding to the data can be defined in advance, and features can be extracted from the data to obtain a preset vector. At least one preset emotion corresponding to at least one preset vector can be determined based on the correspondence between the data and the emotion type. The at least one preset vector obtained and the at least one preset emotion can be matched one-to-one to obtain a preset vector set.

[0091] In another optional embodiment, the query vector may be matched against at least one preset vector in a preset vector set to determine a target vector that successfully matches the query vector. Alternatively, the query vector and the at least one preset vector may be matched based on a similarity between the query vector and the at least one preset vector. If the similarity between the query vector and the at least one preset vector is greater than or equal to a preset similarity, the query vector and the preset vector are successfully matched, and the preset vector is determined to be the target vector. If the similarity between the query vector and the preset vector is less than the preset similarity, the query vector and the preset vector are unmatched.

[0092] Furthermore, if there are multiple preset vectors whose similarities with the query vector are greater than or equal to the preset similarity, the preset vector with the greatest similarity can be determined as the target vector; if there is no preset vector whose similarity with the query vector is greater than or equal to the preset similarity, the preset vector with the greatest similarity can be determined as the target vector, or a prompt message can be output, which can be used to prompt the emotion recognition results that have not obtained the query data.

[0093] In the above embodiment of the present disclosure, the method also includes: obtaining a training data set corresponding to multiple emotion recognition tasks, wherein the training data in the training data set includes: multiple training samples, and an emotion label set corresponding to the training samples, and the emotion label set contains emotion labels corresponding to multiple emotion recognition tasks; based on the emotion categories corresponding to the emotion labels, the training data set is divided into multiple data sets, wherein the emotion categories corresponding to multiple training data contained in the same data set are the same; based on the same data set, the initial recognition model is pre-trained to obtain a target recognition model, wherein the target recognition model is used to identify the emotion category corresponding to the input data; based on the training data set, the target recognition model is pre-trained to obtain an emotion recognition model.

[0094] The aforementioned multiple training samples may include, but are not limited to, image training samples, text training samples, voice training samples, video training samples, and contextual modality samples. The sets of emotion labels corresponding to the aforementioned training samples may be emotion labels corresponding to the training samples for different emotion recognition tasks. Emotion labels may include anger, frustration, sadness, happiness, excitement, and the like.

[0095] The above-mentioned emotion categories can also be called emotion polarity, where the emotion categories can be negative, neutral and positive, but are not limited thereto. The emotion categories can also be set according to actual usage scenarios, where the emotion categories are mainly used for more fine-grained classification of emotion labels.

[0096] The above-mentioned initial recognition model may be a preset neural network model, or an arbitrary model, and no limitation is imposed on the category of the initial recognition model.

[0097] In an optional embodiment, the training data set can be clustered according to the emotion categories corresponding to the emotion labels to obtain multiple data sets, and the multiple data sets correspond to different emotion categories respectively. The emotion categories corresponding to the multiple training data contained in the same data set are the same; the initial recognition model can be pre-trained for targeted emotion classification based on the same data set, so that the obtained target recognition model can obtain preliminary emotion recognition capabilities, thereby being able to identify the emotion categories corresponding to the input data.

[0098] Optionally, multiple data sets and emotion categories corresponding to the multiple data sets can be obtained, and the initial recognition model can be pre-trained based on the multiple data sets and emotion categories corresponding to the multiple data sets to obtain a target recognition model. The target recognition model can be used to obtain the coarse-grained emotion type of the input data.

[0099] Furthermore, the target recognition model can be pre-trained based on multiple training samples in a more fine-grained training data set and the emotion label set corresponding to the training samples to obtain an emotion recognition model, which can be used to obtain fine-grained emotion labels for the input data.

[0100] In order to improve the emotion perception ability of the emotion recognition model, the pre-training can be divided into two stages: the first stage pre-training and the second stage pre-training, so as to guide the model to gradually learn general emotion knowledge.

[0101] The first stage of pre-training is used to pre-train the initial recognition model to obtain the target recognition model.

[0102] In the first stage, the main focus is coarse-grained emotion perception pre-training, including multimodal masked context modeling, emotion polarity prediction, and coarse-grained label contrastive learning. Multimodal masked context modeling is used to represent the use of mask and context information to train the model in order to better understand and process multimodal data. Emotion polarity prediction is used to represent the prediction of the emotion polarity of the data by analyzing the emotion information in text, images, audio and other data, that is, to determine whether the emotion expressed by the data is positive, negative or neutral. Coarse-grained label contrastive learning is used to represent the division of data into several coarse-grained categories through contrastive learning methods. This method can reduce the complexity and computational cost of model training, and at the same time, it can also achieve better performance in some tasks.

[0103] Through the pre-training in the first stage, the target recognition model can acquire preliminary emotion classification capabilities. Inspired by the directed expectation test, it can be considered that emotions are polarity-invariant, that is, the type of emotion is unchanged. Data of the same emotion category can be combined to obtain the above data set, which will not change the emotion category of the data in the data set. Therefore, the training data set can be divided into different data pools according to the emotion category. Different data pools are used to store different data sets. Two data can be randomly selected from the same data pool for combination, so that the initial recognition model can be pre-trained with the combined data to obtain the target recognition model.

[0104] The second stage of pre-training is used to pre-train the target recognition model to obtain the emotion recognition model.

[0105] The second stage of pre-training enables fine-grained emotion perception pre-training, including mask context modeling and cross-task emotion prediction. The goal is to enable the emotion recognition model to acquire fine-grained emotion classification capabilities. Mask context modeling can be used to train the model using both mask and context information.

[0106] In the above embodiment of the present disclosure, the initial recognition model is pre-trained based on the same data set to obtain a target recognition model, including: combining two training samples in the same data set to obtain a first training sample; using an encoder to extract features of the first training sample to obtain a first sample feature corresponding to the first training sample; performing a mask operation on the first sample feature to obtain a second sample feature containing a first mask mark, and using a mask modeling module to predict the mask mark of the second sample feature to obtain a first predicted mask mark in the second sample feature, wherein the first mask mark is used to represent the mark that has been masked in the second sample feature; using the emotion category prediction module to perform emotion prediction on the first sample feature to obtain a predicted emotion category of the first training sample; using a contrastive learning module to determine the first similarity between training samples with the same emotion label in the first training sample and the second similarity between any two training samples in the first training sample based on the first sample feature; adjusting the model parameters of the initial recognition model based on the first mask mark, the first predicted mask mark, the target emotion category corresponding to the same data set, the predicted emotion category, the first similarity and the second similarity to obtain a target recognition model.

[0107] The first mask mark can be a mark for indicating that a mask operation has been performed, such as text, an image, or a number. Optionally, the mask operation can replace some words in the text with a mask so that the subsequent model can predict the words before the replacement.

[0108] In an optional embodiment, training samples with the same sentiment polarity may be combined to obtain new target training samples to enrich the training samples; a mask operation may be performed on the target training samples, and the area subjected to the mask operation may be marked with a first mask mark.

[0109] The first sample feature may be a feature of a target training sample to which the first mask mark is not added, and the second sample feature may be a feature of a target training sample to which the first mask mark is added.

[0110] Furthermore, the mask modeling module can be used to predict the mask label of the second sample feature of the first mask label, so that the mask modeling module can predict the content of the mask operation area to obtain a first predicted mask label; the emotion category prediction module can be used to predict the emotion of the first sample feature to obtain the predicted emotion category of the target training sample, so as to adjust the model parameters according to the predicted emotion category and the target emotion category; the contrastive learning module can be used to determine the first similarity between training samples with the same emotion label and the second similarity between any two training samples in the target training sample based on the first sample feature, and by comparing the first similarity and the second similarity, it can be determined whether the training samples with the same emotion label are more similar, which can be used to adjust the model parameters to obtain a target recognition model.

[0111] Furthermore, a loss value can be constructed based on the first mask mark and the first predicted mask mark, a loss value can be constructed based on the target emotion category and the predicted emotion category corresponding to the same data set, and a loss value can be constructed based on the first similarity and the second similarity. Based on each loss value, the model parameters of the target recognition model can be adjusted to obtain an emotion recognition model.

[0112] In another optional embodiment, the probability of the mask operation can be increased to a preset value, where the preset value can be set by the user. For example, the preset value can be 50%. The probability of the mask operation is increased mainly by randomly setting a preset proportion of the input data to 0, allowing the initial recognition model to predict the value before the mask operation is performed on the input data, thereby achieving efficient self-supervised pre-training.

[0113] The above mask token prediction can be used to represent contextual mask modeling, where random mask tokens can be extracted from input text, acoustic, visual, and contextual modalities to encourage the initial recognition model to predict masked tokens, and the mask ratio can be increased to promote better generalization.

[0114] The above-mentioned sentiment prediction can be used to represent sentiment polarity prediction. In order to encourage the initial recognition model to learn to distinguish different sentiment categories, the fine-grained sentiment labels of the dataset can be mapped to positive, negative, and neutral categories, and the fine-grained sentiment labels can be converted into sentiment polarity labels. Then, in the Sentiment Polarity Prediction task (SPP), the initial recognition model can be trained to predict the sentiment polarity category of the input.

[0115] The sentiment labels corresponding to the two training samples can be used for coarse-grained category labeling (CCL). To encourage the initial recognition model to learn to distinguish different sentiment categories at a coarse level, a coarse-grained label contrastive learning task is proposed. In this task, the encoder output can be used as the feature representation of different training samples, and the Euclidean distance between training samples with the same sentiment label in the batch is calculated to maximize the similarity between samples with the same sentiment label.

[0116] In an optional embodiment, the mask index can be represented by 1≤m≤M, where M is the number of first mask tags. The mask index is used to clearly identify which input vector positions are masked to 0 and which input positions remain unchanged during the model training process.

[0117] The first mask loss values ​​of the first mask mark and the first predicted mask mark can be expressed by the following formula:

[0118] Among them, L MCM (θ) is the loss value of Multimodal Context Mask Modeling (MCM), w m Used to indicate the first mask mark above, w n is used to represent the first predicted mask mark, θ is the model parameter to be updated, P θ Represents the output distribution of the model, and 1≤m≤M represents the mask index.

[0119] The category loss values ​​of the above target emotion category and predicted emotion category can be expressed by the following formula: L SPP (θ)=-log(P θ (s|w n ));

[0120] Among them, L SPP (θ) is the sentiment polarity prediction task, s is used to represent the target sentiment category, and w n Represents the predicted sentiment category.

[0121] The distance between the two training samples and the first label loss value of the sentiment label, that is, the label loss value between the first similarity and the second similarity, can be expressed by the following formula:

[0122] Among them, L CCL(θ) is the coarse-grained contrastive learning loss value, distance(j,k) is used to represent the Euclidean distance between training sample j and training sample k, that is, the second similarity, b represents the batch size, and when sample mask(j,k) is 1, it means that training sample j and training sample k have the same sentiment label, that is, the first similarity, otherwise it is 0.

[0123] In an optional embodiment, weights can be set for the first mask loss value, the category loss value, and the first label loss value, and the first target loss value can be obtained based on the sum of the first mask loss value, the category loss value, and the first label loss value. The user can adjust the weights of each loss value according to needs to obtain the first target loss value.

[0124] Furthermore, the model parameters of the initial recognition model can be adjusted according to the first target loss value, thereby improving the recognition accuracy of the target recognition model for emotion recognition.

[0125] The first target loss value obtained by the above-mentioned first mask loss value, category loss value and first label loss value can be expressed by the following formula: Lstage1=L(MCM)+L(SPP)+L(CCL).

[0126] Among them, Lstage1 is used to represent the first target loss value, Lstage1 is used to represent the first mask loss value, L(SPP) is used to represent the category loss value, and L(CCL) is used to represent the first label loss value.

[0127] In the above embodiments of the present disclosure, a target recognition model is pre-trained based on a training data set to obtain an emotion recognition model, including: using an encoder to extract features of training samples in the training data set to obtain sample features corresponding to the training samples; performing a mask operation on the sample features to obtain target sample features containing a second mask mark, and using a mask modeling module to predict the mask mark on the target sample features to obtain a second predicted mask mark in the target sample features, wherein the second mask mark is used to characterize the mark that has been masked in the target sample features; using an emotion label prediction module to perform emotion prediction on the sample features to obtain a predicted label set of the training samples, wherein the predicted label set includes predicted emotions corresponding to multiple emotion recognition tasks; based on the second mask mark, the second predicted mask mark, the emotion label set and the predicted label set, the model parameters of the target recognition model are adjusted to obtain an emotion recognition model.

[0128] In an optional embodiment, the encoder can be used to directly perform multimodal masking operations on training samples in a multimodal training data set to obtain sample features corresponding to the training samples. Multimodal context mask modeling can randomly mask labels from input text, acoustic, visual and contextual modalities to encourage the model to learn to predict masked labels.

[0129] In another optional embodiment, a mask operation can be directly performed on the training samples in the training data set to obtain target sample features containing a second mask mark, and a mask modeling module can be used to predict the mask mark of the target sample features to predict the content of the mask operation area, thereby obtaining the second predicted mask mark of the target training sample.

[0130] The above masked token operation can be used to represent multimodal contextual mask modeling, which can randomly mask tokens from the multimodal input text, acoustic, visual, and context to encourage the initial recognition model to predict the masked tokens.

[0131] The above-mentioned predicted label set may include the emotion label of the training sample, and may also include the emotion labels of other training samples in the emotion recognition task that are close to the training sample, that is, the predicted emotions corresponding to multiple emotion recognition tasks.

[0132] The above-mentioned emotion prediction can be used to represent cross-task emotion prediction (CEP). The feature output of the encoder can be used as the representation of the target training sample. The training samples of different sub-emotion recognition tasks can be clustered. The distance between the training sample and the label clusters of different sub-tasks can be calculated. The cluster with the smallest distance can be used as the pseudo-label of the training sample. For example, there is a training sample of the ABSA task. The label of this training sample is three-category (negative, neutral, positive). Through clustering, the sample closest to this sample in the other three tasks is found. The label of the closest sample can be used as the pseudo-label of this sample, so that the three pseudo-labels corresponding to the three tasks can be obtained, and the above-mentioned predicted label set can be obtained.

[0133] In the above-mentioned embodiment of the present disclosure, based on the second mask mark, the second predicted mask mark, the emotion label set and the predicted emotion set, the model parameters of the target recognition model are adjusted to obtain the emotion recognition model, including: determining the second mask loss value based on the second mask mark and the second predicted mask mark; determining the second label loss value based on the emotion label set and the predicted emotion set; determining the second target loss value based on the second mask loss value and the second label loss value; adjusting the model parameters of the target recognition model based on the second target loss value to obtain the emotion recognition model.

[0134] The above-mentioned emotion label set and the second label loss value of the predicted emotion set can be expressed by the following formula: L CEP (θ)=-∑D log(P θ (E d |w m );

[0135] Among them, E d Used to represent the emotional label in the emotion recognition task,

[0136] Among them, L CEP (θ) is the loss value of the cross-task sentiment prediction task, E d represents the sentiment label in subtask d∈D, P θ represents the output distribution of the model, w m Denotes the predicted sentiment set, subtask set D = {ABSA, MSA, ERC, CA}.

[0137] In an optional embodiment, weights can be set for the second mask loss value and the second label loss value, and the second target loss value can be obtained based on the sum of the second mask loss value and the second label loss value. The user can adjust the weights of each loss value according to needs to obtain the second target loss value.

[0138] Furthermore, the model parameters of the target recognition model can be adjusted according to the second target loss value, thereby improving the recognition accuracy of the emotion recognition model for emotion recognition.

[0139] The first loss value constructed by the second mask loss value and the second label loss value can be expressed by the following formula: Lstage2 = L(MCM) + L(CEP);

[0140] Among them, Lstage2 is used to represent the second target loss value, L(SPP) is used to represent the category loss value, L(MCM) is used to represent the first mask loss value, and L(CEP) is used to represent the second label loss value.

[0141] In the above-mentioned embodiment of the present disclosure, the method also includes: outputting query data, target emotion recognition task and target emotion recognition result in a generative dialogue interface; responding to a modification instruction acting on the generative dialogue interface, obtaining a new emotion recognition result corresponding to the modification instruction, wherein the new emotion recognition result is an emotion recognition result obtained after modifying the target emotion recognition result based on the query data and the target emotion recognition task; and updating the emotion recognition model based on the query data and the target emotion recognition result.

[0142] In an optional embodiment, the query data, target emotion recognition task and target emotion recognition result can be output in the generative dialogue interface for user viewing. If the user believes that the result is incorrect, the user can generate a modification instruction by operating the generative dialogue interface, obtain a new emotion recognition result corresponding to the modification instruction, and update the model parameters of the emotion recognition model based on the query data and the target emotion recognition result. The model parameters of the emotion recognition model can also be updated according to the target emotion recognition result and the new emotion recognition result.

[0143] In the above embodiment of the present disclosure, obtaining a training data set corresponding to multiple emotion recognition tasks includes: obtaining an original data set corresponding to multiple emotion recognition tasks; and converting the format of the original data set according to a preset template to obtain a training data set.

[0144] The above-mentioned preset template may be a pre-set template, wherein the preset template may include at least one prompt word, and the at least one prompt word may be used to prompt the content that needs to be entered in the preset template.

[0145] In an optional embodiment, to jointly model different subtasks, a unified multi-task prompt word can be proposed to unify the input streams of multiple subtasks, and the multiple subtasks can be converted into generation tasks to unify the output format of the subtasks. In other words, the multimodal raw data set can be formatted according to a preset template, and the data in the raw data set can be placed in corresponding positions according to the prompt words in the preset template, thereby generating a training data set.

[0146] The prompt word can be composed of three parts: task identifier Z, answer set Y and input stream X. The preset template L can be expressed as: L = {Z, Y, X}, where the task identifier Z is composed of special tags, including<task_type> 、<data_id> 、<speaker_id> wait,<task_type> Used to distinguish different subtasks,<data_id> Used to distinguish datasets,<speaker_id> Used to distinguish speaker ID; where the answer set Y is a set of specific labels of the same dataset that guides the model to generate expected results; the input stream X represents the input text, sound, vision, and context.

[0147] The above multi-task unified prompt words standardize the input form and guide the model to generate task-specific results based on the answer set. The following are specific examples:

[0148] Template L = [<task type><data ID><speaker ID>][answer set][<audio><image><text><context>];

[0149] Task identifier Z: [ <erc> <iemocap><speaker 1>];

[0150] Answer set Y: [neutral, angry, frustrated, sad, happy, excited];

[0151] Input stream X: [ <audio>Audio characteristics< / audio> Image features <text> Okay. - But if you fill out this particular form, I'm not telling you to get in line.< / text> <context> Who told you to queue? ...< / context> ].

[0152] In the above input stream X <audio>Used to represent audio, Used to represent images, <text> Used to represent text,< / text> <context>Used to indicate context.

[0153] The text in the template L can be the text of the user's current turn in the conversation, with corresponding audio and images in the input. The context in the template L can be the text of the user's non-current turn in the conversation, such as the previous or next sentence uttered by the user, with no corresponding audio or images in the input.

[0154] Figure 3c is a schematic diagram of a pre-training method based on multimodal emotion joint modeling according to an embodiment of the present disclosure. As shown in Figure 3c, it may include a multi-task unified prompt word input module and a pre-training module for multimodal emotion joint modeling, wherein the pre-training module for multimodal emotion joint modeling includes a multimodal context mask prediction unit, an emotion polarity prediction unit, a coarse-grained label comparison learning unit and a cross-task emotion prediction unit.

[0155] The multi-task unified prompt word input module unifies the prompt words of different subtask input streams, converting them into generation tasks and thus unifying their output format. This enables the model to predict specific tasks based on a given answer set, ensuring the standardization of input data. The multimodal emotion joint modeling pre-training module involves two stages of pre-training, including multimodal context mask prediction, emotion polarity prediction, coarse-grained label comparison learning, and cross-task emotion prediction. These tasks are designed to enhance the model's emotion perception capabilities, enabling it to perform well on multiple emotion-related tasks.

[0156] The above-mentioned method of the present invention can be evaluated on three tasks: ABSA, ERC and MSA. Experiments show that the emotion recognition results of ABSA are improved by 9% compared with the emotion recognition results of the original model, the emotion recognition results of ERC are improved by 7% compared with the emotion recognition results of the original model, and the emotion recognition results of MSA are improved by 1% compared with the emotion recognition results of the original model. The above-mentioned method of the present invention has achieved significant performance improvement in all three tasks, and the average accuracy has been improved by 5.6%.

[0157] The task unified prompt word input design proposed in the present disclosure can unify different sub-task inputs and convert them into generation tasks, thereby providing a method to process multiple tasks simultaneously; multimodal joint pre-training can enable the model to obtain richer emotional knowledge on multimodal data by designing and combining multiple pre-training tasks; the use of multimodal context mask prediction can expand traditional text mask modeling, so that the model can simultaneously learn information in text, acoustic and visual modalities; cross-task emotion prediction can enable the model to share emotional knowledge between different tasks, improving its generalization ability in multi-task environments; in the refined two-stage pre-training, the first stage focuses on coarse-grained emotion perception, and the second stage focuses more on fine-grained emotion classification, ensuring that the model has good performance on emotion tasks at different levels.

[0158] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0159] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0160] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present disclosure.

[0161] Example 2

[0162] According to an embodiment of the present disclosure, a pre-training method for an emotion recognition model is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0163] FIG4 is a flow chart of a method for pre-training an emotion recognition model according to Embodiment 2 of the present disclosure. As shown in FIG4 , the method includes the following steps:

[0164] Step S402: Obtain a plurality of training data sets corresponding to emotion recognition tasks.

[0165] The training data in the training data set includes: multiple training samples, and a set of emotion labels corresponding to the training samples, and the emotion label set includes emotion labels corresponding to multiple emotion recognition tasks.

[0166] Step S404: dividing the training data set into multiple data sets based on the emotion categories corresponding to the emotion tags.

[0167] Among them, the sentiment categories corresponding to multiple training data contained in the same data set are the same.

[0168] Step S406: pre-train the initial recognition model based on the same data set to obtain a target recognition model.

[0169] Among them, the target recognition model is used to identify the emotion category corresponding to the input data.

[0170] Step S408: pre-train the target recognition model based on the training data set to obtain an emotion recognition model.

[0171] Among them, the emotion recognition model is used to perform emotion recognition on the query data to obtain an emotion recognition set of the query data. The emotion recognition set includes emotion recognition results corresponding to multiple emotion recognition tasks.

[0172] The above-mentioned emotion recognition model can be used to identify query data and obtain emotion recognition results corresponding to the query data across multiple emotion recognition tasks. The above-mentioned emotion recognition model can also be used to identify comment data and obtain emotion recognition results corresponding to the comment data across multiple emotion recognition tasks. The above-mentioned emotion recognition results can also be used to identify any data and obtain emotion recognition results corresponding to the data across multiple emotion recognition tasks.

[0173] In an optional embodiment, the initial recognition model can be guided to gradually learn emotional knowledge through two pre-training stages. Through the first stage pre-training, the initial recognition model can be pre-trained based on the same data set, and the initial recognition model can be pre-trained with coarse-grained emotion perception, which can facilitate the target recognition model to identify the emotion category corresponding to the input data; through the second stage pre-training, the target recognition model can be pre-trained based on the training data set, and the target recognition model can be pre-trained with fine-grained emotion perception. Since the target recognition model is already a pre-trained model, the complexity of the second stage pre-training on the target recognition model can be reduced, while improving the accuracy of the emotion recognition model.

[0174] Through the above steps, a training data set corresponding to multiple emotion recognition tasks is obtained, wherein the training data in the training data set includes: multiple training samples, and an emotion label set corresponding to the training samples, and the emotion label set contains emotion labels corresponding to multiple emotion recognition tasks; based on the emotion categories corresponding to the emotion labels, the training data set is divided into multiple data sets, wherein the emotion categories corresponding to multiple training data contained in the same data set are the same; based on the same data set, the initial recognition model is pre-trained to obtain a target recognition model, wherein the target recognition model is used to identify the emotion category corresponding to the input data; based on the training data set, the target recognition model is pre-trained to obtain an emotion recognition model, thereby improving the accuracy of emotion recognition of the data; it is easy to notice that the initial recognition model can be trained in conjunction with multiple emotion recognition tasks, and the training is divided into two stages, the first stage can focus on coarse-grained emotion perception, and the second stage can focus on fine-grained emotion classification, ensuring that the emotion recognition model can have good performance on emotion tasks at different levels, thereby improving the accuracy of the emotion recognition model in emotion recognition of the data, thereby solving the technical problem of low accuracy of emotion recognition of the data in related technologies.

[0175] In the above embodiment of the present disclosure, the initial recognition model includes: an encoder, a mask modeling module connected to the encoder, an emotion category prediction module, a contrastive learning module, and an emotion label prediction module.

[0176] In the above embodiment of the present disclosure, the initial recognition model is pre-trained based on the same data set to obtain a target recognition model, including: combining two training samples in the same data set to obtain a target training sample; using an encoder to extract features of the target training sample to obtain a first sample feature corresponding to the target training sample; performing a mask operation on the first sample feature to obtain a second sample feature containing a first mask label, and using a mask modeling module to predict the mask label of the second sample feature to obtain a first predicted mask label in the second sample feature, wherein the first mask label is used to characterize the label that has been masked in the second sample feature; using an emotion category prediction module to perform emotion prediction on the first sample feature to obtain a predicted emotion category of the target training sample; using a contrastive learning module to determine, based on the first sample feature, a first similarity between training samples with the same emotion label in the target training sample, and a second similarity between any two training samples in the target training sample; adjusting the model parameters of the initial recognition model based on the first mask label, the first predicted mask label, the target emotion category corresponding to the same data set, the predicted emotion category, the first similarity and the second similarity to obtain a target recognition model.

[0177] In the above-mentioned embodiment of the present disclosure, based on the first mask mark, the first predicted mask mark, the target emotion category corresponding to the same data set, the predicted emotion category, the first similarity and the second similarity, the model parameters of the initial recognition model are adjusted to obtain the target recognition model, including: determining the first mask loss value based on the first mask mark and the first predicted mask mark; determining the category loss value based on the target emotion category and the predicted emotion category; determining the first label loss value based on the first similarity and the second similarity; determining the first target loss value based on the first mask loss value, the category loss value and the first label loss value; adjusting the model parameters of the initial recognition model based on the first target loss value to obtain the target recognition model.

[0178] In the above-mentioned embodiment of the present disclosure, a target recognition model is pre-trained based on a training data set to obtain an emotion recognition model, including: using an encoder to extract features of training samples in the training data set to obtain sample features corresponding to the training samples; performing a mask operation on the sample features to obtain target sample features containing a second mask mark, and using a mask modeling module to predict the mask mark on the target sample features to obtain a second predicted mask mark in the target sample features, wherein the second mask mark is used to characterize the mark that has been masked in the target sample features; using an emotion label prediction module to perform emotion prediction on the sample features to obtain a predicted label set of the training samples, wherein the predicted label set includes predicted emotions corresponding to multiple emotion recognition tasks; based on the second mask mark, the second predicted mask mark, the emotion label set and the predicted label set, the model parameters of the target recognition model are adjusted to obtain an emotion recognition model.

[0179] In the above-mentioned embodiment of the present disclosure, based on the second mask mark, the second predicted mask mark, the emotion label set and the predicted emotion set, the model parameters of the target recognition model are adjusted to obtain the emotion recognition model, including: determining the second mask loss value based on the second mask mark and the second predicted mask mark; determining the second label loss value based on the emotion label set and the predicted emotion set; determining the second target loss value based on the second mask loss value and the second label loss value; adjusting the model parameters of the target recognition model based on the second target loss value to obtain the emotion recognition model.

[0180] It should be noted that the preferred implementation scheme involved in the above embodiments of the present disclosure is the same as the solution provided in Example 1, as well as the application scenario and implementation process, but is not limited to the solution provided in Example 1.

[0181] Example 3

[0182] According to an embodiment of the present disclosure, an emotion recognition method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0183] FIG5 is a flow chart of an emotion recognition method according to Embodiment 3 of the present disclosure. As shown in FIG5 , the method includes the following steps:

[0184] Step S502: Obtain data to be identified.

[0185] The data to be recognized includes data of at least one of the following modalities: visual modality, acoustic modality, text modality, and contextual modality.

[0186] The aforementioned data to be recognized can be data for emotion recognition. The data to be recognized can include user query data in a conversational interface, user comment data in a comment section, video data, or audio data. There is no limitation on the type of data to be recognized.

[0187] Step S504: Perform emotion recognition on the data to be recognized to obtain an initial emotion recognition set.

[0188] The initial emotion set includes emotion recognition results corresponding to multiple different emotion recognition tasks.

[0189] In an optional embodiment, emotion recognition may be performed on the data to be recognized, so as to identify emotion recognition results corresponding to the data to be recognized in a plurality of different emotion recognition tasks.

[0190] Step S506: Determine the target emotion recognition result corresponding to the data to be recognized based on the target emotion recognition task corresponding to the data to be recognized.

[0191] The above-mentioned target emotion recognition result can be the emotion label corresponding to the data to be recognized in the target emotion recognition task.

[0192] It should be noted that the preferred implementation scheme involved in the above embodiments of the present disclosure is the same as the solution provided in Example 1, as well as the application scenario and implementation process, but is not limited to the solution provided in Example 1.

[0193] Example 4

[0194] According to an embodiment of the present disclosure, a data generation method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0195] FIG6 is a flow chart of a data generation method according to Embodiment 4 of the present disclosure. As shown in FIG6 , the method includes the following steps:

[0196] Step S602: respond to query data received in the generative dialogue interface.

[0197] The query data includes data of at least one of the following modalities: visual modality, acoustic modality, text modality, and contextual modality.

[0198] In an optional embodiment, if the user enters query data in the generative dialogue interface, the query data can be subjected to emotion recognition in real time by the background server to obtain a target emotion recognition result corresponding to the query data.

[0199] Step S604: perform emotion recognition on the query data to obtain an initial emotion set corresponding to the query data.

[0200] The initial emotion set includes emotion recognition results corresponding to multiple different emotion recognition tasks.

[0201] Step S606: Determine the target emotion recognition result corresponding to the query data based on the target emotion recognition task corresponding to the query data.

[0202] Step S608: Generate a response result that matches the query data based on the target emotion recognition result.

[0203] In an optional embodiment, the query data may correspond to multiple reply results, and the reply result corresponding to the emotion expressed by the query data may be determined from the multiple reply results based on the target emotion recognition result, so that the reply result better meets the user's needs.

[0204] The above-mentioned reply result may be a reply result after adjusting the initial reply result based on the target emotion recognition result.

[0205] In an optional embodiment, if the emotion label in the target emotion recognition result is used to indicate that the user is angry, the initial reply result can be adjusted so that the obtained reply result can alleviate the user's emotions and avoid the user's emotional excitement; if the emotion label in the target emotion recognition result is used to indicate that the user's emotions have not fluctuated, the initial reply result can be directly output as the final reply result.

[0206] Step S610: Display the reply result in the generative dialogue interface.

[0207] In an optional embodiment, after the reply result is generated, the reply result can be directly displayed in the corresponding area in the generative dialogue interface so that the user can see the reply result corresponding to the query data in a timely manner.

[0208] It should be noted that the preferred implementation scheme involved in the above embodiments of the present disclosure is the same as the solution provided in Example 1, as well as the application scenario and implementation process, but is not limited to the solution provided in Example 1.

[0209] Example 5

[0210] According to an embodiment of the present disclosure, a data generation method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0211] FIG7 is a flow chart of a data generation method according to Embodiment 5 of the present disclosure. As shown in FIG7 , the method includes the following steps:

[0212] Step S702: Acquire query data input in the generative dialogue interface in the dialogue mode.

[0213] The query data includes data of at least one of the following modalities: visual modality, acoustic modality, text modality, and contextual modality.

[0214] The above-mentioned dialogue mode can be used to represent the dialogue mode between a user and a robot. The above-mentioned dialogue mode can also be used to represent the dialogue mode between two users or multiple users. The above-mentioned dialogue mode can also be used to represent the dialogue mode between multiple users and a robot. No limitation is made to the above-mentioned dialogue mode here.

[0215] Step S704: perform emotion recognition on the query data to obtain an initial emotion set corresponding to the query data.

[0216] The initial emotion set includes emotion recognition results corresponding to multiple different emotion recognition tasks.

[0217] Step S706: Determine the target emotion recognition result corresponding to the query data based on the target emotion recognition task corresponding to the query data.

[0218] Step S708: Generate a response result that matches the query data based on the target emotion recognition result.

[0219] It should be noted that the preferred implementation scheme involved in the above embodiments of the present disclosure is the same as the solution provided in Example 1, as well as the application scenario and implementation process, but is not limited to the solution provided in Example 1.

[0220] Example 6

[0221] According to an embodiment of the present disclosure, an emotion recognition method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0222] FIG8 is a flow chart of an emotion recognition method according to Embodiment 6 of the present disclosure. As shown in FIG8 , the method includes the following steps:

[0223] Step S802: Acquire query data by calling the first interface.

[0224] The first interface includes a first parameter, a parameter value of the first parameter is query data, and the query data includes data of at least one of the following modalities: visual modality, acoustic modality, text modality, and contextual modality.

[0225] The first interface in the above steps may be an interface for data interaction between the cloud server and the client. The client may pass query data into the interface function as the first parameter of the interface function to achieve the purpose of uploading the query data to the cloud server.

[0226] Step S804: Perform emotion recognition on the query data to obtain an initial emotion recognition set.

[0227] The initial emotion recognition set includes emotion recognition results corresponding to multiple emotion recognition tasks.

[0228] Step S806: Determine the target emotion recognition result corresponding to the query data based on the target emotion recognition task corresponding to the query data.

[0229] Step S808: Output the target emotion recognition result by calling the second interface.

[0230] Among them, the second interface includes a second parameter, and the parameter value of the second parameter is the target emotion recognition result.

[0231] The above-mentioned second interface can be an interface for data interaction between the cloud server and the client. The cloud server can pass the target emotion recognition result into the interface function as the second parameter of the interface function to achieve the purpose of sending the target emotion recognition result to the client.

[0232] It should be noted that the preferred implementation scheme involved in the above embodiments of the present disclosure is the same as the solution provided in Example 1, as well as the application scenario and implementation process, but is not limited to the solution provided in Example 1.

[0233] Example 7

[0234] According to an embodiment of the present disclosure, a data generation method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0235] FIG9 is a flow chart of a data generation method according to Embodiment 7 of the present disclosure. As shown in FIG9 , the method includes the following steps:

[0236] Step S902: Acquire query data by calling the first interface.

[0237] The first interface includes a first parameter, a parameter value of the first parameter is query data, and the query data includes data of at least one of the following modalities: visual modality, acoustic modality, text modality, and contextual modality.

[0238] The first interface in the above steps may be an interface for data interaction between the cloud server and the client. The client may pass query data into the interface function as the first parameter of the interface function to achieve the purpose of uploading the query data to the cloud server.

[0239] Step S904: Perform emotion recognition on the query data to obtain an initial emotion recognition set.

[0240] The initial emotion recognition set includes emotion recognition results corresponding to multiple emotion recognition tasks.

[0241] Step S906: Determine the target emotion recognition result corresponding to the query data based on the target emotion recognition task corresponding to the query data.

[0242] Step S908: Generate a response result that matches the query data based on the target emotion recognition result.

[0243] Step S910: Output the reply result by calling the second interface.

[0244] The second interface includes a second parameter, and the parameter value of the second parameter is the reply result.

[0245] The above-mentioned second interface can be an interface for data interaction between the cloud server and the client. The cloud server can pass the reply result into the interface function as the second parameter of the interface function to achieve the purpose of sending the reply result to the client.

[0246] It should be noted that the preferred implementation scheme involved in the above embodiments of the present disclosure is the same as the solution provided in Example 1, as well as the application scenario and implementation process, but is not limited to the solution provided in Example 1.

[0247] Example 8

[0248] According to an embodiment of the present disclosure, an emotion recognition device for implementing the above-mentioned emotion recognition method is also provided. Figure 10 is a schematic diagram of an emotion recognition device according to an embodiment of the present disclosure. As shown in Figure 10, the device 1000 includes: a receiving module 1002, an identification module 1004, and a determination module 1006.

[0249] Among them, the receiving module is configured to respond to query data received in the generative dialogue interface, wherein the query data includes data of at least one of the following modalities: visual modality, acoustic modality, text modality and contextual modality; the recognition module is configured to perform emotion recognition on the query data to obtain an initial emotion recognition set, wherein the initial emotion recognition set includes emotion recognition results corresponding to multiple emotion recognition tasks; the determination module is configured to determine the target emotion recognition result corresponding to the query data based on the target emotion recognition task corresponding to the query data.

[0250] It should be noted that the receiving module 1002, the identifying module 1004, and the determining module 1006 correspond to steps S202 to S206 in Example 1. The examples and application scenarios implemented by the three modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the modules or units can be hardware components or software components stored in a memory and processed by one or more processors. The modules can also be part of the device and can run in the server 10 provided in Example 1.

[0251] In the above embodiments of the present disclosure, the recognition module is further configured to perform emotion recognition on the query data using the emotion recognition model to obtain an initial emotion recognition set.

[0252] In the above embodiment of the present disclosure, the emotion recognition model includes: an encoder, a mask modeling module connected to the encoder, an emotion category prediction module, a contrastive learning module, and an emotion label prediction module.

[0253] In the above embodiments of the present disclosure, the recognition module is further used to use an encoder to extract features from the query data to obtain a query vector corresponding to the query data; and use the emotion tag prediction module to perform emotion recognition on the query vector to obtain an initial emotion recognition set.

[0254] In the above-mentioned embodiment of the present disclosure, the recognition module is also used to obtain a preset vector set corresponding to multiple emotion recognition tasks, wherein the preset vector set includes at least one preset vector, and the at least one preset vector is used to represent at least one preset emotion corresponding to the emotion recognition task; the query vector is matched with at least one preset vector in the preset vector set respectively, and a target vector in the preset vector set that successfully matches the query vector is determined; and an initial emotion recognition result is obtained based on the preset emotions corresponding to the target vectors in the multiple preset vector sets.

[0255] In the above embodiment of the present disclosure, the device further includes: an acquisition module.

[0256] Among them, the acquisition module is also used to obtain training data sets corresponding to multiple emotion recognition tasks, wherein the training data in the training data set includes: multiple training samples, and an emotion label set corresponding to the training samples, and the emotion label set contains emotion labels corresponding to multiple emotion recognition tasks; based on the emotion categories corresponding to the emotion labels, the training data set is divided into multiple data sets, wherein the emotion categories corresponding to multiple training data contained in the same data set are the same; based on the same data set, the initial recognition model is pre-trained to obtain a target recognition model, wherein the target recognition model is used to identify the emotion category corresponding to the input data; based on the training data set, the target recognition model is pre-trained to obtain an emotion recognition model.

[0257] Among them, the acquisition module is also used to combine two training samples in the same data set to obtain a target training sample; use the encoder to extract features of the target training sample to obtain a first sample feature corresponding to the target training sample; perform a mask operation on the first sample feature to obtain a second sample feature containing a first mask mark, and use the mask modeling module to predict the mask mark of the second sample feature to obtain a first predicted mask mark in the second sample feature, wherein the first mask mark is used to characterize the mark that has been masked in the second sample feature; use the emotion category prediction module to perform emotion prediction on the first sample feature to obtain a predicted emotion category of the target training sample; use the contrastive learning module to determine the first similarity between training samples with the same emotion label in the target training sample based on the first sample feature, and the second similarity between any two training samples in the target training sample; based on the first mask mark, the first predicted mask mark, the target emotion category corresponding to the same data set, the predicted emotion category, the first similarity and the second similarity, adjust the model parameters of the initial recognition model to obtain a target recognition model.

[0258] Among them, the acquisition module is also used to use the encoder to extract features of the training samples in the training data set to obtain sample features corresponding to the training samples; perform a mask operation on the sample features to obtain target sample features containing a second mask mark, and use the mask modeling module to predict the mask mark of the target sample features to obtain a second predicted mask mark in the target sample features, wherein the second mask mark is used to characterize the mark that has been masked in the target sample features; use the emotion label prediction module to perform emotion prediction on the sample features to obtain a predicted label set of the training samples, wherein the predicted label set contains predicted emotions corresponding to multiple emotion recognition tasks; based on the second mask mark, the second predicted mask mark, the emotion label set and the predicted label set, the model parameters of the target recognition model are adjusted to obtain an emotion recognition model.

[0259] In the above embodiment of the present disclosure, the device further includes: an output module.

[0260] Among them, the output module is configured to output query data, target emotion recognition tasks and target emotion recognition results in a generative dialogue interface; respond to modification instructions acting on the generative dialogue interface, and obtain new emotion recognition results corresponding to the modification instructions, wherein the new emotion recognition results are emotion recognition results obtained after modifying the target emotion recognition results based on the query data and the target emotion recognition task; and update the emotion recognition model based on the query data and the target emotion recognition results.

[0261] It should be noted that the preferred implementation scheme involved in the above embodiments of the present disclosure is the same as the solution provided in Example 1, as well as the application scenario and implementation process, but is not limited to the solution provided in Example 1.

[0262] Example 9

[0263] According to an embodiment of the present disclosure, a pre-training device for an emotion recognition model for implementing the above-mentioned pre-training method of the emotion recognition model is also provided. Figure 11 is a schematic diagram of a pre-training device for an emotion recognition model according to an embodiment of the present disclosure. As shown in Figure 11, the device 1100 includes: an acquisition module 1102, a division module 1104, a first pre-training module 1106, and a second pre-training module 1108.

[0264] Among them, the acquisition module is configured to acquire a training data set corresponding to multiple emotion recognition tasks, wherein the training data in the training data set includes: multiple training samples, and an emotion label set corresponding to the training samples, and the emotion label set contains emotion labels corresponding to multiple emotion recognition tasks; the division module is configured to divide the training data set into multiple data sets based on the emotion categories corresponding to the emotion labels, wherein the emotion categories corresponding to multiple training data contained in the same data set are the same; the first pre-training module is configured to pre-train the initial recognition model based on the same data set to obtain a target recognition model, wherein the target recognition model is used to identify the emotion category corresponding to the input data; the second pre-training module is configured, wherein the emotion recognition model is used to perform emotion recognition on the query data to obtain an emotion recognition set of the query data, and the emotion recognition set includes emotion recognition results corresponding to multiple emotion recognition tasks.

[0265] It should be noted that the acquisition module 1102, the division module 1104, the first pre-training module 1106, and the second pre-training module 1108 correspond to steps S502 to S508 in Example 2. The four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors. The above modules can also be run as part of the device in the server 10 provided in Example 1.

[0266] In the above embodiment of the present disclosure, the initial recognition model includes: an encoder, a mask modeling module connected to the encoder, an emotion category prediction module, a contrastive learning module, and an emotion label prediction module.

[0267] In the above-mentioned embodiment of the present disclosure, the first pre-training module is also used to combine two training samples in the same data set to obtain a target training sample; use the encoder to extract features of the target training sample to obtain a first sample feature corresponding to the target training sample; perform a mask operation on the first sample feature to obtain a second sample feature containing a first mask mark, and use the mask modeling module to predict the mask mark of the second sample feature to obtain a first predicted mask mark in the second sample feature, wherein the first mask mark is used to characterize the mark that has been masked in the second sample feature; use the emotion category prediction module to perform emotion prediction on the first sample feature to obtain a predicted emotion category of the target training sample; use the contrastive learning module to determine the first similarity between training samples with the same emotion label in the target training sample based on the first sample feature, and the second similarity between any two training samples in the target training sample; based on the first mask mark, the first predicted mask mark, the target emotion category corresponding to the same data set, the predicted emotion category, the first similarity and the second similarity, adjust the model parameters of the initial recognition model to obtain a target recognition model.

[0268] In the above-mentioned embodiment of the present disclosure, the first pre-training module is also used to determine a first mask loss value based on the first mask mark and the first predicted mask mark; determine a category loss value based on the target emotion category and the predicted emotion category; determine a first label loss value based on the first similarity and the second similarity; determine a first target loss value based on the first mask loss value, the category loss value and the first label loss value; and adjust the model parameters of the initial recognition model based on the first target loss value to obtain a target recognition model.

[0269] In the above-mentioned embodiment of the present disclosure, the second pre-training module is also used to use the encoder to extract features of the training samples in the training data set to obtain sample features corresponding to the training samples; perform a mask operation on the sample features to obtain target sample features containing a second mask mark, and use the mask modeling module to predict the mask mark of the target sample features to obtain a second predicted mask mark in the target sample features, wherein the second mask mark is used to characterize the mark that has been masked in the target sample features; use the emotion label prediction module to perform emotion prediction on the sample features to obtain a predicted label set of the training samples, wherein the predicted label set includes predicted emotions corresponding to multiple emotion recognition tasks; based on the second mask mark, the second predicted mask mark, the emotion label set and the predicted label set, the model parameters of the target recognition model are adjusted to obtain an emotion recognition model.

[0270] In the above embodiments of the present disclosure, the second pre-training module is also used to determine a second mask loss value based on the second mask mark and the second predicted mask mark; determine a second label loss value based on the emotion label set and the predicted emotion set; determine a second target loss value based on the second mask loss value and the second label loss value; and adjust the model parameters of the target recognition model based on the second target loss value to obtain an emotion recognition model.

[0271] It should be noted that the preferred implementation scheme involved in the above embodiments of the present disclosure is the same as the solution provided in Example 1, as well as the application scenario and implementation process, but is not limited to the solution provided in Example 1.

[0272] Example 10

[0273] According to an embodiment of the present disclosure, an emotion recognition device for implementing the above-mentioned emotion recognition method is also provided. Figure 12 is a schematic diagram of another emotion recognition device according to an embodiment of the present disclosure. As shown in Figure 12, the device 1200 includes: an acquisition module 1202, an identification module 1204, and a determination module 1206.

[0274] Among them, the acquisition module is configured to acquire the data to be identified, wherein the data to be identified includes data of at least one of the following modalities: visual modality, acoustic modality, text modality and contextual modality; the recognition module is configured to perform emotion recognition on the data to be identified and obtain an initial emotion recognition set, wherein the initial emotion set includes emotion recognition results corresponding to multiple different emotion recognition tasks; the determination module is configured to determine the target emotion recognition result corresponding to the data to be identified based on the target emotion recognition task corresponding to the data to be identified.

[0275] It should be noted that the acquisition module 1202, identification module 1204, and determination module 1206 correspond to steps S602 to S606 in Example 3. The examples and application scenarios implemented by the three modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors. The above modules can also be run as part of the device in the server 10 provided in Example 1.

[0276] It should be noted that the preferred implementation scheme involved in the above embodiments of the present disclosure is the same as the solution provided in Example 1, as well as the application scenario and implementation process, but is not limited to the solution provided in Example 1.

[0277] Example 11

[0278] According to an embodiment of the present disclosure, a data generation device for implementing the above-mentioned data generation method is also provided. Figure 13 is a schematic diagram of a data generation device according to an embodiment of the present disclosure. As shown in Figure 13, the device 1300 includes: a receiving module 1302, an identification module 1304, a determination module 1306, a generation module 1308, and a display module 1310.

[0279] Among them, the receiving module is configured to respond to query data received in the generative dialogue interface, wherein the query data includes data of at least one of the following modalities: visual modality, acoustic modality, text modality and contextual modality; the recognition module is configured to perform emotion recognition on the query data to obtain an initial emotion set corresponding to the query data, wherein the initial emotion set includes emotion recognition results corresponding to multiple different emotion recognition tasks; the determination module is configured to determine the target emotion recognition result corresponding to the query data based on the target emotion recognition task corresponding to the query data; the generation module is configured to generate a reply result matching the query data based on the target emotion recognition result; the display module is configured to display the reply result in the generative dialogue interface.

[0280] It should be noted that the receiving module 1302, identifying module 1304, determining module 1306, generating module 1308, and displaying module 1310 correspond to steps S702 to S710 in Example 5. The examples and application scenarios implemented by the five modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the modules or units can be hardware components or software components stored in a memory and processed by one or more processors. The modules can also be part of the device and can run in the server 10 provided in Example 1.

[0281] It should be noted that the preferred implementation scheme involved in the above embodiments of the present disclosure is the same as the solution provided in Example 1, as well as the application scenario and implementation process, but is not limited to the solution provided in Example 1.

[0282] Example 12

[0283] According to an embodiment of the present disclosure, a data generation device for implementing the above-mentioned data generation method is also provided. Figure 14 is a schematic diagram of another data generation device according to an embodiment of the present disclosure. As shown in Figure 14, the device 1400 includes: an acquisition module 1402, an identification module 1404, a determination module 1406, and a generation module 1408.

[0284] Among them, the acquisition module is configured to acquire query data input in the generative dialogue interface in dialogue mode, wherein the query data includes data of at least one of the following modalities: visual modality, acoustic modality, text modality and contextual modality; the recognition module is configured to perform emotion recognition on the query data to obtain an initial emotion set corresponding to the query data, wherein the initial emotion set includes emotion recognition results corresponding to multiple different emotion recognition tasks; the determination module is configured to determine the target emotion recognition result corresponding to the query data based on the target emotion recognition task corresponding to the query data; the generation module is configured to generate a response result matching the query data based on the target emotion recognition result.

[0285] It should be noted that the acquisition module 1402, identification module 1404, determination module 1406, and generation module 1408 correspond to steps S802 to S808 in Example 5. The examples and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors. The above modules can also be run as part of the device in the server 10 provided in Example 1.

[0286] It should be noted that the preferred implementation scheme involved in the above embodiments of the present disclosure is the same as the solution provided in Example 1, as well as the application scenario and implementation process, but is not limited to the solution provided in Example 1.

[0287] Example 13

[0288] According to an embodiment of the present disclosure, an emotion recognition device for implementing the above-mentioned emotion recognition method is also provided. Figure 15 is a schematic diagram of another emotion recognition device according to an embodiment of the present disclosure. As shown in Figure 15, the device 1500 includes: an acquisition module 1502, an identification module 1504, a determination module 1506, and an output module 1508.

[0289] Among them, the acquisition module is configured to obtain query data by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter is the query data, and the query data includes data of at least one of the following modalities: visual modality, acoustic modality, text modality and contextual modality; the recognition module is configured to perform emotion recognition on the query data to obtain an initial emotion recognition set, wherein the initial emotion recognition set includes emotion recognition results corresponding to multiple emotion recognition tasks; the determination module is configured to determine the target emotion recognition result corresponding to the query data based on the target emotion recognition task corresponding to the query data; the output module is configured to output the target emotion recognition result by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the target emotion recognition result.

[0290] It should be noted that the acquisition module 1502, identification module 1504, determination module 1506, and output module 1508 correspond to steps S902 to S908 in Example 6. The examples and application scenarios implemented by the four modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the modules or units can be hardware components or software components stored in a memory and processed by one or more processors. The modules can also be part of the device and can run in the server 10 provided in Example 1.

[0291] It should be noted that the preferred implementation scheme involved in the above embodiments of the present disclosure is the same as the solution provided in Example 1, as well as the application scenario and implementation process, but is not limited to the solution provided in Example 1.

[0292] Example 14

[0293] According to an embodiment of the present disclosure, an emotion recognition device for implementing the above-mentioned emotion recognition method is also provided. Figure 16 is a schematic diagram of another emotion recognition device according to an embodiment of the present disclosure. As shown in Figure 16, the device 1600 includes: an acquisition module 1602, an identification module 1604, a determination module 1606, a generation module 1608, and an output module 1610.

[0294] Among them, the acquisition module is configured to obtain query data by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter is the query data, and the query data includes data of at least one of the following modalities: visual modality, acoustic modality, text modality and contextual modality; the recognition module is configured to perform emotion recognition on the query data to obtain an initial emotion recognition set, wherein the initial emotion recognition set includes emotion recognition results corresponding to multiple emotion recognition tasks; the determination module is configured to determine the target emotion recognition result corresponding to the query data based on the target emotion recognition task corresponding to the query data; the generation module is configured to generate a reply result matching the query data based on the target emotion recognition result; the output module is configured to output the reply result by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the reply result.

[0295] It should be noted that the acquisition module 1602, identification module 1604, determination module 1606, generation module 1608, and output module 1610 correspond to steps S1002 to S1010 in Example 7. The examples and application scenarios implemented by the five modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors. The above modules can also be run as part of the device in the server 10 provided in Example 1.

[0296] It should be noted that the preferred implementation scheme involved in the above embodiments of the present disclosure is the same as the solution provided in Example 1, as well as the application scenario and implementation process, but is not limited to the solution provided in Example 1.

[0297] Example 15

[0298] The embodiment of the present disclosure may provide a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal.

[0299] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of a computer network.

[0300] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the emotion recognition method: responding to query data received in the generative dialogue interface, wherein the query data includes data of at least one of the following modalities: visual modality, acoustic modality, text modality and contextual modality; performing emotion recognition on the query data to obtain an initial emotion recognition set, wherein the initial emotion recognition set includes emotion recognition results corresponding to multiple emotion recognition tasks; and determining the target emotion recognition result corresponding to the query data based on the target emotion recognition task corresponding to the query data.

[0301] Optionally, Figure 17 is a block diagram of a computer terminal according to an embodiment of the present disclosure. As shown in Figure 17, the computer terminal A may include: one or more (only one is shown in the figure) processors 102, a memory 104, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0302] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the emotion recognition method and device in the embodiments of the present disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned emotion recognition method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely located relative to the processor, and these remote memories can be connected to the terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0303] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: responding to query data received in the generative dialogue interface, wherein the query data includes data of at least one of the following modalities: visual modality, acoustic modality, text modality and contextual modality; performing emotion recognition on the query data to obtain an initial emotion recognition set, wherein the initial emotion recognition set includes emotion recognition results corresponding to multiple emotion recognition tasks; and determining the target emotion recognition result corresponding to the query data based on the target emotion recognition task corresponding to the query data.

[0304] Using an embodiment of the present disclosure, a response is given to query data received in a generative dialogue interface, wherein the query data includes data of at least one of the following modalities: visual modality, acoustic modality, text modality, and contextual modality; emotion recognition is performed on the query data to obtain an initial emotion recognition set, wherein the initial emotion recognition set includes emotion recognition results corresponding to multiple emotion recognition tasks; based on the target emotion recognition task corresponding to the query data, the target emotion recognition result corresponding to the query data is determined, thereby improving the accuracy of emotion recognition on the data; it is easy to notice that emotion recognition can be performed on the query data in combination with multimodal data to obtain emotion recognition results corresponding to multiple emotion recognition tasks. By combining multiple emotion recognition tasks, more emotion knowledge of the query data can be obtained, thereby making the target emotion recognition result corresponding to the query data more accurate, thereby achieving the technical effect of accurate emotion recognition on the data, and thereby solving the technical problem of low accuracy of emotion recognition on the data in related technologies.

[0305] Those skilled in the art will appreciate that the structure shown in FIG17 is merely illustrative, and the computer terminal may also be a smartphone (such as an Android phone, iOS phone, etc.), a tablet computer, a PDA, a mobile internet device (MID), a PAD, or other terminal device. FIG17 does not limit the structure of the aforementioned electronic devices. For example, the computer terminal A may include more or fewer components (such as a network interface, a display device, etc.) than those shown in FIG17 , or may have a configuration different from that shown in FIG17 .

[0306] A person skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0307] Example 18

[0308] The embodiment of the present disclosure further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the emotion recognition method provided in the above embodiment 1.

[0309] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0310] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: responding to query data received in a generative dialogue interface, wherein the query data includes data of at least one of the following modalities: visual modality, acoustic modality, text modality, and contextual modality; performing emotion recognition on the query data to obtain an initial emotion recognition set, wherein the initial emotion recognition set includes emotion recognition results corresponding to multiple emotion recognition tasks; and determining a target emotion recognition result corresponding to the query data based on a target emotion recognition task corresponding to the query data.

[0311] The serial numbers of the above-mentioned embodiments of the present disclosure are for description only and do not represent the advantages or disadvantages of the embodiments.

[0312] In the above embodiments of the present disclosure, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0313] In the several embodiments provided in this disclosure, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0314] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0315] In addition, the functional units in the various embodiments of the present disclosure may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0316] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0317] The above is only a preferred embodiment of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present disclosure. These improvements and modifications should also be regarded as within the scope of protection of the present disclosure.< / context> < / audio> < / iemocap> < / erc>

Claims

1. An emotion recognition method, comprising: Responding to query data received in the generative dialogue interface, wherein the query data includes data in at least one of the following modalities: a visual modality, an acoustic modality, a textual modality, and a contextual modality; Performing emotion recognition on the query data to obtain an initial emotion recognition set, wherein the initial emotion recognition set includes emotion recognition results corresponding to a plurality of emotion recognition tasks; Based on the target emotion recognition task corresponding to the query data, a target emotion recognition result corresponding to the query data is determined.

2. The method according to claim 1, wherein: Performing emotion recognition on the query data to obtain an initial emotion recognition set, including: The emotion recognition model is used to perform emotion recognition on the query data to obtain the initial emotion recognition set.

3. The method according to claim 2, wherein: The emotion recognition model includes: an encoder, a mask modeling module connected to the encoder, an emotion category prediction module, a contrast learning module, and an emotion label prediction module.

4. The method according to claim 3, wherein: Using the emotion recognition model to perform emotion recognition on the query data to obtain the initial emotion recognition set includes: Using the encoder to perform feature extraction on the query data to obtain a query vector corresponding to the query data; The emotion tag prediction module is used to perform emotion recognition on the query vector to obtain the initial emotion recognition set.

5. The method according to claim 4, wherein: Using the emotion tag prediction module to perform emotion recognition on the query vector to obtain the initial emotion recognition set includes: Acquire a preset vector set corresponding to the plurality of emotion recognition tasks, wherein the preset vector set includes at least one preset vector, and the at least one preset vector is used to represent at least one preset emotion corresponding to the emotion recognition task; Matching the query vector with the at least one preset vector in the preset vector set respectively, and determining a target vector in the preset vector set that successfully matches the query vector; The initial emotion recognition result is obtained based on the preset emotions corresponding to the target vectors in the plurality of preset vector sets.

6. The method according to claim 3, wherein: The method further comprises: Acquire a training data set corresponding to the plurality of emotion recognition tasks, wherein the training data in the training data set includes: a plurality of training samples, and a set of emotion labels corresponding to the training samples, The emotion label set includes emotion labels corresponding to the multiple emotion recognition tasks; Based on the emotion category corresponding to the emotion label, the training data set is divided into multiple data sets, wherein the emotion categories corresponding to multiple training data included in the same data set are the same; Pre-training the initial recognition model based on the same data set to obtain a target recognition model, wherein the target recognition model is used to identify the emotion category corresponding to the input data; The target recognition model is pre-trained based on the training data set to obtain the emotion recognition model.

7. The method according to claim 6, wherein: Pre-training the initial recognition model based on the same data set to obtain a target recognition model includes: Combining two training samples in the same data set to obtain a target training sample; Using the encoder to extract features from the target training sample to obtain a first sample feature corresponding to the target training sample; Performing a mask operation on the first sample feature to obtain a second sample feature including a first mask mark, and using the mask modeling module to predict the mask mark on the second sample feature to obtain a first predicted mask mark in the second sample feature, wherein the first mask mark is used to represent a mark in the second sample feature that has been subjected to a mask operation; Using the emotion category prediction module to perform emotion prediction on the first sample feature to obtain a predicted emotion category of the target training sample; Determine, by using the contrastive learning module based on the first sample feature, a first similarity between training samples with the same sentiment label in the target training samples and a second similarity between any two training samples in the target training samples; Based on the first mask mark, the first predicted mask mark, the target emotion category corresponding to the same data set, the predicted emotion category, the first similarity and the second similarity, the model parameters of the initial recognition model are adjusted to obtain the target recognition model.

8. The method according to claim 6, wherein: Pre-training the target recognition model based on the training data set to obtain the emotion recognition model includes: Using the encoder to extract features from the training samples in the training data set to obtain sample features corresponding to the training samples; Performing a mask operation on the sample feature to obtain a target sample feature including a second mask mark, and using the mask modeling module to predict the mask mark on the target sample feature to obtain a second predicted mask mark in the target sample feature, wherein the second mask mark is used to represent a mark in the target sample feature that has been subjected to a mask operation; The emotion label prediction module is used to perform emotion prediction on the sample features to obtain a predicted label set of the training sample, wherein the predicted label set includes the emotion labels corresponding to the multiple emotion recognition tasks. Predict emotions; Based on the second mask mark, the second predicted mask mark, the emotion label set and the predicted label set, the model parameters of the target recognition model are adjusted to obtain the emotion recognition model.

9. The method according to claim 1, wherein: The method further comprises: Outputting the query data, the target emotion recognition task and the target emotion recognition result in a generative dialogue interface; In response to a modification instruction in the generative dialogue interface, a new emotion recognition result corresponding to the modification instruction is obtained, wherein the new emotion recognition result is an emotion recognition result obtained by modifying the target emotion recognition result based on the query data and the target emotion recognition task; Based on the query data and the target emotion recognition result, the emotion recognition model is updated.

10. A pre-training method for an emotion recognition model, wherein: include: Acquire a training data set corresponding to a plurality of emotion recognition tasks, wherein the training data in the training data set includes: a plurality of training samples, and an emotion label set corresponding to the training samples, and the emotion label set includes emotion labels corresponding to the plurality of emotion recognition tasks; Based on the emotion category corresponding to the emotion label, the training data set is divided into multiple data sets, wherein the emotion categories corresponding to multiple training data included in the same data set are the same; Pre-training the initial recognition model based on the same data set to obtain a target recognition model, wherein the target recognition model is used to identify the emotion category corresponding to the input data; The target recognition model is pre-trained based on the training data set to obtain an emotion recognition model, wherein the emotion recognition model is used to perform emotion recognition on query data to obtain an emotion recognition set of the query data, and the emotion recognition set includes emotion recognition results corresponding to the multiple emotion recognition tasks.

11. The method according to claim 10, wherein: The initial recognition model includes: an encoder, a mask modeling module connected to the encoder, an emotion category prediction module, a contrast learning module, and an emotion label prediction module.

12. The method according to claim 11, wherein: Pre-training the initial recognition model based on the same data set to obtain a target recognition model includes: Combining two training samples in the same data set to obtain a target training sample; Using the encoder to extract features from the target training sample to obtain a first sample feature corresponding to the target training sample; Performing a mask operation on the first sample feature to obtain a second sample feature including a first mask mark, and using the mask modeling module to predict the mask mark on the second sample feature to obtain a first predicted mask mark in the second sample feature, wherein the first mask mark is used to represent a mark in the second sample feature that has been subjected to a mask operation; Using the emotion category prediction module to perform emotion prediction on the first sample feature to obtain a predicted emotion category of the target training sample; Determine, by using the contrastive learning module based on the first sample feature, a first similarity between training samples with the same sentiment label in the target training samples and a second similarity between any two training samples in the target training samples; Based on the first mask mark, the first predicted mask mark, the target emotion category corresponding to the same data set, the predicted emotion category, the first similarity and the second similarity, the model parameters of the initial recognition model are adjusted to obtain the target recognition model.

13. The method according to claim 12, wherein: Based on the first mask mark, the first predicted mask mark, the target emotion category corresponding to the same data set, the predicted emotion category, the first similarity, and the second similarity, adjusting the model parameters of the initial recognition model to obtain the target recognition model, including: Determining a first mask loss value based on the first mask mark and the first predicted mask mark; Determining a category loss value based on the target emotion category and the predicted emotion category; Determining a first label loss value based on the first similarity and the second similarity; Determine a first target loss value based on the first mask loss value, the category loss value, and the first label loss value; The model parameters of the initial recognition model are adjusted based on the first target loss value to obtain the target recognition model.

14. The method according to claim 11, wherein: Pre-training the target recognition model based on the training data set to obtain the emotion recognition model includes: Using the encoder to extract features from the training samples in the training data set to obtain sample features corresponding to the training samples; Performing a mask operation on the sample feature to obtain a target sample feature including a second mask mark, and using the mask modeling module to predict the mask mark on the target sample feature to obtain a second predicted mask mark in the target sample feature, wherein the second mask mark is used to represent a mark in the target sample feature that has been subjected to a mask operation; Using the emotion label prediction module to perform emotion prediction on the sample features to obtain a predicted label set of the training samples, wherein the predicted label set includes predicted emotions corresponding to the multiple emotion recognition tasks; Based on the second mask mark, the second predicted mask mark, the emotion label set and the predicted label set, the model parameters of the target recognition model are adjusted to obtain the emotion recognition model.

15. The method according to claim 14, wherein: Based on the second mask mark, the second predicted mask mark, the emotion tag set and the predicted emotion set, adjusting the model parameters of the target recognition model to obtain the emotion recognition model, including: determining a second mask loss value based on the second mask mark and the second predicted mask mark; Determining a second label loss value based on the emotion label set and the predicted emotion set; Determining a second target loss value based on the second mask loss value and the second label loss value; The model parameters of the target recognition model are adjusted based on the second target loss value to obtain the emotion recognition model.

16. A method for emotion recognition, wherein: include: Acquiring data to be identified, wherein the data to be identified includes data of at least one of the following modalities: visual modality, acoustic modality, text modality, and contextual modality; Performing emotion recognition on the data to be recognized to obtain an initial emotion recognition set, wherein the initial emotion set includes emotion recognition results corresponding to a plurality of different emotion recognition tasks; Based on the target emotion recognition task corresponding to the data to be recognized, a target emotion recognition result corresponding to the data to be recognized is determined.

17. A data generation method, wherein: include: Responding to query data received in the generative dialogue interface, wherein the query data includes data in at least one of the following modalities: a visual modality, an acoustic modality, a textual modality, and a contextual modality; Performing emotion recognition on the query data to obtain an initial emotion set corresponding to the query data, wherein the initial emotion set includes emotion recognition results corresponding to a plurality of different emotion recognition tasks; Based on the target emotion recognition task corresponding to the query data, determining the target emotion recognition result corresponding to the query data; Based on the target emotion recognition result, generating a response result matching the query data; The response result is displayed in the generative dialogue interface.

18. A data generation method, wherein: include: Acquiring query data input in the generative dialogue interface in the dialogue mode, wherein the query data includes data of at least one of the following modalities: visual modality, acoustic modality, text modality, and contextual modality; Performing emotion recognition on the query data to obtain an initial emotion set corresponding to the query data, wherein the initial emotion set includes emotion recognition results corresponding to a plurality of different emotion recognition tasks; Based on the target emotion recognition task corresponding to the query data, determining the target emotion recognition result corresponding to the query data; Based on the target emotion recognition result, a response result matching the query data is generated.

19. An electronic device, wherein: include: A memory storing an executable program; A processor for running the program, wherein the program executes claims 1 to 18 when running. Any one of the methods described in .

20. A computer-readable storage medium, wherein: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 18.

Images