Data processing methods, apparatus, equipment, storage media, and computer program products

Abstract

This application provides a data processing method, apparatus, device, storage medium, and computer program product, applicable to various emotion recognition scenarios such as cloud technology, artificial intelligence, smart transportation, games, and in-vehicle systems. The data processing method includes: predicting the emotion of the information to be recognized in the current round, obtaining the current emotion direction and a first probability for each candidate emotion among multiple candidate emotions, where the current emotion direction is the direction to which the emotion of the information to be recognized belongs; determining a second probability corresponding to each candidate emotion from an emotion transition relationship based on the current emotion direction, where the emotion transition relationship represents the probability of transitioning between historical emotions and candidate emotions based on the emotion direction; integrating the first and second probabilities to obtain a target probability for the candidate emotion; and determining the emotion recognition result of the information to be recognized based on the target probability corresponding to each candidate emotion among multiple candidate emotions. This application can improve the accuracy of emotion recognition.

Description

Technical Field

[0001] This application relates to artificial intelligence technology in the field of computer applications, and more particularly to a data processing method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] Emotion recognition refers to the process of identifying emotions and is a component of affective computing. To achieve emotion recognition, it is typically performed directly on the information to be identified, thus determining the corresponding emotion. However, because this process relies on a single set of data for emotion recognition, its accuracy is compromised. Summary of the Invention

[0003] This application provides a data processing method, apparatus, device, computer-readable storage medium, and computer program product that can improve the accuracy of emotion recognition.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a data processing method, including: Emotion prediction is performed on the information to be identified in the current round to obtain the current emotion direction and the first probability of each of the candidate emotions among a variety of candidate emotions, wherein the current emotion direction is the direction to which the emotion of the information to be identified belongs, and the candidate emotions are pre-set emotions; Based on the current emotional direction, a second probability corresponding to each candidate emotion is determined from the emotional transition relationship, wherein the emotional transition relationship represents the probability of transitioning between historical emotions and candidate emotions in combination with the emotional direction, and the historical emotions are the emotions predicted before the current round; The first probability and the second probability are integrated to obtain the target probability of the candidate emotion; Based on the target probability corresponding to each of the multiple candidate emotions, the emotion recognition result of the information to be identified is determined.

[0005] This application provides a data processing apparatus, including: The emotion prediction module is used to predict the emotion of the information to be identified in the current round, and to obtain the current emotion direction and the first probability of each of the candidate emotions among a variety of candidate emotions, wherein the current emotion direction is the direction to which the emotion of the information to be identified belongs, and the candidate emotions are pre-set emotion types; The jump prediction module is used to determine a second probability for each candidate emotion from the emotion jump relationship based on the current emotion direction, wherein the emotion jump relationship represents the probability of jumping between the historical emotion and the candidate emotion in combination with the emotion direction, and the historical emotion is the emotion predicted before the current round; The probability integration module is used to integrate the first probability and the second probability to obtain the target probability of the candidate emotion; The result determination module is used to determine the emotion recognition result of the information to be identified based on the target probability corresponding to each of the multiple candidate emotions.

[0006] In this embodiment, the data processing device further includes a relationship acquisition module, used to acquire multiple emotion sample sequences and an emotion direction sample for each emotion sample in the emotion sample sequence, wherein the emotion sample sequence refers to N consecutive emotion samples in labeled rounds, where N is a positive integer; based on the multiple emotion sample sequences and the emotion direction sample for each emotion sample, multiple sample tuple sequences are generated, wherein the sample tuple sequence includes multiple sample tuples, each sample tuple including the emotion sample in round n, the emotion sample in round n+1, and the emotion direction sample of the emotion sample in round n+1, where n∈N-1; based on the multiple sample tuple sequences, the probability of a jump between two emotion samples in adjacent rounds for the emotion direction sample is calculated, wherein the emotion jump relationship is determined from the statistical results based on the historical emotions.

[0007] In this embodiment of the application, the probability integration module is further configured to obtain a first integration weight negatively correlated with the current round and a second integration weight positively correlated with the current round, wherein the current round represents the prediction order of the current round; obtain a first integration result of the first probability and the first integration weight; obtain a second integration result of the second probability and the second integration weight; and determine the combination result of the first integration result and the second integration result as the target probability of the candidate emotion.

[0008] In this embodiment of the application, the result determination module is further configured to: sort multiple candidate emotions in reverse order based on the target probability to obtain a sequence of emotions to be screened; screen a target emotion sequence from the sequence of emotions to be screened based on specified screening parameters, wherein the specified screening parameters include one or two of a probability threshold parameter and an emotion quantity parameter; and determine the target emotion sequence and the target probability of each target emotion in the target emotion sequence as the emotion recognition result.

[0009] In this embodiment of the application, the result determination module is further configured to filter a specified number of candidate emotions from a reverse arrangement of multiple candidate emotions based on the target probability; and determine the specified number of candidate emotions as the current candidate emotion sequence, wherein the current candidate emotion sequence is used to determine the historical emotions for the next round of prediction.

[0010] In this embodiment of the application, the emotion prediction module is further configured to perform emotion prediction on the information to be identified in the current round, and obtain a third probability corresponding to each emotion direction; determine the emotion direction with the largest third probability as the baseline emotion direction; determine the emotion direction whose probability difference with the third baseline probability is lower than the probability difference threshold as the candidate emotion direction, wherein the third baseline probability refers to the third probability of the baseline emotion direction; and determine the baseline emotion direction and the candidate emotion direction as the current emotion direction.

[0011] In this embodiment of the application, the emotion prediction module is further configured to extract a semantic string sequence based on the information to be identified in the current round, wherein each semantic string in the semantic string sequence is used to represent emotion semantics; perform feature extraction on the semantic string sequence to obtain a feature to be predicted; and perform semantic emotion prediction on the feature to be predicted to obtain the current emotion direction and the first probability of each of the multiple candidate emotions.

[0012] In this embodiment, the feature extraction and semantic emotion prediction are implemented through an emotion prediction model, wherein the emotion prediction model is trained through the following steps: acquiring identification information samples and acquiring the emotion label and emotion direction label corresponding to the identification information samples; performing emotion prediction on the identification information samples based on the prediction model to be trained to obtain the emotion prediction probability and the emotion direction prediction probability, wherein the prediction model to be trained is a network model to be trained for emotion prediction; and training the prediction model to be trained by combining the difference between the emotion prediction probability and the emotion label, and the difference between the emotion direction prediction probability and the emotion direction label, to obtain the emotion prediction model.

[0013] In this embodiment of the application, the emotion prediction module is further configured to perform text recognition on the audio to be recognized in the current round to obtain the text to be recognized; extract each semantic string from the text to be recognized; and combine each semantic string to obtain the semantic string sequence.

[0014] In this embodiment of the application, the data processing device further includes an emotion application module, which is used to generate facial expression rendering data of the virtual object to be presented based on the emotion recognition result when the information to be identified is the audio to be played of the virtual object to be presented; and to render the facial expression of the virtual object to be presented based on the facial expression rendering data when the audio to be played is played.

[0015] In this embodiment of the application, the emotion application module is further configured to determine the interactive emotional state based on the emotion recognition result when the information to be identified is an interactive message; and to display a prompt message when the interactive emotional state is an emotional state pending review, wherein the prompt message is used to prompt for a change in the interactive emotional state.

[0016] This application provides an electronic device for emotion-based data processing, comprising: Memory is used to store executable instructions for a computer; The processor, when executing computer-executable instructions stored in the memory, implements the data processing method provided in the embodiments of this application.

[0017] This application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the data processing method provided in this application.

[0018] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the data processing method provided in this application.

[0019] The embodiments of this application have at least the following beneficial effects: In the process of emotion prediction, the first probability corresponding to each candidate emotion in a single round and the current emotion direction are predicted based on the current information to be identified. Then, the second probability of jumping from historical emotions to candidate emotions is determined based on the current emotion direction. Finally, the first probability and the second probability are integrated into the target probability of the candidate emotion to obtain the final emotion recognition result. This makes the emotion recognition result not only based on the prediction result of a single round, but also combined with the probability of jumping from historical emotions to candidate emotions. Thus, the data on which the emotion recognition result is obtained is diverse in time and space, thereby improving the accuracy of emotion recognition. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of an exemplary emotion recognition process; Figure 2 This is an exemplary feature acquisition process diagram; Figure 3This is a schematic diagram of the architecture of the emotion processing system provided in the embodiments of this application; Figure 4 This is provided by the embodiments of this application. Figure 3 A schematic diagram of the server's structure; Figure 5 This is a flowchart illustrating the data processing method provided in the embodiments of this application. Figure 1 ; Figure 6 This is a schematic diagram illustrating an exemplary emotion transition relationship provided in an embodiment of this application; Figure 7 This is a flowchart illustrating the data processing method provided in the embodiments of this application. Figure 2 ; Figure 8 This is a flowchart illustrating the data processing method provided in the embodiments of this application. Figure 3 ; Figure 9 This is a flowchart illustrating the data processing method provided in the embodiments of this application. Figure 4 ; Figure 10 This is an exemplary flowchart of an emotion recognition application provided in an embodiment of this application; Figure 11 This is an exemplary schematic diagram of generating virtual human facial expressions provided in an embodiment of this application; Figure 12 This is a flowchart illustrating an exemplary emotion recognition method provided in an embodiment of this application; Figure 13 This is a schematic diagram of the structure of an exemplary depth model provided in an embodiment of this application; Figure 14 This is an exemplary state machine diagram provided in an embodiment of this application; Figure 15 This is an exemplary schematic diagram illustrating the determination of a final result provided in an embodiment of this application; Figure 16 This is a schematic diagram of an exemplary emotion recognition interface provided in an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0023] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0024] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0025] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0026] 1) Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0027] 2) Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills; and how to reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning typically includes techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning.

[0028] 3) Artificial neural networks are mathematical models that mimic the structure and function of biological neural networks. Exemplary structures of artificial neural networks in this application include Graph Convolutional Networks (GCNs, a type of neural network for processing graph-structured data), Deep Neural Networks (DNNs), Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Neural State Machines (NSMs), and Phase-Functioned Neural Networks (PFNNs). The emotion prediction model and the prediction model to be trained involved in this application are all models corresponding to artificial neural networks.

[0029] It should be noted that in order to achieve emotion recognition, the emotion recognition is usually performed directly on the information to be recognized in order to identify the corresponding emotion. However, since the data used for emotion recognition is limited in the above process, the accuracy of emotion recognition is affected.

[0030] Additionally, when the information to be identified is audio, physical information about the information can be incorporated to achieve emotion recognition; see [link to related document]. Figure 1 , Figure 1 This is an exemplary diagram illustrating the emotion recognition process; such as Figure 1 As shown, the exemplary emotion recognition process includes steps 101 to 104, and each step is described below.

[0031] Step 101: Obtain the first and second features of the audio to be recognized.

[0032] It should be noted that the process for obtaining the first and second features is described in [reference needed]. Figure 2 , Figure 2 This is an exemplary feature acquisition process diagram; such as Figure 2 As shown, this exemplary feature acquisition process includes steps 201 and 202. Step 201 refers to extracting a first feature from the audio to be identified using a waveform-to-vector model (Wav2vec model), and step 202 refers to extracting a second feature from the audio to be identified using a speaker classification model. Here, the first feature represents features related to the waveform of the audio to be identified, and the second feature represents features related to the speaker of the audio to be identified.

[0033] Step 102: Input the first feature and the second feature into the speech emotion recognition model to identify the emotion category and obtain the first recognition result.

[0034] It should be noted that the emotion features are decoupled using the first and second features, and the emotion features obtained from the decoupling are used to identify the emotion category.

[0035] Step 103: Use the text emotion recognition model to perform text recognition on the text content corresponding to the audio to be recognized, and obtain the second recognition result.

[0036] Step 104: Weight the first and second identification results to obtain the third identification result.

[0037] because Figure 1 and Figure 2 The emotion recognition described herein is a process of identifying a single emotion from the audio sample. Therefore, the data used for emotion recognition is limited, affecting its accuracy. Furthermore, although... Figure 1 and Figure 2 The emotion recognition described in the article combines speech spectrum features and voiceprint features (referred to as the first feature and the second feature) extracted from audio physical information, as well as text content. However, since the accuracy of audio physical information in emotion recognition is lower than the accuracy threshold, it still affects the accuracy of emotion recognition.

[0038] Based on this, embodiments of this application provide a data processing method, apparatus, device, computer-readable storage medium, and computer program product, which can improve the accuracy of emotion recognition. The following describes exemplary applications of the electronic device (hereinafter referred to as the emotion processing device) provided in this application for emotion-based data processing. The emotion processing device provided in this application can be implemented as various types of terminals such as smartphones, smartwatches, laptops, tablets, desktop computers, smart home appliances, set-top boxes, smart in-vehicle devices, portable music players, personal digital assistants, dedicated messaging devices, intelligent voice interaction devices, portable gaming devices, and smart speakers, or it can be implemented as a server. The following will describe exemplary applications when the emotion processing device is implemented as a server.

[0039] See Figure 3 , Figure 3 This is a schematic diagram of the architecture of the emotion processing system provided in the embodiments of this application; as shown below. Figure 3As shown, to support an emotion processing application, in the emotion processing system 100, terminal 200 (terminals 200-1 and 200-2 are shown as examples) connects to server 400 via network 300. Network 300 can be a wide area network (WAN), a local area network (LAN), or a combination of both. Additionally, the emotion processing system 100 includes a database 500 for providing data support to server 400; and... Figure 3 The example shown illustrates a scenario where the database 500 is independent of the server 400. However, the database 500 can also be integrated into the server 400, and this embodiment does not limit this to any particular case.

[0040] Terminal 200 is configured to receive, via network 300, audio to be played (referred to as identification information) and facial expression rendering data of a virtual object to be presented from server 400, and, while playing the audio to be played, display corresponding text information (e.g., "How can this be?") on a graphical interface (exemplarily shown as graphical interface 210-1 in terminal 200-1), and render the facial expression of the virtual object to be presented based on the facial expression rendering data. Alternatively, it is configured to send interactive information (referred to as identification information) to server 400 via network 300, and receive prompt information sent by server 400 via network 300, and present prompt information (e.g., "Please use polite language!") on a graphical interface (exemplarily shown as graphical interface 210-2 in terminal 200-2).

[0041] Server 400 is used to predict the emotion of the information to be identified in the current round, obtaining the current emotion direction and the first probability of each candidate emotion among multiple candidate emotions, where the current emotion direction is the direction to which the emotion of the information to be identified belongs; based on the current emotion direction, determining the second probability corresponding to each candidate emotion from the emotion jump relationship, where the emotion jump relationship represents the probability of jumping between historical emotions and candidate emotions combined with the emotion direction; integrating the first probability and the second probability to obtain the target probability of the candidate emotion; and determining the emotion recognition result of the information to be identified based on the target probability corresponding to each candidate emotion among multiple candidate emotions. It is also used to generate facial expression rendering data of the virtual object to be presented based on the emotion recognition result, and send the audio to be played and the facial expression rendering data of the virtual object to be presented to terminal 200 through network 300; or, it is also used to receive interactive information sent by terminal 200 through network 300, determine the interactive emotional state based on the emotion recognition result, and send a prompt message to terminal 200 when the interactive emotional state is a pending review state.

[0042] In some embodiments, server 400 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal 200 may be a smartphone, smartwatch, laptop, tablet, desktop computer, smart TV, set-top box, smart in-vehicle device, portable music player, personal digital assistant, dedicated messaging device, portable gaming device, and smart speaker, but is not limited to these. Terminals and servers can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0043] See Figure 4 , Figure 4 This is provided by the embodiments of this application. Figure 3 A schematic diagram of the server's structure. Figure 4 The server 400 shown includes at least one processor 410, memory 450, and at least one network interface 420. The various components in server 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 4 The general labeled all buses as Bus System 440.

[0044] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0045] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0046] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0047] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0048] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, Wi-Fi, and Universal Serial Bus (USB), etc. The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0049] In some embodiments, the data processing apparatus (hereinafter referred to as the data processing apparatus) provided in this application can be implemented in software. Figure 4 A data processing device 455 stored in memory 450 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: a sentiment prediction module 4551, a jump prediction module 4552, a probability integration module 4553, a result determination module 4554, a relationship acquisition module 4555, and a sentiment application module 4556. These modules are logically connected and can therefore be arbitrarily combined or further divided according to their implemented functions. The functions of each module will be described below.

[0050] In some embodiments, the data processing apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data processing method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0051] In some embodiments, the terminal or server can implement the data processing method provided in this application by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), that is, a program that needs to be installed in the operating system to run, such as a live streaming APP, a game APP, a virtual human APP, or an instant messaging APP; it can also be a mini-program, that is, a program that only needs to be downloaded into a browser environment to run; or it can be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin.

[0052] The data processing method provided in this application will be described below with reference to exemplary applications and implementations of the emotion processing device provided in the embodiments of this application. Furthermore, the data processing method provided in the embodiments of this application can be applied to various emotion recognition scenarios such as cloud technology, artificial intelligence, smart transportation, games, and in-vehicle systems.

[0053] See Figure 5 , Figure 5 This is a flowchart illustrating the data processing method provided in the embodiments of this application. Figure 1 , will combine Figure 5 The steps shown are explained.

[0054] Step 501: Perform emotion prediction on the information to be identified in the current round to obtain the current emotion direction and the first probability of each candidate emotion among multiple candidate emotions.

[0055] In this embodiment, when the emotion processing device obtains an object to be identified by emotion, it triggers an emotion-based data processing flow; the object to be identified by emotion is the information to be identified in the current round. Here, when the emotion processing device performs emotion-based data processing on the information to be identified, it first predicts the emotion based on the information itself. At this time, the emotion processing device can obtain the probability corresponding to each of the multiple candidate emotions by predicting the emotion of the information to be identified, which is called the first probability, and can also obtain the direction to which the current emotion belongs, which is called the current emotion direction.

[0056] It should be noted that the information to be identified can be audio (e.g., audio of a sentence), text (e.g., text of a sentence, a message), or a combination of both, etc. This application embodiment does not limit this. Emotion prediction refers to the processing of determining the emotion expressed by the information to be identified. It can be based on the semantics of the information to be identified, or on the physical information of the information to be identified (e.g., audio waveform and timbre), or a combination of both, etc. This application embodiment does not limit this. Here, the emotion processing device pre-sets multiple candidate emotions, each candidate emotion representing a preset emotion, such as normal, anger, disgust, depression, excitement, joy, surprise, etc. Furthermore, the first probability represents the likelihood that the emotion of the information to be identified is each candidate emotion, and the likelihood that the emotion of the information to be identified is each candidate emotion is positively correlated with the first probability; while the current emotion direction is the direction in which the emotion jumps from the historical emotion to the current moment, such as positive (representing the direction corresponding to positive emotions such as excitement, joy, surprise, and smiling), negative (representing the direction corresponding to negative emotions such as anger, disgust, sadness, contempt, irony, and resentment), empty (representing no emotion), etc. Among them, the first probability of each candidate emotion obtained by the emotion processing device in predicting the emotion of the information to be identified is a single-round emotion prediction result.

[0057] For example, the emotion processing device predicts the emotion of "You are very good" (referred to as the information to be identified), and the results are: positive (referred to as the current emotion direction); normal (0.01), angry (0.10), disgust (0.50), depressed (0.02), excited (0.70), joyful (0.90), surprised (0.80) (referred to as the first probability of the candidate emotion).

[0058] Step 502: Based on the current emotional direction, determine the second probability corresponding to each candidate emotion from the emotional transition relationship.

[0059] In this embodiment, the emotion processing device, when performing emotion-based data processing, includes not only single-round emotion prediction but also the acquisition of the probability of historical emotions transitioning to candidate emotions. Here, the emotion processing device acquires the probability of historical emotions transitioning to candidate emotions based on emotion transition relationships. These relationships can be stored in the emotion processing device, obtained from other devices (e.g., databases), or from received information (e.g., emotion recognition requests), etc., and this embodiment does not limit the specifics. Since the emotion transition relationship represents the probability of transitioning between historical and candidate emotions based on the emotion direction, the emotion processing device can determine the probability of transitioning to each candidate emotion based on historical emotions and the current emotion direction. Based on this determined probability, a second probability is obtained, representing the likelihood of transitioning from a historical emotion to that candidate emotion, and the likelihood of transitioning from a historical emotion to that candidate emotion is positively correlated with the second probability.

[0060] It should be noted that "historical emotion" refers to the emotion predicted before the current round, and the historical emotion is the emotion of the context information preceding the information to be identified, which is the emotion identification object before the current round. Here, the historical emotion can be the emotion predicted in the most recent M (M is a positive integer) times before the current moment; and each most recent predicted emotion can be an optimal emotion or a variety of selected emotions, which is not limited in this embodiment; for example, disgust; or disgust and anger, etc.; where the optimal emotion is the emotion with the highest probability among the various historically predicted emotions, and the various selected emotions are the L emotions with the highest probability ranking among the various historically predicted emotions, where L is a positive integer greater than 1. The emotion jump relationship includes each historically predicted emotion to each candidate emotion, and the jump probability under each emotion direction; for example, if the types of emotions predicted in the current round are C1, the types of candidate emotions are C2, and the types of emotion directions are C3, then the emotion jump relationship includes at most C1. C2 C3 types of jump relationships; in addition, each jump relationship in the emotion jump relationship can be represented as a quadruple, including a historically predicted emotion, a candidate emotion, an emotion direction, and a jump probability, which represents the jump probability to a candidate emotion when a historically predicted emotion jumps to an emotion direction; it is easy to see that the historically predicted emotion in this quadruple can include at least one round of historically predicted emotions.

[0061] For example, see Figure 6 , Figure 6 This is a schematic diagram illustrating an exemplary emotion transition relationship provided in an embodiment of this application; as shown below. Figure 6As shown, historical emotions include emotions 6-11 to 6-13, and emotions 6-21 to 6-24 are all candidate emotions. Emotion direction 6-31 is the current emotion direction. Emotions 6-11 to 6-13, emotions 6-21 to 6-24, as well as emotion directions 6-31 and 6-32 correspond to emotion jump relationship 6-4.

[0062] Step 503: Integrate the first probability and the second probability to obtain the target probability of the candidate emotion.

[0063] In the embodiments of this application, after the emotion processing device obtains a first probability and a second probability for each candidate emotion, it integrates the first probability and the second probability and determines the obtained integrated result as the final probability of the candidate emotion, which is referred to here as the target probability.

[0064] It should be noted that when the emotion processing device integrates the first probability and the second probability into the target probability, it can do so by accumulation, by weighted summation, by averaging, etc. This application embodiment does not limit this.

[0065] Step 504: Based on the target probability corresponding to each candidate emotion among multiple candidate emotions, determine the emotion recognition result of the information to be identified.

[0066] In this embodiment, after obtaining the target probability corresponding to each candidate emotion for multiple candidate emotions, the emotion processing device can directly determine the multiple target probabilities corresponding to multiple candidate emotions as the emotion recognition result. It can also filter the emotion recognition result from the reverse order of the target probabilities corresponding to multiple candidate emotions based on thresholds and quantity, etc. This embodiment does not limit this.

[0067] It should be noted that the emotion recognition result represents the emotion expressed by the information to be recognized. It can be an optimal (i.e., the most likely) candidate emotion and its corresponding target probability, or it can be at least two candidate emotions with a target probability greater than a specified threshold and their respective target probabilities, or it can be the target probabilities corresponding to all candidate emotions.

[0068] Understandably, in the process of emotion prediction, the first probability of each candidate emotion in a single round and the current emotion direction are predicted based on the current information to be identified. Then, based on the current emotion direction, the second probability of jumping from historical emotions to candidate emotions is determined. Finally, the first and second probabilities are integrated into the target probability of the candidate emotion to obtain the final emotion recognition result. This makes the emotion recognition result not only based on the prediction result of a single round, but also combined with the probability of jumping from historical emotions to candidate emotions. Therefore, the data used to obtain the emotion recognition result is diverse in time and space, thus improving the accuracy of emotion recognition.

[0069] See Figure 7 , Figure 7 This is a flowchart illustrating the data processing method provided in the embodiments of this application. Figure 2 ;like Figure 7 As shown in the embodiment of this application, steps 505 to 507 are included before step 502; that is, before the emotion processing device determines the second probability corresponding to each candidate emotion from the emotion jump relationship based on the current emotion direction, the data processing method further includes steps 505 to 507, and each step is described below.

[0070] Step 505: Obtain the emotion sample sequence and the emotion direction sample of each emotion sample in the emotion sample sequence.

[0071] In this embodiment, the emotion sample sequence refers to a series of consecutively labeled emotion samples. Each emotion sample represents an emotion and is sample data. In other words, an emotion sample refers to the emotion of the sample data. The emotion direction sample is the direction of the emotion obtained by labeling each emotion sample, such as positive, negative, or empty (indicating no emotion). Here, the emotion processing device obtains the emotion sample sequence and the emotion direction sample of each emotion sample in the emotion sample sequence by labeling the emotions and emotion directions of multiple consecutive pieces of information.

[0072] Step 506: Generate multiple sample tuples based on the emotion sample sequence and the emotion direction sample of each emotion sample.

[0073] In this embodiment, the emotion processing device selects two adjacent emotion samples from the emotion sample sequence, thus obtaining the Nth round emotion sample and the N+1th round emotion sample. Then, the emotion processing device combines the Nth round emotion sample, the N+1th round emotion sample, and the emotion direction sample of the N+1th round emotion sample into a sample tuple, thereby enabling the emotion processing device to obtain multiple sample tuples. Therefore, each sample tuple includes the Nth round emotion sample, the N+1th round emotion sample, and the emotion direction sample of the N+1th round emotion sample, where N is a positive integer.

[0074] Step 507: Based on multiple sample tuples, calculate the probability of a jump between two emotion samples in adjacent rounds, where the emotion jump relationship is determined from the statistical results based on historical emotions.

[0075] In this embodiment, the emotion processing device uses multiple sample tuples as sample data to statistically determine the probability of a jump between two emotion samples in adjacent rounds, based on the emotion direction sample. This yields the probability of a jump from all emotions in the previous round to all emotions in the next round, combined with the emotion direction; this is called the statistical result. Since historical emotions are known, the emotion processing device filters jump relationships from the statistical result based on historical emotions and the current emotion direction, thus obtaining the emotion jump relationships. Furthermore, the statistical result can be represented by a state machine, which describes the transition relationships between different states. Therefore, when using a state machine to represent the statistical result, emotion samples can be used as nodes in the state machine, the probability of a jump between emotion samples can be used as the output of the jump arc, and the emotion direction sample corresponding to the target node of the jump can be used as the input of the jump arc.

[0076] It should be noted that the emotion processing device is based on multiple sample tuples. It counts the probability of jumping to an emotion direction sample between two emotion samples in adjacent rounds. It can first count the total number of jumps corresponding to each emotion in the previous round based on multiple sample tuples, then count the number of sub-jumps to an emotion in the next round based on each emotion in the previous round combined with each emotion direction sample, and finally determine the probability of jumping by the ratio of the sub-jumps to the total number.

[0077] In this embodiment of the application, the emotion transition relationship can also be obtained through a neural network model; in this case, the input of the neural network model is the historical emotion, the current emotion direction and the candidate emotion, and the output is the second probability.

[0078] See Figure 8 , Figure 8 This is a flowchart illustrating the data processing method provided in the embodiments of this application. Figure 3 ;like Figure 8 As shown in the embodiment of this application, step 503 can be implemented through steps 5031 to 5033. That is, the emotion processing device integrates the first probability and the second probability to obtain the target probability of the candidate emotion, including steps 5031 to 5033. Each step will be described below.

[0079] Step 5031: Obtain the first integration weight that is negatively correlated with the current round and the second integration weight that is positively correlated with the current round.

[0080] In this embodiment, when the emotion processing device integrates the first probability and the second probability using a weighted summation method, it can use fixed weights to integrate the first probability and the second probability, or it can use dynamic weights based on the current round. This embodiment does not limit the specific method used. When the emotion processing device uses dynamic weights to integrate the first probability and the second probability, it can first determine the current round and then dynamically determine the integration weights based on the current round. The current round represents the prediction order of the current round, for example, 1, 4.

[0081] It should be noted that the first integration weight is negatively correlated with the current round, while the second integration weight is positively correlated with the current round; furthermore, the first integration weight and the second integration weight can also be mutually exclusive, such as the sum of the two being a specified value (for example, the sum of the first integration weight and the second integration weight is 1).

[0082] In this embodiment, the emotion processing device obtains a first integrated weight negatively correlated with the current round and a second integrated weight positively correlated with the current round. This includes: the emotion processing device obtaining a previous first integrated weight corresponding to the previous round, and lowering this previous first integrated weight by a specified adjustment range to obtain a first integrated weight negatively correlated with the current round; and obtaining a previous second integrated weight corresponding to the previous round, and raising this previous second integrated weight by a specified adjustment range to obtain a second integrated weight positively correlated with the current round. The specified adjustment range for different prediction rounds can be the same or different, and this embodiment does not limit this. Furthermore, when the current round is the first round, both the previous first integrated weight and the previous second integrated weight are specified weight values.

[0083] For example, in the current round 1, the first integration weight can be 0.3 and the second integration weight can be 0.7; in the current round 2, the first integration weight can be 0.35 and the second integration weight can be 0.65; in the current round 3, the first integration weight can be 0.38 and the second integration weight can be 0.62; ... the later the round, the greater the weight of the second probability and the smaller the weight of the first probability.

[0084] Step 5032: Obtain the first integration result of the first probability and the first integration weight, and obtain the second integration result of the second probability and the second integration weight.

[0085] In this embodiment, the emotion processing device fuses a first probability with a first integration weight to obtain a first fusion result; for example, the emotion processing device determines the first integration result as the product of the first probability and the first integration weight. Similarly, the emotion processing device fuses a second probability with a second integration weight to obtain a second fusion result; for example, the emotion processing device determines the second integration result as the product of the second probability and the second integration weight.

[0086] Step 5033: Combine the first integration result and the second integration result to determine the target probability of the candidate emotion.

[0087] In this embodiment, after obtaining the first integration result and the second integration result, the emotion processing device combines the first integration result and the second integration result to obtain the target probability of the candidate emotion. For example, the emotion processing device determines the target probability of the candidate emotion by the sum of the first integration result and the second integration result.

[0088] In this embodiment, when the emotion processing device obtains the second probability, if there are multiple jump paths for a candidate emotion, the emotion processing device integrates the multiple path probabilities corresponding to the multiple jump paths into a second probability. Furthermore, the integration process can also determine a path probability integration weight that is positively correlated with the probability of historical emotions in the jump path, based on the probability of historical emotions in the jump path. Additionally, the probability integrated from multiple path probabilities can be adjusted based on the number of jump paths. For example, the adjustment direction is positively correlated with the number of jump paths; the integrated probability is increased when the number is greater than a threshold, and decreased when the number is less than a threshold. The jump path probability can be the product of all jump probabilities in that jump path.

[0089] In this embodiment of the application, when the emotion processing device filters out the emotion recognition result from the target probabilities corresponding to multiple candidate emotions based on at least one of a threshold and a quantity, step 504, in which the emotion processing device determines the emotion recognition result based on the target probability corresponding to each candidate emotion among the multiple candidate emotions, includes: the emotion processing device sorts the multiple candidate emotions in reverse order based on the target probability to obtain a sequence of emotions to be filtered; and filters a target emotion sequence from the sequence of emotions to be filtered based on specified filtering parameters; finally, the target emotion sequence and the target probability of each target emotion in the target emotion sequence are determined as the emotion recognition result.

[0090] It should be noted that the specified filtering parameters include one or both of probability threshold parameters and emotion quantity parameters, or the specified filtering parameters may be at least one of threshold and quantity. Here, when the specified filtering parameters include probability threshold parameters, the emotion processing device can identify all candidate emotions with a target probability greater than the probability threshold parameter as the target emotion sequence. When the specified filtering parameters include emotion quantity parameters, the emotion processing device can determine the maximum target probability of each emotion quantity parameter and combine the corresponding candidate emotions into the target emotion sequence. Here, the probability threshold parameter is a specified probability threshold representing the minimum probability used to filter candidate emotions, such as 0.8; the emotion quantity parameter is a specified quantity threshold representing the maximum number of candidate emotions used to filter, such as 5.

[0091] Following step 504 in this embodiment, the process further includes the emotion processing device determining subsequent historical emotions based on the emotion recognition result. That is, after the emotion processing device determines the emotion recognition result based on the target probability corresponding to each of the multiple candidate emotions, the data processing method further includes: the emotion processing device filtering a specified number of candidate emotions from a reverse order of the multiple candidate emotions based on the target probability; and determining the specified number of candidate emotions as the current candidate emotion sequence, wherein the current candidate emotion sequence is used to determine the historical emotions predicted in the next round. At this point, the emotion recognition result includes multiple candidate emotions and the target probability corresponding to each candidate emotion.

[0092] It should be noted that the reverse order of multiple candidate emotions constitutes the aforementioned sequence of emotions to be screened. Here, the emotion processing device can use all multiple candidate emotions to determine the historical emotion for the next round of prediction, or it can select a specified number of candidate emotions from the multiple candidate emotions to determine the historical emotion for the next round of prediction; this embodiment does not limit this. Furthermore, the specified number can be the same as or different from the emotion quantity parameter; this embodiment does not limit this.

[0093] Understandably, in the process of determining the historical emotions for the next round of prediction by combining target probabilities to filter a specified number of candidate emotions from multiple candidate emotions, the number of historical emotions is reduced while ensuring accuracy, thus reducing the amount of emotion-based data processing and improving the efficiency of emotion recognition.

[0094] In step 501 of this application embodiment, the emotion processing device performs emotion prediction on the information to be identified to obtain the current emotion direction, including: the emotion processing device performs emotion prediction on the information to be identified to obtain the third probability corresponding to each emotion direction; and determines the emotion direction with the highest third probability as the baseline emotion direction; and determines the emotion direction whose probability difference with the third probability of the baseline emotion direction is lower than the probability difference threshold as the candidate emotion direction; finally, the baseline emotion direction and the candidate emotion direction are determined as the current emotion direction.

[0095] It should be noted that when predicting the information to be identified, the emotion processing device predicts a third probability for each of several emotion directions. These multiple emotion directions are preset, and each direction can correspond to at least one candidate emotion. For example, candidate emotions corresponding to a positive emotion direction include excitement, happiness, and liking; candidate emotions corresponding to an empty emotion direction include normal; and candidate emotions corresponding to a negative emotion direction include anger, disgust, sadness, and irony. Next, based on the third probability, the emotion processing device selects the emotion direction with the highest third probability from the reverse order of the multiple emotion directions, calling it the baseline emotion direction, and directly determines the baseline emotion direction as the current emotion direction. Alternatively, it continues to judge from the reverse order of the multiple emotion directions to determine whether there are emotion directions whose probability difference with the baseline emotion direction is less than a probability difference threshold (e.g., 0.001). These are called candidate emotion directions, and if the judgment result is yes, the candidate emotion direction and the baseline emotion direction are used together as the current emotion direction.

[0096] For example, the emotion processing device predicts the emotion of the information to be identified, and obtains the third probability corresponding to each emotion direction as: positive (0.9), empty (0.01), negative (0.89). When the probability difference threshold is 0.001, the baseline emotion direction is positive and the emotion direction to be selected is negative, so the current emotion direction includes positive and negative.

[0097] See Figure 9 , Figure 9 This is a flowchart illustrating the data processing method provided in the embodiments of this application. Figure 4 ;like Figure 9 As shown in the embodiment of this application, step 501 can be implemented by steps 5011 to 5013; that is, the emotion processing device performs emotion prediction on the information to be identified in the current round, and obtains the current emotion direction and the first probability of each candidate emotion among multiple candidate emotions, including steps 5011 to 5013. Each step is described below.

[0098] Step 5011: Extract the semantic string sequence based on the information to be identified in the current round.

[0099] In this embodiment, when the emotion processing device performs emotion prediction on the information to be identified based on semantics, it first extracts the semantics of the information to be identified, thus extracting a sequence of semantic strings. Here, the emotion processing device can extract the semantics of the information to be identified based on a preset semantic string table to filter out characters and strings without semantic meaning. Specifically, when the information to be identified is in text format, the emotion processing device directly extracts semantic strings from the information to be identified and combines the extracted semantic strings into a speech string sequence. When the information to be identified is audio, the emotion processing device extracts a sequence of semantic strings based on the information to be identified, including: the emotion processing device first performs text recognition on the audio to be identified in the current round to obtain the text to be identified; then extracts each semantic string from the text to be identified; and finally, combines the semantic strings into a sequence of semantic strings. Each semantic string in the sequence of semantic strings is used to represent emotional semantics.

[0100] It should be noted that text recognition is used to convert non-text information such as audio into text information, which can be achieved through a pre-set Automatic Speech Recognition (ASR) system.

[0101] Step 5012: Extract features from the semantic string sequence to obtain the features to be predicted.

[0102] In this embodiment, after obtaining a semantic string, the emotion processing device extracts features from the semantic string, thus obtaining the features to be predicted. Here, the emotion processing device can first represent the semantic string sequence as a vector, and then perform embedded representation on the vector representation result. The obtained embedded representation result is the feature to be predicted. Feature extraction includes vector representation and embedded representation.

[0103] Step 5013: Perform semantic emotion prediction on the feature to be predicted to obtain the current emotion direction and the first probability of each candidate emotion among multiple candidate emotions.

[0104] In this embodiment, the emotion processing device performs fine-grained semantic emotion prediction based on the features to be predicted, thus obtaining the first probability of each candidate emotion among multiple candidate emotions; the emotion processing device also performs coarse-grained semantic emotion prediction based on the features to be predicted, thereby obtaining the current emotion direction. The category granularity of the fine-grained semantic emotion prediction is finer than that of the coarse-grained semantic emotion prediction.

[0105] In this embodiment, feature extraction and semantic emotion prediction can be achieved through an emotion prediction model. The emotion prediction model is trained through the following steps: acquiring identification information samples and obtaining the corresponding emotion labels and emotion direction labels; performing emotion prediction on the identification information samples based on the prediction model to be trained, obtaining the emotion prediction probability and emotion direction prediction probability, where the prediction model to be trained is a network model to be trained for emotion prediction; and training the prediction model to be trained by combining the differences between the emotion prediction probability and the emotion label, and the differences between the emotion direction prediction probability and the emotion direction label, to obtain the emotion prediction model. The emotion prediction model is used to perform semantic-based emotion prediction on the identification information, and the prediction result includes a first probability of the candidate emotion and a third probability of the emotion direction; the emotion label is the actual emotion, a type of annotation information; and the emotion direction label is the actual emotion direction, also a type of annotation information.

[0106] In this embodiment, the emotion processing device includes a prediction model to be trained, or the emotion processing device can obtain the prediction model to be trained from other devices (e.g., pre-training devices, storage devices, etc.). After obtaining the identification information samples, the emotion processing device uses the identification information samples to train the prediction model. During model training, the emotion processing device combines the difference between the emotion prediction probability and the emotion label, and the difference between the emotion direction prediction probability and the emotion direction label, to obtain the loss function value for model training. Then, based on this loss function value, backpropagation is performed in the prediction model to be trained to adjust the parameters in the prediction model, thereby achieving the training of the prediction model.

[0107] It should also be noted that the training of the prediction model to be trained can be carried out iteratively. The training ends when the training termination condition is met, and the emotion prediction model is obtained. Here, the training termination condition can be reaching the accuracy index threshold, the number of training sessions threshold, the training duration threshold, or a combination of the above, etc. This application does not limit this.

[0108] Following step 504 in this embodiment, a process of applying the emotion recognition result is also included. For example, it can be applied to the expression rendering of virtual objects, and also to the information detection of interactive messages. When the emotion processing device applies the emotion recognition result to the expression rendering of virtual objects, the information to be recognized is the audio to be played for the virtual object to be presented; at this time, the emotion processing device generates expression rendering data of the virtual object to be presented based on the emotion recognition result; and when playing the audio to be played, it renders the expression of the virtual object to be presented based on the expression rendering data.

[0109] It should be noted that the facial expression rendering data can be the facial expression value determined based on the target probability of candidate emotions in the emotion recognition results. After obtaining the facial expression rendering data, the emotion processing device can render the facial expression of the virtual object based on the facial expression rendering data when playing the audio to be played, so as to realize the synchronization between the audio playback and the facial expression of the virtual object.

[0110] In this embodiment of the application, when the emotion processing device applies the emotion recognition result to the information detection of the interactive message, the information to be recognized is the interactive message; at this time, the emotion processing device determines the interactive emotional state based on the emotion recognition result; and when the interactive emotional state is an emotional state pending review, a prompt message is displayed, wherein the prompt message is used to prompt the change of the interactive emotional state; and when the interactive emotional state is a preset normal state, no prompt message is displayed.

[0111] The following describes an exemplary application of the embodiments of this application in a real-world application scenario. This exemplary application describes the process of generating facial expressions for a virtual human (referred to as the virtual object to be presented) through emotion recognition.

[0112] See Figure 10 , Figure 10 This is an exemplary flowchart of an emotion recognition application provided in an embodiment of this application; as shown... Figure 10 As shown, in this exemplary emotion recognition application, the input audio or text 10-1 (referred to as the information to be recognized) is first received. Then, combined with the historical emotions 10-4, emotion recognition is performed on the audio or text 10-1 to obtain emotion 10-2 (referred to as the emotion recognition result). Finally, the virtual human's facial expression 10-3 is determined based on emotion 10-2, and emotion 10-2 is updated to the historical emotions 10-4.

[0113] The description of determining virtual human facial expressions based on emotion 10-2 10-3 is as follows: Figure 11 , Figure 11 This is an exemplary schematic diagram of generating virtual human facial expressions provided in an embodiment of this application; as shown... Figure 11 As shown, through Figure 10 In the emotion section 10-2, adjust the values ​​of each control in the control area 11-11 of page 11-1 to complete the generation of the virtual human's facial expression.

[0114] The following example illustrates the process of emotion recognition, using audio (also known as the current audio) as the received information to be recognized.

[0115] See Figure 12 , Figure 12 This is a flowchart illustrating an exemplary emotion recognition method provided in an embodiment of this application; as shown below. Figure 12As shown, in this exemplary emotion recognition process, the current audio 12-1 undergoes text processing to obtain a text segmentation sequence 12-2 (referred to as a semantic string sequence). Then, a deep model 12-3 (referred to as an emotion prediction model) is used to perform a single-round emotion prediction on the text segmentation sequence 12-2, resulting in a single-round emotion prediction result 12-4 (referred to as the current emotion direction and the first probability corresponding to each candidate emotion). Next, based on the obtained historical emotions 12-5, state machine 12-6, and the single-round emotion prediction result 12-4, multiple rounds of emotion prediction are performed to obtain historical emotion prediction results 12-7. Finally, by continuously searching the historical emotion prediction result 12-7 and the single-round emotion prediction result 12-4, the optimal emotion 12-8 is obtained, which is the [emotion name / position]. Figure 10 Emotions in the heart 10-2.

[0116] It should be noted that during the text processing of the current audio 12-1, since the current audio 12-1 is in audio format, the text processing includes audio-to-text conversion and preprocessing. Preprocessing further includes word segmentation and serialization. Audio-to-text conversion refers to using ASR (Answer Sequencing Recognition) to perform text recognition on the current audio 12-1, obtaining the corresponding text. Word segmentation refers to segmenting the text using a pre-defined semantic vocabulary (called a semantic string table) to remove characters outside the semantic vocabulary (including semantically meaningless pause words). Serialization refers to the process of converting the word segmentation results into the text word segmentation sequence 12-2.

[0117] The single-round emotion prediction results 12-4 include two parts: the probability distribution corresponding to explicit emotion states (called multiple candidate emotions) and the probability distribution corresponding to implicit emotion states (called multiple emotion directions); among them, explicit emotion states represent the clear category of emotion, and implicit emotion states represent the direction of emotion change; as shown in Table 1.

[0118] Table 1

[0119] The deep model 12-3 includes an embedding layer, an encoding layer, an explicit multilayer perceptron (MLP) layer, and a hidden multilayer perceptron layer. The corresponding input and output features are shown in Table 2.

[0120] Table 2

[0121] See Figure 13 , Figure 13 This is a schematic diagram of the structure of an exemplary depth model provided in an embodiment of this application; as shown... Figure 13As shown, sparse features 13-1 is the text segmentation sequence. After passing through the embedding layer 13-2, sparse features 13-1 yields vector representation 13-3. Then, in one branch, vector representation 13-3 passes through the encoding layer 13-41 and the explicit multilayer perceptron layer 13-51 (Output Layer, e.g., Softmax layer) to obtain the prediction information 13-91 of the explicit emotional state (called the probability distribution). In another branch, vector representation 13-3 passes through the encoding layer 13-42 and the implicit multilayer perceptron layer 13-52 to obtain the prediction information 13-92 of the implicit emotional state (called the probability distribution). The coding layer 13-41 includes module 13-411, self-attention layer 13-412, module 13-413, normalization layer 13-414, feedforward layer 13-415, and normalization layer 13-416; the coding layer 13-42 includes module 13-421, self-attention layer 13-422, module 13-423, normalization layer 13-424, feedforward layer 13-425, and normalization layer 13-426; and the outputs of modules 13-411 and 13-421 are also used for positional coding. The self-attention layers 13-412 and 13-422 are in the same module, the normalization layers 13-414 and 13-424 are in the same module, and the normalization layers 13-416 and 13-426 share model parameters.

[0122] Therefore, the process of obtaining the single-round emotion prediction result 12-4 is shown in Table 3.

[0123] Table 3

[0124] The following section explains the multi-round sentiment prediction process.

[0125] It should be noted that multi-round sentiment prediction is achieved through a sequence prediction model, which is used for prediction. The probability distribution of ; where, h represents explicit emotion, n represents implicit emotion, and n represents the previous round. This represents the current round; it can be described by equation (1), which is shown below: (1); in, Indicates based on Predicted explicit emotional state The probability (called the second probability). For example, when "Normal" When it is "positive", the prediction is The probability of being "excited" is 0.9, predicting... The probability of being "happy" is 0.8, predicting... The probability of "sadness" is 0.02, etc.

[0126] In the embodiments of this application, for the trained sequence prediction model of explicit and implicit emotional states, a state machine is obtained by using explicit emotional states as nodes, implicit emotional states as inputs to transition arcs, and predicted probabilities as outputs of transition arcs.

[0127] See Figure 14 , Figure 14 This is an exemplary state machine diagram provided in an embodiment of this application; as shown below. Figure 14 As shown, state machine 14-1 is illustrated; state machine 14-1 includes 11 emotion nodes (emotion node 14-21 to emotion node 14-211) and 12 transition arcs (transition arc 14-31 to transition arc 14-312). Here, we will use emotion node 14-21 and transition arc 14-31 as examples. Emotion node 14-21 represents the starting emotion node, and the explicit emotion state S it represents is excitement; transition arc 14-31 indicates that when the emotion direction h is positive, the probability of transitioning from emotion node 14-21 to emotion node 14-22 is 0.9.

[0128] It should be noted that in the state machine, the explicit emotional state is a node, the implicit emotional state is the input of the transition arc, and the predicted probability is the output of the transition arc; and, in the state machine, for any node... traverse by node The jump arc originating from the node, when the jump arc reaches the node At that time, the input corresponding to the jump arc is The output is the probability. Therefore, based on the nodes of the previous round... and the input of the arc It is possible to obtain at least one sequence prediction result, wherein at least one sequence prediction result is as shown in equation (2).

[0129] (2); At this point, we have obtained the single-round sentiment prediction result 12-4 and at least one sequence prediction result. Here, we use a continuous sequence prediction method (such as the Beam Search algorithm) to combine the single-round sentiment prediction result 12-4 and at least one sequence prediction result to calculate the optimal prediction result.

[0130] It should be noted that the implementation process of the continuous sequence prediction method is shown in Table 4.

[0131] Table 4

[0132] For example, see Figure 15 , Figure 15 This is an exemplary schematic diagram illustrating the determination of a final result provided in an embodiment of this application; as shown... Figure 15 The diagram illustrates the process of determining the optimal emotion for the current round during predictions from rounds t0 to t5. Specifically, during round t1, the final probability (called the target probability) of each explicit emotion state (S11 to S15 in the stack) is calculated based on the transition probability from explicit emotion state S0 to each explicit emotion state in round t1, and the probability of each explicit emotion state predicted by the deep model. This final probability is then used to determine the emotion recognition result for round t1; for example, the optimal emotion is explicit emotion state S11. In round t2 prediction, the final probability of each explicit emotional state is calculated based on the jump probability from explicit emotional states S11 to S15 to each explicit emotional state in round t2 (S21 to S25 in the stack), and the probability of each explicit emotional state predicted by the deep model. This final probability is then used to determine the emotion recognition result for round t2. For example, if the optimal emotion is explicit emotional state S21, and since S21 jumps from S12, the optimal emotion in round t1 can be corrected to explicit emotional state S11. Similarly, the optimal emotion in round t3 is predicted to be explicit emotional state S31, in round t4 it is S41, and in round t5 it is S5. Thus, the optimal emotion jump relationship is obtained as shown in path 15-1.

[0133] The following describes an exemplary application of the embodiments of this application in another practical application scenario. This exemplary application describes the process of detecting player emotions in a game scene through emotion recognition.

[0134] See Figure 16 , Figure 16 This is a schematic diagram of an exemplary emotion recognition interface provided in an embodiment of this application; as shown... Figure 16As shown, message interface 16-1 describes a chat scene within a game, including message box 16-11 and game scene 16-12. When a chat message (referred to as interactive message) 16-111 ("You're just a...") is obtained through the message input box in message box 16-111, the data processing method provided in this application embodiment performs emotion recognition on the chat message 16-111. When it is determined that the recognition result indicates that the current player's emotion is a preset emotion (e.g., anger, or other emotions that trigger public opinion), the current player's emotional state is determined to be a pending review emotional state, thereby displaying prompt message 16-112 ("Please use polite language!") to alleviate the current player's emotions. The process of performing emotion recognition on the chat message 16-111 using the data processing method provided in this application embodiment is related to... Figures 12 to 15 The corresponding descriptions are similar, and the embodiments in this application will not be repeated here.

[0135] It is understood that the embodiments of this application realize the prediction of explicit emotional states and implicit emotional states, and also combine implicit emotional states with a state machine to realize the process of comprehensively predicting the current optimal emotional state by combining historical emotional states, thereby improving the accuracy of emotional prediction; and achieving smoothness and continuity in emotional prediction.

[0136] The following description continues to illustrate the exemplary structure of the data processing apparatus 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 4 As shown, the software modules stored in the data processing device 455 of the memory 450 may include: The emotion prediction module 4551 is used to predict the emotion of the information to be identified in the current round, and obtain the current emotion direction and the first probability of each of the candidate emotions among a variety of candidate emotions, wherein the current emotion direction is the direction to which the emotion of the information to be identified belongs, and the candidate emotion is a pre-set emotion type; The jump prediction module 4552 is used to determine a second probability for each candidate emotion from the emotion jump relationship based on the current emotion direction, wherein the emotion jump relationship represents the probability of jumping between the historical emotion and the candidate emotion in combination with the emotion direction, and the historical emotion is the emotion predicted before the current round; The probability integration module 4553 is used to integrate the first probability and the second probability to obtain the target probability of the candidate emotion; The result determination module 4554 is used to determine the emotion recognition result of the information to be identified based on the target probability corresponding to each of the multiple candidate emotions.

[0137] In this embodiment, the data processing device 455 further includes a relationship acquisition module 4555, used to acquire multiple emotion sample sequences and an emotion direction sample for each emotion sample in the emotion sample sequence, wherein the emotion sample sequence refers to N consecutive emotion samples in labeled rounds, where N is a positive integer; based on the multiple emotion sample sequences and the emotion direction sample for each emotion sample, multiple sample tuple sequences are generated, wherein the sample tuple sequence includes multiple sample tuples, and each sample tuple includes the emotion sample in round n, the emotion sample in round n+1, and the emotion direction sample of the emotion sample in round n+1, where n∈N-1; based on the multiple sample tuple sequences, the probability of a jump between two emotion samples in adjacent rounds for the emotion direction sample is calculated, wherein the emotion jump relationship is determined from the statistical results based on the historical emotions.

[0138] In this embodiment of the application, the probability integration module 4553 is further configured to obtain a first integration weight negatively correlated with the current round and a second integration weight positively correlated with the current round, wherein the current round represents the prediction order of the current round; obtain a first integration result of the first probability and the first integration weight; obtain a second integration result of the second probability and the second integration weight; and determine the combination result of the first integration result and the second integration result as the target probability of the candidate emotion.

[0139] In this embodiment of the application, the result determination module 4554 is further configured to: sort multiple candidate emotions in reverse order based on the target probability to obtain a sequence of emotions to be screened; screen a target emotion sequence from the sequence of emotions to be screened based on specified screening parameters, wherein the specified screening parameters include one or two of a probability threshold parameter and an emotion quantity parameter; and determine the target emotion sequence and the target probability of each target emotion in the target emotion sequence as the emotion recognition result.

[0140] In this embodiment of the application, the result determination module 4554 is further configured to filter a specified number of candidate emotions from a reverse arrangement of multiple candidate emotions based on the target probability; and determine the specified number of candidate emotions as the current candidate emotion sequence, wherein the current candidate emotion sequence is used to determine the historical emotions for the next round of prediction.

[0141] In this embodiment of the application, the emotion prediction module 4551 is further configured to perform emotion prediction on the information to be identified in the current round, and obtain a third probability corresponding to each emotion direction; determine the emotion direction with the largest third probability as the baseline emotion direction; determine the emotion direction with a probability difference lower than the probability difference threshold with the third baseline probability as the emotion direction to be selected, wherein the third baseline probability refers to the third probability of the baseline emotion direction; and determine the baseline emotion direction and the emotion direction to be selected as the current emotion direction.

[0142] In this embodiment of the application, the emotion prediction module 4551 is further configured to extract a semantic string sequence based on the information to be identified in the current round, wherein each semantic string in the semantic string sequence is used to represent emotion semantics; perform feature extraction on the semantic string sequence to obtain a feature to be predicted; and perform semantic emotion prediction on the feature to be predicted to obtain the current emotion direction and the first probability of each of the multiple candidate emotions.

[0143] In this embodiment, the feature extraction and semantic emotion prediction are implemented through an emotion prediction model, wherein the emotion prediction model is trained through the following steps: acquiring identification information samples and acquiring the emotion label and emotion direction label corresponding to the identification information samples; performing emotion prediction on the identification information samples based on the prediction model to be trained to obtain the emotion prediction probability and the emotion direction prediction probability, wherein the prediction model to be trained is a network model to be trained for emotion prediction; and training the prediction model to be trained by combining the difference between the emotion prediction probability and the emotion label, and the difference between the emotion direction prediction probability and the emotion direction label, to obtain the emotion prediction model.

[0144] In this embodiment of the application, the emotion prediction module 4551 is further configured to perform text recognition on the audio to be recognized in the current round to obtain the text to be recognized; extract each semantic string from the text to be recognized; and combine each semantic string to obtain the semantic string sequence.

[0145] In this embodiment of the application, the data processing device 455 further includes an emotion application module 4556, which is used to generate facial expression rendering data of the virtual object to be presented based on the emotion recognition result when the information to be identified is the audio to be played of the virtual object to be presented; and to render the facial expression of the virtual object to be presented based on the facial expression rendering data when the audio to be played is played.

[0146] In this embodiment of the application, the emotion application module 4556 is further configured to determine the interactive emotion state based on the emotion recognition result when the information to be identified is an interactive message; and to display a prompt message when the interactive emotion state is an emotion state pending review, wherein the prompt message is used to prompt for changing the interactive emotion state.

[0147] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an emotion processing device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the emotion processing device to perform the data processing method described in this application.

[0148] This application provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are executed by a processor, they cause the processor to perform the data processing method provided in this application. For example, ... Figure 5 The data processing method is shown.

[0149] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0150] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0151] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0152] As an example, computer-executable instructions can be deployed to execute on a single electronic device (in which case, this single electronic device is an emotion processing device), or to execute on multiple electronic devices located in one location (in which case, the multiple electronic devices located in one location are emotion processing devices), or to execute on multiple electronic devices distributed across multiple locations and interconnected via a communication network (in which case, the multiple electronic devices distributed across multiple locations and interconnected via a communication network are emotion processing devices).

[0153] It is understood that in the embodiments of this application, data related to interactive information is involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0154] In summary, the embodiments of this application, in the process of emotion prediction, first predict the first probability corresponding to each candidate emotion in a single round and the current emotion direction based on the current information to be identified. Then, based on the current emotion direction, determine the second probability of jumping from historical emotions to candidate emotions. Finally, by integrating the first and second probabilities into the target probability of the candidate emotion, the final emotion recognition result is obtained. This makes the emotion recognition result not only based on the prediction result of a single round, but also combined with the probability of jumping from historical emotions to candidate emotions. Therefore, the data used to obtain the emotion recognition result is diverse in time and space, thus improving the accuracy of emotion recognition.

[0155] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, The method includes: Emotion prediction is performed on the information to be identified in the current round to obtain the current emotion direction and the first probability of each of the candidate emotions among a variety of candidate emotions, wherein the current emotion direction is the direction to which the emotion of the information to be identified belongs, and the candidate emotions are pre-set emotions; Based on the current emotional direction, a second probability corresponding to each candidate emotion is determined from the emotional transition relationship, wherein the emotional transition relationship represents the probability of transitioning between historical emotions and candidate emotions in combination with the emotional direction, and the historical emotions are the emotions predicted before the current round; Obtain a first integrated weight that is negatively correlated with the current round and a second integrated weight that is positively correlated with the current round, wherein the current round represents the prediction order of the current round; Obtain a first integration result of the first probability and the first integration weight; obtain a second integration result of the second probability and the second integration weight; determine the combined result of the first integration result and the second integration result as the target probability of the candidate emotion; Based on the target probability corresponding to each of the multiple candidate emotions, the emotion recognition result of the information to be identified is determined.

2. The method according to claim 1, characterized in that, Before determining the second probability corresponding to each candidate emotion from the emotion transition relationship based on the current emotion direction, the method further includes: Obtain multiple emotion sample sequences and the emotion direction sample of each emotion sample in the emotion sample sequence, wherein the emotion sample sequence refers to N consecutive emotion samples in the labeled rounds, where N is a positive integer; Based on multiple emotion sample sequences and the emotion direction sample of each emotion sample, multiple sample tuple sequences are generated, wherein the sample tuple sequence includes multiple sample tuples, and the sample tuples include the emotion sample of the nth round, the emotion sample of the (n+1)th round, and the emotion direction sample of the (n+1)th round, where n∈N-1; Based on multiple sample tuple sequences, the probability of a jump to the emotion direction sample between two emotion samples in adjacent rounds is calculated, wherein the emotion jump relationship is determined from the statistical results based on the historical emotions.

3. The method according to any one of claims 1 to 2, characterized in that, The step of determining the emotion recognition result of the information to be identified based on the target probability corresponding to each of the multiple candidate emotions includes: Based on the target probability, the candidate emotions are arranged in reverse order to obtain the emotion sequence to be screened. Based on specified filtering parameters, a target emotion sequence is filtered from the emotion sequence to be filtered, wherein the specified filtering parameters include one or both of probability threshold parameters and emotion quantity parameters; The target emotion sequence and the target probability of each target emotion in the target emotion sequence are determined as the emotion recognition result of the information to be identified.

4. The method according to any one of claims 1 to 2, characterized in that, After determining the combined result of the first integration result and the second integration result as the target probability of the candidate emotion, the method further includes: Based on the target probability, a specified number of candidate emotions are selected from a reverse arrangement of the various candidate emotions; The specified number of candidate emotions are determined as the current candidate emotion sequence, wherein the current candidate emotion sequence is used to determine the historical emotions for the next round of prediction.

5. The method according to any one of claims 1 to 2, characterized in that, The step of predicting the emotion of the information to be identified in the current round to obtain the current emotion direction includes: Emotion prediction is performed on the information to be identified in the current round to obtain a third probability corresponding to each emotion direction; The emotional direction with the highest third probability is determined as the baseline emotional direction; The emotion direction whose probability difference with the third benchmark probability is less than the probability difference threshold is determined as the emotion direction to be selected, wherein the third benchmark probability refers to the third probability of the benchmark emotion direction; The baseline emotion direction and the emotion direction to be selected are determined as the current emotion direction.

6. The method according to any one of claims 1 to 2, characterized in that, The step of performing emotion prediction on the information to be identified in the current round to obtain the current emotion direction and the first probability of each of the multiple candidate emotions includes: Based on the information to be identified in the current round, a semantic string sequence is extracted, wherein each semantic string in the semantic string sequence is used to represent emotional semantics; Feature extraction is performed on the semantic string sequence to obtain the features to be predicted; Semantic emotion prediction is performed on the feature to be predicted to obtain the current emotion direction and the first probability of each of the multiple candidate emotions.

7. The method according to claim 6, characterized in that, The feature extraction and semantic sentiment prediction are achieved through a sentiment prediction model, which is trained through the following steps: Obtain identification information samples, and obtain the emotion tags and emotion direction tags corresponding to the identification information samples; Based on the prediction model to be trained, emotion prediction is performed on the identification information sample to obtain the emotion prediction probability and the emotion direction prediction probability, wherein the prediction model to be trained is a network model to be trained for emotion prediction. By combining the difference between the emotion prediction probability and the emotion label, and the difference between the emotion direction prediction probability and the emotion direction label, the training prediction model is trained to obtain the emotion prediction model.

8. The method according to claim 6, characterized in that, When the information to be identified is audio to be identified, the step of extracting a semantic string sequence based on the information to be identified in the current round includes: Perform text recognition on the audio to be recognized in the current round to obtain the text to be recognized; Extract each semantic string from the text to be identified; The semantic strings are combined to obtain the semantic string sequence.

9. The method according to any one of claims 1 to 2, characterized in that, After determining the emotion recognition result of the information to be identified based on the target probability corresponding to each of the multiple candidate emotions, the method further includes: When the information to be identified is the audio to be played for a virtual object to be presented, based on the emotion recognition result, the facial expression rendering data of the virtual object to be presented is generated; While the audio to be played is being played, the facial expression of the virtual object to be presented is rendered based on the facial expression rendering data.

10. The method according to any one of claims 1 to 2, characterized in that, After determining the emotion recognition result of the information to be identified based on the target probability corresponding to each of the multiple candidate emotions, the method further includes: When the information to be identified is an interactive message, the interactive emotional state is determined based on the emotion recognition result; When the interactive emotional state is a pending review state, a prompt message is displayed, wherein the prompt message is used to suggest changing the interactive emotional state.

11. A data processing apparatus, characterized in that, The data processing device includes: The emotion prediction module is used to predict the emotion of the information to be identified in the current round, and obtain the current emotion direction and the first probability of each of the candidate emotions among a variety of candidate emotions, wherein the current emotion direction is the direction to which the emotion of the information to be identified belongs, and the candidate emotions are preset emotions; The jump prediction module is used to determine a second probability for each candidate emotion from the emotion jump relationship based on the current emotion direction, wherein the emotion jump relationship represents the probability of jumping between the historical emotion and the candidate emotion in combination with the emotion direction, and the historical emotion is the emotion predicted before the current round; The probability integration module is used to obtain a first integration weight negatively correlated with the current round and a second integration weight positively correlated with the current round, wherein the current round represents the prediction order of the current round; obtain a first integration result of the first probability and the first integration weight; obtain a second integration result of the second probability and the second integration weight; and determine the combined result of the first integration result and the second integration result as the target probability of the candidate emotion. The result determination module is used to determine the emotion recognition result of the information to be identified based on the target probability corresponding to each of the multiple candidate emotions.

12. The apparatus according to claim 11, characterized in that, The device further includes: The relationship acquisition module is used to acquire multiple emotion sample sequences and emotion direction samples of each emotion sample in the emotion sample sequence before determining the second probability corresponding to each candidate emotion from the emotion jump relationship based on the current emotion direction. The emotion sample sequence refers to N consecutive emotion samples labeled in rounds, where N is a positive integer. Based on multiple emotion sample sequences and the emotion direction sample of each emotion sample, multiple sample tuple sequences are generated, wherein the sample tuple sequence includes multiple sample tuples, and the sample tuples include the emotion sample of the nth round, the emotion sample of the (n+1)th round, and the emotion direction sample of the (n+1)th round, where n∈N-1; Based on multiple sample tuple sequences, the probability of a jump to the emotion direction sample between two emotion samples in adjacent rounds is calculated, wherein the emotion jump relationship is determined from the statistical results based on the historical emotions.

13. The apparatus according to any one of claims 11 to 12, characterized in that, The result determination module is further configured to sort the various candidate emotions in reverse order based on the target probability to obtain a sequence of emotions to be screened. Based on specified filtering parameters, a target emotion sequence is filtered from the emotion sequence to be filtered, wherein the specified filtering parameters include one or both of probability threshold parameters and emotion quantity parameters; The target emotion sequence and the target probability of each target emotion in the target emotion sequence are determined as the emotion recognition result of the information to be identified.

14. The apparatus according to any one of claims 11 to 12, characterized in that, The result determination module is further configured to, after determining the combination result of the first integration result and the second integration result as the target probability of the candidate emotion, filter a specified number of candidate emotions from a reverse arrangement of multiple candidate emotions based on the target probability. The specified number of candidate emotions are determined as the current candidate emotion sequence, wherein the current candidate emotion sequence is used to determine the historical emotions for the next round of prediction.

15. The apparatus according to any one of claims 11 to 12, characterized in that, The emotion prediction module is also used to predict the emotion of the information to be identified in the current round and obtain a third probability corresponding to each emotion direction. The emotional direction with the highest third probability is determined as the baseline emotional direction; The emotion direction whose probability difference with the third benchmark probability is less than the probability difference threshold is determined as the emotion direction to be selected, wherein the third benchmark probability refers to the third probability of the benchmark emotion direction; The baseline emotion direction and the emotion direction to be selected are determined as the current emotion direction.

16. The apparatus according to any one of claims 11 to 12, characterized in that, The emotion prediction module is further configured to extract a semantic string sequence based on the information to be identified in the current round, wherein each semantic string in the semantic string sequence is used to represent emotion semantics; Feature extraction is performed on the semantic string sequence to obtain the features to be predicted; Semantic emotion prediction is performed on the feature to be predicted to obtain the current emotion direction and the first probability of each of the multiple candidate emotions.

17. The apparatus according to claim 16, characterized in that, The feature extraction and semantic sentiment prediction are achieved through a sentiment prediction model, which is trained through the following steps: Obtain identification information samples, and obtain the emotion tags and emotion direction tags corresponding to the identification information samples; Based on the prediction model to be trained, emotion prediction is performed on the identification information sample to obtain the emotion prediction probability and the emotion direction prediction probability, wherein the prediction model to be trained is a network model to be trained for emotion prediction. By combining the difference between the emotion prediction probability and the emotion label, and the difference between the emotion direction prediction probability and the emotion direction label, the training prediction model is trained to obtain the emotion prediction model.

18. The apparatus according to claim 16, characterized in that, When the information to be identified is audio to be identified. The emotion prediction module is also used to perform text recognition on the audio to be recognized in the current round to obtain the text to be recognized; Extract each semantic string from the text to be identified; The semantic strings are combined to obtain the semantic string sequence.

19. The apparatus according to any one of claims 11 to 12, characterized in that, The device further includes: The emotion application module is used to determine the emotion recognition result of the information to be recognized based on the target probability corresponding to each of the multiple candidate emotions, and when the information to be recognized is the audio to be played for a virtual object to be presented, generate the facial expression rendering data of the virtual object to be presented based on the emotion recognition result. While the audio to be played is being played, the facial expression of the virtual object to be presented is rendered based on the facial expression rendering data.

20. The apparatus according to any one of claims 11 to 12, characterized in that, The emotion application module is used to determine the emotion recognition result of the information to be identified after determining the target probability corresponding to each of the multiple candidate emotions, and when the information to be identified is an interactive message, to determine the interactive emotion state based on the emotion recognition result. When the interactive emotional state is a pending review state, a prompt message is displayed, wherein the prompt message is used to suggest changing the interactive emotional state.

21. An electronic device for processing data based on emotion, characterized in that, The electronic device includes: Memory is used to store executable instructions for a computer; A processor, when executing computer-executable instructions stored in the memory, implements the data processing method according to any one of claims 1 to 10.

22. A computer-readable storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed by the processor, they implement the data processing method according to any one of claims 1 to 10.

23. A computer program product, comprising a computer program or computer-executable instructions, characterized in that, When the computer program or computer-executable instructions are executed by a processor, the data processing method according to any one of claims 1 to 10 is implemented.

Images