Emotion recognition method

Abstract

The invention discloses an emotion recognition method. The method comprises the following steps: acquiring an audio to be recognized; inputting the to-be-recognized audio into a trained emotion recognition model, and recognizing an emotion tag corresponding to the to-be-recognized audio; wherein the emotion recognition model is obtained by training a first corpus constructed on the basis of standard mood and a second corpus constructed on the basis of personalized mood, the first corpus comprises a plurality of first audio recognition texts, and each first audio recognition text corresponds to at least one emotion label; each emotion tag corresponds to at least one audio feature, the second corpus comprises a plurality of second audio recognition texts, each second audio recognition text corresponds to one emotion tag, and each emotion tag corresponds to one audio feature.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and in particular, relates to an emotion recognition method. Background Art

[0002] In the field of sentiment analysis, the user's emotions can usually be reflected in their voice expressions. Traditional emotion recognition methods have low accuracy and are difficult to accurately identify voice emotions in complex environments, which cannot meet the needs of actual interactive application scenarios. Therefore, how to improve the accuracy of voice emotion recognition is a technical problem to be solved by this application. Summary of the invention

[0003] The embodiment of the present application provides an emotion recognition method, which can improve the accuracy of speech emotion recognition.

[0004] In a first aspect, an embodiment of the present application provides an emotion recognition method, the method comprising: obtaining audio to be recognized; identifying an emotion tag corresponding to the audio to be recognized by inputting the audio to be recognized into a trained emotion recognition model; wherein the emotion recognition model is trained using a first corpus constructed based on a standard tone and a second corpus constructed based on a personalized tone, the first corpus comprising a plurality of first audio recognition texts, each of the first audio recognition texts corresponding to at least one emotion tag, and each of the emotion tags corresponding to at least one audio feature, the second corpus comprising a plurality of second audio recognition texts, each of the second audio recognition texts corresponding to an emotion tag, and each of the emotion tags corresponding to an audio feature.

[0005] In a second aspect, an embodiment of the present application provides an emotion recognition device, which includes: an acquisition module for acquiring audio to be recognized; a recognition module for identifying an emotion label corresponding to the audio to be recognized by inputting the audio to be recognized into a trained emotion recognition model; wherein the emotion recognition model is trained using a first corpus constructed based on a standard tone and a second corpus constructed based on a personalized tone, the first corpus includes a plurality of first audio recognition texts, each of the first audio recognition texts corresponds to at least one emotion label, and each of the emotion label corresponds to at least one audio feature, the second corpus includes a plurality of second audio recognition texts, each of the second audio recognition texts corresponds to an emotion label, and each of the emotion label corresponds to an audio feature.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0008] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0009] In an embodiment of the present application, after obtaining the audio to be recognized, the emotion label corresponding to the audio to be recognized is recognized by inputting the audio to be recognized into a trained emotion recognition model, wherein the emotion recognition model is trained using a first corpus constructed based on a standard tone and a second corpus constructed based on a personalized tone, the first corpus includes a plurality of first audio recognition texts, each of which corresponds to at least one emotion label, and each emotion label corresponds to at least one audio feature, and the second corpus includes a plurality of second audio recognition texts, each of which corresponds to an emotion label, and each emotion label corresponds to an audio feature. In this way, by combining a diverse corpus to train the emotion recognition model, the emotion recognition model can learn and understand the emotional changes expressed under different tones and audio features, so that the emotion recognition model can adapt to the tone characteristics of different users, thereby improving the accuracy and sensitivity of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a flow chart of an emotion recognition method provided by an embodiment of the present application;

[0011] Figure 2a is a schematic diagram of the structure of the first corpus provided in an embodiment of the present application;

[0012] Figure 2b It is a schematic diagram of calculating cosine similarity provided in an embodiment of the present application;

[0013] Figure 3 is a sampling schematic diagram of an audio feature provided in an embodiment of the present application;

[0014] Figure 4 is a training diagram of an emotion recognition model provided in an embodiment of the present application;

[0015] Figure 5 is a schematic diagram of the structure of an emotion recognition device provided in an embodiment of the present application;

[0016] Figure 6 is a schematic diagram of the structure of another emotion recognition device provided in an embodiment of the present application;

[0017] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] In the following, in conjunction with the accompanying drawings, an emotion recognition method provided by an embodiment of the present application is described in detail through specific embodiments and their application scenarios.

[0020] Figure 1 An embodiment of the present application provides an emotion recognition method, which can be performed by an electronic device, and the electronic device can include: a server and / or a terminal device. In other words, the method can be performed by software or hardware installed in the electronic device, and the method includes the following steps:

[0021] S110: Acquire audio to be recognized.

[0022] The type of the audio to be identified may include but is not limited to: conversations, interviews, video audio, songs, etc. That is, the audio to be identified may be extracted from related audio or video, or may be collected in real time or in advance by a sound collector.

[0023] Optionally, if the audio to be identified includes audio of multiple objects, the audio to be identified can be parsed first to extract the audio corresponding to each object, and then emotion recognition is performed for each object according to the corresponding audio. Exemplarily, assuming that the audio to be identified is "This video is so exciting! I almost laughed to death. Well, I think this video is boring", then the audio to be identified includes the audio of object A and the audio of object B. Therefore, the audio of object A and the audio of object B can be extracted respectively: {Object A: This video is so funny! I almost laughed to death}, {Object B: Well, I think this video is boring}.

[0024] S120: Identify the emotion label corresponding to the audio to be recognized by inputting the audio to be recognized into a trained emotion recognition model.

[0025] Among them, the emotion recognition model is trained using a first corpus constructed based on a standard tone and a second corpus constructed based on a personalized tone, the first corpus includes multiple first audio recognition texts, each of the first audio recognition texts corresponds to at least one emotion tag, and each of the emotion tags corresponds to at least one audio feature, the second corpus includes multiple second audio recognition texts, each of the second audio recognition texts corresponds to an emotion tag, and each of the emotion tags corresponds to an audio feature.

[0026] It is understandable that the first corpus constructed based on the standard tone provides the tone features of standard pronunciation for different emotions in an ideal environment, and the second corpus constructed based on the personalized tone provides the tone features of personalized pronunciation for different emotions in different environments. For example, for the emotion of anger, in a certain scenario, the standard tone may be manifested as slightly faster speech speed and slightly higher pitch, while the personalized tone is related to the actual voice characteristics of the user when angry, such as a stronger tone, a specific voice rhythm, etc., to more realistically convey the current emotional state. Since users of different age groups and users of different attributes have different tone characteristics, the emotion recognition model trained only with the first corpus constructed based on the standard tone cannot accurately recognize the emotions of different users. During the recognition process, it may cause the emotion recognition model to produce irreversible deviations when self-adjusting, thereby reducing the emotion recognition ability. Then, during the training process, the first corpus constructed based on the standard tone can be adjusted and optimized according to the second corpus constructed based on the personalized tone. In this way, the standardization and universal characteristics of the standard tone can be retained, and the personalized tone of the user can be adapted to adapt to specific application scenarios and different users, thereby effectively improving the performance of the sentiment analysis model.

[0027] In addition, the audio features may include but are not limited to acoustic features, semantic features, and waveform features. Among them, acoustic features are used to describe the acoustic properties of audio signals, such as intonation, speech speed, pitch change, etc.; semantic features are used to describe the semantic information of audio, such as the vocabulary, syntactic structure and tone used by the speaker; waveform features are used to describe the curve shape and amplitude change of the audio signal's amplitude over time.

[0028] Optionally, before S120, noise reduction processing may be performed on the audio to be recognized.

[0029] Optionally, in order to better understand the first corpus, in one implementation, an embodiment of the present application provides a structural diagram, as shown in FIG2 , the first corpus can be represented by a tree structure, each first audio recognition text is a root node, the root node corresponds to at least one branch node, i.e., an emotion tag, and each branch node corresponds to at least one leaf node, i.e., an audio feature. In another implementation, the first corpus can be represented by a feature set, each feature set including a first audio recognition text, an emotion tag corresponding to the first audio recognition text, and at least one audio feature corresponding to the emotion tag. Exemplarily, continuing to refer to the example corresponding to FIG2 , a plurality of feature sets may include:

[0030] (i-th text to be identified - emotion label i1 - audio feature i11);

[0031] …

[0032] (i-th text to be identified - emotion label i1 - audio feature i1a);

[0033] (i-th text to be identified - emotion label i2 - audio feature i21);

[0034] …

[0035] (i-th text to be identified - emotion label i2 - audio feature i2b);

[0036] (the i-th text to be identified - emotion label in - audio feature in1).

[0037] For example, (wow-surprised-0.02|0.03|0.05|0.08|0.95), (sigh-disappointed-0.04|0.03|0.02|0.02|0.01).

[0038] As for the second corpus, its structure and representation are similar to those of the first corpus and will not be described in detail here.

[0039] In an embodiment of the present application, after obtaining the audio to be recognized, the emotion label corresponding to the audio to be recognized is recognized by inputting the audio to be recognized into a trained emotion recognition model, wherein the emotion recognition model is trained using a first corpus constructed based on a standard tone and a second corpus constructed based on a personalized tone, the first corpus includes a plurality of first audio recognition texts, each of which corresponds to at least one emotion label, and each emotion label corresponds to at least one audio feature, and the second corpus includes a plurality of second audio recognition texts, each of which corresponds to an emotion label, and each emotion label corresponds to an audio feature. In this way, by combining a diverse corpus to train the emotion recognition model, the emotion recognition model can learn and understand the emotional changes expressed under different tones and audio features, so that the emotion recognition model can adapt to the tone characteristics of different users, thereby improving the accuracy and sensitivity of emotion recognition.

[0040] In one implementation, before identifying the emotion label corresponding to the audio to be recognized in the emotion recognition model trained by inputting the audio to be recognized, the method further includes the following steps:

[0041] Step 1: Obtain an initial corpus, wherein the initial corpus includes a standard audio library, a tone audio library and a personalized audio library, the standard audio library includes at least one first audio, the tone audio library includes at least one second audio corresponding to at least one tone word, and the personalized audio library includes at least one third audio.

[0042] Among them, regarding the standard audio library, standard audio, i.e., the first audio, in various emotional states can be generated by speech synthesis technology, or manually recorded and collected. For example, multiple professional voice actors are asked to record different sentences in different tones, and emotional tags are manually marked to obtain a standard audio library. Optionally, the formats of each sentence can be consistent or inconsistent.

[0043] Regarding the tone audio library, the standard tone audio, i.e., the second audio, under various emotional states can also be generated through speech synthesis technology, or it can be manually recorded and collected. For example, multiple professional voice actors can record different tone words with different tones, and manually mark the emotional labels to obtain the tone audio library. Among them, different tones can also be recorded for the same tone word. For example, the tone word "ah" may express different emotions in different tones, ā means surprise or admiration, á means question, ǎ means slight surprise or uncertainty, and à means surprise or admiration.

[0044] The personalized audio library may include third audios for different emotions corresponding to the same user, or third audios for different emotions corresponding to different users. In other words, if the personalized audio library consists of only third audios for different emotions corresponding to one user, then the recognition parameters of the trained emotion recognition model are adaptable to the user; if the personalized audio library consists of third audios for different emotions corresponding to different users, then the recognition parameters of the trained emotion recognition model are universally adaptable, that is, emotion recognition can be performed for different users.

[0045] In addition, since different users have different speaking speeds, timbres, and tones, when constructing a personalized audio library, users can enter audio with different emotions based on specific texts, or audio for different users with different emotions can be synthesized using relevant tools to collect personalized audio.

[0046] It should be noted that the modal particles in different language environments are different. For example, Chinese modal particles may include: yes, good, okay, um, okay, no problem, eh, wow, ah, alas, oops, indeed, very, etc., and English modal particles may include: Wow, Oh no, amazing, Wonderful, etc. Therefore, when obtaining the standard audio library, tone audio library and personalized audio library, relevant audio can be obtained according to the specific language usage environment.

[0047] Step 2: performing a first processing on the standard audio library according to the tone audio library to obtain the first corpus, and performing a second processing on the personalized audio library according to the tone audio library to obtain the second corpus.

[0048] It is understandable that the same sentence or the same modal particle may have different corresponding emotional intentions when spoken with different tones or voices. In this step, the standard audio library is first processed based on the voice audio library, and the personalized audio library is second processed, so that the audio of the modal particle is used as an important emotional guide to improve the accuracy of emotion recognition. For example, if an audio includes multiple sighs, it can be judged that the user's emotion is low at this time.

[0049] Step 3: Use the first corpus and the second corpus to train the emotion recognition model.

[0050] In this implementation, based on the standard audio library and the personalized audio library, weights are assigned to the first audio and the third audio in combination with the audio of the modal particles, so that the modal particles will not be removed as stop words, but will serve as important emotional guides, further improving the accuracy of recognition.

[0051] In one implementation, before the emotion recognition model is trained using the first corpus and the second corpus, the method further includes: constructing the emotion recognition model based on a naive Bayesian network algorithm. It is understandable that the emotion recognition model is constructed based on a naive Bayesian network algorithm, which is simple to implement, fast in calculation, can be used for real-time recognition, and is suitable for processing large-scale data sets. Even when the amount of data is small, the naive Bayesian algorithm can provide accurate recognition results. In addition, the emotion recognition model constructed based on the naive Bayesian network algorithm can support incremental training, that is, it can be continuously updated and improved based on new data without retraining from scratch.

[0052] In one implementation, the method of identifying the emotion label corresponding to the audio to be recognized by inputting the audio to be recognized into a trained emotion recognition model includes: determining target recognition text and target audio features corresponding to the audio to be recognized; determining the features to be recognized corresponding to the audio to be recognized based on the target recognition text and the target audio features; and determining the emotion label corresponding to the audio to be recognized based on the features to be recognized.

[0053] It can be understood that the trained emotion recognition model has good emotion recognition capabilities. Therefore, it is only necessary to input the audio to be recognized into the emotion recognition model to determine the target recognition text and target audio features corresponding to the audio to be recognized, determine the features to be recognized based on the target recognition text and target audio features, and finally perform emotion recognition based on the features to be recognized.

[0054] Wherein, in another implementation, based on the target recognition text and the target audio feature, determining the feature to be recognized corresponding to the audio to be recognized may include: performing word segmentation processing on the target recognition text and determining the semantic vector corresponding to each word segmentation; performing discretization processing on the target audio feature to obtain a discrete vector corresponding to the target audio feature; and obtaining the feature to be recognized by fusing the semantic vector and the discrete vector. Wherein, word segmentation refers to dividing a continuous text into meaningful units, such as words, phrases or subwords. Word segmentation can reduce the dimension of the text and improve the efficiency of subsequent processing. After word segmentation, the semantic vector corresponding to each word segmentation can be determined by word embedding, and the semantic vector captures the semantic information of the corresponding word segmentation. Optionally, the word embedding method includes but is not limited to Word2Vec, GloVe and FastText. Discretization processing refers to converting a continuous audio signal into a discrete feature representation. In audio processing tasks, if continuous features are directly operated, a large amount of computing resources and complex algorithms may be required. Therefore, the complexity and cost of calculation can be reduced by discretization processing. Finally, the semantic vector and discrete vector are used to obtain the features to be identified through vector fusion. The fusion refers to associating the semantic information of the target text with the feature information of the audio to be identified.

[0055] In one implementation, after the emotion tag corresponding to the audio to be recognized is identified by inputting the audio to be recognized into the trained emotion recognition model, the method further includes: updating the parameter value of the emotion recognition model according to the emotion tag corresponding to the audio to be recognized. It can be understood that updating the parameters according to the actual recognition result feedback can reduce the prediction error, thereby improving the recognition accuracy.

[0056] In one implementation, before the emotion recognition model that has been trained by inputting the audio to be recognized is trained and the emotion label corresponding to the audio to be recognized is recognized, it may also include: obtaining the training audio input by the target user; and performing incremental training on the emotion recognition model according to the training audio. That is, before the user uses the emotion recognition model to perform emotion analysis, training data may be pre-input to help the emotion recognition model learn to adapt to the personalized parameters of the user, thereby improving the accuracy of emotion recognition for the user. In this implementation, the parameters and structure of the original model can be effectively utilized through incremental training, and the model can be updated and optimized quickly and effectively while maintaining the stability and performance of the model, so that the trained emotion recognition model can adapt to changing application scenarios and user needs.

[0057] In one implementation, the using the first corpus and the second corpus to train the emotion recognition model may include the following steps:

[0058] Step 1: Determine an offset parameter according to the first corpus and the second corpus, wherein the offset parameter is used to characterize a personalized tone feature of a user.

[0059] Step 2: According to the offset parameter and the loss of training the emotion recognition model based on the first corpus and the second corpus, and adjusting the parameter value of the emotion recognition model based on the loss until the emotion recognition model converges.

[0060] It is understandable that the emotion recognition model has an initial parameter value, which can be determined based on the first corpus training or preset. In order to adapt to the personalized tone of different types or different users, it is necessary to adjust the initial model parameter value during the training process, that is, compare the emotion recognition results of the second corpus with the associated emotion tags in the first corpus to obtain the offset result, and perform weighted averaging on the offset result to obtain the offset parameter. Then, based on the offset parameter and the model training loss, adjust the parameter value of the emotion recognition model until the emotion recognition model converges, so that the emotion recognition model can better adapt to the personalized tone characteristics of the user, thereby improving the accuracy of emotion recognition.

[0061] Optionally, the first audio in the first corpus associated with a third audio in the second corpus can be determined by cosine similarity. Optionally, the cosine similarity between the audio features of the first audio and the audio features of the third audio can be obtained by the Euclidean dot product formula, where the Euclidean dot product formula is a·b=‖a‖‖b‖cosθ, then: like Figure 2b As shown, given two attribute vectors: a and b, the cosine similarity θ is given by the dot product and the vector length. The cosine similarity range is [-1,1], where -1 means that the two vectors point in opposite directions, 1 means that the two vectors point in exactly the same direction, and 0 means that the two vectors are independent of each other. Other values ​​except -1, 0 and 1 represent the similarity of the two vectors. For audio matching, the attribute vectors A and B can be the audio features of one audio and the audio features of a third audio.

[0062] In one implementation, performing a first processing on the standard audio library according to the tone audio library to obtain the first corpus includes the following steps:

[0063] Step 1: Obtain a standard emotion feature library by performing feature analysis on each of the first audios.

[0064] It is understandable that if only the semantics of the text is analyzed or only the waveform of the audio is analyzed or weights are used to distinguish the influence of the text audio when training the emotion recognition model, the trained emotion recognition model will not be able to accurately reflect the emotions corresponding to the relevant audio. Therefore, the feature analysis in this application includes text recognition, emotion labeling and audio feature discretization. The standard emotion feature library includes the feature analysis results of each first audio. Taking the audio feature as a waveform as an example, after performing feature analysis on a first audio, the first audio recognition text, emotion label and waveform discrete vector corresponding to the first audio can be obtained, exemplarily: (I am so angry-angry-0.02|0.03|0.05|0.08|0.95), where "I am so angry" is the first audio recognition text, the emotion label is "angry", and "0.02|0.03|0.05|0.08|0.95" is a waveform discrete vector.

[0065] Optionally, when performing audio feature discretization processing, the sampling step can be determined according to the training situation. The sampling step refers to the time interval between two adjacent sampling points. The smaller the step, the more accurate the waveform can be obtained, such as Figure 3 As shown, the sampling step is 1.

[0066] In addition, the audio waveforms of many emotions are similar. For example, the waveforms of angry and happy speech may both have an upward curve. Therefore, the following three factors can be used for disambiguation:

[0067] (1) The starting decibel value of the audio waveform. For example, the starting decibel value of anger is higher.

[0068] (2) The rapidity of the audio. This rapidity can be directly reflected in the derivative of the waveform, that is, the degree of change of the waveform per unit time.

[0069] (3) Application scenarios of tone text. For example, "ha" is used to express happiness, and "ah" is used to express anger or surprise.

[0070] Step 2: Obtain a tone feature library by performing feature analysis on each of the second audios.

[0071] The process of performing feature analysis on each of the second audios is the same as the process of performing feature analysis on the first audio in the above step 1, and will not be repeated here.

[0072] Step 3: Obtain the first corpus by fusing the standard emotion feature library and the tone feature library.

[0073] It can be understood that the feature fusion process is to use the tone feature as an important emotion guide to update the standard emotion feature library, thereby obtaining the first corpus. Exemplarily, it is assumed that the standard emotion feature library includes feature sets such as w01-n1-x01, w02-n2-x02, w03-n3-x03...w0m-nm-x0m, where w0m is the first audio recognition text, nm is the classification label, and x0m is the waveform discrete vector; and it is assumed that the tone feature library includes: a01-y1-z01, a02-y2-z02, ax03-y3-x03...a0b-yb-z0b, where a0b is a modal particle, yb is an emotion label, and z0b is a waveform discrete vector. When a certain modal particle a0b is included in a first audio recognition text w0m, it is further determined whether the emotion label corresponding to the w0m is consistent with that of the modal particle a0b. If they are consistent, a weight value is assigned to the audio feature x0m corresponding to the first audio recognition text w0m based on the audio feature z0b corresponding to the modal particle a0b, thereby realizing feature fusion.

[0074] In this implementation, modal particles are given greater weight in emotional expression, so that emotional expression can be quantified through modal particles, further improving the accuracy of emotion recognition. In addition, since the recognized text of each audio may be ambiguous based on semantic analysis, it can be combined with the modal audio library to achieve disambiguation.

[0075] Furthermore, in another implementation, the second processing is performed on the personalized audio library according to the tone audio library to obtain the second corpus, including: obtaining a personalized emotion feature library by performing feature analysis on each of the third audios; and obtaining the second corpus by performing feature fusion of the personalized emotion feature library and the tone feature library.

[0076] It should be noted that for the specific description of this implementation, reference can be made to the previous implementation, and the process is the same and will not be repeated here.

[0077] Optionally, based on the above embodiments, the present application provides a flow chart of a training method for an emotion recognition model, such as Figure 4 As shown, the following steps may be included:

[0078] Step 1: Create a standard audio library and a tone audio library.

[0079] By using a standard fixed sentence format and having multiple professional voice actors record with different emotional tones, and then manually classifying and labeling them, a standard audio library can be obtained. Since modal particles play an important role in the tone of emotional expression, the same method can be used to generate a tone audio library by having voice actors dub with different tones for modal particles.

[0080] Step 2: Perform feature analysis on the standard audio library and the tone audio library respectively to obtain a standard emotion waveform library and a tone waveform library.

[0081] For the generated standard audio library and tone audio library, after audio waveform analysis and combined with the manually labeled emotion types, the waveforms are discretized to obtain the standard emotion waveform library and tone waveform library.

[0082] Step 3: Perform feature fusion on the standard emotion waveform library and the tone waveform library to obtain a parallel corpus.

[0083] Step 4: Build a personalized audio library.

[0084] Since different users speak at different speeds, timbres, and tones, when users use the emotion recognition model, they can record audio based on specific text.

[0085] Step 5: Use the naive Bayesian network algorithm to tune parameters.

[0086] According to the feature set of the personalized audio library: "text-emotion-waveform bar" and the feature set of the parallel corpus "text-emotion-waveform library", parameters are tuned through the naive Bayesian network classification algorithm.

[0087] In this embodiment, modal particles are used as important emotion guides, and dubbing annotations are made for the modal particles separately to generate a personalized text-audio parallel corpus, thereby achieving accurate and multi-level emotion division and improving the accuracy of emotion recognition.

[0088] Optionally, the present application also provides a structural schematic diagram of an emotion recognition device, such as Figure 5 As shown, it includes a platform side 510 and a terminal side 520, wherein the platform side includes: a data collection module 511, a label module 512, a corpus training module 513 and a model parameter preset module 514. The terminal side 520 includes: a parameter feedback upload module 521, a real-time information collection module 522, a model parameter adaptation module 523 and a sentiment analysis module 524.

[0089] The data collection module 511 is used to collect audio data. Data collection can include three types:

[0090] (1) Professional voice actors record audio with standardized emotional tones, and use professional audio processing tools to generate a standard audio library.

[0091] (2) Professional voice actors record modal particles separately with standardized emotional tones, and use professional audio processing tools to generate a tone audio library.

[0092] (3) Before using the terminal side for emotion recognition, the user makes a pre-setting, which allows the user to record audio using different emotional tones in natural language habits, and the audio is used as the basis for generating personalized parameters.

[0093] The label module 512 is used to use the standard tone and emotion of the dubbing actor, and in a manual labeling method, use the file system to perform preliminary processing on the directory and the audio file name to generate a label. In other words, the emotional type label of the audio can be obtained based on the file attributes.

[0094] The corpus training module 513 is used to extract audio features based on the target learning library and generate a general mlmodel model file in combination with the audio library. In this module, the preliminary classification of the input audio can be tested based on the audio features alone, but since only the audio features are used, the user's current emotional value cannot be accurately determined, so it needs to be further processed through the model parameter preset module.

[0095] Model parameter preset module 514: Based on the user input audio, the threshold offset is adjusted through the audio fitting process. When the user uses natural audio habits to train 12 types of audio, the sampling results are used to compare the result offset of the standard model, and then the user's personalized offset parameters are obtained through weighted averaging.

[0096] The real-time information collection module 522 is used to perform real-time speech transcription to obtain the user's speech text, and to use audio input to obtain the user's real-time "text-emotion-waveform bar" feature set.

[0097] The sentiment analysis module 524 is used to utilize the user's real-time "text-sentiment-waveform bar" feature set to further extract the features of modal particles and assign high weights to modal particles, so that the modal features serve as important sentiment guides.

[0098] The model parameter adaptive module 523 is used to update the model parameter configuration required at the next moment by downloading the parameters trained on the platform side.

[0099] The parameter feedback upload module 521 is used to provide feedback to the platform side after obtaining the sentiment analysis result through the information collected by the real-time information collection module 522.

[0100] Figure 6 A schematic diagram of the structure of an emotion recognition device provided by an embodiment of the present application is shown as follows: Figure 6 As shown, the emotion recognition device 600 may include: an acquisition module 610 and a recognition module 620 .

[0101] In this embodiment, the acquisition module 610 is used to acquire the audio to be recognized; the recognition module 620 is used to identify the emotion label corresponding to the audio to be recognized by inputting the audio to be recognized into a trained emotion recognition model; wherein the emotion recognition model is trained using a first corpus constructed based on a standard tone and a second corpus constructed based on a personalized tone, the first corpus includes a plurality of first audio recognition texts, each of the first audio recognition texts corresponds to at least one emotion label, and each of the emotion label corresponds to at least one audio feature, the second corpus includes a plurality of second audio recognition texts, each of the second audio recognition texts corresponds to an emotion label, and each of the emotion label corresponds to an audio feature.

[0102] In one implementation, the emotion recognition device 600 may also include a training module, which is used to: obtain an initial corpus, wherein the initial corpus includes a standard audio library, a tone audio library and a personalized audio library, the standard audio library includes at least one first audio, the tone audio library includes at least one second audio corresponding to at least one tone word, and the personalized audio library includes at least one third audio; perform a first processing on the standard audio library according to the tone audio library to obtain the first corpus, and perform a second processing on the personalized audio library according to the tone audio library to obtain the second corpus; and use the first corpus and the second corpus to train the emotion recognition model.

[0103] In one implementation, the method of identifying the emotion label corresponding to the audio to be recognized by inputting the audio to be recognized into a trained emotion recognition model includes: determining target recognition text and target audio features corresponding to the audio to be recognized; determining the features to be recognized corresponding to the audio to be recognized based on the target recognition text and the target audio features; and determining the emotion label corresponding to the audio to be recognized based on the features to be recognized.

[0104] In one implementation, the using the first corpus and the second corpus to train the emotion recognition model includes: determining an offset parameter based on the first corpus and the second corpus, wherein the offset parameter is used to characterize the personalized tone characteristics of the user; according to the offset parameter and the loss of training the emotion recognition model based on the first corpus and the second corpus, and adjusting the parameter value of the emotion recognition model based on the loss until the emotion recognition model converges.

[0105] In one implementation, the first processing is performed on the standard audio library according to the tone audio library to obtain the first corpus, including: obtaining a standard emotion feature library by performing feature analysis on each of the first audios; obtaining a standard emotion feature library by performing feature analysis on each of the first audios; obtaining a tone feature library by performing feature analysis on each of the second audios; and obtaining the first corpus by performing feature fusion of the standard emotion feature library and the tone feature library.

[0106] In one implementation, the second processing of the personalized audio library according to the tone audio library to obtain the second corpus includes: obtaining a personalized emotion feature library by performing feature analysis on each of the third audios; and obtaining the second corpus by performing feature fusion of the personalized emotion feature library and the tone feature library.

[0107] In one implementation, the feature to be recognized corresponding to the audio to be recognized is determined based on the target recognition text and the target audio feature, including: performing word segmentation on the target recognition text and determining a semantic vector corresponding to each word segmentation; discretizing the target audio feature to obtain a discrete vector corresponding to the target audio feature; and obtaining the feature to be recognized by fusing the semantic vector and the discrete vector.

[0108] In one implementation, the emotion recognition device 600 may further include an updating module, configured to update a parameter value of the emotion recognition model according to an emotion tag corresponding to the audio to be recognized.

[0109] In one implementation, the emotion recognition device 600 may further include a construction module for: constructing the emotion recognition model based on a naive Bayesian network algorithm.

[0110] In one implementation, the training module is further used to: obtain training audio input by a target user; and perform incremental training on the emotion recognition model according to the training audio.

[0111] An emotion recognition device provided in an embodiment of the present application can implement each process implemented in the emotion recognition method embodiment, and will not be described again here to avoid repetition.

[0112] The emotion recognition device in the embodiment of the present application may be a device, or a component, an integrated circuit, or a chip in an electronic device. The embodiment of the present application is not specifically limited.

[0113] The emotion recognition device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0114] Optional, such as Figure 7 As shown, an embodiment of the present application also provides an electronic device 700, including a processor 710, a memory 720, and a program or instruction stored in the memory 720 and executable on the processor 710. When the program or instruction is executed by the processor 710, each process of the above-mentioned emotion recognition method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0115] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned emotion recognition method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0116] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0117] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned emotion recognition method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0118] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0119] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0120] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0121] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An emotion recognition method, characterized in that: include: Get the audio to be recognized; By inputting the audio to be recognized into a trained emotion recognition model, an emotion label corresponding to the audio to be recognized is recognized; Among them, the emotion recognition model is trained using a first corpus constructed based on a standard tone and a second corpus constructed based on a personalized tone, the first corpus includes multiple first audio recognition texts, each of the first audio recognition texts corresponds to at least one emotion tag, and each of the emotion tags corresponds to at least one audio feature, the second corpus includes multiple second audio recognition texts, each of the second audio recognition texts corresponds to an emotion tag, and each of the emotion tags corresponds to an audio feature.

2. The method according to claim 1, characterized in that Before identifying the emotion label corresponding to the audio to be recognized in the emotion recognition model trained by inputting the audio to be recognized, the method further includes: Acquire an initial corpus, wherein the initial corpus includes a standard audio library, a tone audio library, and a personalized audio library, the standard audio library includes at least one first audio, the tone audio library includes at least one second audio corresponding to at least one tone particle, and the personalized audio library includes at least one third audio; Performing a first process on the standard audio library according to the tone audio library to obtain the first corpus, and performing a second process on the personalized audio library according to the tone audio library to obtain the second corpus; The emotion recognition model is trained using the first corpus and the second corpus.

3. The method according to claim 1, characterized in that The step of inputting the audio to be recognized into a trained emotion recognition model to recognize an emotion label corresponding to the audio to be recognized includes: Determining target recognition text and target audio features corresponding to the audio to be recognized; Based on the target recognition text and the target audio feature, determining a feature to be recognized corresponding to the audio to be recognized; Based on the feature to be identified, an emotion tag corresponding to the audio to be identified is determined.

4. The method according to claim 2, characterized in that: The using the first corpus and the second corpus to train the emotion recognition model comprises: Determining an offset parameter according to the first corpus and the second corpus, wherein the offset parameter is used to characterize a personalized tone feature of the user; According to the offset parameter and the loss of training the emotion recognition model based on the first corpus and the second corpus, and adjusting the parameter value of the emotion recognition model based on the loss until the emotion recognition model converges.

5. The method according to claim 2, characterized in that: The first processing of the standard audio library according to the tone audio library to obtain the first corpus includes: By performing feature analysis on each of the first audios, a standard emotion feature library is obtained; By performing feature analysis on each of the second audios, a tone feature library is obtained; The first corpus is obtained by fusing the standard emotion feature library and the tone feature library.

6. The method according to claim 5, characterized in that The performing a second processing on the personalized audio library according to the tone audio library to obtain the second corpus includes: Obtaining a personalized emotion feature library by performing feature analysis on each of the third audios; The second corpus is obtained by fusing the personalized emotion feature library with the tone feature library.

7. The method according to claim 2, characterized in that: Before using the first corpus and the second corpus to train the emotion recognition model, the method further includes: The emotion recognition model is constructed based on the Naive Bayesian network algorithm.

8. The method according to claim 3, characterized in that The determining, based on the target recognition text and the target audio feature, the feature to be recognized corresponding to the audio to be recognized comprises: Performing word segmentation processing on the target recognition text and determining a semantic vector corresponding to each word segmentation; Discretize the target audio feature to obtain a discrete vector corresponding to the target audio feature; The feature to be identified is obtained by fusing the semantic vector and the discrete vector.

9. The method according to claim 3, characterized in that: Before identifying the emotion label corresponding to the audio to be recognized in the emotion recognition model trained by inputting the audio to be recognized, the method further includes: Obtain training audio input by the target user; Incrementally train the emotion recognition model according to the training audio.

10. The method according to claim 1, characterized in that After identifying the emotion label corresponding to the audio to be identified by inputting the audio to be identified into the trained emotion recognition model, the method further includes: The parameter values ​​of the emotion recognition model are updated according to the emotion label corresponding to the audio to be recognized.

Images