Server and speech emotion recognition method

Abstract

The embodiment of the application provides a kind of server and voice sentiment recognition method, received voice is identified as text data, the vector representation of text data is acquired;According to sentiment polarity word table, sentiment word in text data is marked using remote supervision, sentiment word is mapped to random vector space, and the vector representation of sentiment word is acquired;The vector representation of text data and sentiment word is spliced, and bottom layer shared parameter is obtained;Bottom layer shared parameter is input to multi-task learning model, and sentiment analysis result and sentiment keyword are obtained.The vector of the word corresponding to the sentiment word is spliced with the sentence vector to which it belongs, so that the sentiment word in the sentence can be focused on, to improve the extraction accuracy of sentiment keyword, and to improve the accuracy of sentiment recognition.In addition, the server continuously supplements the sentiment polarity word table using the extracted sentiment keyword, so that subsequent remote supervision of sentiment words in text data can be more comprehensive according to the sentiment polarity word table.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet, and particularly relates to a server and a voice emotion recognition method. BACKGROUND

[0002] The display device is a kind of television product that can realize bidirectional man-machine interaction function, integrates video, audio, entertainment, data and other functions. In order to meet the diversified needs of users, the display device is installed with video, audio, entertainment and other applications, and interacts with users through user interface and exchanges information.

[0003] With the continuous development of human-computer interaction, voice is being reshaped as a new paradigm of human-computer interaction. The current display device has dialogue and question and answer functions. For example, the user can send "I want to see XX" to the display device, and the display device can obtain video content related to XX from the server and display and recommend it to the user. When the user browses the video content recommended by the display device, the user can give a voice response to the display device to express whether he is satisfied with the recommended result of the display device. In order to obtain the satisfaction degree of the user to the recommended result, the display device can perform emotion recognition on the voice of the user to judge the emotional state of the user.

[0004] At present, voice emotion recognition is mainly based on the text content itself, and the accuracy is low. For example, in Chinese, the position of some emotional words in the whole sentence and the different collocations with other words may lead to different emotion recognition results. SUMMARY

[0005] The present application provides a server and a voice emotion recognition method to solve the technical problem of low accuracy of voice emotion recognition in the prior art.

[0006] In a first aspect, the present application provides a server, which is configured to:

[0007] recognize the received voice as text data, obtain a vector representation of the text data;

[0008] label the emotional words in the text data according to the emotion polarity word table using remote supervision, map the emotional words to a random vector space, and obtain a vector representation of the emotional words;

[0009] splice the vector representation of the text data and the vector representation of the emotional words to obtain bottom layer shared parameters;

[0010] input the bottom layer shared parameter into a trained multi-task learning model to obtain a sentiment analysis result and a sentiment keyword, wherein the multi-task learning model comprises a sentiment analysis task and a sentiment keyword extraction task, and the sentiment keyword is used to supplement the sentiment polarity lexicon.

[0011] In a second aspect, the present application provides a voice sentiment recognition method, which comprises:

[0012] recognizing the received voice as text data, and obtaining a vector representation of the text data;

[0013] labeling a sentiment word in the text data according to a sentiment polarity lexicon by using remote supervision, mapping the sentiment word to a random vector space, and obtaining a vector representation of the sentiment word;

[0014] concatenating the vector representation of the text data and the vector representation of the sentiment word to obtain a bottom layer shared parameter;

[0015] inputting the bottom layer shared parameter into a trained multi-task learning model to obtain a sentiment analysis result and a sentiment keyword, wherein the multi-task learning model comprises a sentiment analysis task and a sentiment keyword extraction task, and the sentiment keyword is used to supplement the sentiment polarity lexicon.

[0016] Compared with the prior art, the present application has the following beneficial effects:

[0017] The present application provides a server and a voice sentiment recognition method. The server obtains voice input by a user, recognizes the voice as text data, and converts the text data into a corresponding sentence vector. Meanwhile, the server can extract a sentiment word in the text data by using a sentiment polarity lexicon, and convert the sentiment word into a word vector. The server concatenates the sentence vector corresponding to the text data and the word vector corresponding to the sentiment word to obtain a bottom layer shared parameter. The server inputs the bottom layer shared parameter into a multi-task learning model, so that the multi-task learning model outputs a sentiment analysis result and a sentiment keyword. In the present application, the server concatenates the word vector corresponding to the sentiment word and the sentence vector of the text data in which the sentiment word is located, so that the multi-task learning model pays more attention to the vector corresponding to the sentiment word in the bottom layer shared parameter, thereby improving the extraction accuracy of the sentiment keyword in the multi-task learning model and the accuracy of sentiment recognition. In addition, the server constantly supplements the sentiment polarity lexicon with the sentiment keyword output by the multi-task learning model, so that subsequent remote supervision of the sentiment word in the text data can be performed more comprehensively according to the sentiment polarity lexicon. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 Schematic diagram showing a system architecture of a speech recognition method and a speech recognition device according to some embodiments;

[0020] Figure 2 exemplarily shows a hardware configuration block diagram of a smart device 200 according to some embodiments;

[0021] Figure 3 Schematic diagram of the configuration of the smart device 200 according to some embodiments is exemplarily shown in FIG.

[0022] Figure 4 exemplarily shows a schematic diagram of a voice interaction network architecture according to some embodiments;

[0023] Figure 5 hereinafter is a schematic diagram illustrating a training process of a multi-task learning model according to some embodiments;

[0024] Figure 6 FIG. 4 exemplarily shows a schematic diagram of obtaining an initial sentiment polarity vocabulary according to some embodiments;

[0025] Figure 7 exemplarily shows a training diagram of a multi-task learning model according to some embodiments;

[0026] Figure 8 exemplarily shows a schematic diagram of a network structure of a multi-task learning model according to some embodiments;

[0027] Figure 9 hereinafter is a flow chart showing a method for speech emotion recognition according to some embodiments;

[0028] Figure 10 exemplarily shows another flow chart of the speech emotion recognition method according to some embodiments. DETAILED DESCRIPTION

[0029] In order to make the purpose and implementation of this application clearer, the exemplary implementation of this application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only part of the embodiments of this application, not all of the embodiments.

[0030] It should be noted that the brief descriptions of terms in this application are only for the purpose of facilitating the understanding of the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and usual meanings.

[0031] In the specification and claims of this application and the accompanying drawings, the terms "first," "second," "third," etc. are used to distinguish similar or similar objects or entities, and are not necessarily intended to limit a particular order or sequence, unless otherwise noted. It should be understood that the terms used in this manner are interchangeable under appropriate circumstances.

[0032] Figure 1 FIG. 1 shows an exemplary system architecture to which the speech recognition method and speech recognition device of the present application can be applied. Figure 1 As shown, 10 is a server and 200 is a terminal device, which exemplarily includes (smart TV 200a, mobile device 200b, smart speaker 200c).

[0033] In this application, the server 10 and the smart device 200 communicate data via various communication methods. The smart device 200 may be connected to a local area network (LAN), a wireless local area network (WLAN), or other networks. The server 10 may provide various content and interactions to the terminal device 20. For example, the smart device 200 and the server 10 may send and receive information, as well as receive software program updates.

[0034] The server 10 can be a server that provides various services, such as a backend server that supports audio data collected by the smart device 200. The backend server can analyze and process the received audio data and other data, and feed back the processing results (such as endpoint information) to the terminal device. The server 10 can be a server cluster or multiple server clusters, and can include one or more types of servers.

[0035] The smart device 200 can be hardware or software. When the smart device 200 is hardware, it can be various electronic devices with sound collection functions, including but not limited to smart speakers, smart phones, TVs, tablets, e-book readers, smart watches, players, computers, AI devices, robots, smart vehicles, etc. When the smart devices 200, 201, and 202 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, to provide sound collection services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0036] It should be noted that the speech emotion recognition method provided in the embodiment of the present application can be executed by the server 10, or by the terminal device 20, or by both the server 10 and the terminal device 20, and this application does not limit this.

[0037] Figure 2 FIG. 2 shows a hardware configuration block diagram of the smart device 200 according to an exemplary embodiment. Figure 2 The illustrated smart device 200 includes at least one of a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280. The controller includes a central processing unit, an audio processor, a graphics processor, RAM, ROM, and first to nth interfaces for input / output.

[0038] The display 260 includes a display screen component for presenting images, and a driving component for driving image display, a component for receiving image signals output from a controller, and a component for displaying video content, image content, and a menu control interface and a user control UI interface.

[0039] The display 260 may be a liquid crystal display, an OLED display, or a projection display, and may also be a projection device and a projection screen.

[0040] Communicator 220 is a component used to communicate with external devices or servers using various communication protocols. For example, the communicator may include at least one of a Wi-Fi module, a Bluetooth module, a wired Ethernet module, or other network communication protocol chip, a near-field communication protocol chip, and an infrared receiver. Smart device 200 can send and receive control signals and data signals with server 10 via communicator 220.

[0041] The user interface can be used to receive external control signals.

[0042] Detector 230 is used to collect signals from the external environment or external interactions. For example, detector 230 includes a light receiver, a sensor for collecting ambient light intensity; or detector 230 includes an image collector, such as a camera, for collecting external environmental scenes, user attributes, or user interaction gestures; or detector 230 includes a sound collector, such as a microphone, for receiving external sounds.

[0043] The sound collector can be a microphone, also known as a "microphone" or "microphone," which can be used to receive the user's voice and convert the sound signal into an electrical signal. The smart device 200 can be equipped with at least one microphone. In other embodiments, the smart device 200 can be equipped with two microphones, which can not only collect sound signals but also implement noise reduction functions. In other embodiments, the smart device 200 can be equipped with three, four, or more microphones to collect sound signals, reduce noise, identify sound sources, implement directional recording functions, etc.

[0044] In addition, the microphone may be built into the smart device 200, or the microphone may be connected to the smart device 200 via a wired or wireless manner. Of course, the embodiment of the present application does not limit the position of the microphone on the smart device 200. Alternatively, the smart device 200 may not include a microphone, that is, the above-mentioned microphone is not provided in the smart device 200. The smart device 200 may be connected to an external microphone (also referred to as a microphone) through an interface (such as a USB interface 130). The external microphone may be fixed to the smart device 200 by an external fixing member (such as a camera holder with a clip).

[0045] The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in the memory. The controller 250 controls the overall operation of the smart device 200.

[0046] Exemplarily, the controller includes at least one of a central processing unit (CPU), an audio processor, a graphics processing unit (GPU), RAM Random Access Memory (RAM), ROM (Read-Only Memory, ROM), a first interface to an nth interface for input / output, a communication bus, etc.

[0047] In some examples, the operating system of the smart device is an Android system, for example, Figure 3 As shown, the smart TV 200 - 1 can be logically divided into an application layer (abbreviated as “application layer”) 21 , a kernel layer 22 and a hardware layer 23 .

[0048] Among them, such as Figure 3 As shown, the hardware layer may include Figure 2The controller 250, communicator 220, detector 230, etc. are shown. The application layer 21 includes one or more applications. The applications can be system applications or third-party applications. For example, the application layer 21 includes a voice recognition application that can provide a voice interaction interface and services for connecting the smart TV 200-1 to the server 10.

[0049] The kernel layer 22 serves as a software middleware between the hardware layer and the application layer 21 and is used to manage and control hardware and software resources.

[0050] In some examples, the kernel layer 22 includes a detector driver, which is used to send voice data collected by the detector 230 to a voice recognition application. For example, when the voice recognition application in the smart device 200 is activated and a communication connection is established between the smart device 200 and the server 10, the detector driver is used to send the user-entered voice data collected by the detector 230 to the voice recognition application. The voice recognition application then sends a query message containing the voice data to the intent recognition module 202 in the server. The intent recognition module 202 is used to input the voice data sent by the smart device 200 into an intent recognition model.

[0051] To clearly illustrate the embodiments of the present application, Figure 4 A speech emotion recognition network architecture provided in an embodiment of the present application is described.

[0052] See also Figure 4 , Figure 4 A schematic diagram of a voice interaction network architecture provided in an embodiment of the present application. Figure 4 In the example, the smart device is used to receive input information and output the processing results of the information. The speech recognition module is deployed with a speech recognition service for recognizing audio as text; the semantic understanding module is deployed with a semantic understanding service for semantically parsing text; the business management module is deployed with a business instruction management service for providing business instructions; the language generation module is deployed with a language generation service (NLG) for converting instructions for the smart device to execute into text language; the speech synthesis module is deployed with a speech synthesis (TTS) service for processing the text language corresponding to the instruction and sending it to the speaker for broadcast. In one embodiment, Figure 4 The illustrated architecture may include multiple physical service devices deployed with different business services, or one or more physical service devices may integrate one or more functional services.

[0053] In some embodiments, the following Figure 4 The process of processing information input into a smart device in the illustrated architecture is described by way of example, taking the case where the information input into the smart device is a query statement input via voice as an example:

[0054] [Speech recognition]

[0055] After receiving the query sentence through the voice input, the intelligent device can perform noise reduction processing and feature extraction on the audio of the query sentence. The noise reduction processing here can include steps such as removing echo and environmental noise.

[0056] [Semantic understanding]

[0057] The acoustic model and the language model are used to perform natural language understanding on the recognized candidate text and the associated context information, to parse the text into structured, machine-readable information, such as business domain, intent, word slot, and semantic expression. An executable intent determination intent confidence score is obtained, and the semantic understanding module selects one or more candidate executable intents based on the determined intent confidence score,

[0058] [Business management]

[0059] The semantic understanding module issues a query instruction to the corresponding business management module according to the semantic parsing result of the text of the query sentence, to obtain a query result given by the business service, and to perform an action required to complete the user's final request, and feeds back a device execution instruction corresponding to the query result.

[0060] It should be noted that, Figure 4 The architecture shown is only an example and does not limit the scope of protection of the present application. In the embodiments of the present application, other architectures can also be used to implement similar functions, for example: all or part of the above processes can be completed by the intelligent terminal, which will not be described here.

[0061] With the continuous development of human-computer interaction, the corresponding intelligent device is controlled by voice commands, which is favored by the majority of users. When the user queries the function through the intelligent device, the intelligent device will obtain the corresponding recommended result from the server 10 and feed back to the user. The user can give voice feedback on the recommended result of the intelligent device to express whether he is satisfied, for example: the user says: "You recommend a good movie for me, and the previous recommendation is not liked by me". When receiving such a query feedback from the user, it is necessary to identify that the user is not satisfied with the previous round of recommended results, to identify the current user's emotional state as negative, negative, etc., and to analyze the dialogue or the recommended, query result in a timely manner. At present, voice emotion recognition is mainly based on the text content itself for recognition, and the accuracy is low. In order to improve the accuracy of voice emotion recognition, the present application provides a server in some embodiments, which is configured to perform a voice emotion recognition process. The voice emotion recognition process will be described below in conjunction with the drawings.

[0062] In some embodiments, the voice emotion recognition process performed by the server 10 can be performed in another device. Hereinafter, the server 10 is taken as an example.

[0063] In some embodiments, a multi-task learning model can be trained before performing speech emotion recognition. In some embodiments, the multi-task learning model includes two tasks: sentiment analysis and sentiment keyword extraction. The sentiment keyword extraction task extracts words with sentiment polarity from user speech, while the sentiment analysis task determines whether the user's speech is positive or negative.

[0064] The following is an introduction to the training process of the multi-task learning model with the help of the accompanying figures.

[0065] Figure 5 FIGURE 2 shows an exemplary diagram of a training process of a multi-task learning model according to some embodiments. Figure 5 , the training process of the multi-task learning model is as follows:

[0066] S501: Acquire an initial sentiment polarity vocabulary and training data according to user log data, wherein the initial sentiment polarity vocabulary is the sentiment polarity vocabulary before being supplemented with sentiment keywords.

[0067] In some embodiments, the server 10 can use the user log data as a basic corpus. After obtaining the user log data, it can perform manual annotation by sampling part of the user log data. The server 10 uses the PMI algorithm (point mutual information algorithm) to calculate the first word used to represent the emotion in the user log data and the second word with a clear emotional polarity, and calculates the PMI value between the two. The server 10 confirms whether the emotional polarity between the two words is the same through the PMI value between the two words. Taking positive emotional words as an example, that is, by calculating the probability of two words appearing together in the corpus, when the PMI value of a word and a word with positive emotion reaches a certain threshold, it can be determined that the word is also a positive emotional word, and the same is true for negative.

[0068] In some embodiments, if a sentiment transition occurs in a sentence, even if two words appear simultaneously in the same sentence, they cannot be determined to have the same polarity. Therefore, the server 10 can use a dependency parsing algorithm to determine whether a sentiment transition occurs in the sentence containing the first word. If a sentiment transition occurs in the sentence containing the first word, the PMI values ​​of the first and second words will not increase, and it can be directly determined that the two words do not have the same polarity. If there is no sentiment transition in the sentence containing the first word, the PMI values ​​of the first and second words will increase, and it can also be determined that the two words have the same polarity.

[0069] Figure 6 FIG. 4 shows an exemplary diagram of obtaining an initial sentiment polarity vocabulary according to some embodiments. Figure 6As shown, the point mutual information algorithm is performed on the common emotional polarity words and the user log data, and the dependency syntax analysis algorithm is used to correct the calculation results. Through the above process and manual annotation of the sampled part of the log data, the initial emotional polarity word table and the emotional classification training data can be obtained.

[0070] S502: Obtain input layer data according to the initial emotional polarity word table and the training data.

[0071] Figure 7 The training schematic diagram of the multi-task learning model according to some embodiments is exemplarily shown in FIG. 2. In combination with Figure 7 After obtaining the training data and the initial emotional polarity word table, for the sentence "I am very happy, really good" in the training data, the server 10 labels the emotional words "happy" and "good" in the sentence through the initial emotional polarity word table.

[0072] In some embodiments, the server 10 inputs the sentence into the BERT (Bidirectional Encoder Representation from Transformers) model, and outputs the vector representation of the sentence via the BERT model. At the same time, the server 10 maps the labeled emotional words into an extra-feature embedding space to obtain the vector representation of the emotional words, splices the vector representation of the emotional words with the BERT output of the whole sentence to obtain a sentence vector representation with additional information, i.e., the input layer data, which is the bottom shared parameter.

[0073] S503: Input the input layer data into a plurality of expert networks to obtain first features, and input the input layer data into a gate network to obtain weights of the plurality of expert networks.

[0074] S504: Weight the first features according to the weights of the plurality of expert networks to obtain second features corresponding to the tasks.

[0075] S505: Input the second features into corresponding Tower networks to obtain output layer data.

[0076] Figure 8 The network structure schematic diagram of the multi-task learning model according to some embodiments is exemplarily shown in FIG. 3. In combination with Figure 8The multi-task learning model includes multiple expert networks, a number of gate networks equal to the number of tasks, and a number of Tower networks equal to the number of tasks. The first expert network, the second expert network, and the third expert network are used to extract features from input layer data. The first gate network is used to calculate the weights of the multiple expert networks for the sentiment analysis task, and the second gate network is used to calculate the weights of the multiple expert networks for the sentiment keyword extraction task. The server 10 inputs the second features weighted by the first gate network into the first Tower network for sentiment analysis, and inputs the second features weighted by the second gate network into the second Tower network for sentiment keyword extraction.

[0077] In some embodiments, the server 10 controls the multi-task learning model to iteratively train the above process. The server 10 obtains the output layer data generated by the multi-task learning model each time during the iterative training process, and the output layer data includes the extracted emotional keywords. The server 10 filters the emotional keywords in the output layer data according to the meaningless word list, wherein the meaningless word list contains words that are clearly not emotionally polar, such as "ah", "ya" and other modal particles. The server 10 supplements the initial emotional polarity word list based on the emotional keywords in the filtered output layer data, and the initial emotional polarity word list after continuous supplementation is the emotional polarity table actually to be applied in the subsequent multi-task learning model.

[0078] The following describes the speech emotion recognition process provided by some embodiments of the present application in conjunction with the accompanying drawings.

[0079] Figure 9 FIGURE 1 shows a flow chart of a method for speech emotion recognition according to some embodiments. Figure 9 As shown, the method includes the following steps:

[0080] S901: Recognize the received speech as text data, and obtain a vector representation of the text data.

[0081] In some embodiments, a user inputs voice into a smart device, which can then send the received voice to server 10. Server 10 then recognizes the user's voice input as text data using ASR. Alternatively, the smart device can directly convert the user's voice into text data using ASR, and then send the text data to server 10. In some embodiments, server 10 further uses a BERT model to obtain a vector representation of the text data.

[0082] S902: tagging sentiment words in the text data using remote supervision according to a sentiment polarity word list, mapping the sentiment words to a random vector space, and obtaining vector representations of the sentiment words.

[0083] In some embodiments, during the iterative training of the multi-task learning model, the server 10 continuously uses the acquired new sentiment keywords to populate the initial sentiment polarity vocabulary, thereby obtaining a sentiment polarity vocabulary with a relatively rich sentiment vocabulary. The server 10 remotely supervises the annotation of sentiment words in the text data through the sentiment polarity vocabulary and further obtains vector representations of the sentiment words.

[0084] S903: Concatenate the vector representation of the text data and the vector representation of the sentiment words to obtain underlying shared parameters.

[0085] In some embodiments, the vector representation of the text data and the vector representation of the sentiment word are added together to obtain the underlying shared parameters of the multi-task learning model. Figure 7 .

[0086] S904: Input the underlying shared parameters into a trained multi-task learning model to obtain sentiment analysis results and sentiment keywords, wherein the multi-task learning model includes a sentiment analysis task and a sentiment keyword extraction task, and the sentiment keywords are used to supplement the sentiment polarity vocabulary.

[0087] In some embodiments, after obtaining only the sentiment analysis results and sentiment keywords from the multi-task learning model, the server 10 can continue to supplement the sentiment polarity word list with the obtained sentiment keywords, regardless of whether the sentiment analysis results are positive or negative. Of course, the content added to the sentiment polarity word list must be new sentiment words, that is, sentiment words that do not exist in the previous table.

[0088] In some embodiments, during the emotional keyword extraction task, server 10 utilizes a pointer network to convert the task into multiple binary classification networks, predicting head and tail pointers. Specifically, server 10 decodes the second feature weighted by the second gate network and outputs a probability distribution of whether it is the start or end position of the keyword, also known as a pointer. Server 10 extracts the characters between the head and tail pointers as emotional keywords.

[0089] In some embodiments, considering that the length of the segment predicted by the pointer network may be too long and does not conform to the task idea of ​​emotional keyword extraction, the server 10 also needs to perform post-processing, that is, control the interval length between the head pointer and the tail pointer to be less than a preset value, for example, less than 3, to avoid the length of the extracted emotional keywords being longer.

[0090] In some embodiments, when server 10 decodes the second feature weighted by the second gate network, it sets corresponding processing functions for each result, including one, multiple, or no decoding, to ensure that the corresponding emotional keyword is ultimately extracted. For example, if no decoding result is obtained, that is, if no emotional keyword is obtained, server 10 may select the next largest value in the probability distribution as the head pointer and tail pointer.

[0091] In some embodiments, in a sentiment analysis task, the server 10 inputs the second feature weighted by the first gate network into the fully connected layer and then performs binary classification.

[0092] In some embodiments, when the server 10 learns from the sentiment analysis result that the user's sentiment is negative, it needs to update the recommendation strategy in a timely manner to re-recommend media content to the user.

[0093] The following further introduces the above-mentioned speech emotion recognition process with reference to the accompanying drawings.

[0094] Figure 10 FIG. 4 shows another flow chart of a method for speech emotion recognition according to some embodiments. Figure 10 The server 10 needs to use the sentiment polarity word list to perform remote supervision on the existing small amount of annotated data as a priori features for sentiment classification and sentiment keyword extraction, and map it to a new vector space. Then, the sentence vectors of the original text are spliced ​​together to train a multi-task learning model that simultaneously performs sentiment classification and sentiment keyword extraction, namely the initial model. Here, the sentiment polarity word list used initially is the initial sentiment polarity word list, which can also be regarded as a table composed of common sentiment polarity words. After obtaining the initial model, the server 10 continuously uses user log data for prediction, performing N rounds of iterations to obtain a phased model. When the server 10 uses the initial model to analyze user log data, it can obtain new sentiment polarity words, add them to the sentiment polarity word list as remote supervision for new training. After several rounds of iterations, a more accurate sentiment classification model for common user reactions can be generated. Here, after several rounds of iterations, when the number of new sentiment polarity words obtained by the model gradually decreases to a certain number, it can be determined that the model has stabilized and can be put into use. When the phased model is trained, real-time user speech can be obtained and sentiment keyword extraction and sentiment analysis can be performed using the phased model. New sentiment-polarity words extracted by the model can continue to supplement the sentiment-polarity word list. That is, after the sentiment classification model is put into use, the server 10 can also collect some newly emerging sentiment-polarity words. When a certain number is reached, the server 10 can retrain and update the model, achieving semi-automatic optimization. Finally, if the model analyzes that the user's sentiment is negative, the server 10 needs to update the recommendation strategy.

[0095] In this application, the server splices the word vectors corresponding to the sentiment words with the sentence vectors of the text data where the sentiment words are located, so that the multi-task learning model focuses more on the vectors corresponding to the sentiment words in the underlying shared parameters, so as to improve the extraction accuracy of the sentiment keywords in the multi-task learning model, and is conducive to improving the accuracy of sentiment recognition. In addition, the server continuously uses the sentiment keywords output by the multi-task learning model to supplement the sentiment polarity vocabulary, so that the sentiment words in the text data can be more comprehensively remotely supervised based on the sentiment polarity vocabulary. The server achieves the goal of finer-grained and more accurate sentiment recognition through remote supervision, thereby providing direction and ideas for product optimization, providing real examples of changes in user emotional states, and providing basic technical and data guarantees for the continuous optimization of products.

[0096] Corresponding to the above-mentioned server, the present application also provides a method for speech emotion recognition, which includes: the server 10 recognizes the received speech as text data and obtains a vector representation of the text data. The emotion words in the text data are marked using remote supervision according to the emotion polarity vocabulary, and the server 10 maps the emotion words to a random vector space to obtain the vector representation of the emotion words. The server 10 splices the vector representation of the text data and the vector representation of the emotion words to obtain underlying shared parameters, and inputs the underlying shared parameters into a trained multi-task learning model to obtain emotion analysis results and emotion keywords, wherein the multi-task learning model includes emotion analysis tasks and emotion keyword extraction tasks, and the emotion keywords are used to supplement the emotion polarity vocabulary.

[0097] In some embodiments, during the training process of the multi-task learning model, the method includes: the server 10 obtains an initial sentiment polarity vocabulary and training data based on user log data, wherein the initial sentiment polarity vocabulary is the sentiment polarity vocabulary before being supplemented by sentiment keywords. The server 10 obtains input layer data based on the initial sentiment polarity vocabulary and the training data. The server 10 inputs the input layer data into multiple expert networks to obtain a first feature, and inputs the input layer data into a gate network to obtain weights of the multiple expert networks. The server 10 weights the first feature according to the weights of the multiple expert networks to obtain a second feature corresponding to the task. The server 10 inputs the second feature into the corresponding Tower network to obtain output layer data.

[0098] In some embodiments, during the training of the multi-task learning model, the method further includes: the server 10 obtaining the output layer data generated by the multi-task learning model during the iterative training process. The server 10 filters the sentiment keywords in the output layer data according to a meaningless word list, wherein the meaningless word list includes words that are clearly not sentimentally polarized. The server 10 supplements the initial sentiment polarity word list based on the sentiment keywords in the filtered output layer data.

[0099] Since the above embodiments are all described by reference in combination with other embodiments, different embodiments have the same parts, and the same and similar parts between the various embodiments in this specification can be referred to each other. No further detailed explanation is given here.

[0100] It should be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a circuit structure, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such circuit structure, article or device. In the absence of further restrictions, the presence of an element defined by the phrase "includes a..." does not exclude the presence of other identical elements in the circuit structure, article or device comprising the element.

[0101] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the disclosure of the invention herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0102] The above embodiments of the present application do not constitute a limitation on the protection scope of the present application.

Claims

1. A server, characterized in that: The server is configured to: Recognize the received speech as text data and obtain a vector representation of the text data; Tagging sentiment words in the text data using remote supervision according to a sentiment polarity word list, mapping the sentiment words to a random vector space, and obtaining vector representations of the sentiment words; Concatenate the vector representation of the text data and the vector representation of the sentiment word to obtain underlying shared parameters; The underlying shared parameters are input into a trained multi-task learning model to obtain sentiment analysis results and sentiment keywords, wherein the multi-task learning model includes a sentiment analysis task and a sentiment keyword extraction task, the sentiment keywords are used to supplement the sentiment polarity vocabulary, and the sentiment keyword extraction task is to extract words with sentiment polarity from the voice uttered by the user.

2. The server according to claim 1, wherein: During the training process of the multi-task learning model, the server is configured to: Acquire an initial sentiment polarity vocabulary and training data according to user log data, wherein the initial sentiment polarity vocabulary is the sentiment polarity vocabulary before being supplemented with sentiment keywords; Obtaining input layer data according to the initial sentiment polarity vocabulary and the training data; Inputting the input layer data into a plurality of expert networks to obtain a first feature, and inputting the input layer data into a gate network to obtain weights of the plurality of expert networks; Weighting the first features according to the weights of the plurality of expert networks to obtain a second feature corresponding to the task; The second feature is input into the corresponding Tower network to obtain the output layer data.

3. The server according to claim 2, wherein: During the training process of the multi-task learning model, the server is further configured to: Obtaining the output layer data generated by the multi-task learning model during iterative training; Filtering sentiment keywords in the output layer data according to a meaningless word list, wherein the meaningless word list includes words that are clearly not sentimentally polar; The initial sentiment polarity vocabulary is supplemented according to the sentiment keywords in the filtered output layer data.

4. The server according to claim 2, wherein: In the step of obtaining an initial sentiment polarity vocabulary according to user log data, the server is configured to: Obtain user log data, and calculate PMI values ​​of a first word and a second word using a PMI algorithm, wherein the first word is a word used to represent emotion in the user log data, and the second word is a word with a clear emotional polarity, and the PMI value is used to confirm whether the emotional polarity of the first word and the second word is the same; Determining whether an emotional transition occurs in the sentence containing the first word using a dependency parsing algorithm; If there is an emotional transition in the sentence containing the first word, the PMI value does not increase; If there is no emotional transition in the sentence containing the first word, the PMI value increases.

5. The server according to claim 1, wherein: After inputting the underlying shared parameters into the trained multi-task learning model to obtain sentiment analysis results and sentiment keywords, the server is further configured to: If the sentiment analysis result indicates that the user's sentiment is positive, then supplementing the sentiment polarity vocabulary according to the sentiment keywords; If the sentiment analysis result indicates that the user's sentiment is negative, the sentiment polarity word list is supplemented according to the sentiment keywords, and a recommendation strategy is updated, wherein the recommendation strategy is used to recommend media content to the user.

6. The server according to claim 1, wherein: In the emotion keyword extraction task, the server is configured to: The sentiment keyword extraction task is converted into multiple binary classification networks using a pointer network to predict the head pointer and the tail pointer respectively; The characters between the head pointer and the tail pointer are extracted as sentiment keywords.

7. The server according to claim 6, wherein: The server is configured to: The interval length between the head pointer and the tail pointer is controlled to be smaller than a preset value.

8. A speech emotion recognition method, characterized in that: The method comprises: Recognize the received speech as text data and obtain a vector representation of the text data; Tagging sentiment words in the text data using remote supervision according to a sentiment polarity word list, mapping the sentiment words to a random vector space, and obtaining vector representations of the sentiment words; Concatenate the vector representation of the text data and the vector representation of the sentiment word to obtain underlying shared parameters; The underlying shared parameters are input into a trained multi-task learning model to obtain sentiment analysis results and sentiment keywords, wherein the multi-task learning model includes a sentiment analysis task and a sentiment keyword extraction task, the sentiment keywords are used to supplement the sentiment polarity vocabulary, and the sentiment keyword extraction task is to extract words with sentiment polarity from the voice uttered by the user.

9. The speech emotion recognition method according to claim 8, characterized in that During the training process of the multi-task learning model, the method includes: Acquire an initial sentiment polarity vocabulary and training data according to user log data, wherein the initial sentiment polarity vocabulary is the sentiment polarity vocabulary before being supplemented with sentiment keywords; Obtaining input layer data according to the initial sentiment polarity vocabulary and the training data; Inputting the input layer data into a plurality of expert networks to obtain a first feature, and inputting the input layer data into a gate network to obtain weights of the plurality of expert networks; Weighting the first features according to the weights of the plurality of expert networks to obtain a second feature corresponding to the task; The second feature is input into the corresponding Tower network to obtain the output layer data.

10. The speech emotion recognition method according to claim 9, characterized in that: During the training of the multi-task learning model, the method further includes: Obtaining the output layer data generated by the multi-task learning model during iterative training; Filtering sentiment keywords in the output layer data according to a meaningless word list, wherein the meaningless word list includes words that are clearly not sentimentally polar; The initial sentiment polarity vocabulary is supplemented according to the sentiment keywords in the filtered output layer data.

Images