Sentiment analysis method, related device and readable storage medium

Abstract

The present application discloses a sentiment analysis method, related equipment and readable storage medium. After acquiring the voice conversation data to be analyzed, for each sentence in the voice conversation data, the audio data and text data of the sentence are determined, the audio data and text data of the sentence are interactively fused and encoded to obtain the audio features and text features of the sentence, and the sentence is sentimentally analyzed based on the audio features and text features of the sentence to determine the sentiment type of the sentence. Since the voice conversation data, text information and audio information can express the user's emotional information in different dimensions, the sentiment type of the sentence determined based on the audio features and text features of the sentence is more accurate.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and more specifically, to a sentiment analysis method, related equipment and readable storage medium. Background Art

[0002] Natural Language Processing (NLP) is an important branch of artificial intelligence, among which sentiment analysis is an important topic in NLP research, involving multiple fields. For example, in the field of outbound calls or navigation, sentiment analysis of user input is required during multi-round interactions, and the multi-round interaction system is optimized based on the sentiment analysis results to improve user satisfaction. It can be seen that the accuracy of sentiment analysis results plays a vital role in user satisfaction.

[0003] Therefore, how to obtain highly accurate sentiment analysis results has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0004] In view of the above problems, this application proposes a sentiment analysis method, related equipment and readable storage medium. The specific scheme is as follows:

[0005] A sentiment analysis method, the method comprising:

[0006] Acquire voice conversation data to be analyzed;

[0007] For each sentence in the voice dialogue data, determine the audio data and text data of the sentence; interactively fuse and encode the audio data and text data of the sentence to obtain the audio features and text features of the sentence; and determine the emotion type of the sentence based on the audio features and text features of the sentence.

[0008] Optionally, interactively fusing and encoding the audio data and text data of the sentence to obtain audio features and text features of the sentence; and determining the emotion type of the sentence based on the audio features and text features of the sentence includes:

[0009] The audio data and text data of the sentence are input into a sentiment analysis model; the interactive fusion coding module of the sentiment analysis model performs interactive fusion coding on the audio data and text data of the sentence to obtain the audio features and text features of the sentence; the sentiment analysis module of the sentiment analysis model determines the sentiment type of the sentence based on the audio features and text features of the sentence.

[0010] Optionally, the interactive fusion coding module of the sentiment analysis model performs interactive fusion coding on the audio data and text data of the sentence to obtain the audio features and text features of the sentence, including:

[0011] The audio processing submodule in the interactive fusion encoding module performs interactive fusion encoding on the audio data and text data of the sentence to obtain the audio features of the sentence;

[0012] The text processing submodule in the interactive fusion encoding module performs interactive fusion encoding on the audio data and text data of the sentence to obtain the text features of the sentence.

[0013] Optionally, the audio processing submodule includes an audio coding layer, a first audio interactive fusion coding layer, and a second audio interactive fusion coding layer; the text processing submodule includes a text coding layer, a first text interactive fusion coding layer, and a second text interactive fusion coding layer; the interactive fusion coding module of the sentiment analysis model interactively fusion codes the audio data and text data of the sentence to obtain the audio features and text features of the sentence, including:

[0014] The audio coding layer encodes the audio data of the sentence to obtain initial audio features of the sentence;

[0015] The text encoding layer encodes the text data of the sentence to obtain the initial text features of the sentence;

[0016] The first audio interactive fusion coding layer interactively fusion codes the initial audio features of the sentence and the initial text features of the sentence to obtain a first audio interactive feature of the sentence;

[0017] The first text interactive fusion coding layer interactively fuses the initial audio features of the sentence and the initial text features of the sentence to obtain a first text interactive feature of the sentence;

[0018] The second audio interaction fusion coding layer interactively fusion codes the first audio interaction feature of the sentence and the first text interaction feature of the sentence to obtain the audio feature of the sentence;

[0019] The second text interactive fusion coding layer interactively fusion codes the first audio interactive feature of the sentence and the first text interactive feature of the sentence to obtain the text feature of the sentence.

[0020] Optionally, the sentiment analysis model is trained as follows:

[0021] Acquire a first true sample data set, wherein the first true sample data set includes a plurality of first true sample data, each of which includes audio data, text data, and an emotion type label of each sentence in a set of true conversation data;

[0022] Generate a fake sample data set according to the first true sample data set, wherein the fake sample data set includes a plurality of fake sample data, each of the fake sample data includes audio data, text data and emotion type label of each sentence in a set of fake conversation data;

[0023] The pre-trained model is trained based on the fake sample data set to obtain a sentiment analysis model.

[0024] Optionally, generating the fake sample data set according to the first real sample data set includes:

[0025] For each first true sample data in the first true sample data set, determining an emotional logic factor of the first true sample data;

[0026] A false sample data set is generated based on each first true sample data and the sentiment logic factor of each first true sample data in the first true sample data set.

[0027] Optionally, determining the sentiment logic factor of the first true sample data includes:

[0028] The text data of each sentence in the first true sample data is input into the sentiment logic factor generation model, and the sentiment logic factor generation model processes the text data of each sentence in the first true sample data to obtain the sentiment logic factor of the first true sample data.

[0029] Optionally, the sentiment logic factor generation model processes the text data of each sentence in the first true sample data to obtain the sentiment logic factor of the first true sample data, including:

[0030] The semantic feature extraction module in the sentiment logic factor generation model extracts semantic features from each sentence in the first true sample data to obtain semantic features of each sentence in the first true sample data;

[0031] The sentiment feature extraction module in the sentiment logic factor generation model extracts sentiment features from the semantic features of each sentence in the first true sample data to obtain the sentiment features of each sentence in the first true sample data;

[0032] The cause feature extraction module in the sentiment logic factor generation model extracts cause features from the semantic features of each sentence in the first true sample data to obtain cause features of each sentence in the first true sample data;

[0033] The emotion probability prediction module in the emotion logic factor generation model performs emotion probability prediction on the emotion feature of each sentence in the first true sample data to obtain the emotion probability of each sentence in the first true sample data;

[0034] The cause probability prediction module in the emotional logic factor generation model performs cause probability prediction on the cause feature of each sentence in the first true sample data to obtain the cause probability of each sentence in the first true sample data;

[0035] The emotional logic factor generation module in the emotional logic factor generation model fuses the emotional features, emotional probabilities, cause features, cause probabilities of each sentence in the first true sample data and the positional relationship encoding vectors of the sentences in the first true sample data to obtain the emotional logic factors of the first true sample data.

[0036] Optionally, the emotion logic factor generation model is trained based on preset sliding window parameters and a second true sample data set; the preset sliding window parameters are used to indicate the number of sentences included in a sentence group with a high possibility of an emotion logic relationship; the second true sample data set includes multiple second true sample data, each of the second true sample data includes text data, reason labels, emotion type labels and emotion logic factors of each sentence in a group of true dialogue data, and the reason labels are used to indicate whether the sentence has an emotion logic relationship with other sentences.

[0037] Optionally, the semantic feature extraction module in the sentiment logic factor generation model extracts semantic features from each sentence in the first true sample data to obtain the semantic features of each sentence in the first true sample data, including:

[0038] For each sentence in the first true sample data, the overall semantic features of the sentence and other sentences in the same sliding window as the sentence are extracted as the semantic features of the sentence, and the sliding window is determined based on the preset sliding window parameters.

[0039] Optionally, the generating the fake sample data set based on each first true sample data and the sentiment logic factor of each first true sample data in the first true sample data set includes:

[0040] For each true sample data in the first true sample data set, the first true sample data and the emotional logic factor of the first true sample data are input into N generators of the data generation model, and the N generators of the data generation model output the fake sample data set, where N is the same as the number of emotional type labels.

[0041] Optionally, the data generation model is trained as follows:

[0042] Acquire a third true sample data set, wherein the third true sample data set includes a plurality of third true sample data, each of the third true sample data includes audio data, text data, an emotion type label of each sentence in a set of true conversation data, and an emotion logic factor of the third true sample data;

[0043] Constructing a discriminator, wherein the discriminator is an N+1 classifier;

[0044] The data generation model is trained using the third true sample data set as a training sample, with the cumulative penalty items of the N generators being minimized and the judgment result of the discriminator on the output of the data model being close to the judgment result of the discriminator on the third true sample data as the training target.

[0045] A sentiment analysis device, comprising:

[0046] An acquisition unit, used to acquire the voice conversation data to be analyzed;

[0047] The sentiment analysis unit is used to determine the audio data and text data of each sentence in the voice dialogue data; interactively fuse and encode the audio data and text data of the sentence to obtain the audio features and text features of the sentence; and determine the sentiment type of the sentence based on the audio features and text features of the sentence.

[0048] Optionally, the sentiment analysis unit includes:

[0049] The sentiment analysis model processing unit is used to input the audio data and text data of the sentence into the sentiment analysis model, and the sentiment analysis model includes an interactive fusion encoding module and a sentiment analysis module. The interactive fusion encoding module is used to perform interactive fusion encoding on the audio data and text data of the sentence to obtain the audio features and text features of the sentence; the sentiment analysis module is used to determine the sentiment type of the sentence based on the audio features and text features of the sentence.

[0050] Optionally, the interactive fusion encoding module of the sentiment analysis model includes an audio processing submodule and a text processing submodule:

[0051] The audio processing submodule is used to interactively fuse and encode the audio data and text data of the sentence to obtain the audio features of the sentence;

[0052] The text processing submodule is used to interactively fuse and encode the audio data and text data of the sentence to obtain the text features of the sentence.

[0053] Optionally, the audio processing submodule includes an audio coding layer, a first audio interactive fusion coding layer, and a second audio interactive fusion coding layer, and the text processing submodule includes a text coding layer, a first text interactive fusion coding layer, and a second text interactive fusion coding layer;

[0054] The audio encoding layer is used to encode the audio data of the sentence to obtain the initial audio features of the sentence;

[0055] The text encoding layer is used to encode the text data of the sentence to obtain the initial text features of the sentence;

[0056] The first audio interactive fusion coding layer is used to interactively fuse and encode the initial audio features of the sentence and the initial text features of the sentence to obtain the first audio interactive features of the sentence;

[0057] The first text interactive fusion coding layer is used to interactively fuse the initial audio features of the sentence and the initial text features of the sentence to obtain the first text interactive features of the sentence;

[0058] The second audio interactive fusion coding layer is used to interactively fuse the first audio interactive feature of the sentence and the first text interactive feature of the sentence to obtain the audio feature of the sentence;

[0059] The second text interactive fusion coding layer is used to interactively fuse the first audio interactive feature of the sentence and the first text interactive feature of the sentence to obtain the text feature of the sentence.

[0060] Optionally, the device further includes a sentiment analysis model training unit, wherein the sentiment analysis model training unit includes:

[0061] A first true sample data set acquisition unit, used to acquire a first true sample data set, wherein the first true sample data set includes a plurality of first true sample data, each of which includes audio data, text data, and an emotion type label of each sentence in a set of true conversation data;

[0062] A fake sample data set generating unit, configured to generate a fake sample data set according to the first real sample data set, wherein the fake sample data set includes a plurality of fake sample data, each of which includes audio data, text data and an emotion type label of each sentence in a set of fake conversation data;

[0063] The sentiment analysis model training unit is used to train the pre-training model based on the fake sample data set to obtain the sentiment analysis model.

[0064] Optionally, the fake sample data set generating unit includes:

[0065] An emotional logic factor determination unit, used to determine, for each first true sample data in the first true sample data set, an emotional logic factor of the first true sample data;

[0066] The fake sample data set generating subunit is used to generate a fake sample data set based on each first real sample data in the first real sample data set and the emotional logic factor of each first real sample data.

[0067] Optionally, the emotional logic factor determination unit includes:

[0068] The sentiment logic factor generation model processing unit is used to input the text data of each sentence in the first true sample data into the sentiment logic factor generation model, and the sentiment logic factor generation model processes the text data of each sentence in the first true sample data to obtain the sentiment logic factor of the first true sample data.

[0069] Optionally, the sentiment logic factor generation model includes a semantic feature extraction module, a sentiment feature extraction module, a cause feature extraction module, a sentiment probability prediction module, a cause probability prediction module and a sentiment logic factor generation module:

[0070] The semantic feature extraction module in the sentiment logic factor generation model extracts semantic features from each sentence in the first true sample data to obtain semantic features of each sentence in the first true sample data;

[0071] The sentiment feature extraction module in the sentiment logic factor generation model extracts sentiment features from the semantic features of each sentence in the first true sample data to obtain the sentiment features of each sentence in the first true sample data;

[0072] The cause feature extraction module in the sentiment logic factor generation model extracts cause features from the semantic features of each sentence in the first true sample data to obtain cause features of each sentence in the first true sample data;

[0073] The emotion probability prediction module in the emotion logic factor generation model performs emotion probability prediction on the emotion feature of each sentence in the first true sample data to obtain the emotion probability of each sentence in the first true sample data;

[0074] The cause probability prediction module in the emotional logic factor generation model performs cause probability prediction on the cause feature of each sentence in the first true sample data to obtain the cause probability of each sentence in the first true sample data;

[0075] The emotional logic factor generation module in the emotional logic factor generation model fuses the emotional features, emotional probabilities, cause features, cause probabilities of each sentence in the first true sample data and the positional relationship encoding vectors of the sentences in the first true sample data to obtain the emotional logic factors of the first true sample data.

[0076] Optionally, the emotion logic factor generation model is trained based on preset sliding window parameters and a second true sample data set; the preset sliding window parameters are used to indicate the number of sentences included in a sentence group with a high possibility of an emotion logic relationship; the second true sample data set includes multiple second true sample data, each of the second true sample data includes text data, reason labels, emotion type labels and emotion logic factors of each sentence in a group of true dialogue data, and the reason labels are used to indicate whether the sentence has an emotion logic relationship with other sentences.

[0077] Optionally, the semantic feature extraction module in the sentiment logic factor generation model is further used for:

[0078] For each sentence in the first true sample data, the overall semantic features of the sentence and other sentences in the same sliding window as the sentence are extracted as the semantic features of the sentence, and the sliding window is determined based on the preset sliding window parameters.

[0079] Optionally, the fake sample data set generating subunit includes:

[0080] A data generation model processing unit is used to input the first true sample data and the emotional logic factor of the first true sample data into N generators of the data generation model for each true sample data in the first true sample data set, and the N generators of the data generation model output the fake sample data set, where N is the same as the number of emotional type labels.

[0081] Optionally, the data generation model is trained as follows:

[0082] Acquire a third true sample data set, wherein the third true sample data set includes a plurality of third true sample data, each of the third true sample data includes audio data, text data, an emotion type label of each sentence in a set of true conversation data, and an emotion logic factor of the third true sample data;

[0083] Constructing a discriminator, wherein the discriminator is an N+1 classifier;

[0084] The data generation model is trained using the third true sample data set as a training sample, with the cumulative penalty items of the N generators being minimized and the judgment result of the discriminator on the output of the data model being close to the judgment result of the discriminator on the third true sample data as the training target.

[0085] A sentiment analysis device, comprising a memory and a processor;

[0086] The memory is used to store programs;

[0087] The processor is used to execute the program to implement the various steps of the sentiment analysis method described above.

[0088] A readable storage medium stores a computer program, and when the computer program is executed by a processor, the various steps of the sentiment analysis method described above are implemented.

[0089] By means of the above technical scheme, the present application discloses a sentiment analysis method, related equipment and readable storage medium. After acquiring the voice conversation data to be analyzed, for each sentence in the voice conversation data, the audio data and text data of the sentence are determined, the audio data and text data of the sentence are interactively fused and encoded to obtain the audio features and text features of the sentence, and sentiment analysis is performed on the sentence based on the audio features and text features of the sentence to determine the sentiment type of the sentence. Since the voice conversation data, text information and audio information can express the user's emotional information in different dimensions, the sentiment type of the sentence determined based on the audio features and text features of the sentence is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0091] Figure 1 A schematic diagram of the flow of the sentiment analysis method disclosed in the embodiment of the present application;

[0092] Figure 2 A schematic diagram of a sentiment analysis model structure disclosed in an embodiment of the present application;

[0093] Figure 3 A schematic diagram of a sentiment analysis model structure disclosed in an embodiment of the present application;

[0094] Figure 4 A schematic diagram of a sentiment analysis model structure disclosed in an embodiment of the present application;

[0095] Figure 5 A schematic diagram of the structure of an interactive fusion coding module disclosed in an embodiment of the present application;

[0096] Figure 6 A schematic diagram of the structure of an emotional logic factor generation model disclosed in an embodiment of the present application;

[0097] Figure 7 A schematic diagram of the structure of an emotional logic factor generation model disclosed in an embodiment of the present application;

[0098] Figure 8 A schematic diagram of the structure of a sentiment analysis device disclosed in an embodiment of the present application;

[0099] Fig. 9 This is a hardware structure block diagram of the sentiment analysis device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0100] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0101] In order to obtain highly accurate sentiment analysis results, the inventors of this case conducted research and found that there are two main existing sentiment analysis methods, one is a sentiment analysis method based on sentiment dictionaries, and the other is a sentiment analysis method based on machine learning.

[0102] Among them, the sentiment analysis method based on the sentiment dictionary needs to pre-build the sentiment dictionary, which contains a positive word dictionary, a negative word dictionary, a degree word dictionary, and a negation word dictionary. In each dictionary, each entry consists of a word and a corresponding weight. When performing sentiment analysis on the user's input based on the sentiment dictionary, it is necessary to match the user's input with the sentiment dictionary, and obtain the sentiment analysis result according to the weight of the matched entry.

[0103] However, the sentiment analysis method based on the sentiment dictionary only uses simple keyword matching and linear superposition of sentiment word weights to obtain sentiment analysis results, and its accuracy is low. For example, "I started to think that you were very bad, slow progress, and rubbish efficiency, but after getting to know you, I found you were very good." This sentence actually expresses a kind of praise, but the result of sentiment analysis based on the sentiment dictionary is not very good.

[0104] Sentiment analysis methods based on machine learning require training RNNs (Recurrent Neural Networks) such as LSTM (Long Short-Term Memory) or GRU (gated recurrent unit) based on a large amount of sentiment data to obtain a sentiment analysis model, where the sentiment data is mainly text data. When sentiment analysis methods based on machine learning perform sentiment analysis on user input, the text data corresponding to the user's input needs to be input into the sentiment analysis model to obtain the sentiment analysis result.

[0105] However, existing sentiment analysis methods based on machine learning only use text data corresponding to user input for sentiment analysis. When the user input is voice or video, it is impossible to consider the user's emotions from the perspective of the user's voice, resulting in low accuracy of its sentiment analysis results.

[0106] In view of the problems existing in the above methods, the inventors of this case conducted in-depth research and finally proposed a sentiment analysis method.

[0107] Next, the sentiment analysis method provided by this application is introduced through the following embodiments.

[0108] Reference Figure 1 , Figure 1 This is a flow chart of the sentiment analysis method disclosed in the embodiment of the present application. The method may include:

[0109] Step S101: Acquire voice conversation data to be analyzed.

[0110] In the present application, the voice conversation data to be analyzed can be voice conversation data in any scenario, and the voice conversation data contains multiple sentences. For example, in the outbound call field, multiple rounds of voice interaction data between users and customer service or agent systems. This application does not impose any restrictions on this.

[0111] In the present application, the voice conversation data to be analyzed may be original voice conversation data, or may be voice conversation data after preprocessing (such as noise reduction) of the original voice conversation data, and the present application does not impose any limitation on this.

[0112] Step S102: for each sentence in the voice dialogue data, determine the audio data and text data of the sentence; interactively fuse and encode the audio data and text data of the sentence to obtain the audio features and text features of the sentence; and determine the emotion type of the sentence based on the audio features and text features of the sentence.

[0113] Conventional sentiment analysis techniques only use text data, but for voice conversation data, audio information can actually also express the user's emotional information. Therefore, in this application, for each sentence in the voice conversation data, the audio data and text data of the sentence are determined, and sentiment analysis is performed on the sentence based on the audio data and text data of the sentence, which can improve the accuracy of the sentiment analysis result.

[0114] It should be noted that, in the present application, each sentence in the voice conversation data can be processed by voice recognition to obtain audio data and text data of the sentence. This method can adopt the existing voice recognition technology, which will not be described in detail here.

[0115] In addition, it should be noted that interactive fusion encoding of the audio data and text data of a sentence can make the obtained audio features and text features contain the semantic features of the entire sentence, making the emotion type of the sentence determined based on the audio features and text features of the sentence more accurate.

[0116] In the present application, a neural network model can be used to interactively fuse and encode the audio data and text data of the sentence to obtain the audio features and text features of the sentence. This will be explained in detail through subsequent embodiments and will not be described in detail here.

[0117] In this embodiment, a sentiment analysis method is disclosed. In this method, after obtaining the voice conversation data to be analyzed, for each sentence in the voice conversation data, the audio data and text data of the sentence are determined, the audio data and text data of the sentence are interactively fused and encoded to obtain the audio features and text features of the sentence, and the sentence is sentimentally analyzed based on the audio features and text features of the sentence to determine the sentiment type of the sentence. Since the voice conversation data, text information and audio information can express the user's emotional information in different dimensions, the sentiment type of the sentence determined based on the audio features and text features of the sentence is more accurate.

[0118] In another embodiment of the present application, a specific implementation method of interactively fusion encoding the audio data and text data of the sentence in the above step S102 to obtain the audio features and text features of the sentence, and determining the emotion type of the sentence based on the audio features and text features of the sentence is described in detail.

[0119] In the present application, a sentiment analysis model may be pre-constructed and trained, and the above step S102 may be implemented based on the trained sentiment analysis model.

[0120] Please refer to Figure 2 , Figure 2 Schematic diagram of a sentiment analysis model structure disclosed in the embodiment of this application. Figure 2As shown, the sentiment analysis model includes an interactive fusion encoding module and a sentiment analysis module.

[0121] based on Figure 2 The sentiment analysis model shown in the present application can input the audio data and text data of the sentence into the sentiment analysis model, and the interactive fusion encoding module of the sentiment analysis model performs interactive fusion encoding on the audio data and text data of the sentence to obtain the audio features and text features of the sentence; the sentiment analysis module of the sentiment analysis model determines the sentiment type of the sentence based on the audio features and text features of the sentence.

[0122] In the sentiment analysis model, the interactive fusion coding module of the sentiment analysis model needs to interactively fuse the audio data and text data of the sentence, and the audio data and the text data are two different modal data. Encoding the audio data and the text data together will inevitably cause confusion. Therefore, in this application, another sentiment analysis model is proposed, which is described in detail in the following content.

[0123] Please refer to Figure 3 , Figure 3 Schematic diagram of a sentiment analysis model structure disclosed in the embodiment of this application. Figure 3 As shown, the sentiment analysis model includes an interactive fusion encoding module and a sentiment analysis module, and the interactive fusion encoding module includes an audio processing submodule and a text processing submodule.

[0124] based on Figure 3 The sentiment analysis model shown in the present application, the interactive fusion coding module of the sentiment analysis model interactively fusion codes the audio data and text data of the sentence to obtain the audio features and text features of the sentence. The specific implementation method may include: the audio processing submodule in the interactive fusion coding module interactively fusion codes the audio data and text data of the sentence to obtain the audio features of the sentence; the text processing submodule in the interactive fusion coding module interactively fusion codes the audio data and text data of the sentence to obtain the text features of the sentence.

[0125] Please refer to Figure 4 , Figure 4 Schematic diagram of a sentiment analysis model structure disclosed in the embodiment of this application. Figure 4 As shown, the sentiment analysis model includes an interactive fusion coding module and a sentiment analysis module, and the interactive fusion coding module includes an audio processing submodule and a text processing submodule. The audio processing submodule includes an audio coding layer, a first audio interactive fusion coding layer, and a second audio interactive fusion coding layer, and the text processing submodule includes a text coding layer, a first text interactive fusion coding layer, and a second text interactive fusion coding layer.

[0126] based on Figure 4 The sentiment analysis model shown in the figure, in the present application, the interactive fusion coding module of the sentiment analysis model performs interactive fusion coding on the audio data and text data of the sentence, and the specific implementation method of obtaining the audio features and text features of the sentence may include:

[0127] The audio coding layer encodes the audio data of the sentence to obtain initial audio features of the sentence;

[0128] The text encoding layer encodes the text data of the sentence to obtain the initial text features of the sentence;

[0129] The first audio interactive fusion coding layer interactively fusion codes the initial audio features of the sentence and the initial text features of the sentence to obtain a first audio interactive feature of the sentence;

[0130] The first text interactive fusion coding layer interactively fuses the initial audio features of the sentence and the initial text features of the sentence to obtain a first text interactive feature of the sentence;

[0131] The second audio interaction fusion coding layer interactively fusion codes the first audio interaction feature of the sentence and the first text interaction feature of the sentence to obtain the audio feature of the sentence;

[0132] The second text interactive fusion coding layer interactively fusion codes the first audio interactive feature of the sentence and the first text interactive feature of the sentence to obtain the text feature of the sentence.

[0133] Considering that only one interactive fusion coding can extract only the shallow information of text and audio, and three or more interactive fusion coding will increase the model parameters sharply, this solution performs two interactive fusion codings based on comprehensive considerations. The first interactive fusion coding mainly interacts with the shallow information of "audio" and "text" and extracts features, focusing on the learning of word pronunciation and word shape; the second interactive fusion coding mainly interacts with the deep features of "voice" and "text" and extracts features, focusing on the learning of the overall sentence voice features and the deep understanding of the entire sentence semantics.

[0134] It should be noted that the sentiment analysis model can be any neural network structure that can realize the function of "interactively fusion encoding the audio data and text data of the sentence to obtain the audio features and text features of the sentence; based on the audio features and text features of the sentence, determining the sentiment type of the sentence", and this application does not impose any limitations on this.

[0135] As an implementable method, the structure of the interactive fusion encoding module can be constructed with reference to the Multi-BERT (Multi-bidirectional encoder representations from transformers) model. The sentiment analysis module can be a structure of an attention layer followed by a softmax layer.

[0136] As an implementable method, in this application, a specific implementation of an interactive fusion coding module structure is disclosed, please refer to Figure 5 , Figure 5 This is a schematic diagram of the structure of an interactive fusion coding module disclosed in an embodiment of the present application, which is constructed with reference to the Multi-BERT (Multi-bid irectional encoder representations from transformers) model.

[0137] like Figure 5 As shown in the figure, in the interactive fusion coding module, the left module is the audio processing submodule, and the right module is the text processing submodule. Figure 5 , the specific implementation method of "the audio processing submodule in the interactive fusion encoding module interactively fusion encodes the audio data and text data of the sentence to obtain the audio features of the sentence" is described as follows:

[0138] In the audio processing submodule, first, the text data after word segmentation is processed by w1, w2, w3, …, w n The audio segment corresponding to w1 is denoted by t1, the audio segment corresponding to w2 is denoted by t2, ..., w n The corresponding audio clip is t n The corresponding audio data can be represented by t1, t2, t3, …, t n Represents. The audio data t1, t2, t3, ..., t n The acoustic features are extracted through the pre-trained MFCC (Mel-scale Frequency Cepstral Coefficients) module to obtain the acoustic feature vectors v1, v2, v3, …, v n Then, the acoustic feature vectors v1, v2, v3, …, v are encoded by the embedding module. nEncode to obtain the feature vector after embedding encoding, and pass the feature vector after embedding encoding and the feature vector output by the first Transformer module in the text processing submodule to the first co-transformer module in the audio processing submodule. The first co-transformer module and the first Transformer module in the audio processing submodule jointly implement the first interactive fusion encoding of the audio data and the text data. Similarly, the second co-transformer module and the second Transformer module in the audio processing submodule jointly implement the second interactive fusion encoding of the audio data and the text data to obtain the audio feature h of the sentence. v1 ,h v2 ,h v3 ,…,h vn .

[0139] In the text processing submodule, the text data is segmented and then used as w1, w2, w3, ..., w n It is then encoded through the embedding encoding module and the first Transformer module to obtain the feature vector output by the first Transformer module. The feature vector and the feature vector after embedding encoding of the audio processing submodule are passed to the first co-transformer module in the text processing submodule. The first co-transformer module and the second Transformer module in the text processing submodule jointly implement the first interactive fusion encoding of the audio data and the text data. Similarly, the second co-transformer module and the third Transformer module in the text processing submodule jointly implement the second interactive fusion encoding of the audio data and the text data, and obtain the text feature h of the sentence. w1 ,h w2 ,h w3 ,…,h wn .

[0140] In the present application, the interactive fusion coding module can be pre-trained based on a large amount of "speech-text" data to obtain a pre-trained model, and the training of the sentiment analysis model can be performed on the basis of the pre-trained model.

[0141] In another embodiment of the present application, a training method for the sentiment analysis model is described in detail.

[0142] As an implementable embodiment, the training method of the sentiment analysis model may include: obtaining a first true sample data set, the first true sample data set including multiple first true sample data, each of the first true sample data including audio data, text data and sentiment type label of each sentence in a set of true conversation data; training a pre-trained model based on the first true sample data set to obtain a sentiment analysis model.

[0143] It should be noted that the emotion type label can be set based on the specific application scenario, and this application does not impose any restrictions on this. For example, in the field of outbound calls, considering business needs, the emotion type label can include five types: pleasure, praise, neutral, complaint, and anger. Among them, pleasure and praise are positive emotions, and complaints and anger are negative emotions.

[0144] Since in actual tasks, except for neutral emotions, the proportion of negative and positive emotional data is relatively small, in order to ensure the training effect of the sentiment analysis model, as another feasible implementation method, the training method of the sentiment analysis model may include:

[0145] Step S201: Obtain a first true sample data set.

[0146] The first true sample data set includes multiple first true sample data, and each of the first true sample data includes audio data, text data and emotion type labels of each sentence in a set of true conversation data.

[0147] Step S202: Generate a fake sample data set according to the first real sample data set.

[0148] The fake sample data set includes multiple fake sample data, and each of the fake sample data includes audio data, text data and emotion type labels of each sentence in a set of fake conversation data.

[0149] In recent years, GAN (Generative Adversarial Net) has been widely used in the field of image data generation, so in this application, a fake sample data set can be generated based on GAN. The specific method will be described in detail through the following embodiments and will not be described here.

[0150] Step S203: training the pre-trained model based on the fake sample data set to obtain a sentiment analysis model.

[0151] It should be noted that in this step, the pre-trained model can be trained only based on the fake sample data set to obtain the sentiment analysis model, or the pre-trained model can be trained based on both the first real sample data set and the fake sample data set to obtain the sentiment analysis model. This application does not impose any limitations on this.

[0152] It is worth noting that in voice conversations, the user's current emotion has an obvious logical relationship with the previous and subsequent conversation content, that is, a causal relationship. In order to ensure the quality of the fake sample data, in this application, the emotional logical relationship between sentences is taken into consideration when generating fake sample data. Specifically, in this application, the emotional logic factor parameter is introduced into the voice conversation data to indicate the degree of logical association between every two sentences in the voice conversation data.

[0153] Based on this, in one embodiment of the present application, a specific implementation method of generating a fake sample data set according to the first real sample data set in step S202 is introduced. The method may include the following steps:

[0154] Step S301: for each first true sample data in the first true sample data set, determine the sentiment logic factor of the first true sample data.

[0155] It should be noted that the sentiment logic factor is in matrix form. Assuming that the first true sample data contains d sentences, the sentiment logic factor is a matrix of size d×d, and each element represents the degree of logical association between two sentences. The value range of each element can be [0,1]. For example, for sentences A and B, the larger the value of the sentiment logic factor element, the greater the degree of logical association between the two sentences, that is, it is more reasonable to deduce sentence B from sentence A; conversely, if the value of the sentiment logic factor between sentences A and B is smaller, it means that the degree of logical association between the two sentences is smaller, that is, the probability of logically deriving sentence B from sentence A is smaller.

[0156] In the present application, a sentiment logic factor generation model can be pre-constructed and trained, and the above step S301 can be implemented based on the trained sentiment logic factor generation model. Specifically, the text data of each sentence in the first true sample data can be input into the sentiment logic factor generation model, and the sentiment logic factor generation model processes the text data of each sentence in the first true sample data to obtain the sentiment logic factor of the first true sample data. The construction and training methods of the sentiment logic factor generation model will be described in detail through the following embodiments, and will not be described in detail here.

[0157] Step S302: Generate a false sample data set based on each first true sample data and the sentiment logic factor of each first true sample data in the first true sample data set.

[0158] In the present application, a data generation model can be pre-built and trained, and the above step S302 is implemented based on the trained data generation model. The specific implementation method will be described in detail through the following embodiments, and will not be described in detail here.

[0159] In another embodiment of the present application, the construction and training process of the emotional logic factor generation model is introduced in detail, as follows:

[0160] Please refer to Figure 6 , Figure 6 This is a schematic diagram of the structure of a sentiment logic factor generation model disclosed in an embodiment of the present application, such as Figure 6 As shown, the emotional logic factor generation model includes: a semantic feature extraction module, an emotional feature extraction module, a cause feature extraction module, an emotional probability prediction module, a cause probability prediction module and an emotional logic factor generation module.

[0161] As an implementable method, the emotional logic factor generation model can be trained based on a second real sample data set; the second real sample data set includes multiple second real sample data, each of the second real sample data includes text data, reason labels, emotional type labels and emotional logic factors of each sentence in a group of real conversation data, and the reason labels are used to indicate whether the sentence has an emotional logic relationship with other sentences.

[0162] Based on this, the text data of each sentence in the first true sample data is input into the sentiment logic factor generation model, and the sentiment logic factor generation model processes the text data of each sentence in the first true sample data to obtain the sentiment logic factor of the first true sample data. The specific implementation method can be as follows:

[0163] The semantic feature extraction module in the sentiment logic factor generation model extracts semantic features from each sentence in the first true sample data to obtain semantic features of each sentence in the first true sample data;

[0164] The sentiment feature extraction module in the sentiment logic factor generation model extracts sentiment features from the semantic features of each sentence in the first true sample data to obtain the sentiment features of each sentence in the first true sample data;

[0165] The cause feature extraction module in the sentiment logic factor generation model extracts cause features from the semantic features of each sentence in the first true sample data to obtain cause features of each sentence in the first true sample data;

[0166] The emotion probability prediction module in the emotion logic factor generation model performs emotion probability prediction on the emotion feature of each sentence in the first true sample data to obtain the emotion probability of each sentence in the first true sample data;

[0167] The cause probability prediction module in the emotional logic factor generation model performs cause probability prediction on the cause feature of each sentence in the first true sample data to obtain the cause probability of each sentence in the first true sample data;

[0168] The emotional logic factor generation module in the emotional logic factor generation model fuses the emotional features, emotional probabilities, cause features, cause probabilities of each sentence in the first true sample data and the positional relationship encoding vector of the sentences in the first true sample data to obtain the emotional logic factors of the first true sample data.

[0169] It should be noted that the sentiment logic factor can be represented by a two-dimensional matrix, where the sentiment logic factor element in the i-th row and j-th column is It is used to represent the emotional logical relationship between sentence i and sentence j. Specifically, it can be calculated by the following formula:

[0170]

[0171] Among them, r i emo represents the sentiment feature of sentence i, represents the sentiment probability of sentence i, represents the causal feature of sentence j, represents the cause probability of sentence j, rpe i,j Represents the positional relationship encoding vector between sentence i and sentence j.

[0172] In this application, the emotional logic factor generation model can be implemented based on a multi-layer neural network. As an implementable method, the semantic feature extraction module can be implemented using the Bi-LSTM model and the Attention mechanism, the emotional feature extraction module and the cause feature extraction module can be implemented using the Bi-LSTM model, and the emotional probability prediction module, the cause probability prediction module and the emotional logic factor generation module can be implemented using the softmax network.

[0173] For easier understanding, please refer to Figure 7 , Figure 7 Schematic diagram of the structure of a sentiment logic factor generation model disclosed in the embodiment of the present application, such as Figure 7 As shown, the input text data is D = {c1,...,c d}, where c d represents the dth sentence in the input text data, r i emo represents the sentiment feature of sentence i, r i cau represents the cause feature of sentence i, represents the sentiment probability of sentence i, Represents the probability of the cause of sentence i. The final output of the model is Represents the sentiment logic factor element in the i-th row and j-th column of the sentiment logic factor.

[0174] In the actual human-computer interaction process, the sentences around the current sentence are more likely to have an emotional logical relationship with it, and the sentences farther away are less likely to have an emotional logical relationship with it. Therefore, in this application, a sliding window parameter is preset, and the sliding window parameter is used to indicate the number of sentences included in the sentence group with a high possibility of an emotional logical relationship. The sliding window can be determined based on the sliding window parameter. For example, if the sliding window parameter is 3, each sliding window is a 3×3 matrix size.

[0175] As another possible implementation method, the emotional logic factor generation model is trained based on preset sliding window parameters and a second true sample data set.

[0176] Based on this, the semantic feature extraction module in the sentiment logic factor generation model extracts semantic features from each sentence in the first true sample data to obtain the semantic features of each sentence in the first true sample data, including:

[0177] For each sentence in the first true sample data, the overall semantic features of the sentence and other sentences in the same sliding window as the sentence are extracted as the semantic features of the sentence.

[0178] In this method, the contextual relationship of other sentences in the same sliding window as the current sentence can be considered at the same time, and the semantic features of the current sentence obtained are integrated with the features of sentences that are likely to have emotional logical relationships with the current sentence, so that the emotional logical factor of the first true sample data finally obtained is more accurate.

[0179] In another embodiment of the present application, a detailed description is given of an implementation method for generating a fake sample data set based on each first true sample data in the first true sample data set and the emotional logic factor of each first true sample data, as follows:

[0180] As mentioned in the above content, in this application, a fake sample data set can be generated based on GAN. GAN uses the discriminator D (Discriminator) to guide the generator G (Generator) to generate data. In this application, the data generation model is constructed and trained by optimizing the model structure of GAN. The details will be explained in detail in the following content.

[0181] As an implementable embodiment, the data generation model in the present application includes N generators, where N is the same as the number of emotion type labels.

[0182] Taking the five emotion type labels including happy emotion type label, praise emotion type label, neutral emotion type label, complain emotion type label and angry emotion type label as examples, the data generation model includes 5 generators, each of which generates data with happy emotion type label, praise emotion type label, neutral emotion type label, complain emotion type label and angry emotion type label respectively. Using multiple generators can generate data with different emotion type labels, and at the same time can increase the richness of generated data, rather than generating a small amount of similar data that is not much different from the original data.

[0183] Based on this, as an implementable method, the fake sample data set is generated based on each first true sample data and the emotional logic factor of each first true sample data in the first true sample data set, including: for each true sample data in the first true sample data set, the first true sample data and the emotional logic factor of the first true sample data are input into N generators of the data generation model, and the N generators of the data generation model output the fake sample data set.

[0184] As an implementable method, the data generation model is trained as follows:

[0185] Acquire a third true sample data set, wherein the third true sample data set includes a plurality of third true sample data, each of the third true sample data includes audio data, text data, an emotion type label of each sentence in a set of true conversation data, and an emotion logic factor of the third true sample data;

[0186] Construct a discriminator, which is an N+1 classifier; where N is the same as the number of emotion type labels;

[0187] The data generation model is trained with the third true sample data set as the training sample, and the cumulative penalty items of the N generators are minimized, and the judgment result of the discriminator on the output of the data model is close to the judgment result of the discriminator on the third true sample data as the training goal. The penalty item of each generator can be obtained through Monte Carlo search and the discriminator.

[0188] It should be noted that in this application, the discriminator adopts multi-classification, which can guide a generator to generate data while staying away from other emotion type labels. In addition, when training the data generation model, adding the emotion logic factor of the real sample data can enable the data generation model to learn the emotion logic characteristics of the real sample data, making the generated fake sample data closer to the real sample data.

[0189] As an implementation method, each generator can use LSTM as a network layer, and the discriminator can use CNN as a network layer.

[0190] The fake sample data set generated by the data generation model can solve the problem of uneven distribution of data of various emotion types in the real sample data set.

[0191] The sentiment analysis device disclosed in the embodiment of the present application is described below. The sentiment analysis device described below and the sentiment analysis method described above can be referenced to each other.

[0192] Reference Figure 8 , Figure 8 Schematic diagram of the structure of a sentiment analysis device disclosed in the embodiment of this application. Figure 8 As shown, the sentiment analysis device may include:

[0193] An acquisition unit 81, used to acquire the voice conversation data to be analyzed;

[0194] The sentiment analysis unit 82 is used to determine the audio data and text data of each sentence in the voice dialogue data; interactively fuse and encode the audio data and text data of the sentence to obtain the audio features and text features of the sentence; and determine the sentiment type of the sentence based on the audio features and text features of the sentence.

[0195] As an implementable embodiment, the sentiment analysis unit includes:

[0196] The sentiment analysis model processing unit is used to input the audio data and text data of the sentence into the sentiment analysis model, and the sentiment analysis model includes an interactive fusion encoding module and a sentiment analysis module. The interactive fusion encoding module is used to perform interactive fusion encoding on the audio data and text data of the sentence to obtain the audio features and text features of the sentence; the sentiment analysis module is used to determine the sentiment type of the sentence based on the audio features and text features of the sentence.

[0197] As an implementable method, the interactive fusion encoding module of the sentiment analysis model includes an audio processing submodule and a text processing submodule:

[0198] The audio processing submodule is used to interactively fuse and encode the audio data and text data of the sentence to obtain the audio features of the sentence;

[0199] The text processing submodule is used to interactively fuse and encode the audio data and text data of the sentence to obtain the text features of the sentence.

[0200] As an implementable method, the audio processing submodule includes an audio coding layer, a first audio interactive fusion coding layer, and a second audio interactive fusion coding layer, and the text processing submodule includes a text coding layer, a first text interactive fusion coding layer, and a second text interactive fusion coding layer;

[0201] The audio encoding layer is used to encode the audio data of the sentence to obtain the initial audio features of the sentence;

[0202] The text encoding layer is used to encode the text data of the sentence to obtain the initial text features of the sentence;

[0203] The first audio interactive fusion coding layer is used to interactively fuse and encode the initial audio features of the sentence and the initial text features of the sentence to obtain the first audio interactive features of the sentence;

[0204] The first text interactive fusion coding layer is used to interactively fuse the initial audio features of the sentence and the initial text features of the sentence to obtain the first text interactive features of the sentence;

[0205] The second audio interactive fusion coding layer is used to interactively fuse the first audio interactive feature of the sentence and the first text interactive feature of the sentence to obtain the audio feature of the sentence;

[0206] The second text interactive fusion coding layer is used to interactively fuse the first audio interactive feature of the sentence and the first text interactive feature of the sentence to obtain the text feature of the sentence.

[0207] As an implementable embodiment, the device further includes a sentiment analysis model training unit, and the sentiment analysis model training unit includes:

[0208] A first true sample data set acquisition unit, used to acquire a first true sample data set, wherein the first true sample data set includes a plurality of first true sample data, each of which includes audio data, text data, and an emotion type label of each sentence in a set of true conversation data;

[0209] A fake sample data set generating unit, configured to generate a fake sample data set according to the first real sample data set, wherein the fake sample data set includes a plurality of fake sample data, each of which includes audio data, text data and an emotion type label of each sentence in a set of fake conversation data;

[0210] The sentiment analysis model training unit is used to train the pre-training model based on the fake sample data set to obtain the sentiment analysis model.

[0211] As an implementable embodiment, the fake sample data set generating unit includes:

[0212] An emotional logic factor determination unit, used to determine, for each first true sample data in the first true sample data set, an emotional logic factor of the first true sample data;

[0213] The fake sample data set generating subunit is used to generate a fake sample data set based on each first real sample data in the first real sample data set and the emotional logic factor of each first real sample data.

[0214] As an implementable embodiment, the emotional logic factor determination unit includes:

[0215] The sentiment logic factor generation model processing unit is used to input the text data of each sentence in the first true sample data into the sentiment logic factor generation model, and the sentiment logic factor generation model processes the text data of each sentence in the first true sample data to obtain the sentiment logic factor of the first true sample data.

[0216] As an implementable method, the emotional logic factor generation model includes a semantic feature extraction module, an emotional feature extraction module, a cause feature extraction module, an emotional probability prediction module, a cause probability prediction module and an emotional logic factor generation module:

[0217] The semantic feature extraction module in the sentiment logic factor generation model extracts semantic features from each sentence in the first true sample data to obtain semantic features of each sentence in the first true sample data;

[0218] The sentiment feature extraction module in the sentiment logic factor generation model extracts sentiment features from the semantic features of each sentence in the first true sample data to obtain the sentiment features of each sentence in the first true sample data;

[0219] The cause feature extraction module in the sentiment logic factor generation model extracts cause features from the semantic features of each sentence in the first true sample data to obtain cause features of each sentence in the first true sample data;

[0220] The emotion probability prediction module in the emotion logic factor generation model performs emotion probability prediction on the emotion feature of each sentence in the first true sample data to obtain the emotion probability of each sentence in the first true sample data;

[0221] The cause probability prediction module in the emotional logic factor generation model performs cause probability prediction on the cause feature of each sentence in the first true sample data to obtain the cause probability of each sentence in the first true sample data;

[0222] The emotional logic factor generation module in the emotional logic factor generation model fuses the emotional features, emotional probabilities, cause features, cause probabilities of each sentence in the first true sample data and the positional relationship encoding vectors of the sentences in the first true sample data to obtain the emotional logic factors of the first true sample data.

[0223] As an implementable method, the emotional logic factor generation model is trained based on preset sliding window parameters and a second true sample data set; the preset sliding window parameters are used to indicate the number of sentences included in a sentence group with a high possibility of emotional logic relationship; the second true sample data set includes multiple second true sample data, each of the second true sample data includes text data, reason labels, emotional type labels and emotional logic factors of each sentence in a group of real dialogue data, and the reason labels are used to indicate whether the sentence has an emotional logic relationship with other sentences.

[0224] As an implementable method, the semantic feature extraction module in the sentiment logic factor generation model is also used for:

[0225] For each sentence in the first true sample data, the overall semantic features of the sentence and other sentences in the same sliding window as the sentence are extracted as the semantic features of the sentence, and the sliding window is determined based on the preset sliding window parameters.

[0226] As an implementable embodiment, the fake sample data set generating subunit includes:

[0227] A data generation model processing unit is used to input the first true sample data and the emotional logic factor of the first true sample data into N generators of the data generation model for each true sample data in the first true sample data set, and the N generators of the data generation model output the fake sample data set, where N is the same as the number of emotional type labels.

[0228] As an implementable method, the data generation model is trained as follows:

[0229] Acquire a third true sample data set, wherein the third true sample data set includes a plurality of third true sample data, each of the third true sample data includes audio data, text data, an emotion type label of each sentence in a set of true conversation data, and an emotion logic factor of the third true sample data;

[0230] Constructing a discriminator, wherein the discriminator is an N+1 classifier;

[0231] The data generation model is trained using the third true sample data set as a training sample, with the cumulative penalty items of the N generators being minimized and the judgment result of the discriminator on the output of the data model being close to the judgment result of the discriminator on the third true sample data as the training target.

[0232] Reference Fig. 9 , Fig. 9 The hardware structure diagram of the sentiment analysis device provided in the embodiment of the present application is shown in FIG. Fig. 9 ,The hardware structure of the sentiment analysis device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;

[0233] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;

[0234] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;

[0235] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0236] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to:

[0237] Acquire voice conversation data to be analyzed;

[0238] For each sentence in the voice dialogue data, determine the audio data and text data of the sentence; interactively fuse and encode the audio data and text data of the sentence to obtain the audio features and text features of the sentence; and determine the emotion type of the sentence based on the audio features and text features of the sentence.

[0239] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0240] The embodiment of the present application further provides a readable storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:

[0241] Acquire voice conversation data to be analyzed;

[0242] For each sentence in the voice dialogue data, determine the audio data and text data of the sentence; interactively fuse and encode the audio data and text data of the sentence to obtain the audio features and text features of the sentence; and determine the emotion type of the sentence based on the audio features and text features of the sentence.

[0243] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0244] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0245] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0246] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A sentiment analysis method, characterized in that: The method comprises: Acquire voice conversation data to be analyzed; For each sentence in the voice dialogue data, determine the audio data and text data of the sentence; perform interactive fusion encoding on the audio data and text data of the sentence using an interactive fusion encoding module in a pre-built sentiment analysis model to obtain audio features and text features of the sentence; determine the sentiment type of the sentence based on the audio features and text features of the sentence; Wherein, the interactive fusion coding module includes an audio processing submodule and a text processing submodule; The audio processing submodule in the interactive fusion encoding module performs interactive fusion encoding on the audio data and text data of the sentence to obtain the audio features of the sentence; The text processing submodule in the interactive fusion encoding module performs interactive fusion encoding on the audio data and text data of the sentence to obtain the text features of the sentence.

2. The method according to claim 1, characterized in that: Interactively fusing and encoding the audio data and text data of the sentence to obtain audio features and text features of the sentence; Determining the emotion type of the sentence based on the audio features and text features of the sentence includes: The audio data and text data of the sentence are input into a sentiment analysis model; the interactive fusion coding module of the sentiment analysis model performs interactive fusion coding on the audio data and text data of the sentence to obtain the audio features and text features of the sentence; the sentiment analysis module of the sentiment analysis model determines the sentiment type of the sentence based on the audio features and text features of the sentence.

3. The method according to claim 1, characterized in that The audio processing submodule includes an audio coding layer, a first audio interactive fusion coding layer, and a second audio interactive fusion coding layer, and the text processing submodule includes a text coding layer, a first text interactive fusion coding layer, and a second text interactive fusion coding layer; The interactive fusion coding module of the sentiment analysis model performs interactive fusion coding on the audio data and text data of the sentence to obtain the audio features and text features of the sentence, including: The audio coding layer encodes the audio data of the sentence to obtain initial audio features of the sentence; The text encoding layer encodes the text data of the sentence to obtain the initial text features of the sentence; The first audio interactive fusion coding layer interactively fusion codes the initial audio features of the sentence and the initial text features of the sentence to obtain a first audio interactive feature of the sentence; The first text interactive fusion coding layer interactively fuses the initial audio features of the sentence and the initial text features of the sentence to obtain a first text interactive feature of the sentence; The second audio interaction fusion coding layer interactively fusion codes the first audio interaction feature of the sentence and the first text interaction feature of the sentence to obtain the audio feature of the sentence; The second text interactive fusion coding layer interactively fusion codes the first audio interactive feature of the sentence and the first text interactive feature of the sentence to obtain the text feature of the sentence.

4. The method according to claim 2, characterized in that: The training method of the sentiment analysis model is as follows: Acquire a first true sample data set, wherein the first true sample data set includes a plurality of first true sample data, each of which includes audio data, text data, and an emotion type label of each sentence in a set of true conversation data; Generate a fake sample data set according to the first true sample data set, wherein the fake sample data set includes a plurality of fake sample data, each of the fake sample data includes audio data, text data and emotion type label of each sentence in a set of fake conversation data; The pre-trained model is trained based on the fake sample data set to obtain a sentiment analysis model.

5. The method according to claim 4, characterized in that The generating a fake sample data set according to the first real sample data set comprises: For each first true sample data in the first true sample data set, determining an emotional logic factor of the first true sample data; A false sample data set is generated based on each first true sample data and the sentiment logic factor of each first true sample data in the first true sample data set.

6. The method according to claim 5, characterized in that The determining of the sentiment logic factor of the first true sample data includes: The text data of each sentence in the first true sample data is input into the sentiment logic factor generation model, and the sentiment logic factor generation model processes the text data of each sentence in the first true sample data to obtain the sentiment logic factor of the first true sample data.

7. The method according to claim 6, characterized in that The sentiment logic factor generation model processes the text data of each sentence in the first true sample data to obtain the sentiment logic factor of the first true sample data, including: The semantic feature extraction module in the sentiment logic factor generation model extracts semantic features from each sentence in the first true sample data to obtain semantic features of each sentence in the first true sample data; The sentiment feature extraction module in the sentiment logic factor generation model extracts sentiment features from the semantic features of each sentence in the first true sample data to obtain the sentiment features of each sentence in the first true sample data; The cause feature extraction module in the sentiment logic factor generation model extracts cause features from the semantic features of each sentence in the first true sample data to obtain cause features of each sentence in the first true sample data; The emotion probability prediction module in the emotion logic factor generation model performs emotion probability prediction on the emotion feature of each sentence in the first true sample data to obtain the emotion probability of each sentence in the first true sample data; The cause probability prediction module in the emotional logic factor generation model performs cause probability prediction on the cause feature of each sentence in the first true sample data to obtain the cause probability of each sentence in the first true sample data; The emotional logic factor generation module in the emotional logic factor generation model fuses the emotional features, emotional probabilities, cause features, cause probabilities of each sentence in the first true sample data and the positional relationship encoding vectors of the sentences in the first true sample data to obtain the emotional logic factors of the first true sample data.

8. The method according to claim 7, characterized in that The emotional logic factor generation model is obtained by training based on preset sliding window parameters and a second true sample data set; the preset sliding window parameters are used to indicate the number of sentences included in a sentence group with a high possibility of emotional logic relationship; The second true sample data set includes multiple second true sample data, each of which includes text data, a reason label, an emotion type label and an emotion logic factor of each sentence in a group of true conversation data, and the reason label is used to indicate whether the sentence has an emotion logic relationship with other sentences.

9. The method according to claim 8, characterized in that The semantic feature extraction module in the sentiment logic factor generation model extracts semantic features from each sentence in the first true sample data to obtain the semantic features of each sentence in the first true sample data, including: For each sentence in the first true sample data, the overall semantic features of the sentence and other sentences in the same sliding window as the sentence are extracted as the semantic features of the sentence, and the sliding window is determined based on the preset sliding window parameters.

10. The method according to claim 5, characterized in that The generating of the fake sample data set based on each first true sample data in the first true sample data set and the sentiment logic factor of each first true sample data includes: For each true sample data in the first true sample data set, the first true sample data and the emotional logic factor of the first true sample data are input into N generators of the data generation model, and the N generators of the data generation model output the fake sample data set, where N is the same as the number of emotional type labels.

11. The method according to claim 10, characterized in that The data generation model is trained as follows: Acquire a third true sample data set, wherein the third true sample data set includes a plurality of third true sample data, each of the third true sample data includes audio data, text data, an emotion type label of each sentence in a set of true conversation data, and an emotion logic factor of the third true sample data; Constructing a discriminator, wherein the discriminator is an N+1 classifier; The data generation model is trained using the third true sample data set as a training sample, with the cumulative penalty items of the N generators being minimized and the judgment result of the discriminator on the output of the data model being close to the judgment result of the discriminator on the third true sample data as the training target.

12. A sentiment analysis device, characterized in that: The device comprises: An acquisition unit, used to acquire the voice conversation data to be analyzed; The sentiment analysis unit is used to determine the audio data and text data of each sentence in the voice dialogue data; interactively fuse the audio data and text data of the sentence using the interactive fusion coding module in the pre-built sentiment analysis model to obtain the audio features and text features of the sentence; and determine the sentiment type of the sentence based on the audio features and text features of the sentence; Wherein, the interactive fusion coding module includes an audio processing submodule and a text processing submodule; The audio processing submodule is used to interactively fuse and encode the audio data and text data of the sentence to obtain the audio features of the sentence; The text processing submodule is used to interactively fuse and encode the audio data and text data of the sentence to obtain the text features of the sentence.

13. A sentiment analysis device, characterized in that: including memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the sentiment analysis method according to any one of claims 1 to 11.

14. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the sentiment analysis method according to any one of claims 1 to 11 is implemented.

Images