Emotional state information acquisition method and emotional state information acquisition model training method

Abstract

The invention provides an emotional state information acquisition method, an emotional state information acquisition model training method, an emotional state information acquisition model training device, equipment and a medium. Relates to the technical field of artificial intelligence. The emotional state information acquisition method comprises the following steps: acquiring historical behavior data and dialogue audio data in a target scene; inputting the historical behavior data and the dialogue audio data into a pre-trained emotional state information acquisition model; extracting first feature data according to the historical behavior data through the pre-trained emotional state information acquisition model, extracting second feature data according to audio information in the dialogue audio data, and extracting third feature data according to semantic information in the dialogue audio data; performing feature fusion on the first feature data, the second feature data and the third feature data to obtain a fused feature; and acquiring emotional state information based on the fused features. The emotional state information obtaining effect can be improved to a certain extent.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular, to a method and device for acquiring emotional state information, a method and device for training an emotional state information acquisition model, an electronic device, and a computer-readable storage medium. Background Art

[0002] Usually, a person's speech content often contains his emotional state information. Therefore, when communicating with a user, the user's emotional state information can be obtained based on the user's language and other related information to enhance the user's communication experience.

[0003] In some existing technologies, the emotional state information of the user is manually determined based on the user's speech content; in other existing technologies, the emotional state information of the user is automatically obtained based on the user's speech content using a network model. However, there is still room for improvement in the effect of obtaining emotional state information.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the embodiments of the present disclosure is to provide a method and device for acquiring emotional state information, a method and device for training an emotional state information acquisition model, an electronic device, and a computer-readable storage medium, thereby improving the effect of acquiring emotional state information at least to a certain extent.

[0006] According to a first aspect of the present disclosure, a method for acquiring emotional state information is provided, comprising:

[0007] Obtain historical behavior data and conversation audio data in the target scenario;

[0008] Inputting the historical behavior data and the conversation audio data into a pre-trained emotional state information acquisition model;

[0009] Extracting first feature data according to the historical behavior data, extracting second feature data according to the audio information in the conversation audio data, and extracting third feature data according to the semantic information in the conversation audio data through the pre-trained emotional state information acquisition model;

[0010] Performing feature fusion on the first feature data, the second feature data, and the third feature data to obtain a fused feature;

[0011] Emotional state information is acquired based on the fused features.

[0012] According to a second aspect of the present disclosure, a method for training an emotional state information acquisition model is provided, comprising:

[0013] Obtain historical behavior data, historical conversation audio data, and annotation data during telephone communication;

[0014] Inputting the historical behavior data, historical conversation audio data and annotated data into the emotional state information acquisition model to be trained;

[0015] Extracting first feature data according to the historical behavior data, extracting second feature data according to the audio information in the historical conversation audio data, and extracting third feature data according to the semantic information in the historical conversation audio data through the emotional state information acquisition model to be trained;

[0016] Performing feature fusion on the first feature data, the second feature data, and the third feature data to obtain a fused feature;

[0017] The emotional state information acquisition model to be trained is trained using a loss function to obtain a pre-trained emotional state information acquisition model.

[0018] According to a third aspect of the present disclosure, there is provided an emotional state information acquisition device, comprising:

[0019] A data acquisition module is used to acquire historical behavior data and conversation audio data in the target scenario;

[0020] A data input module, used to input the historical behavior data and conversation audio data into a pre-trained emotional state information acquisition model;

[0021] A feature extraction module, configured to extract first feature data according to the historical behavior data, extract second feature data according to the audio information in the conversation audio data, and extract third feature data according to the semantic information in the conversation audio data through the pre-trained emotional state information acquisition model;

[0022] A feature fusion module, used for performing feature fusion on the first feature data, the second feature data and the third feature data to obtain a fusion feature;

[0023] The emotion acquisition module is used to acquire emotion state information based on the fusion feature.

[0024] According to a fourth aspect of the present disclosure, there is provided a training device for an emotional state information acquisition model, comprising:

[0025] The data acquisition module is used to obtain historical behavior data, historical conversation audio data and annotation data during telephone communication;

[0026] A data input module, used to input the historical behavior data, historical conversation audio data and annotated data into the emotional state information acquisition model to be trained;

[0027] A feature extraction module, configured to extract first feature data according to the historical behavior data, extract second feature data according to the audio information in the historical conversation audio data, and extract third feature data according to the semantic information in the historical conversation audio data through the emotional state information acquisition model to be trained;

[0028] A feature fusion module, used for performing feature fusion on the first feature data, the second feature data and the third feature data to obtain a fusion feature;

[0029] The model training module is used to train the emotional state information acquisition model to be trained using a loss function to obtain a pre-trained emotional state information acquisition model.

[0030] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned methods by executing the executable instructions.

[0031] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, any one of the methods described above is implemented.

[0032] Exemplary embodiments of the present disclosure may have some or all of the following beneficial effects:

[0033] In the method for acquiring emotional state information provided by the exemplary embodiment of the present disclosure, a pre-trained emotional state information acquisition model can be used to extract first feature data based on historical behavior data, extract second feature data based on audio information in conversation audio data, and extract third feature data based on semantic information in conversation audio data; then, the first feature data, the second feature data, and the third feature data are subjected to feature fusion to obtain fusion features; finally, emotional state information is acquired based on the fusion features. On the one hand, feature data is extracted not only based on historical behavior data, but also by using conversation audio data to extract feature data, which can improve the integrity of feature data to a certain extent; on the other hand, feature fusion is performed on the first feature data, the second feature data, and the third feature data to obtain fusion features, so that the fusion features not only contain historical behavior information, but also contain audio information and semantic information, thereby improving the representation of the fusion features to a certain extent; on the other hand, acquiring emotional state information based on fusion features can make the obtained emotional state information more accurate; thereby, the acquisition effect of emotional state information is improved to a certain extent.

[0034] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0036] Figure 1 The following schematically shows a schematic diagram of a system architecture of an application scenario provided by an embodiment of the present disclosure;

[0037] Figure 2 The flowchart of the method for acquiring emotional state information provided in accordance with an embodiment of the present disclosure is schematically shown;

[0038] Figure 3 The flowchart of extracting first feature data provided according to an embodiment of the present disclosure is schematically shown;

[0039] Figure 4 The flowchart of extracting the second characteristic data provided according to the embodiment of the present disclosure is schematically shown;

[0040] Figure 5 The flowchart of extracting the third characteristic data provided according to the embodiment of the present disclosure is schematically shown;

[0041] Figure 6 A schematic diagram of recording a conversation text according to an embodiment of the present disclosure is schematically shown;

[0042] Figure 7 The following is a schematic diagram schematically showing a process of acquiring emotional state information according to an embodiment of the present disclosure;

[0043] Figure 8 The flowchart schematically shows a method for training an emotional state information acquisition model according to an embodiment of the present disclosure;

[0044] Fig. 9 The block diagram of the device for acquiring emotional state information provided according to the embodiment of the present disclosure is schematically shown;

[0045] Fig.10 A block diagram schematically shows a training device for an emotional state information acquisition model provided in accordance with an embodiment of the present disclosure;

[0046] Fig.11The structure diagram of the computer system of the electronic device provided according to the embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0047] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0048] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0049] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0050] Figure 1 A schematic diagram of a system architecture according to an application scenario provided by an embodiment of the present disclosure is schematically shown.

[0051] like Figure 1As shown, the system architecture 100 may include a terminal device, a network, and a server. The network may be used to provide a transmission medium for a communication link between the terminal device and the server, and may also be used to transmit conversation audio data to the terminal device during telephone communication between a customer service staff and a target user. The network may include various connection types, such as wired or wireless communication links or optical fiber cables. The terminal device may be a variety of electronic devices, including but not limited to desktop computers, portable computers, smart phones, and tablet computers. It should be understood that Figure 1 The number of terminal devices, networks, servers, users and customer service personnel in the embodiment is only for illustration, and any number of terminal devices, networks, servers, users and customer service personnel may be provided according to the implementation requirements. For example, the server may be a server cluster composed of multiple servers.

[0052] For example, in an exemplary embodiment, a pre-trained emotional state information acquisition model may be stored in a server, and during telephone communication between a customer service staff and a user, the conversation audio data is transmitted to a terminal device via a network; then, the terminal device transmits the user's historical behavior data and the conversation audio data during telephone communication with the customer service staff to the server via a network; after receiving the historical behavior data and the conversation audio data, the server inputs the historical behavior data and the conversation audio data into the pre-trained emotional state information acquisition model, and then, the pre-trained emotional state information acquisition model extracts a first feature data based on the historical behavior data, extracts a second feature data based on the audio information in the conversation audio data, extracts a third feature data based on the semantic information in the conversation audio data, and performs feature fusion on the first feature data, the second feature data, and the third feature data to obtain a fused feature, and finally, obtains emotional state information based on the fused feature; the server sends the user's emotional state information to the terminal device via a network, and the customer service staff adjusts the communication method with the user based on the emotional state information presented on the terminal device to improve the user experience.

[0053] In an exemplary embodiment, the pre-trained emotional state information acquisition model can also be stored in the terminal device, and during the telephone communication between the customer service personnel and the user, the conversation audio data is transmitted to the terminal device through the network; then, the terminal device inputs the user's historical behavior data and the conversation audio data during the telephone communication with the customer service personnel into the pre-trained emotional state information acquisition model, and then, the pre-trained emotional state information acquisition model extracts the first feature data according to the historical behavior data, extracts the second feature data according to the audio information in the conversation audio data, extracts the third feature data according to the semantic information in the conversation audio data, and performs feature fusion on the first feature data, the second feature data and the third feature data to obtain the fusion feature, and finally, obtains the emotional state information based on the fusion feature; the customer service personnel adjusts the communication method with the user according to the emotional state information presented on the terminal device to improve the user experience. However, it is easy for those skilled in the art to understand that the above application scenario is only used for example, and this exemplary embodiment is not limited to this.

[0054] The technical solutions of the exemplary embodiments of the present disclosure are described in detail below:

[0055] This exemplary embodiment provides a method for acquiring emotional state information. The method for acquiring emotional state information can be applied to the above-mentioned server, or to the above-mentioned terminal device; it can also be applied to the above-mentioned terminal device and the above-mentioned server at the same time; this is not particularly limited in this exemplary embodiment. Figure 2 As shown, the emotional state information acquisition method may include the following steps:

[0056] Step S210, obtaining historical behavior data and conversation audio data in the target scenario;

[0057] Step S220, inputting the historical behavior data and the conversation audio data into a pre-trained emotional state information acquisition model;

[0058] Step S230, extracting first feature data according to the historical behavior data, extracting second feature data according to the audio information in the conversation audio data, and extracting third feature data according to the semantic information in the conversation audio data through the pre-trained emotional state information acquisition model;

[0059] Step S240, performing feature fusion on the first feature data, the second feature data and the third feature data to obtain a fusion feature;

[0060] Step S250, acquiring emotional state information based on the fusion feature.

[0061] Understandably, in some scenarios, customer service staff usually reach a large number of users for performance indicators. However, for users, unfamiliar and large numbers of calls can easily cause negative emotions. In addition, as people's privacy awareness becomes stronger, if users express strong dissatisfaction or even anger when communicating with customer service staff over the phone, it is easy to cause complaints from users, and a large number of complaints can easily cause unnecessary public opinion risks for enterprises and institutions, increasing their operational burden.

[0062] Therefore, in order to reduce customer complaints, ensure user experience to a certain extent, and achieve certain promotion, reminder and other purposes, the user's emotional state information can be obtained through the emotional state information acquisition model, so that the way or words of communication with the user can be adjusted according to the user's emotional state information, so as to minimize user dissatisfaction.

[0063] In the method for acquiring emotional state information provided by this exemplary embodiment, on the one hand, feature data is extracted not only based on historical behavior data, but also by using historical conversation audio data to extract feature data, which can improve the integrity of feature data to a certain extent; on the other hand, feature fusion is performed on the first feature data, the second feature data and the third feature data to obtain fused features, so that the fused features not only include historical behavior information, but also audio information and semantic information, thereby improving the representativeness of the fused features to a certain extent; on the other hand, acquiring emotional state information based on the fused features can make the obtained emotional state information more accurate; thereby, improving the effect of acquiring emotional state information to a certain extent.

[0064] In another exemplary embodiment, the above steps are described in detail below.

[0065] In step S210, historical behavior data and conversation audio data in the target scenario are obtained.

[0066] Among them, the target scenario in the present disclosure may be a scenario where a customer service staff communicates with a target user by phone to achieve certain promotion, reminder, etc. Specifically, the target scenario may be when an enterprise conducts telephone promotion to a target user, when a financial institution makes a telephone reminder to a target user, or when a customer service staff answers an after-sales call.

[0067] The historical behavior data may be the target user's previous communication records and economic situation data, which can represent the user profile of the user to a certain extent. After collecting the historical behavior data, the present disclosure will de-identify the data through technical means.

[0068] In an exemplary embodiment, the historical behavior data may include historical call data, historical label data, historical online shopping data, historical loan data, etc. Among them, the historical call data may be the number of historical calls between the target user and the customer service staff, the number of historical calls refused to answer, the number of historical calls agreed to answer, etc. The historical label data may be the label data manually made for the target user, for example, the collection customer service makes a mark based on the user's reply and attitude when communicating with the target user by phone, and the label data may be "promise to repay", "to continue communication", etc.; the historical label data may also be the label data made for the target user based on the intention recognition model, for example, when the customer service staff and the target user communicate by phone, the intention recognition model makes label data for the target user's intention based on the call audio data, and the label data may be "first attitude", "second attitude", "no willingness to repay", etc. The historical online shopping data may be the historical browsing data and shopping data of the target user on the online shopping platform, which may reflect the economic situation of the target user to a certain extent. The historical loan data may be the historical borrowing data and lending data of the target user, which may reflect the debt situation and repayment ability of the target user to a certain extent.

[0069] In the present disclosure, the conversation audio data may be the audio data generated during the telephone communication between the customer service personnel and the target user, which includes audio information and semantic information, etc. After collecting the audio data, the present disclosure will de-identify the data through technical means.

[0070] In step S220, the historical behavior data and the conversation audio data are input into a pre-trained emotional state information acquisition model.

[0071] Among them, the pre-trained emotional state information acquisition model can extract the emotional state information of the target user based on historical behavior data and conversation audio data.

[0072] In step S230, the pre-trained emotional state information acquisition model extracts first feature data based on the historical behavior data, extracts second feature data based on the audio information in the conversation audio data, and extracts third feature data based on the semantic information in the conversation audio data.

[0073] Among them, the first feature data is structured data, the second feature data and the third feature data are unstructured data; the audio information represents digitized sound data, etc.; the semantic information represents information provided by language and text, etc.

[0074] In an exemplary embodiment, referring to Figure 3 As shown, the first feature data can be extracted through steps S310 to S330. Wherein:

[0075] In step S310, the historical behavior data is aggregated according to multiple categories of time windows to obtain aggregated data.

[0076] The multi-category time window may be a time window of the past week, past month, past three months, past six months, past year, etc. The summary data obtained by summarizing the historical behavior data according to the multi-category time window may reflect the historical behavior information of the target user in different time periods.

[0077] In step S320, the summary data is aggregated according to the data distribution characteristics to obtain statistical characteristics.

[0078] Among them, the data distribution characteristics can reflect the degree of dispersion and concentration trend of the data to a certain extent, while the statistical characteristics can reflect the quantitative characteristics and quantitative relationships of the summary data to a certain extent. Exemplarily, the aggregation processing of the summary data according to the data distribution characteristics can be to aggregate the summary data according to the data distribution characteristics such as the mean, minimum value, maximum value, standard deviation, count, etc. Among them, the count can represent the number of specific behaviors of the target user. For example, if the target user is counted to have borrowed a total of 3 times in the time window of the past 3 months, the count result is 3. Of course, the summary data can also be processed by data distribution characteristics such as mode, median, range, mean difference, etc. This exemplary embodiment does not specifically limit the type of data distribution characteristics.

[0079] In step S330, the statistical features are screened using a feature screening model to obtain the first feature data.

[0080] Among them, the feature screening model refers to a model that can be used to screen statistical features, which can eliminate useless and redundant features in statistical features to a certain extent. For example, statistical features can be screened through the LightGBM (LightGradient Boosting Machine) model; statistical features can also be screened through the Logistics model; of course, statistical features can also be screened through the XGBoost (ExtremeGradient Boosting) model.

[0081] Specifically, the feature screening of statistical features through the lightweight gradient boosting tree model can be to divide the gain value of the statistical feature by the total gain to obtain the gain ratio of the feature, sort them in descending order according to the gain ratio, and screen the statistical features with a gain ratio greater than 90%, where the gain represents the relative contribution of the feature to the model. The higher the gain ratio, the more important the feature is for generating predictions. Of course, the feature screening of statistical features can also be performed through feature importance indicators such as "weight", that is, the number of times the feature is selected as a split feature, and "cover", that is, the coverage rate of the feature.

[0082] After extracting features from the statistical data using the feature screening model, the first feature data is obtained for subsequent feature fusion. Because the historical behavior data of the target user is structured data, the first feature data obtained is also structured feature data. The first feature data can be used struct If X struct The expression formula is as follows:

[0083] X struct =x1,x2,x3,…x n

[0084] Among them, x1 to x2 represent the structural features after extraction and screening.

[0085] In an exemplary embodiment, referring to Figure 4 As shown, the second feature data can be extracted through steps S410 to S430.

[0086] In step S410, the audio information in the conversation audio data is sampled and framed to obtain a signal vector in the time domain.

[0087] Framing is to divide several audio sampling points into a frame. In this frame, the characteristics of the speech signal can be considered stable. The frame length is usually 10-30 milliseconds. The signal vector in the time domain can represent the audio information of the corresponding frame.

[0088] In step S420, the time series features of the audio information are extracted based on the signal vector in the time domain.

[0089] The length of the time series feature is the number of frames. In an exemplary embodiment, the time series feature of the audio information can be extracted by the following method: extracting multiple audio features based on the signal vector in the time domain, wherein the multiple audio features may include Mel cepstral coefficient features, root mean square energy features, short-time zero-crossing rate features, spectrum centroid features, etc.; using the multiple audio features as time series features with a length of the number of frames.

[0090] Specifically, Mel-Frequency Cepstral Coefficients (MFCCs) are the coefficients that make up the Mel-Frequency Cepstral, which concisely describe the overall shape of the spectrum envelope and can simulate the characteristics of human voice audio; Root Mean Square Energy represents the comprehensive information of all sample points in a single-frame audio signal; Zero Crossing Rate represents the number of times the waveform of a single-frame audio signal crosses the horizontal axis, i.e., the zero level; Spectral Centroid is one of the important physical parameters that describe audio information, and is the center of gravity of the frequency component, which represents the frequency weighted by energy within a certain frequency range.

[0091] In the present disclosure, the Mel-frequency cepstral coefficient feature can be extracted based on the signal vector in the time domain by the following method. First, before the audio information in the historical conversation audio data is framed, the audio data can also be pre-processed to remove the completely soundless segments in the audio data caused by non-human factors, and the pre-processed audio data is pre-emphasized to enhance the high-frequency part of the audio data and highlight the high-frequency resonance peaks.

[0092] Then, after the audio data is framed, the framed speech data is windowed to obtain a signal vector in the time domain. The windowing is because the subsequent method requires a Fast Fourier Transform (FFT), and the Fast Fourier Transform requires the audio data to be from negative infinity to positive infinity or periodic, while the framed audio data is non-periodic, so it needs to be windowed to make it periodic.

[0093] Subsequently, the signal vector in the time domain obtained after the windowing process is processed by fast Fourier transform to obtain the spectrum of each frame of audio data. The spectrum of each frame is stacked up in time to obtain the spectrogram; then, after applying the Mel filter to the spectrogram, the logarithm is taken to obtain the Mel spectrogram, and the frequency of the spectrogram is converted into the "Mel" scale, that is, it is linearly distributed below 1000Hz and increases logarithmically above 1000Hz.

[0094] Finally, the Mel-spectrogram is subjected to cepstrum analysis. Due to the characteristics of human pronunciation, the most useful information is in the filter, that is, the vocal cavity. Therefore, the sound source and the filter need to be separated. By performing cepstrum analysis on the Mel-spectrogram, the speech signal can be decomposed, and the 2nd to 13th coefficients representing the filter information are taken. The 12 coefficients obtained are used to represent the Mel-cepstrum coefficient characteristics.

[0095] In other exemplary embodiments of the present disclosure, other audio features such as pitch features, intonation features, speaking speed features and pause features of audio data can also be extracted based on the time-series signal vector, and then the multiple audio features can be used as time-series features with a length of the number of frames, which also fall within the protection scope of the present disclosure.

[0096] In step S430, feature extraction is performed on the time series feature through a long short-term memory network to obtain the second feature data.

[0097] Among them, the Long Short-Term Memory (LSTM) network is a time recurrent neural network that can solve the long-term dependency problem of general recurrent neural networks to a certain extent.

[0098] Specifically, the time series features, i.e., multiple audio features, can be spliced ​​using existing feature splicing technology, and then input into the long short-term memory network, and the vector output from the last layer is extracted as the second feature data. The second feature data can be obtained by X audio If X audio The acquisition process can be expressed by the following formula:

[0099] X audio =LSTM hidden[-1] (F mel (x), F zero (x), ..., F centroid (x)

[0100] Among them, F mel 、F zero 、F centroid etc. represent the functions of the audio feature calculation process such as Mel cepstral coefficient feature, short-time zero-crossing rate feature, spectrum centroid feature, etc. extracted from the signal vector. LSTM hidden[-1] It means concatenating the above audio features and inputting them into the long short-term memory network to take the output of the last layer of vectors.

[0101] In an exemplary embodiment, referring to Figure 5 As shown, the third characteristic data can be extracted through steps S510 to S540.

[0102] In step S510, speech recognition is performed on the conversation audio data to obtain the semantic information.

[0103] Specifically, in this exemplary embodiment, speech recognition can be performed on each of the audio data to obtain semantic information through one or more models of a deep neural network model, a hidden Markov model, and a Gaussian mixture model. For example, the time series information of the audio data can be modeled through a hidden Markov model. After a state of the hidden Markov model is given, the probability distribution of the speech feature vector belonging to the state is modeled based on the Gaussian mixture model by a maximum expectation value algorithm or other methods; after the modeling is successful, speech recognition can be performed on the speech data to obtain the corresponding semantic information. Of course, in other exemplary embodiments of the present invention, speech recognition can also be performed by combining context information (ContextDependent) or by other means, and this is not specifically limited in this exemplary embodiment. In addition, the semantic information can represent the text content of the historical conversation audio data.

[0104] In step S520, the semantic information is segmented to obtain a plurality of segmented words.

[0105] Specifically, taking Conditional Random Field (CRF) word segmentation as an example, the subtitle text can be segmented based on the conditional random field theory by comprehensively considering the frequency of the word segmentation in the subtitle text and the contextual relationship of the word segmentation. Of course, it is easy for those skilled in the art to understand that the above word segmentation processing can also be implemented in any possible word segmentation method, such as JIEBA word segmentation (i.e., Jieba word segmentation), NLPIR word segmentation, LTP (Language Technology Platform) word segmentation, or THULAC (THU Lexical Analyzer for Chinese) word segmentation, etc.

[0106] In step S530, feature extraction is performed on the multiple word segments using a BERT model to obtain a conversation text word vector, wherein the BERT model is fine-tuned using a one-dimensional convolutional neural network.

[0107] Among them, the BERT (Bidirectional Encoder Representation from Transformers) model can be used to extract word embeddings from conversation text. It pre-trains word embeddings or sentence embeddings in an unsupervised manner based on a huge corpus, and by additionally introducing position vectors, sentence vectors, and attention mechanisms, the model can better capture semantic information in different contexts.

[0108] A word vector is a dense mathematical representation of a discrete character or word. The more related the characters or words are, the higher the cosine similarity of the word vector will be.

[0109] Model fine-tuning can save computing resources and time, and improve computing rate and accuracy to a certain extent. Specifically, model fine-tuning can be to take the embedding vector of the last layer of the BERT model, connect it to a one-dimensional convolutional neural network, and pass it through a fully connected network.

[0110] In addition, since BERT has a limit on the length of the input text, for a longer text, its important information usually appears at the beginning, so the first 512 words can be intercepted, and modal particles can be retained to enhance the expressiveness of special semantic environments. After each sentence is processed by BERT, a 768-dimensional embedding vector representation of the entire sentence text can be obtained, and finally a sequence data of the number of sentence texts × 768 can be obtained.

[0111] In step S540, the probability value of the dialogue text word vector being able to obtain its corresponding label is determined through a classification function, and the probability value is used as the third feature data.

[0112] Specifically, the classification function can be a softmax function; the label can be whether the user will file a complaint within a specific time period in the future. The third feature data obtained is one-dimensional and can be expressed as Score. te+t Indicates that Score text The acquisition process can be expressed by the following formula:

[0113] Score text =softmax(Conv1D(BERT embedding (text)))

[0114] Among them, text represents the semantic information after speech recognition, BERT embedding represents the feature vector output by the fine-tuned BERT, Conv1D represents a one-dimensional convolutional network, ScoTe text It indicates that the semantic information is passed through the softmax function to obtain the probability value of the corresponding label. This probability value is used as the final feature of the semantic information of the conversation audio data for fusion.

[0115] In an exemplary embodiment, considering that the triggering of user complaints requires not only identifying the emotional changes of customers, but also considering how the customer service staff responds to the emotional changes of customers, after performing speech recognition on the audio data of the conversation between the customer service staff and the target user, the conversation texts of the customer service staff and the target user can be put together for modeling. Figure 6As shown, a schematic diagram of recording a conversation text according to an embodiment of the present disclosure is schematically shown, and an identity identifier "AGENT" or "USER" is added before the conversation text of the customer service staff and the target user as a prefix to identify which party sent the current text. Since the pre-training stage of the BERT model also has conversation text as input, under this input format design, the BERT model will capture the conversation features of this text.

[0116] In step S240, feature fusion is performed on the first feature data, the second feature data, and the third feature data to obtain a fused feature.

[0117] Feature fusion refers to the use of complementarity between features and the fusion of advantages between features when given features of different attributes, thereby improving the performance of the model. Because the first feature data can represent the user's historical behavior information, the second feature data can represent the audio information in the conversation audio data, and the third feature data can represent the semantic information in the conversation audio data, the fusion feature is a multimodal feature that combines three modalities, which can better represent the emotional state information of the target user.

[0118] In an exemplary embodiment, the first feature data, the second feature data and the third feature data are subjected to feature fusion to obtain the fused feature. The first feature data, the second feature data and the third feature data can be input into a lightweight gradient boosting tree model to obtain the fused feature.

[0119] Among them, the lightweight gradient boosting tree model is a framework for implementing the GBDT (Gradient Boosting DecisionTree) algorithm. It supports efficient parallel training and has the advantages of faster training speed, lower memory consumption, better accuracy, support for distribution, and the ability to quickly process massive data. It can achieve good results in common classification and regression problems.

[0120] Specifically, the first feature data and the third feature data are multi-dimensional data, while the second feature data is one-dimensional data. These three feature data can be concatenated and then input into the lightweight gradient boosting tree model for feature extraction. The feature fusion process can be expressed by the following formula:

[0121] Model = LightGBM(concat[X struct , Score text , X audio ])

[0122] Among them, the structured feature after screening, that is, the first feature data X struct, the second feature data X represented by the last hidden layer vector of the long short-term memory network audio , the third feature data Score output by the one-dimensional convolutional neural network and the fully connected network text These three features are concatenated and merged, and then a lightweight gradient boosting tree model is used for feature fusion.

[0123] In addition, compared with only using the first feature data, namely historical behavior data, to obtain the user's emotional state information, the result of predicting the user's emotional state information by combining historical behavior data, audio information and semantic information of the conversation audio data is more accurate, and the second feature data and third feature data corresponding to the audio information and semantic information of the conversation audio data, respectively, are ranked higher in feature importance.

[0124] In step S250, emotional state information is acquired based on the fusion feature.

[0125] Among them, the emotional state information can be whether the target user will make a complaint or not in a certain period of time in the future. When the output of the user emotion recognition model is whether the target user will make a complaint, the user emotion recognition model can be regarded as a binary classification model, and its target variable is determined by whether the target user will make a complaint in a specific period of time in the future. Therefore, when training the user emotion recognition model, the annotation in the training set can be whether the user has made a complaint, and the emotional state information obtained based on the fusion feature is whether the target user will make a complaint in a specific period of time in the future.

[0126] In addition, after obtaining the user's emotional state information, customer service personnel can adjust the way they communicate with the user based on the emotional state information, for example, reducing the frequency of outbound calls, stopping outbound calls and instead sending text messages, changing communication language, etc., thereby improving the user experience to a certain extent.

[0127] In an exemplary embodiment, referring to Figure 7 As shown, the emotional state information of the target user can be obtained through the illustrated method.

[0128] First, the historical behavior data, conversation audio data, and text data converted by ASR (Automatic Speech Recognition) of the conversation audio data are input into the emotional state information acquisition model. Then, the emotional state information acquisition model groups the historical behavior data according to the time window, and extracts the first feature data after aggregation and feature screening; the conversation audio data is sampled and framed to obtain the signal vector in the time domain, and the time series features are extracted based on the signal vector in the time domain, and the time series features are extracted through the long short-term memory network to obtain the second feature data; the text data is segmented to obtain multiple segmented words, and the multiple segmented words are feature extracted through the BERT model to obtain the conversation text word vector, and the BERT model is fine-tuned, and the probability value of the conversation text word vector that can obtain its corresponding label is determined by the classification function, and the probability value is used as the third feature data. Finally, the first feature data, the second feature data, and the third feature data are feature fused to obtain the fused feature, and the emotional state information is obtained based on the fused feature.

[0129] Based on the emotional state information acquisition method in the above exemplary embodiment, the emotional state information acquisition model can be used to automatically obtain the emotional state information of the target user based on the historical behavior data of the target user in the target scenario and the audio data of the conversation between the target user and the customer service staff, so that the customer service staff, enterprises and institutions can understand the emotional state information of the target user in real time to adjust the communication method and ensure the user experience as much as possible. In addition, feature extraction based on historical behavior data can mine the long-term behavior habits and emotional state of the target user, and feature extraction based on conversation audio data can capture the semantic information in the conversation in real time, identify negative words, and can capture the emotional changes and emotional expressions of the target user during the conversation in real time, thereby improving the acquisition effect of emotional state information. The method for acquiring emotional state information provided by the above exemplary embodiment, on the one hand, not only extracts feature data based on historical behavior data, but also uses historical conversation audio data to extract feature data, which can improve the integrity of feature data to a certain extent; on the other hand, the first feature data, the second feature data and the third feature data are subjected to feature fusion to obtain fused features, so that the fused features not only include historical behavior information, but also include audio information and semantic information, thereby improving the representativeness of the fused features to a certain extent; on the other hand, acquiring emotional state information based on the fused features can make the obtained emotional state information more accurate; thereby, improving the effect of acquiring emotional state information to a certain extent.

[0130] In an exemplary embodiment of the present disclosure, a training method for an emotional state information acquisition model is also provided. Figure 8As shown, the training method of the emotional state information acquisition model may include the following steps S810 to S850:

[0131] Step S810, obtaining historical behavior data, historical conversation audio data and annotation data during telephone communication;

[0132] Step S820, inputting the historical behavior data, historical conversation audio data and annotated data into the emotional state information acquisition model to be trained;

[0133] Step S830, extracting first feature data according to the historical behavior data, extracting second feature data according to the audio information in the historical conversation audio data, and extracting third feature data according to the semantic information in the historical conversation audio data through the emotional state information acquisition model to be trained;

[0134] Step S840, performing feature fusion on the first feature data, the second feature data and the third feature data to obtain a fusion feature;

[0135] Step S850, using the loss function to train the emotional state information acquisition model to be trained to obtain a pre-trained emotional state information acquisition model.

[0136] Specifically, the labeled data may be whether the user has made a complaint within a certain period of time. For the sample data, it may be that the user has made a complaint within a certain period of time as a positive sample, and the user has not made a complaint within a certain period of time as a negative sample. The certain period of time may be the time from the last call record between the user and the customer service staff to the next week or half a month. After collecting the labeled data, the present disclosure will de-identify the data through technical means.

[0137] The model used to train the emotional state information acquisition model can be composed of historical behavior data, historical conversation audio data and annotated data. When the annotated data is whether the user has made a complaint within a certain period of time, the emotional state information acquisition model can be regarded as a binary classification model. The model can extract the emotional state information of the target user based on the target user's historical behavior data and conversation audio data and classify it, that is, determine whether the target user will make a complaint within a certain period of time in the future.

[0138] In addition, before training the model, the sample data can be divided into a training set, a development set, and a test set. The training set and the development set maintain the same time range, are randomly split q at a ratio of 8:2, and maintain independent and identical distribution. The time range of the test set is later than that of the training set and the development set, and is not independent and identically distributed with the training set and the development set. This is to pay more attention to its cross-time generalization ability when evaluating the model effect.

[0139] In addition, the loss function is an operation function used to measure the difference between the predicted value and the true value of the model. It is a non-negative real-valued function. The smaller the loss function, the better the robustness of the model. In the present disclosure, when training the emotional state information acquisition model to be trained, the loss function can be Cross-Entropy LOSS (cross entropy loss function), and the loss function can also be BCE Loss (Binary Cross Entropy Loss). Of course, the loss function can also be NLL LOSS (Negative Log Likelihood Loss) and other types of classification problem loss functions. This exemplary embodiment is not specifically limited here.

[0140] In an exemplary embodiment, the above-mentioned emotional state information acquisition model training method can also optimize and evaluate the state information acquisition model to be trained according to the AUC index and the KS index.

[0141] Among them, the AUC (Area Under the Curve) indicator refers to the area under the ROC curve, and the ROC curve is obtained based on the true category and predicted probability of the sample. Its x-axis is the True Positive Rate and the y-axis is the False Positive Rate. The essence of AUC is to reflect the ability to select a positive sample and a negative sample from the data set and put the positive sample in front of the negative sample. When calculating the AUC indicator, the following formula can be used. Assume that M is the number of positive samples and N is the number of negative samples. Therefore, there are a total of M×N combinations. It is necessary to find a combination in which the output result of the positive sample is greater than that of the negative sample.

[0142]

[0143] in,

[0144]

[0145] In addition, the KS (Kolmogorov Smirnov) indicator is an indicator used to measure the risk differentiation ability of the model. It calculates the difference between the cumulative distribution of good and bad samples. It usually sorts the output results in ascending order and performs equal frequency binning. The number of cumulative good samples and cumulative bad samples is calculated for each bin. The greater the cumulative difference between good and bad samples, the greater the KS indicator, and the stronger the model's differentiation ability. Among them, the KS indicator can be calculated by the following formula:

[0146]

[0147] It should be noted that although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0148] Furthermore, in an exemplary embodiment, a device for acquiring emotional state information is also provided. Fig. 9 As shown, the emotional state information acquisition device 900 may include a data acquisition module 910, a data input module 920, a feature extraction module 930, a feature fusion module 940 and an emotional acquisition module 950. Among them:

[0149] The data acquisition module 910 may be used to acquire historical behavior data and conversation audio data in a target scenario;

[0150] The data input module 920 may be used to input the historical behavior data and the conversation audio data into the pre-trained emotional state information acquisition model;

[0151] The feature extraction module 930 may be used to extract first feature data according to the historical behavior data, extract second feature data according to the audio information in the historical conversation audio data, and extract third feature data according to the semantic information in the historical conversation audio data through the pre-trained emotional state information acquisition model;

[0152] The feature fusion module 940 may be used to perform feature fusion on the first feature data, the second feature data and the third feature data to obtain a fusion feature;

[0153] The emotion acquisition module 950 may be configured to acquire emotion state information based on the fusion feature.

[0154] As another aspect, in an exemplary embodiment, a training device for an emotional state information acquisition model is also provided. Fig.10 As shown, the training device 1000 for acquiring the emotional state information model may include a data acquisition module 1010, a data input module 1020, a feature extraction module 1030, a feature fusion module 1040 and a model training module 1050. Among them:

[0155] The data acquisition module 1010 can be used to acquire historical behavior data, historical conversation audio data and annotation data during telephone communication;

[0156] The data input module 1020 may be used to input the historical behavior data, historical conversation audio data and annotated data into the emotional state information acquisition model to be trained;

[0157] The feature extraction module 1030 may be used to extract first feature data according to the historical behavior data, extract second feature data according to the audio information in the historical conversation audio data, and extract third feature data according to the semantic information in the historical conversation audio data through the emotional state information acquisition model to be trained;

[0158] The feature fusion module 1040 may be used to perform feature fusion on the first feature data, the second feature data and the third feature data to obtain a fused feature;

[0159] The model training module 1050 can be used to train the emotional state information acquisition model to be trained using a loss function to obtain a pre-trained emotional state information acquisition model.

[0160] As another aspect, the present disclosure further provides an electronic device, Fig.11 The structure diagram of the computer system of the electronic device provided according to the embodiment of the present disclosure is schematically shown.

[0161] It should be noted that Fig.11 The computer system 1000 of the electronic device shown is only an example and should not bring any limitation to the functions and application scope of the embodiments of the present disclosure.

[0162] like Fig.11 As shown, the computer system 1100 includes a central processing unit (CPU) 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage part 1108 into a random access memory (RAM) 1103. Various programs and data required for system operation are also stored in the RAM 1103. The CPU 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0163] The following components are connected to the I / O interface 1105: an input section 1106 including a keyboard, a mouse, etc.; an output section 1107 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN card, a modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1111 is also connected to the I / O interface 1105 as needed. A removable medium 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1110 as needed, so that a computer program read therefrom is installed into the storage section 1108 as needed.

[0164] In particular, according to an embodiment of the present disclosure, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 1101, and / or installed from a removable medium 1111. When the computer program is executed by the central processing unit (CPU) 1101, various functions defined in the method and apparatus of the present disclosure are executed. In some embodiments, the computer system 1100 may also include a data processor for processing computing operations on data obtained from emotional state information.

[0165] As another aspect, the present disclosure further provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiments. For example, the electronic device may implement the following Figure 2 to Figure 8 The various steps shown, etc.

[0166] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0167] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

Claims

1. A method for acquiring emotional state information, It is characterized in that include: Obtain historical behavior data and conversation audio data in the target scenario; Inputting the historical behavior data and the conversation audio data into a pre-trained emotional state information acquisition model; Extracting first feature data according to the historical behavior data, extracting second feature data according to the audio information in the conversation audio data, and extracting third feature data according to the semantic information in the conversation audio data through the pre-trained emotional state information acquisition model; Performing feature fusion on the first feature data, the second feature data, and the third feature data to obtain a fused feature; Emotional state information is acquired based on the fused features.

2. The method according to claim 1, It is characterized in that The extracting first feature data according to the historical behavior data includes: Aggregating the historical behavior data according to multiple categories of time windows to obtain aggregated data; Aggregating the summary data according to data distribution characteristics to obtain statistical characteristics; The statistical features are screened using a feature screening model to obtain the first feature data.

3. The method according to claim 1, It is characterized in that The extracting second feature data according to the audio information in the conversation audio data comprises: Sampling and framing the audio information in the conversation audio data to obtain a signal vector in the time domain; Extracting the time series features of the audio information based on the signal vector in the time domain; The time series features are extracted through a long short-term memory network to obtain the second feature data.

4. The method according to claim 3, It is characterized in that The extracting the time series feature of the audio data based on the signal vector in the time domain comprises: Extracting a plurality of audio features based on the signal vector in the time domain, wherein the plurality of audio features include one or more of a Mel-frequency cepstral coefficient feature, a root mean square energy feature, a short-time zero-crossing rate feature, and a spectrum centroid feature; The multiple audio features are taken as time series features with a length of the number of frames.

5. The method according to claim 1, It is characterized in that The step of extracting the third feature data according to the semantic information in the conversation audio data comprises: Performing speech recognition on the audio data to obtain the semantic information; Performing word segmentation processing on the semantic information to obtain multiple word segments; Performing feature extraction on the multiple word segmentations through a BERT model to obtain a conversation text word vector, wherein the BERT model is fine-tuned through a one-dimensional convolutional neural network; The probability value of the dialogue text word vector being able to obtain its corresponding label is determined through a classification function, and the probability value is used as the third feature data.

6. The method according to claim 1, It is characterized in that The step of fusing the first feature data, the second feature data, and the third feature data to obtain a fused feature includes: The first feature data, the second feature data and the third feature data are input into a lightweight gradient boosting tree model to obtain a fusion feature.

7. The method according to claim 1, It is characterized in that The historical behavior data includes one or more of historical call data, historical tag data, historical online shopping data, and historical loan data.

8. A training method for an emotional state information acquisition model, It is characterized in that include: Obtain historical behavior data, historical conversation audio data, and annotation data during telephone communication; Inputting the historical behavior data, historical conversation audio data and annotated data into the emotional state information acquisition model to be trained; Extracting first feature data according to the historical behavior data, extracting second feature data according to the audio information in the historical conversation audio data, and extracting third feature data according to the semantic information in the historical conversation audio data through the emotional state information acquisition model to be trained; Performing feature fusion on the first feature data, the second feature data, and the third feature data to obtain a fused feature; The emotional state information acquisition model to be trained is trained using a loss function to obtain a pre-trained emotional state information acquisition model.

9. The method according to claim 8, It is characterized in that The method further comprises: The state information acquisition model to be trained is optimized and evaluated according to the AUC index and the KS index.

10. An emotional state information acquisition device, It is characterized in that include: A data acquisition module is used to acquire historical behavior data and conversation audio data in the target scenario; A data input module, used to input the historical behavior data and conversation audio data into a pre-trained emotional state information acquisition model; A feature extraction module, configured to extract first feature data according to the historical behavior data, extract second feature data according to the audio information in the conversation audio data, and extract third feature data according to the semantic information in the conversation audio data through the pre-trained emotional state information acquisition model; A feature fusion module, used for performing feature fusion on the first feature data, the second feature data and the third feature data to obtain a fusion feature; The emotion acquisition module is used to acquire emotion state information based on the fusion feature.

11. A training device for an emotional state information acquisition model, It is characterized in that include: The data acquisition module is used to obtain historical behavior data, historical conversation audio data and annotation data during telephone communication; A data input module, used to input the historical behavior data, historical conversation audio data and annotated data into the emotional state information acquisition model to be trained; A feature extraction module, configured to extract first feature data according to the historical behavior data, extract second feature data according to the audio information in the historical conversation audio data, and extract third feature data according to the semantic information in the historical conversation audio data through the emotional state information acquisition model to be trained; A feature fusion module, used for performing feature fusion on the first feature data, the second feature data and the third feature data to obtain a fusion feature; The model training module is used to train the emotional state information acquisition model to be trained using a loss function to obtain a pre-trained emotional state information acquisition model.

12. An electronic device, It is characterized in that include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1-9 by executing the executable instructions.

13. A computer readable medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, each step of the method according to any one of claims 1 to 9 is implemented.

Images