Emotion-based voice conversion method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an emotion-based voice conversion method and device, equipment and a storage medium, and belongs to the technical field of artificial intelligence and financial science and technology. The method comprises the following steps: acquiring voice sample data and text sample data; training a preset original decoder according to the voice sample data and the text sample data to obtain a target voice decoder; acquiring original voice data; performing emotion attribute prediction according to the original voice data to obtain an emotion attribute embedding vector; performing emotion style coding on the original voice data to obtain an emotion style embedding vector; and performing voice reconstruction on the emotion attribute embedded vector and the emotion style embedded vector based on the target voice decoder to obtain target voice data, thereby realizing voice conversion of comprehensive emotion and improving the conversion effect.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and financial technology, and in particular to an emotion-based speech conversion method and apparatus, device, and storage medium. Background Art

[0002] Voice Conversion: Voice conversion is a voice conversion technology that transfers one person's voice to another person's speech. Usually, VC is applied to neutral emotions, that is, to emotionless speech. Taking the financial scenario as an example, when the smart insurance agent gives an intelligent response to the insured, the tone is relatively stiff and the emotion is relatively neutral, or when the smart customer service provides relevant services to the customer, the tone is relatively neutral, which gives the user a poor experience. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose an emotion-based speech conversion method and apparatus, device, and storage medium, aiming to perform speech conversion based on emotion and improve the speech conversion effect.

[0004] To achieve the above object, a first aspect of an embodiment of the present application proposes an emotion-based speech conversion method, the method comprising:

[0005] Acquire speech sample data and text sample data; wherein the speech sample data has a sample emotional style, and the text sample data is the speech content of the speech sample data;

[0006] Training a preset original decoder according to the speech sample data and the text sample data to obtain a target speech decoder;

[0007] Acquire original voice data; wherein the original voice data has an original emotional style;

[0008] Performing emotional attribute prediction based on the original speech data to obtain an emotional attribute embedding vector; wherein the emotional attribute embedding vector is used to characterize the difference between the original emotional style and the sample emotional style;

[0009] Performing emotion style encoding on the original speech data to obtain an emotion style embedding vector;

[0010] Based on the target speech decoder, speech reconstruction is performed on the emotion attribute embedding vector and the emotion style embedding vector to obtain target speech data.

[0011] In some embodiments, the speech sample data includes sample speech features, the sample initial speech features are used to characterize the sample emotional style, and the emotional attribute prediction is performed based on the original speech data to obtain the emotional attribute embedding vector, including:

[0012] Extracting features from the original speech data to obtain initial speech features; wherein the initial speech features are used to characterize the original emotional style;

[0013] Performing emotion difference prediction based on the initial speech features to obtain emotion difference data; wherein the emotion difference data is used to characterize the difference between the initial speech features and the sample speech features;

[0014] An embedding vector conversion is performed according to the emotion difference data to obtain the emotion attribute embedding vector.

[0015] In some embodiments, performing embedding vector conversion according to the emotion difference data to obtain the emotion attribute embedding vector includes:

[0016] Get the preset sentiment ranking function;

[0017] Sorting the sentiment difference data according to the sentiment ranking function to obtain sentiment ranking data;

[0018] The sentiment ranking data is projected through a preset fully connected layer to obtain the sentiment attribute embedding vector.

[0019] In some embodiments, the step of performing emotion difference prediction based on the initial speech features to obtain emotion difference data includes:

[0020] Performing weight estimation on the initial speech features based on a support vector machine to obtain feature weight data;

[0021] Performing product processing on the initial speech feature and the feature weight data to obtain initial speech feature data;

[0022] The emotion difference data is obtained according to the initial voice feature data and the preset sample voice feature data.

[0023] In some embodiments, obtaining the emotion difference data according to the initial voice feature data and the preset sample voice feature data includes:

[0024] Performing weight estimation on the sample speech features based on the support vector machine to obtain sample weight data;

[0025] Performing product processing on the speech sample feature and the sample weight data to obtain sample speech feature data;

[0026] A difference calculation is performed between the initial voice feature data and the sample voice feature data to obtain the emotion difference data.

[0027] In some embodiments, the training of a preset original decoder according to the speech sample data and the text sample data to obtain a target speech decoder includes:

[0028] Performing text encoding on the text sample data based on a preset text encoder to obtain a sample text embedding vector;

[0029] Obtaining a preset language coding model; wherein the language coding model includes a language encoder and a language decoder;

[0030] Performing language encoding on the speech sample data based on the language encoder to obtain a preliminary language vector;

[0031] Performing language decoding on the preliminary language vector based on the language decoder to obtain a sample language embedding vector;

[0032] The original decoder is trained according to the sample text embedding vector and the sample language embedding vector to obtain the target speech decoder.

[0033] In some embodiments, the step of performing emotion style encoding on the original speech data to obtain an emotion style embedding vector includes:

[0034] Get the pre-trained target emotion encoding model;

[0035] Performing emotion style encoding on the original speech data based on the target emotion encoding model to obtain an emotion style vector;

[0036] The emotional style vector is projected based on the target emotional coding model to obtain the emotional style embedding vector.

[0037] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes an emotion-based speech conversion device, the device comprising:

[0038] A sample data acquisition module, used to acquire voice sample data and text sample data; wherein the voice sample data has a sample emotional style, and the text sample data is the voice content of the voice sample data;

[0039] A decoder training module, used for training a preset original decoder according to the speech sample data and the text sample data to obtain a target speech decoder;

[0040] An original voice data acquisition module, used to acquire original voice data; wherein the original voice data has an original emotional style;

[0041] An emotional attribute prediction module, used to predict emotional attributes according to the original speech data to obtain an emotional attribute embedding vector; wherein the emotional attribute embedding vector is used to characterize the difference between the original emotional style and the sample emotional style;

[0042] An emotion style encoding module, used to perform emotion style encoding on the original speech data to obtain an emotion style embedding vector;

[0043] The speech reconstruction module is used to perform speech reconstruction on the emotion attribute embedding vector and the emotion style embedding vector based on the target speech decoder to obtain target speech data.

[0044] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0045] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium, and the storage medium stores a computer program. When the computer program is executed by a processor, the method described in the first aspect is implemented.

[0046] The emotion-based speech conversion method, apparatus, device, and storage medium proposed in the embodiments of the present application can be applied to the fields of artificial intelligence and financial technology. It obtains speech sample data, text sample data, and original speech data, trains a preset original decoder according to the speech sample data and the text sample data to obtain a target speech decoder, predicts emotion attributes according to the original speech data to obtain an emotion attribute embedding vector; performs emotion style encoding on the original speech data to obtain an emotion style embedding vector; performs speech reconstruction on the emotion attribute embedding vector and the emotion style embedding vector based on the target speech decoder to obtain target speech data, thereby realizing emotion-based speech conversion and improving speech conversion effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flow chart of an emotion-based speech conversion method provided in an embodiment of the present application;

[0048] Figure 2 yes Figure 1 Flowchart of step 102 in FIG.

[0049] Figure 3 yes Figure 1Flowchart of step 104 in;

[0050] Figure 4 yes Figure 3 Flowchart of step 302 in FIG.

[0051] Figure 5 yes Figure 4 Flow chart of step 403 in;

[0052] Figure 6 yes Figure 3 Flow chart of step 303 in;

[0053] Figure 7 yes Figure 1 Flowchart of step 105 in;

[0054] Figure 8 is a schematic diagram of the structure of an emotion-based speech conversion device provided in an embodiment of the present application;

[0055] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0059] First, some nouns involved in this application are analyzed:

[0060] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0061] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, information analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0062] Support Vector Machine (SVM): SVM is a supervised learning method and also a classifier. The core of SVM is to separate data by constructing a segmentation surface.

[0063] Voice Conversion: Voice conversion is a voice conversion technology. Voice conversion is the transfer of one person's voice to another person's speech. Usually, VC is applied to neutral emotions, that is, to speech without emotion. For example, in financial scenarios, when intelligent insurance agents give intelligent responses to policyholders, their tone is stiff and their emotions are neutral, or when intelligent customer service provides relevant services to customers, their tone is neutral, which gives users a poor experience. At present, there are some emotional style VC applications that express the same speaker's words with different emotions, but this application uses clearly defined emotions as the basis for conversion, such as happiness, sadness, anger, etc. In actual application scenarios, people's voice emotions are often comprehensive, such as anger with sadness. However, most emotional speech databases divide speech into several discrete categories (such as happiness, sadness, anger, etc.), which limits the learning of richer emotional style descriptions.

[0064] Based on this, the embodiments of the present application provide an emotion-based speech conversion method and apparatus, device, and storage medium, aiming to improve the efficiency and effect of speech conversion, realize comprehensive emotion speech conversion, and enhance user experience.

[0065] The emotion-based speech conversion method, apparatus, device, and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the emotion-based speech conversion method in the embodiments of the present application is described.

[0066] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0067] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0068] The emotion-based speech conversion method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The emotion-based speech conversion method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements an emotion-based speech conversion method, etc., but is not limited to the above forms.

[0069] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0070] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data such as user audio, user voice, user behavior, user historical data, and user attribute information and other data related to user identity or characteristics, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or jump to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0071] Figure 1 is an optional flow chart of the emotion-based speech conversion method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps 101 to 106.

[0072] Step 101, obtaining voice sample data and text sample data; wherein the voice sample data has a sample emotional style, and the text sample data is the voice content of the voice sample data;

[0073] Step 102, training a preset original decoder according to the speech sample data and the text sample data to obtain a target speech decoder;

[0074] Step 103, obtaining original voice data; wherein the original voice data has an original emotional style;

[0075] Step 104, predicting the emotion attribute according to the original speech data to obtain an emotion attribute embedding vector; wherein the emotion attribute embedding vector is used to characterize the difference between the original emotion style and the sample emotion style;

[0076] Step 105, performing emotion style encoding on the original speech data to obtain an emotion style embedding vector;

[0077] Step 106, based on the target speech decoder, the emotion attribute embedding vector and the emotion style embedding vector are reconstructed to obtain target speech data.

[0078] Steps 101 to 106 shown in the embodiment of the present application are obtained by acquiring speech sample data, text sample data, and original speech data, training the preset original decoder according to the speech sample data and text sample data to obtain a target speech decoder, and predicting the emotional attributes according to the original speech data to obtain the emotional attribute embedding vector, encoding the emotional style of the original speech data to obtain the emotional style embedding vector; based on the target speech decoder, the emotional attribute embedding vector and the emotional style embedding vector are voice reconstructed to obtain the target speech data, and speech conversion is realized. For example, in the financial technology scenario, banks, insurance companies, and other financial institutions usually record customer service and sales calls, which may contain emotional information about customers. Through the speech conversion technology of the embodiment of the present application, the voice of a customer service representative can be converted to another emotion to improve the customer experience. For example, if a customer service representative sounds unfriendly or impatient when serving a customer, the voice conversion method of the embodiment of the present application can convert the voice of the customer service representative into a more friendly and patient voice, thereby improving customer satisfaction.

[0079] The speech sample data, text sample data, and original speech data in the above-mentioned embodiments are all tensors, wherein the speech sample data and original speech data are both speech data and are one-dimensional data.

[0080] In step 101 of some embodiments, voice sample data can be obtained from sample objects, and the sample objects can be insurance agents, financial advisors, agents, etc. in financial scenarios, or doctors, nurses, etc. who conduct online consultations in medical scenarios. In the financial technology scenario, banks, insurance companies and other financial institutions usually record customer service and sales calls, and voice sample data can come from these calls. Voice sample data is the voice data of sample objects, for example, it can be the voice data of insurance agents answering policyholders' questions about insurance, it can also be the voice data of financial advisors answering financial users' questions about financial management, or it can be the voice data of agents recommending credit card services to the outside. Text sample data is the voice content of voice sample data. Taking the financial technology scenario as an example, text sample data can be: when the bank's credit card center makes an outbound call service, the agent recommends the credit card business to the outside with the following words: "Hello, Mr. Zhang, I am the customer service of the credit card channel center of XX Bank, with the work number 952M. The following call will be recorded. You have a credit card with the last number 1234 in our bank. In view of your good usage record, we now provide you with a cash limit of 100,000 yuan."

[0081] The speech sample data has a sample emotional style, and the sample emotional style is used to represent the emotion of the sample object, such as happiness, sadness, anger, etc.

[0082] See also Figure 2 In some embodiments, step 102 may include but is not limited to steps 201 to 205.

[0083] Step 201, encoding the text sample data based on a preset text encoder to obtain a sample text embedding vector;

[0084] Step 202, obtaining a preset language coding model; wherein the language coding model includes a language encoder and a language decoder;

[0085] Step 203, language encoding the speech sample data based on the language encoder to obtain a preliminary language vector;

[0086] Step 204, performing language decoding on the preliminary language vector based on the language decoder to obtain a sample language embedding vector;

[0087] Step 205: Train the original decoder according to the sample text embedding vector and the sample language embedding vector to obtain a target speech decoder.

[0088] In step 201 of some embodiments, the text encoder is obtained in advance. In an application scenario, the text encoder uses a 3-layer 1D convolutional neural network (CNN), the text encoder kernel size is 5, and the channel is 512. The sample text embedding vector obtained by text encoding the text sample data through the text encoder is a text embedding sequence, and the original decoder is supervised by the sample text embedding vector to improve the training effect. The sample text embedding vector obtained after the text encoder of step 201 performs text encoding is two-dimensional, one dimension is time series, and the other dimension is vector space.

[0089] In an application scenario, for example, text sample data is represented as: [batchSize, sentence], and after text encoding by the text encoder, [batchSize, seq_len, dim] is obtained, generating a tensor with a length of seq_len and a dimension of dim for each word.

[0090] In step 202 of some embodiments, the language encoder includes a built-in encoder and a 2-layer 256-unit BLSTM, the language decoder includes a built-in decoder, a 1-layer 512-unit BLSTM, an attention layer, and a fully connected (FC) layer with an output channel of 512, and the language encoding model also includes a fully connected (FC) layer with an output channel of 512, that is, the language encoding model includes an encoder, a 2-layer 256-unit BLSTM, a decoder, a 1-layer 512-unit BLSTM, an attention layer, and a fully connected (FC) layer with an output channel of 512.

[0091] In some embodiments, in steps 203 to 204, the speech sample data is language encoded based on the language encoder to obtain a preliminary language vector, and the preliminary language vector is decoded and projected based on the language decoder to obtain a sample language embedding vector. The sample language embedding vector obtained after the language encoder of step 203 performs language encoding and the language decoder of step 204 performs language decoding is two-dimensional, one dimension is time series, and the other dimension is vector space.

[0092] In an application scenario, for example, the speech sample data is [batchSize, y], y represents the amplitude of each sampling point, and after the language encoder in step 203 performs language encoding and the language decoder in step 204 performs language decoding, [bS, len, audio_dim] is obtained, which represents the generation of a tensor with a length of len and a dimension of audio_dim for each word.

[0093] In step 205 of some embodiments, the sample text embedding vector and the sample language embedding vector are input into the original decoder for training to obtain a target speech decoder. In an application scenario, contrastive loss can be used to ensure the similarity between the sample text embedding vector and the sample language embedding vector, and a classifier can be used for adversarial training to eliminate residual emotional information in the sample language embedding vector. Specifically, contrastive loss is an unsupervised method that can be used to learn the similarity between embeddings. In contrastive loss, a pair of sample text embedding vectors and sample language embedding vectors from the same group of samples are used as positive examples, and sample text embedding vectors and sample language embedding vectors from different samples are used as negative examples. Contrastive loss can be calculated by the following formula:

[0094] L=1 / 2N*sum(y*d^2+(1-y)*max(margin-d,0)^2)

[0095] Among them, N is the number of sample pairs, a pair of sample text embedding vector and sample language embedding vector is a sample pair, y is the similarity label (1 for similarity and 0 for dissimilarity), d is the embedding distance, and margin is a hyperparameter used to control the gap between the distance between similar samples and the distance between dissimilar samples.

[0096] Through steps 201 to 205 of the above embodiment, the original decoder can be trained, and the target speech decoder finally trained has the same model architecture as Tacotron. Tacotron is an end-to-end text-to-speech (Speech Synthesis) model based on neural networks. It is an end-to-end TTS (text to speech) deep neural network model. Tacotron is an attention-based seq2seq model. Tacotron is used to convert text to high-quality speech conversion without manual adjustment of parameters or rules.

[0097] The difference between the initial speech feature data and other sample speech feature data can be obtained, which is convenient for subsequent emotion-based speech conversion.

[0098] In step 103 of some embodiments, the original voice data can be obtained from the target object. In actual application scenarios, the target object can be an insurance agent, financial advisor, seat staff, etc. in a financial scenario, or a doctor, nurse, etc. who conducts online consultation in a medical scenario. In the financial technology scenario, banks, insurance companies and other financial institutions usually record customer service and sales calls, and the original voice data can come from these calls. The original voice data is similar to the above-mentioned voice sample data. Specifically, the original voice data can be the voice data of an insurance agent replying to an insured person's insurance consultation, or the voice data of a financial advisor replying to a financial user's financial consultation, or the voice data of a seat staff recommending credit card services to the outside. Taking the financial technology scenario as an example, the original voice data and text sample data can be: when the bank's credit card center makes an outbound call service, the seat staff recommends the credit card business to the outside as follows: "Hello, Mr. Zhang, I am a customer service representative of the credit card channel center of XX Bank, with a work number of 952M. The following call will be recorded. You have a credit card with a last number of 1234 in our bank. In view of your good usage record, we now provide you with a cash limit of 100,000 yuan."

[0099] The original speech data has an original emotional style, which is used to represent the emotions of the target object, such as anger, sadness, happiness, etc.

[0100] See also Figure 3 In some embodiments, step 104 may include but is not limited to steps 301 to 303 .

[0101] Step 301, extracting features from the original speech data to obtain initial speech features; wherein the initial speech features are used to characterize the original emotional style;

[0102] Step 302, predicting the emotion difference according to the initial speech features to obtain emotion difference data; wherein the emotion difference data is used to characterize the difference between the initial speech features and the sample speech features;

[0103] Step 303: Perform embedding vector conversion according to the sentiment difference data to obtain a sentiment attribute embedding vector.

[0104] In step 301 of some embodiments, the initial speech feature obtained by feature extraction of the original speech data can be represented by Yn, where Yn is the nth initial speech feature, and can represent, for example, the happy feature of the sample object 1 (e.g., an insurance agent), for example, the happy feature that characterizes the happy emotional style of the target object 1 (e.g., a sales customer service representative), in which case the happy emotional style is the original emotional style of the target object 1; for another example, the initial speech feature can also be the angry feature that characterizes the angry emotional style of the target object 2 (e.g., a financial advisor), and the angry emotional style is the original emotional style of the target object 2.

[0105] See also Figure 4 In some embodiments, step 302 may include but is not limited to steps 401 to 403.

[0106] Step 401, weight estimation of initial speech features based on a support vector machine to obtain feature weight data;

[0107] Step 402, performing product processing according to the initial speech feature and the feature weight data to obtain initial speech feature data;

[0108] Step 403, obtaining emotion difference data according to the initial speech feature data and the preset sample speech feature data.

[0109] In step 401 of some embodiments, the feature weight data is a weight matrix, and the weight matrix W1 is estimated by solving a support vector machine problem.

[0110] In step 402 of some embodiments, the initial speech feature data obtained by multiplying the initial speech feature and the feature weight data can be expressed as W1*Yn, where Yn is the initial speech feature.

[0111] The speech sample data includes sample speech features. The sample initial speech features are used to characterize the sample emotional style. The sample speech features can be represented by Xn, where Xn is the nth sample speech feature, for example, it can represent the happiness feature of sample object 1 (such as an insurance agent).

[0112] See also Figure 5 In some embodiments, step 403 may include but is not limited to steps 501 to 503.

[0113] Step 501, weight estimation of sample speech features based on a support vector machine to obtain sample weight data;

[0114] Step 502, performing product processing according to the speech sample feature and the sample weight data to obtain sample speech feature data;

[0115] Step 503: perform a difference calculation between the initial speech feature data and the sample speech feature data to obtain emotion difference data.

[0116] In step 501 of some embodiments, the sample weight data is a weight matrix, and the sample weight data can be represented as W2; it should be understood that the method of weight estimation for sample speech features is similar to the method of weight estimation for initial speech features in step 401 above, and will not be repeated here.

[0117] In step 501 of some embodiments, the sample speech feature data may be represented as W2*Xn; it should be understood that the principle of multiplying the speech sample features and the sample weight data is similar to the principle of step 402 above and will not be described in detail here.

[0118] In step 503 of some embodiments, a difference calculation is performed between W1*Yn and W2*Xn. Taking the financial scenario as an example, the initial voice feature data is an anger feature, and the sample voice feature data includes a happy feature, a sad feature, and a happy feature. The emotional difference data between the anger feature and the happy feature is 0.9, the emotional difference data between the anger feature and the sad feature is 0.5, and the emotional difference data between the anger feature and the happy feature is 0.8.

[0119] Through steps 501 to 503 of the above embodiment, the emotional difference between the initial voice feature data and the sample voice feature data can be calculated, thereby obtaining the emotional difference data

[0120] Through steps 401 to 403 of the above embodiment, the difference between the initial speech feature data and other sample speech feature data can be obtained, which is convenient for subsequent emotion-based speech conversion.

[0121] See also Figure 6 In some embodiments, step 303 may include but is not limited to steps 501 to 603.

[0122] Step 601, obtaining a preset sentiment ranking function;

[0123] Step 602, sorting the sentiment difference data according to the sentiment ranking function to obtain sentiment ranking data;

[0124] Step 603: Project the sentiment ranking data through a preset fully connected layer to obtain a sentiment attribute embedding vector.

[0125] In some embodiments, before executing step 601, the sentiment ranking function needs to be pre-trained. The embodiment of the present application does not limit the step of pre-training the sentiment ranking function. The sentiment ranking function can be expressed as f(x).

[0126] In step 602 of some embodiments, the value of the emotion difference data is [0, 1], and the emotion difference data includes: f1(x) = 0.9, f2(x) = 0.6, f3(x) = 0.7, f4(x) = 0.8. The emotion ranking data obtained after sorting the emotion difference data is: 0.9, 0.8, 0.7, 0.6. The higher the value in the emotion ranking data, the greater the difference between the two speech features. If there are two data with equal values ​​in the emotion ranking data, the two equal data will be eliminated.

[0127] In step 603 of some embodiments, the above numerical emotion difference data is projected to obtain an emotion attribute embedding vector.

[0128] Through steps 601 to 603 of the above embodiment, the difference between the two speech features can be obtained, and the difference between the two speech features can be represented by the emotion attribute embedding vector, which facilitates the subsequent emotion-based speech conversion.

[0129] Through steps 301 to 303 of the above embodiment, the difference between the initial speech feature data and other sample speech feature data can be obtained, which is convenient for subsequent emotion-based speech conversion.

[0130] In step 105 of some embodiments, the original speech data is encoded with an emotion style by a pre-trained target emotion encoding model. In one application scenario, the target emotion encoding model includes a 2-layer 128-unit BLSTM, an FC layer with 64 output channels, and a classifier, which is a 4-layer FC with channels {512, 512, 512, 5}.

[0131] See also Figure 7 , in some embodiments 105 may include but is not limited to steps 601 to 703 .

[0132] Step 701, obtaining a pre-trained target emotion encoding model;

[0133] Step 702, performing emotion style encoding on the original speech data based on the target emotion encoding model to obtain an emotion style vector;

[0134] Step 703: Project the emotional style vector based on the target emotional coding model to obtain an emotional style embedding vector.

[0135] In some embodiments, before executing step 701, it is necessary to pre-train the target emotion coding model. In the embodiment of the present application, the step of training the target emotion coding model is not limited. The target emotion coding model includes a built-in emotion encoder and a fully connected layer. The built-in emotion encoder is used to implement steps 602 and 603.

[0136] In step 702 of some embodiments, the original emotional style in the original speech data is encoded by an emotional style encoder based on a built-in emotional encoder of a target emotional encoding model to obtain an emotional style vector.

[0137] In step 703 of some embodiments, the emotion style vector is projected based on the fully connected layer of the target emotion encoding model to obtain the emotion style embedding vector.

[0138] In an application scenario, for example, the original speech data is [batchSize1, y1], y1 represents the amplitude of each sampling point; after encoding and decoding in step 102, [bS1, len1, audio_dim1] is obtained, indicating that a tensor with a length of len1 and a dimension of audio_dim1 for each word is generated; after emotion encoding by the target emotion encoding model in step 105, [BS, emo_dim1] is obtained, indicating that the dimension of the obtained emotion style embedding vector is a tensor of emo_dim1.

[0139] Through steps 701 to 703 of the above embodiment, the emotional style of the original speech data can be encoded, and the obtained emotional style embedding vector is convenient for subsequent speech reconstruction to achieve emotion-based speech conversion.

[0140] In step 106 of some embodiments, since the target speech decoder has been pre-trained and has language encoding capabilities, after the emotion attribute embedding vector and the emotion style embedding vector are input into the target speech decoder, the target speech decoder can realize speech reconstruction. The target speech data obtained after speech reconstruction can realize emotion-based speech conversion, and can convert unfriendly or impatient-sounding speech into patient and gentle speech.

[0141] In the FinTech scenario, banks, insurance companies, and other financial institutions often record customer service and sales calls, which may contain emotional information about customers. Through the voice conversion technology of the embodiment of the present application, the voice of a customer service representative can be converted into another emotion to improve the customer experience. For example, if a customer service representative sounds unfriendly or impatient when serving a customer, the voice conversion method of the embodiment of the present application can convert the customer service representative's voice into a more friendly and patient voice, thereby improving customer satisfaction. In addition, combined with professional knowledge in the financial field, a more in-depth analysis of the call can be performed to understand the customer's investment preferences, financial status, and other relevant information, so as to better provide personalized services.

[0142] For example, when the bank's credit card center makes outbound calls, the agent recommends the credit card business using the following words: "Hello, Mr. Zhang. I am a customer service representative from the credit card channel center of XX Bank. My employee number is 952M. The following call will be recorded. You have a credit card with the last number 1234 in our bank. In view of your good usage record, we are now providing you with a cash limit of 100,000 yuan." Such words will cause certain emotional fluctuations in the voice call every time a call is made, that is, the emotional expression will be different due to the customer's workload and daily life. Through the embodiment of the present application, the voice called out by the agent is emotionally converted so that every call with the customer can have better emotional expression. The specific process is as follows:

[0143] The agent's speech content will be passed through the language encoding model to obtain the speech content encoding latent variables, and then the emotions to be expressed will be emotionally encoded based on the target emotion encoding model. The emotion difference data will be sorted through the emotion ranking function to calculate the difference between the two emotion styles, and an emotion attribute embedding vector will be obtained through the fully connected layer. The greater the difference between the two emotions, the higher the ranking result, and the value range is [0 to 1]. Finally, the pre-trained target speech decoder can realize speech reconstruction. The target speech data obtained after speech reconstruction can realize emotion-based speech conversion, which can convert unfriendly or impatient voice into patient and gentle voice.

[0144] The emotion-based speech conversion method provided in the embodiment of the present application obtains speech sample data, text sample data, and original speech data, trains a preset original decoder according to the speech sample data and the text sample data to obtain a target speech decoder, predicts emotion attributes according to the original speech data to obtain an emotion attribute embedding vector, and performs emotion style encoding on the original speech data to obtain an emotion style embedding vector; based on the target speech decoder, speech is reconstructed on the emotion attribute embedding vector and the emotion style embedding vector to obtain the target speech data, which can improve the speech conversion efficiency and enhance the speech conversion effect.

[0145] See also Figure 8 The embodiment of the present application further provides an emotion-based speech conversion device, which can implement the above-mentioned emotion-based speech conversion method, and the device includes:

[0146] A sample data acquisition module, used to acquire voice sample data and text sample data;

[0147] A decoder training module, used for training a preset original decoder according to the speech sample data and the text sample data to obtain a target speech decoder;

[0148] An original voice data acquisition module, used to acquire original voice data;

[0149] An emotion attribute prediction module, used to predict emotion attributes according to the original speech data to obtain an emotion attribute embedding vector;

[0150] An emotion style encoding module, used to perform emotion style encoding on the original speech data to obtain an emotion style embedding vector;

[0151] The speech reconstruction module is used to perform speech reconstruction on the emotion attribute embedding vector and the emotion style embedding vector based on the target speech decoder to obtain target speech data.

[0152] In some embodiments, the decoder training module may be specifically used to implement:

[0153] Encode the text sample data based on a preset text encoder to obtain a sample text embedding vector;

[0154] Obtaining a preset language coding model; wherein the language coding model includes a language encoder and a language decoder;

[0155] Performing language encoding on the speech sample data based on the language encoder to obtain a preliminary language vector;

[0156] Perform language encoding on the preliminary language vector based on the language decoder to obtain a sample language embedding vector;

[0157] The original decoder is trained according to the sample text embedding vector and the sample language embedding vector to obtain the target speech decoder.

[0158] Specifically, the decoder training module can be used to implement the above steps 201 to 205, which will not be described in detail here.

[0159] In some embodiments, the sentiment attribute prediction module may be used to implement:

[0160] Extracting features from the original speech data to obtain initial speech features, wherein the initial speech features are used to characterize the original emotional style;

[0161] Predicting the emotion difference based on the initial speech features to obtain emotion difference data; wherein the emotion difference data is used to characterize the difference between the initial speech features and the sample speech features;

[0162] The embedding vector is converted according to the sentiment difference data to obtain the sentiment attribute embedding vector.

[0163] Specifically, the sentiment attribute prediction module can be used to implement the above steps 301 to 303, which will not be described in detail here.

[0164] In one embodiment, the emotion attribute prediction module is used to implement "predicting emotion differences according to initial speech features to obtain emotion difference data", specifically including:

[0165] Estimating the weight of the initial speech features based on the support vector machine to obtain feature weight data;

[0166] Performing product processing on the initial speech feature and the feature weight data to obtain initial speech feature data;

[0167] Emotional difference data is obtained based on the initial voice feature data and the preset sample voice feature data.

[0168] Furthermore, the sentiment attribute prediction module can be used to implement the above steps 401 to 403, which will not be described in detail here.

[0169] In one embodiment, the emotion attribute prediction module is used to implement "predicting emotion differences according to initial speech features to obtain emotion difference data", specifically including:

[0170] Step 501, weight estimation of sample speech features based on a support vector machine to obtain sample weight data;

[0171] Step 502, performing product processing according to the speech sample feature and the sample weight data to obtain sample speech feature data;

[0172] Step 503: perform a difference calculation between the initial speech feature data and the sample speech feature data to obtain emotion difference data.

[0173] Furthermore, the sentiment attribute prediction module can be used to implement the above steps 501 to 503, which will not be described in detail here.

[0174] Specifically, the sentiment attribute prediction module is used to implement "converting the embedding vector according to the sentiment difference data to obtain the sentiment attribute embedding vector", which specifically includes:

[0175] Get the preset sentiment ranking function;

[0176] Sort the sentiment difference data according to the sentiment ranking function to obtain sentiment ranking data;

[0177] The sentiment ranking data is projected through a preset fully connected layer to obtain the sentiment attribute embedding vector.

[0178] Furthermore, the sentiment attribute prediction module can be used to implement the above steps 601 to 603, which will not be described in detail here.

[0179] In some embodiments, the emotion style encoding module can be specifically used to implement:

[0180] Get the pre-trained target emotion encoding model;

[0181] Based on the target emotion coding model, the original speech data is encoded with emotion style to obtain the emotion style vector;

[0182] The emotional style vector is projected based on the target emotional encoding model to obtain the emotional style embedding vector.

[0183] Specifically, the emotion style encoding module can be used to implement the above steps 701 to 703, which will not be described in detail here.

[0184] The specific implementation of the emotion-based speech conversion device is basically the same as the specific implementation of the emotion-based speech conversion method described above, and will not be repeated here.

[0185] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned emotion-based speech conversion method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0186] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0187] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0188] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the emotion-based speech conversion method of the embodiment of this application;

[0189] Input / output interface 903, used to implement information input and output;

[0190] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0191] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0192] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 95 .

[0193] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned emotion-based speech conversion method is implemented.

[0194] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0195] The emotion-based speech conversion method, apparatus, device, and storage medium provided in the embodiments of the present application obtain speech sample data, text sample data, and original speech data, train a preset original decoder according to the speech sample data and text sample data to obtain a target speech decoder, predict emotion attributes based on the original speech data to obtain an emotion attribute embedding vector, and perform emotion style encoding on the original speech data to obtain an emotion style embedding vector; perform speech reconstruction on the emotion attribute embedding vector and the emotion style embedding vector based on the target speech decoder to obtain the target speech data, thereby realizing speech conversion based on comprehensive emotions, improving speech conversion efficiency, and enhancing speech conversion effects.

[0196] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0197] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0198] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0199] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0200] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0201] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0202] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0203] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0204] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0205] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0206] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A speech conversion method based on emotion, characterized in that: The method comprises: Acquire speech sample data and text sample data; wherein the speech sample data has a sample emotional style, and the text sample data is the speech content of the speech sample data; Training a preset original decoder according to the speech sample data and the text sample data to obtain a target speech decoder; Acquire original voice data; wherein the original voice data has an original emotional style; Performing emotional attribute prediction based on the original speech data to obtain an emotional attribute embedding vector; wherein the emotional attribute embedding vector is used to characterize the difference between the original emotional style and the sample emotional style; Performing emotion style encoding on the original speech data to obtain an emotion style embedding vector; Based on the target speech decoder, speech reconstruction is performed on the emotion attribute embedding vector and the emotion style embedding vector to obtain target speech data.

2. The method according to claim 1, characterized in that: The speech sample data includes sample speech features, the sample initial speech features are used to characterize the sample emotional style, and the emotional attribute prediction is performed according to the original speech data to obtain the emotional attribute embedding vector, including: Extracting features from the original speech data to obtain initial speech features; wherein the initial speech features are used to characterize the original emotional style; Performing emotion difference prediction based on the initial speech features to obtain emotion difference data; wherein the emotion difference data is used to characterize the difference between the initial speech features and the sample speech features; An embedding vector conversion is performed according to the emotion difference data to obtain the emotion attribute embedding vector.

3. The method according to claim 2, characterized in that The step of performing embedding vector conversion according to the emotion difference data to obtain the emotion attribute embedding vector comprises: Get the preset sentiment ranking function; Sorting the sentiment difference data according to the sentiment ranking function to obtain sentiment ranking data; The sentiment ranking data is projected through a preset fully connected layer to obtain the sentiment attribute embedding vector.

4. The method according to claim 2, characterized in that: The step of performing emotion difference prediction according to the initial speech features to obtain emotion difference data includes: Performing weight estimation on the initial speech features based on a support vector machine to obtain feature weight data; Performing product processing on the initial speech feature and the feature weight data to obtain initial speech feature data; The emotion difference data is obtained according to the initial voice feature data and the preset sample voice feature data.

5. The method according to claim 4, characterized in that The obtaining of the emotion difference data according to the initial voice feature data and the preset sample voice feature data includes: Performing weight estimation on the sample speech features based on the support vector machine to obtain sample weight data; Performing product processing on the speech sample feature and the sample weight data to obtain sample speech feature data; A difference calculation is performed between the initial voice feature data and the sample voice feature data to obtain the emotion difference data.

6. The method according to any one of claims 1 to 5, characterized in that: The step of training a preset original decoder according to the speech sample data and the text sample data to obtain a target speech decoder includes: Performing text encoding on the text sample data based on a preset text encoder to obtain a sample text embedding vector; Acquire a preset language coding model; wherein the language coding model includes a language encoder and a language decoder; Performing language encoding on the speech sample data based on the language encoder to obtain a preliminary language vector; Performing language decoding on the preliminary language vector based on the language decoder to obtain a sample language embedding vector; The original decoder is trained according to the sample text embedding vector and the sample language embedding vector to obtain the target speech decoder.

7. The method according to any one of claims 1 to 4, characterized in that: The step of performing emotional style encoding on the original speech data to obtain an emotional style embedding vector includes: Get the pre-trained target emotion encoding model; Performing emotion style encoding on the original speech data based on the target emotion encoding model to obtain an emotion style vector; The emotional style vector is projected based on the target emotional coding model to obtain the emotional style embedding vector.

8. An emotion-based speech conversion device, characterized in that: The device comprises: A sample data acquisition module, used to acquire voice sample data and text sample data; wherein the voice sample data has a sample emotional style, and the text sample data is the voice content of the voice sample data; A decoder training module, used for training a preset original decoder according to the speech sample data and the text sample data to obtain a target speech decoder; An original voice data acquisition module, used to acquire original voice data; wherein the original voice data has an original emotional style; An emotional attribute prediction module, used to predict emotional attributes according to the original speech data to obtain an emotional attribute embedding vector; wherein the emotional attribute embedding vector is used to characterize the difference between the original emotional style and the sample emotional style; An emotion style encoding module, used to perform emotion style encoding on the original speech data to obtain an emotion style embedding vector; The speech reconstruction module is used to perform speech reconstruction on the emotion attribute embedding vector and the emotion style embedding vector based on the target speech decoder to obtain target speech data.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images