A voice conversion method, device, equipment and storage medium thereof

Abstract

The embodiment of the application belongs to the technical field of speech synthesis, is applied to a sound conversion scene, and relates to a speech conversion method, device and equipment and a storage medium thereof, and comprises the following steps: obtaining speech to be converted in pronunciation and reference speech of a target pronunciation user; inputting a pre-trained feature decoupling model to obtain decoupled features of the speech to be converted in pronunciation and the reference speech of the target pronunciation user respectively; inputting text features of the speech to be converted in pronunciation and acoustic features of the reference speech of the target pronunciation user into a pre-trained feature fusion model to perform feature fusion, obtain a feature fusion result, and decode to obtain speech whose pronunciation conversion is completed. Through the feature decoupling model and the feature fusion model, speech conversion is quickly realized, and the speech conversion method is applied to the financial customer service industry, so that more high-quality and pleasant sound quality service functions can be provided for customers.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech synthesis, and is applied to a sound conversion scene, and in particular relates to a speech conversion method and device, equipment and a storage medium thereof. BACKGROUND

[0002] With the rapid development of the Internet, especially the rapid development of the self-media industry and the financial industry. The traditional speech synthesis is more and more developing in the direction of AI intelligent speech synthesis, and often needs to use speech conversion technology, that is, to output a specified speech in the pronunciation of another character object.

[0003] The existing speech conversion task is divided into several mainstream ideas, the first one adopts instance standardization, decouples and extracts the content and timbre information of the input speech information, the second one adopts to split the speech into content, rhythm, pitch, timbre and other features, and then uses a supervised technology and a random sampling method to extract the corresponding features, but the above-mentioned first instance standardization scheme cannot complete the real feature decoupling, and the synthesis effect is not good in the speech conversion task; the above-mentioned second scheme increases the complexity of model training, and the feature selection is easy to cause a large amount of partial repetition, which also leads to a more complex processing process in the subsequent speech conversion task, therefore, the prior art still has the problems of being unable to quickly realize speech conversion or complex speech conversion in the speech conversion. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a speech conversion method, device, equipment and storage medium, to solve the problem that the prior art still has the problems of being unable to quickly realize speech conversion or complex speech conversion in the speech conversion.

[0005] In order to solve the above technical problems, the speech conversion method provided by the embodiments of the present application adopts the technical scheme as follows:

[0006] A speech conversion method, comprising the following steps:

[0007] Obtaining first speech data and second speech data, wherein the first speech data comprises speech to be converted in pronunciation, and the second speech data comprises reference speech of a target pronunciation user;

[0008] Inputting the first speech data and the second speech data into a pre-trained feature decoupling model to obtain decoupling features of the first speech data and the second speech data respectively, wherein the decoupling features comprise text features and acoustic features;

[0009] input the text features corresponding to the first voice data and the acoustic features corresponding to the second voice data into a pre-trained feature fusion model to perform feature fusion and obtain a feature fusion result;

[0010] input the feature fusion result into a preset voice decoder to obtain a voice with pronunciation conversion completed.

[0011] Further, before the step of inputting the first voice data and the second voice data into the pre-trained feature decoupling model, the method further comprises:

[0012] obtaining a voice training set, wherein the voice training set contains a plurality of voice data to be decoupled in feature classification;

[0013] inputting the voice training set into a pre-constructed feature decoupling model to perform feature decoupling model training and obtain a trained feature decoupling model; and

[0014] obtaining text features and acoustic features decoupled and output by the trained feature decoupling model.

[0015] Further, the pre-constructed feature decoupling model comprises a feature extraction layer, a pooling layer, and a full connection layer, and the step of inputting the voice training set into the pre-constructed feature decoupling model to perform feature decoupling model training and obtain a trained feature decoupling model specifically comprises:

[0016] Step 301, performing feature extraction on the plurality of voice data to be decoupled in feature classification in the voice training set respectively through the feature extraction layer to obtain corresponding feature extraction sequences;

[0017] Step 302, performing a pooling operation on the feature extraction sequences using the pooling layer, and performing feature classification prediction on the pooling operation result through the full connection layer to obtain a feature classification prediction result;

[0018] Step 303, comparing the feature classification prediction result with a pre-labeled feature classification result, and calculating a classification accuracy and a recall rate of the feature classification prediction result compared with the pre-labeled feature classification result according to the comparison result;

[0019] Step 304, if either the classification accuracy or the recall rate does not meet a preset requirement threshold, constantly adjusting parameters of the feature decoupling model through a back propagation algorithm, and repeatedly performing steps 301 to 303 until the classification accuracy and the recall rate both meet the preset requirement threshold;

[0020] Step 305, if the classification accuracy and the recall rate both meet the preset requirement threshold, a trained feature decoupling model is obtained.

[0021] Further, after performing the step of obtaining the text features and acoustic features decoupled output by the training completed feature decoupling model, the method further comprises:

[0022] According to different voice data entries, the text features and acoustic features are marked respectively, specifically, the marking operation comprises: the text features and acoustic features corresponding to the same entry of voice data are marked with the same identifier, and the text features and acoustic features corresponding to different entries of voice data are marked with different identifiers.

[0023] Further, before performing the step of inputting the text features corresponding to the first voice data and the acoustic features corresponding to the second voice data into the pre-trained feature fusion model, the method further comprises:

[0024] Obtaining text features and acoustic features to be fused, wherein the text features and acoustic features to be fused include the text features and acoustic features decoupled output by the training completed feature decoupling model;

[0025] Inputting the text features and acoustic features to be fused into a pre-constructed feature fusion model to perform feature fusion model training, and obtaining a training completed feature fusion model.

[0026] Further, the feature fusion model comprises a text feature hidden state vector extraction component based on streaming and an acoustic feature hidden state vector extraction component based on posterior encoding, and the step of inputting the text features and acoustic features to be fused into a pre-constructed feature fusion model to perform feature fusion model training, and obtaining a training completed feature fusion model, specifically comprises:

[0027] Step 401, extracting a hidden state vector representation result of the text features through the text feature hidden state vector extraction component based on streaming;

[0028] Step 402, extracting a hidden state vector representation result of the acoustic features through the acoustic feature hidden state vector extraction component based on posterior encoding;

[0029] Step 403, fusing the hidden state vector representation results corresponding to the text features and acoustic features with the same identifier respectively to obtain a feature fusion result;

[0030] Step 404, decoding the feature fusion result using a preset speech decoder to obtain output speech;

[0031] Step 405, according to the preset loss function, the loss value of the output speech compared with the batch speech data in the speech training set is calculated;

[0032] Step 406, if the loss value does not satisfy the preset loss threshold, the model is optimized by adjusting the parameters of the feature fusion model, and steps 401 to 405 are repeatedly executed until the loss value satisfies the preset loss threshold;

[0033] Step 407, if the loss value satisfies the preset loss threshold, a trained feature fusion model is obtained.

[0034] Further, the step of inputting the text features corresponding to the first speech data and the acoustic features corresponding to the second speech data into the pre-trained feature fusion model for feature fusion to obtain a feature fusion result specifically includes:

[0035] extracting, by a stream-based text feature hidden state vector extraction component of the trained feature fusion model, a hidden state vector representation result of the text features corresponding to the first speech data;

[0036] extracting, by a posteriori coding-based acoustic feature hidden state vector extraction component of the trained feature fusion model, a hidden state vector representation result of the acoustic features corresponding to the second speech data;

[0037] fusing the hidden state vector representation result of the text features corresponding to the first speech data and the hidden state vector representation result of the acoustic features corresponding to the second speech data to obtain the feature fusion result.

[0038] To solve the above technical problems, the embodiment of the application also provides a speech conversion device, which adopts the technical scheme as follows:

[0039] A speech conversion device, comprising:

[0040] A speech data acquisition module is configured to acquire first speech data and second speech data, wherein the first speech data includes speech to be converted in pronunciation, and the second speech data includes reference speech of a target pronunciation user;

[0041] A speech feature decoupling module is configured to input the first speech data and the second speech data into a pre-trained feature decoupling model to obtain decoupled features of the first speech data and the second speech data, respectively, wherein the decoupled features include text features and acoustic features;

[0042] a voice feature fusion module, configured to input text features corresponding to the first voice data and acoustic features corresponding to the second voice data into a pre-trained feature fusion model for feature fusion to obtain a feature fusion result;

[0043] a voice feature decoding module, configured to input the feature fusion result into a preset voice decoder to obtain a voice after pronunciation conversion.

[0044] To solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the technical scheme as follows:

[0045] A computer device includes a memory and a processor, the memory stores computer readable instructions, and the processor executes the computer readable instructions to realize the steps of the voice conversion method described above.

[0046] To solve the above technical problems, the embodiment of the present application further provides a computer readable storage medium, which adopts the technical scheme as follows:

[0047] A computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by a processor to realize the steps of the voice conversion method described above.

[0048] Compared with the prior art, the embodiment of the present application has the following beneficial effects:

[0049] The voice conversion method described in the embodiment of the present application obtains the voice to be converted and the reference voice of the target pronunciation user, inputs a pre-trained feature decoupling model to obtain the decoupled features of the voice to be converted and the reference voice of the target pronunciation user, respectively; inputs the text features of the voice to be converted and the acoustic features of the reference voice of the target pronunciation user into a pre-trained feature fusion model for feature fusion to obtain a feature fusion result, and decodes to obtain a voice after pronunciation conversion. Through the feature decoupling model and the feature fusion model, voice conversion is quickly realized, and the voice conversion method is applied to the financial customer service industry to provide customers with more high-quality and pleasant sound quality service functions. BRIEF DESCRIPTION OF DRAWINGS

[0050] To more clearly illustrate the schemes in the present application, the following will briefly introduce the drawings needed in the description of the embodiments of the present application. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0051] Figure 1is an exemplary system architecture diagram in which the present application can be applied;

[0052] Figure 2 is a flow chart of one embodiment of the voice conversion method according to the present application;

[0053] Figure 3 is a flow chart of one specific embodiment of the training of the feature decoupling model in the voice conversion method according to the present application;

[0054] Figure 4 is a flow chart of one specific embodiment of the training of the feature fusion model in the voice conversion method according to the present application;

[0055] Figure 5 is a flow chart of one specific embodiment of the step 203 shown in Figure 2

[0056] Figure 6 is a structural schematic diagram of one embodiment of the voice conversion apparatus according to the present application;

[0057] Figure 7 is a structural schematic diagram of one specific embodiment of the feature decoupling training module 605;

[0058] Figure 8 is a structural schematic diagram of one specific embodiment of the feature fusion training module 606;

[0059] Figure 9 is a structural schematic diagram of one embodiment of the computer device according to the present application. DETAILED DESCRIPTION

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application; the description and claims herein and the above description of drawings herein use the term "comprising" and "including" and their derivatives, which are intended to be open-ended and to mean "including, but not limited to"; the description and claims herein and the above description of drawings herein use the term "first", "second", and the like, which are used for distinguishing between similar objects having a same and / or a different function in the specification. It is to be noted that the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise.

[0061] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase "in an embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily all directed to the same embodiment, or to one or more feature, structure, or characteristic of an embodiment. It will be explicitly understood that the embodiments described herein can be combined with other embodiments in one or more combinations.​

[0062] In order to make the person skilled in the art better understand the scheme of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings.

[0063] As shown in Figure 1 , the system architecture 100 can include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a communication link medium between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0064] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0065] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing, in addition to a notebook computer, a tablet computer, or a mobile phone, the terminal device 101 can also be an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer, and a desktop computer, etc.

[0066] The server 105 can be a server providing various services, such as a background server supporting the page displayed on the terminal devices 101, 102, 103.

[0067] It should be noted that the voice conversion method provided by the embodiments of the present application is generally executed by a terminal device, and correspondingly, the voice conversion apparatus is generally arranged in a terminal device.

[0068] It should be understood that Figure 1 the number of terminal devices, networks, and servers in is only illustrative. According to the needs of implementation, there can be any number of terminal devices, networks, and servers.

[0069] Continuing to refer to Figure 2 , a flowchart of one embodiment of the voice conversion method according to the present application is shown. The voice conversion method includes the following steps:

[0070] Step 201, obtaining first voice data and second voice data, wherein the first voice data includes voice to be converted, and the second voice data includes reference voice of a target voice user.

[0071] Specifically, the first voice data and the second voice data are, for example, a voice-over of Shui Tiaogoutou by a certain voice actor, the first voice data is actually the voice text content of Shui Tiaogoutou, and the second voice data is actually the reference voice of the voice actor; for another example, a character A recites a section of Shui Tiaogoutou, and intends to convert the recited part into a voice recited by a character B, the first voice data is actually the voice part recited by the character A, and the second voice data is the reference voice of the character B. The use scenarios of the present application are mostly in sound beautification or converting a section of existing voice data into voice output of another target character.

[0072] The voice conversion task described in the present application is mostly used in AI intelligent voice synthesis scenarios, for example, inputting a section of text or recording a section of voice, outputting the text or the recorded voice in the reference voice of a target character, or applying to the financial field customer service industry, pre-providing script text content or script voice content, combining the reference voice of a target character or a specified character, and outputting the script text content or the script voice content to ensure providing high-quality and pleasant sound quality service functions for customers.

[0073] Step 202, inputting the first voice data and the second voice data into a pre-trained feature decoupling model to obtain decoupling features of the first voice data and the second voice data, respectively, wherein the decoupling features include text features and acoustic features.

[0074] In the embodiment, before the step of inputting the first voice data and the second voice data into the pre-trained feature decoupling model, the method further includes: obtaining a voice training set, wherein the voice training set contains a batch of voice data to be decoupled in feature classification which has been labeled; inputting the voice training set into a pre-constructed feature decoupling model to perform feature decoupling model training, obtaining a trained feature decoupling model, and obtaining text features and acoustic features decoupled and output by the trained feature decoupling model.

[0075] By using the batch of voice data to be decoupled in feature classification which has been labeled, the feature decoupling model can be verified when the feature decoupling model is trained, the feature decoupling model is trained in a supervised learning manner, and the usability of the feature decoupling model is ensured.

[0076] In this embodiment, the pre-built feature decoupling model includes a speech feature decoupling model based on the Xi-Vector framework, and the pre-built feature decoupling model includes a feature extraction layer, a pooling layer, and a full connection layer. The feature extraction layer can adopt a convolutional neural network (CNN), the pooling layer can adopt a local connection or a fully connected pooling layer, and the full connection layer is mainly used for classification prediction of features after the pooling operation. Correspondingly, a classification prediction component can be introduced into the full connection layer.

[0077] With reference to the foregoing Figure 3 , Figure 3 is a flowchart of a specific embodiment of the feature decoupling model training method in the speech conversion method described in the present application, including the following steps:

[0078] In step 301, the feature extraction layer is used to extract features from batch speech data in the speech training set, and a corresponding feature extraction sequence is obtained.

[0079] In step 302, the pooling layer is used to perform a pooling operation on the feature extraction sequence, and the full connection layer is used to perform feature classification prediction on the pooling operation result, and a feature classification prediction result is obtained.

[0080] In step 303, the feature classification prediction result is compared with a pre-labeled feature classification result, and the classification accuracy and recall rate of the feature classification prediction result compared with the pre-labeled feature classification result are calculated according to the comparison result.

[0081] In step 304, if either the classification accuracy or the recall rate does not meet the preset requirement threshold, the parameters of the feature decoupling model are adjusted through a back propagation algorithm, and steps 301 to 303 are repeatedly executed until the classification accuracy and the recall rate both meet the preset requirement threshold.

[0082] In step 305, if the classification accuracy and the recall rate both meet the preset requirement threshold, a trained feature decoupling model is obtained.

[0083] The feature decoupling model is trained by using a supervised learning method, which ensures the availability of the feature decoupling model in subsequent actual speech conversion scenarios.

[0084] In the embodiment, after the step of obtaining the text features and the acoustic features decoupled and output by the trained feature decoupling model, the method further comprises: performing a marking operation on the text features and the acoustic features respectively according to different voice data entries. Specifically, the marking operation comprises: marking the text features and the acoustic features corresponding to the voice data of the same entry with the same identifier, and marking the text features and the acoustic features corresponding to the voice data of different entries with different identifiers.

[0085] The marking operation on the text features and the acoustic features respectively according to different voice data entries enables the text features and the acoustic features corresponding to the voice data of the same entry to be fused according to the same identifier in subsequent feature fusion model training, thereby playing a role in reconstructing the voice data in the voice training set.

[0086] In step 203, the text features corresponding to the first voice data and the acoustic features corresponding to the second voice data are input into the pre-trained feature fusion model for feature fusion, and a feature fusion result is obtained.

[0087] In the embodiment, before the step of inputting the text features corresponding to the first voice data and the acoustic features corresponding to the second voice data into the pre-trained feature fusion model, the method further comprises: obtaining text features and acoustic features to be fused, wherein the text features and the acoustic features to be fused include the text features and the acoustic features decoupled and output by the trained feature decoupling model; and inputting the text features and the acoustic features to be fused into a pre-constructed feature fusion model for feature fusion model training, and obtaining a trained feature fusion model.

[0088] By directly using the text features and the acoustic features decoupled and output by the trained feature decoupling model as the text features and the acoustic features to be fused, it is not necessary to collect or prepare training data again, and the voice training set obtained once is applied to the training of the feature decoupling model and the feature fusion model, thereby reducing the time consumption of training data collection to a certain extent.

[0089] In the embodiment, the feature fusion model comprises a text feature hidden state vector extraction component based on streaming and an acoustic feature hidden state vector extraction component based on posterior encoding.

[0090] Specifically, the text feature hidden state vector extraction component based on streaming can obtain the corresponding continuous speech text content through a linear projection layer after obtaining the language information (text feature), and then use the k-means algorithm to obtain discrete speech text content, and use streaming to obtain the hidden state vector representation corresponding to all speech text content from the discrete speech text content one by one, that is, to obtain the hidden state vector representation of the text feature.

[0091] Specifically, the acoustic feature hidden state vector extraction component based on posterior encoding can be implemented by using a pre-constructed posterior encoder, wherein the posterior encoder is constructed using a WaveGlow acoustic prosody generation component and a WaveNet residual block in a Glow-TTS text-to-speech generation model. The WaveNet residual block includes multiple layers of dilated convolution and gated activation units, and a skip connection. The last linear projection layer of these modules generates the mean and variance of the standard posterior distribution. For the case of multiple speakers, a global condition is used on the residual block to increase the embedding of the speaker's text feature and acoustic feature hidden state vector representation.

[0092] The WaveGlow acoustic prosody generation component is a network that synthesizes high-quality speech from a mel-spectrogram by relying on streaming. It generates fast, good, and high-quality acoustic prosody without the need for an autoregressive approach. The implementation only requires a single network, which is trained using only a single cost function. The training data is maximized to make the training process of the feature fusion model simple and stable. Glow-TTS is an efficient text-to-speech generation model. On the basis of the WaveGlow acoustic prosody generation component, it combines the WaveNet residual block of the Glow-TTS text-to-speech generation model to ensure that the training process of the feature fusion model is simple and stable. It can also use a global condition on the residual block to increase the embedding of the speaker's text feature and acoustic feature hidden state vector representation, making the trained feature fusion model more intelligent.

[0093] Continuing to refer to Figure 4 , Figure 4 is a flowchart of a specific embodiment of the feature fusion model training method described in the present application, which includes:

[0094] Step 401: Extract the hidden state vector representation of the text feature by the text feature hidden state vector extraction component based on streaming.

[0095] Step 402: Extract the hidden state vector representation of the acoustic feature by the acoustic feature hidden state vector extraction component based on posterior encoding.

[0096] Step 403, fusing the hidden state vector representation results corresponding to the text features and the acoustic features of the same identity respectively to obtain a feature fusion result;

[0097] Step 404, decoding the feature fusion result by using a preset speech decoder to obtain an output speech;

[0098] Step 405, calculating a loss value of the output speech compared with batch speech data in the speech training set according to a preset loss function;

[0099] Step 406, if the loss value does not satisfy a preset loss threshold, model tuning is performed by adjusting parameters of the feature fusion model, and steps 401 to 405 are repeatedly executed until the loss value satisfies the preset loss threshold;

[0100] Step 407, if the loss value satisfies the preset loss threshold, a trained feature fusion model is obtained.

[0101] In the embodiment, the preset loss function includes a cross-entropy loss function, a reconstruction loss function, an adversarial loss function, and a comprehensive loss function composed of the cross-entropy loss function, the reconstruction loss function, and the adversarial loss function. Here, the model training requirement and the actual situation can be freely set by a training operator. For example, the cross-entropy loss function can calculate the loss value of speech feature extraction, i.e., text feature extraction and acoustic feature extraction, the reconstruction loss function can calculate the loss value of the output speech compared with the speech training set after feature decoupling and feature fusion, and the adversarial loss function can calculate the loss value of the output speech compared with the speech training set after the feature decoupling model and the feature fusion model are used as a generator of an adversarial neural network.

[0102] By training the feature fusion model, the credibility of the feature fusion result in subsequent actual use is ensured. In addition, the text features are processed in a streaming manner and the acoustic features are processed in a posterior encoding manner in the feature fusion model, so that more hidden state vector representation results participate in the final speech synthesis encoding operation, the trained feature fusion model is more in line with the high fusion requirement, and the final output speech data in actual use is more credible.

[0103] Referring back to Figure 5 , Figure 5 is Figure 2 a flowchart of one specific embodiment of step 203, including:

[0104] Step 501, extracting a hidden state vector representation result of text features corresponding to the first speech data by using a streaming-based text feature hidden state vector extraction component of the trained feature fusion model;

[0105] Step 502, extracting the hidden state vector representation result of the acoustic feature corresponding to the second speech data by the posteriori coding-based acoustic feature hidden state vector extraction component of the trained feature fusion model;

[0106] Step 503, fusing the hidden state vector representation result of the text feature corresponding to the first speech data and the hidden state vector representation result of the acoustic feature corresponding to the second speech data, to obtain the feature fusion result.

[0107] Step 204, inputting the feature fusion result to a preset speech decoder to obtain the speech after pronunciation conversion.

[0108] The application obtains the speech to be converted and the reference speech of the target pronunciation user, inputs a pre-trained feature decoupling model, and obtains the decoupled features of the speech to be converted and the reference speech of the target pronunciation user, respectively. The text features of the speech to be converted and the acoustic features of the reference speech of the target pronunciation user are input into a pre-trained feature fusion model for feature fusion to obtain a feature fusion result, and the speech after pronunciation conversion is obtained by decoding. Through the feature decoupling model and the feature fusion model, the speech conversion is quickly realized, and the speech conversion method is applied to the financial customer service industry to provide customers with more high-quality and pleasant sound quality service functions.

[0109] The embodiments of the application can acquire and process related data based on artificial intelligence technology. Artificial intelligence (AI) is the use of digital computers or computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0110] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The software technology of artificial intelligence mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0111] In the embodiment of the present application, by obtaining the speech to be converted and the reference speech of the target pronunciation user, the pre-trained feature decoupling model is inputted to obtain the decoupling features of the speech to be converted and the reference speech of the target pronunciation user respectively; the text features of the speech to be converted and the acoustic features of the reference speech of the target pronunciation user are inputted into the pre-trained feature fusion model for feature fusion to obtain the feature fusion result, and the pronunciation conversion completed speech is obtained by decoding. Through the feature decoupling model and the feature fusion model, the speech conversion is quickly realized, and the speech conversion method is applied to the financial customer service industry to provide more high-quality and pleasant sound quality service functions for customers.

[0112] Further reference Figure 6 , as the implementation of the method shown in the above Figure 2 , the present application provides an embodiment of a speech conversion device, which corresponds to the method embodiment shown in Figure 2 , and the device can be applied to various electronic devices.

[0113] As shown in Figure 6 , the speech conversion device 600 described in the embodiment includes a speech data acquisition module 601, a speech feature decoupling module 602, a speech feature fusion module 603, and a speech feature decoding module 604. Among them:

[0114] The speech data acquisition module 601 is configured to acquire first speech data and second speech data, wherein the first speech data includes speech to be converted, and the second speech data includes reference speech of a target pronunciation user.

[0115] The speech feature decoupling module 602 is configured to input the first speech data and the second speech data into a pre-trained feature decoupling model to obtain decoupling features of the first speech data and the second speech data respectively, wherein the decoupling features include text features and acoustic features.

[0116] The speech feature fusion module 603 is configured to input the text features corresponding to the first speech data and the acoustic features corresponding to the second speech data into a pre-trained feature fusion model for feature fusion to obtain a feature fusion result.

[0117] The speech feature decoding module 604 is configured to input the feature fusion result into a preset speech decoder to obtain a pronunciation conversion completed speech by decoding.

[0118] In some specific embodiments of the present application, the speech conversion device 600 further includes a feature decoupling training module 605, which continues to refer to Figure 7 , Figure 7is a structural diagram of one specific embodiment of the feature decoupling training module 605. The feature decoupling training module 605 includes a feature extraction submodule 701, a feature classification prediction submodule 702, a comparison calculation submodule 703, a first tuning submodule 704, and a first training completion submodule 705. Among them:

[0119] The feature extraction submodule 701 is configured to perform feature extraction on the batch of voice data to be decoupled in feature in the voice training set respectively through the feature extraction layer, and obtain the corresponding feature extraction sequence.

[0120] The feature classification prediction submodule 702 is configured to perform a pooling operation on the feature extraction sequence by using the pooling layer, and perform feature classification prediction on the pooling operation result through the full connection layer, to obtain a feature classification prediction result.

[0121] The comparison calculation submodule 703 is configured to compare the feature classification prediction result with the pre-labeled feature classification result, and calculate the classification accuracy and recall rate of the feature classification prediction result compared with the pre-labeled feature classification result according to the comparison result.

[0122] The first tuning submodule 704 is configured to, if either the classification accuracy or the recall rate does not meet the preset requirement threshold, constantly adjust the parameters of the feature decoupling model through a back propagation algorithm, and repeatedly execute steps 301 to 303.

[0123] The first training completion submodule 705 is configured to, until the classification accuracy and the recall rate both meet the preset requirement threshold, obtain a trained feature decoupling model.

[0124] In some specific embodiments of the present application, the voice conversion device 600 further includes a feature fusion training module 606, which continues to refer to Figure 8 , Figure 8 is a structural diagram of one specific embodiment of the feature fusion training module 606. The feature fusion training module 606 includes a hidden state vector first extraction submodule 801, a hidden state vector second extraction submodule 802, a feature fusion submodule 803, a decoding output submodule 804, a loss value calculation submodule 805, a second tuning submodule 806, and a second training completion submodule 807. Among them:

[0125] The hidden state vector first extraction submodule 801 is configured to extract the hidden state vector representation result of the text feature through the streaming-based text feature hidden state vector extraction component.

[0126] The hidden state vector second extraction submodule 802 is configured to extract a hidden state vector representation result of the acoustic feature by using the hidden state vector extraction component based on the posteriori coding acoustic feature.

[0127] The feature fusion submodule 803 is configured to fuse the hidden state vector representation results of the text feature and the acoustic feature corresponding to the same identifier to obtain a feature fusion result.

[0128] The decoding output submodule 804 is configured to decode the feature fusion result by using a preset speech decoder to obtain an output speech.

[0129] The loss value calculation submodule 805 is configured to calculate a loss value of the output speech compared with the batch speech data in the speech training set according to a preset loss function.

[0130] The second tuning submodule 806 is configured to, if the loss value does not satisfy a preset loss threshold, perform model tuning by adjusting parameters of the feature fusion model, and repeatedly execute steps 401 to 405.

[0131] The second training completion submodule 807 is configured to, until the loss value satisfies the preset loss threshold, obtain a trained feature fusion model.

[0132] The application obtains the speech to be converted and the reference speech of the target pronunciation user, inputs a pre-trained feature decoupling model, and obtains the decoupled features of the speech to be converted and the reference speech of the target pronunciation user, respectively. The text feature of the speech to be converted and the acoustic feature of the reference speech of the target pronunciation user are input into a pre-trained feature fusion model for feature fusion to obtain a feature fusion result, and the converted speech is obtained by decoding. Through the feature decoupling model and the feature fusion model, the speech conversion is quickly realized. The speech conversion method is applied to the financial customer service industry, and can provide more high-quality and pleasant sound quality service functions for customers.

[0133] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by computer readable instructions instructing related hardware, and the computer readable instructions can be stored in a computer readable storage medium. When the program is executed, the processes of the above-mentioned embodiments can be included. The storage medium can be a non-volatile storage medium such as a magnetic disc, an optical disc, a read-only memory (ROM), or a random access memory (RAM).

[0134] It should be understood that although each step in the flowchart of the accompanying drawings is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless explicitly stated herein, the execution of these steps is not strictly limited in sequence, and they can be executed in other sequences. Moreover, at least part of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or sub-steps or stages of other steps.

[0135] To solve the above technical problems, the embodiments of the present application further provide a computer device. For details, please refer to Figure 9 , Figure 9 The basic structure block diagram of the computer device of the present embodiment is shown in FIG. 1.

[0136] The computer device 9 includes a memory 9a, a processor 9b, and a network interface 9c, which are connected to each other through a system bus. It should be noted that only the computer device 9 with components 9a-9c is shown in the figure, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented. Among them, those skilled in the art can understand that the computer device here is a device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0137] The computer device can be a desktop computer, a notebook computer, a palm computer, a cloud server, and other computing devices. The computer device can interact with the user through a keyboard, a mouse, a remote controller, a touchpad, a voice control device, and other means.

[0138] The memory 9a includes at least one type of readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 9a can be an internal storage unit of the computer device 9, such as a hard disk or a memory of the computer device 9. In other embodiments, the memory 9a can also be an external storage device of the computer device 9, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 9. Of course, the memory 9a can also include both an internal storage unit and an external storage device of the computer device 9. In this embodiment, the memory 9a is generally used to store an operating system and various application software installed on the computer device 9, such as computer readable instructions of a voice conversion method, etc. In addition, the memory 9a can also be used to temporarily store various data that have been output or will be output.

[0139] The processor 9b can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 9b is generally used to control the overall operation of the computer device 9. In this embodiment, the processor 9b is used to run computer readable instructions or process data stored in the memory 9a, such as computer readable instructions of the voice conversion method.

[0140] The network interface 9c can include a wireless network interface or a wired network interface, and is generally used to establish a communication connection between the computer device 9 and other electronic devices.

[0141] The computer device provided in the embodiment belongs to the technical field of speech synthesis and is applied to a sound conversion scene. The voice to be converted and the reference voice of a target pronunciation user are acquired, a pre-trained feature decoupling model is input, and decoupled features of the voice to be converted and the reference voice of the target pronunciation user are obtained respectively. The text features of the voice to be converted and the acoustic features of the reference voice of the target pronunciation user are input into a pre-trained feature fusion model, feature fusion is performed, a feature fusion result is obtained, and the voice after pronunciation conversion is obtained through decoding. Through the feature decoupling model and the feature fusion model, voice conversion is quickly realized. The voice conversion method is applied to the financial customer service industry, and more high-quality and pleasant voice service functions can be provided for customers.

[0142] The application also provides another implementation, namely providing a computer readable storage medium, the computer readable storage medium stores computer readable instructions, the computer readable instructions can be executed by a processor to make the processor execute the steps of the voice conversion method as described above.

[0143] The computer readable storage medium provided in the embodiment belongs to the technical field of speech synthesis and is applied to a sound conversion scene. The voice to be converted and the reference voice of a target pronunciation user are acquired, a pre-trained feature decoupling model is input, and decoupled features of the voice to be converted and the reference voice of the target pronunciation user are obtained respectively. The text features of the voice to be converted and the acoustic features of the reference voice of the target pronunciation user are input into a pre-trained feature fusion model, feature fusion is performed, a feature fusion result is obtained, and the voice after pronunciation conversion is obtained through decoding. Through the feature decoupling model and the feature fusion model, voice conversion is quickly realized. The voice conversion method is applied to the financial customer service industry, and more high-quality and pleasant voice service functions can be provided for customers.

[0144] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and a general hardware platform, of course, can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, an optical disk), and includes a plurality of instructions for making a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device) execute the methods described in the embodiments of the application.

[0145] Obviously, the above-described embodiments are only some embodiments but not all the embodiments of the present application, the preferred embodiments of the present application are shown in the drawings, but do not limit the patent scope of the present application. The present application can be implemented in many different forms, and conversely, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent replacements to some technical features therein. Any equivalent structure made by using the content of the specification and drawings, directly or indirectly applied to other related technical fields, is also within the patent protection scope of the present application.

Claims

1. A speech conversion method, characterized in that, Includes the following steps: Acquire first voice data and second voice data, wherein the first voice data includes the voice to be converted into speech, and the second voice data includes the reference voice of the target user; The first speech data and the second speech data are input into a pre-trained feature decoupling model to obtain decoupling features of the first speech data and the second speech data, respectively. The decoupling features include text features and acoustic features. The text features corresponding to the first speech data and the acoustic features corresponding to the second speech data are input together into a pre-trained feature fusion model to perform feature fusion and obtain the feature fusion result, specifically including: The latent state vector representation of the text features corresponding to the first speech data is extracted using the streaming text feature latent state vector extraction component of the trained feature fusion model. The latent state vector representation of the acoustic features corresponding to the second speech data is extracted using the acoustic feature latent state vector extraction component based on posterior coding of the trained feature fusion model. The feature fusion result is obtained by fusing the latent state vector representation of the text features corresponding to the first speech data and the latent state vector representation of the acoustic features corresponding to the second speech data. The feature fusion result is input into a preset speech decoder to decode and obtain the speech after pronunciation conversion.

2. The speech conversion method according to claim 1, characterized in that, Before performing the step of inputting the first speech data and the second speech data into the pre-trained feature decoupling model, the method further includes: Obtain a speech training set, wherein the speech training set contains a batch of speech data to be decoupled from features, which have been labeled and classified. The speech training set is input into a pre-constructed feature decoupling model to train the feature decoupling model, thereby obtaining the trained feature decoupling model; and Obtain the text features and acoustic features of the decoupled output of the trained feature decoupling model.

3. The speech conversion method according to claim 2, characterized in that, The pre-built feature decoupling model includes a feature extraction layer, a pooling layer, and a fully connected layer. The step of inputting the speech training set into the pre-built feature decoupling model for training to obtain the trained feature decoupling model specifically includes: Step 301: The feature extraction layer performs feature extraction on the speech data to be decoupled in batches in the speech training set to obtain the corresponding feature extraction sequence. Step 302: The pooling layer is used to perform a pooling operation on the feature extraction sequence, and the pooling operation result is then processed by the fully connected layer to perform feature classification prediction, thereby obtaining the feature classification prediction result. Step 303: Compare the feature classification prediction results with the pre-labeled feature classification results, and calculate the classification accuracy and recall of the feature classification prediction results compared with the pre-labeled feature classification results based on the comparison results; Step 304: If either the classification accuracy or recall does not meet the preset threshold, the parameters of the feature decoupling model are continuously adjusted through the backpropagation algorithm, and steps 301 to 303 are repeated until both the classification accuracy and recall meet the preset threshold. Step 305: If the classification accuracy and recall both meet the preset threshold requirements, then the trained feature decoupling model is obtained.

4. The speech conversion method according to claim 2, characterized in that, After performing the step of obtaining the text features and acoustic features of the decoupled output of the trained feature decoupling model, the method further includes: According to the different speech data entries, the text features and acoustic features are marked separately. Specifically, the marking operation includes: marking the text features and acoustic features corresponding to the same speech data entry with the same identifier, and marking the text features and acoustic features corresponding to different speech data entries with different identifiers.

5. The speech conversion method according to claim 4, characterized in that, Before performing the step of inputting the text features corresponding to the first speech data and the acoustic features corresponding to the second speech data into the pre-trained feature fusion model, the method further includes: Obtain text features and acoustic features to be fused, wherein the text features and acoustic features to be fused include the text features and acoustic features decoupled from the trained feature decoupling model; The text features and acoustic features to be fused are input into a pre-constructed feature fusion model, and the feature fusion model is trained to obtain a trained feature fusion model.

6. The speech conversion method according to claim 5, characterized in that, The feature fusion model includes a streaming-based text feature latent state vector extraction component and a posterior coding-based acoustic feature latent state vector extraction component. The step of inputting the text features and acoustic features to be fused into a pre-constructed feature fusion model, training the feature fusion model, and obtaining the trained feature fusion model specifically includes: Step 401: Extract the hidden state vector representation of the text features using the streaming-based text feature hidden state vector extraction component; Step 402: Extract the latent state vector representation of the acoustic features using the acoustic feature latent state vector extraction component based on posterior coding; Step 403: Fuse the latent state vector representations corresponding to the text features and acoustic features with the same identifier to obtain the feature fusion result; Step 404: Decode the feature fusion result using a preset speech decoder to obtain the output speech; Step 405: Calculate the loss value of the output speech relative to the batch speech data in the speech training set according to the preset loss function; Step 406: If the loss value does not meet the preset loss threshold, the model is tuned by adjusting the parameters of the feature fusion model, and steps 401 to 405 are repeated until the loss value meets the preset loss threshold. Step 407: If the loss value meets the preset loss threshold, the trained feature fusion model is obtained.

7. A voice conversion device, characterized in that, include: The voice data acquisition module is used to acquire first voice data and second voice data, wherein the first voice data includes the voice to be converted into speech, and the second voice data includes the reference voice of the target user. The speech feature decoupling module is used to input the first speech data and the second speech data into a pre-trained feature decoupling model to obtain decoupling features of the first speech data and the second speech data respectively, wherein the decoupling features include text features and acoustic features; The speech feature fusion module is used to input the text features corresponding to the first speech data and the acoustic features corresponding to the second speech data into a pre-trained feature fusion model to perform feature fusion and obtain the feature fusion result, specifically including: The latent state vector representation of the text features corresponding to the first speech data is extracted using the streaming text feature latent state vector extraction component of the trained feature fusion model. The latent state vector representation of the acoustic features corresponding to the second speech data is extracted using the acoustic feature latent state vector extraction component based on posterior coding of the trained feature fusion model. The feature fusion result is obtained by fusing the latent state vector representation of the text features corresponding to the first speech data and the latent state vector representation of the acoustic features corresponding to the second speech data. The speech feature decoding module is used to input the feature fusion result to a preset speech decoder to decode and obtain the speech after pronunciation conversion.

8. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the speech conversion method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the speech conversion method as described in any one of claims 1 to 6.

Images