Speech conversion methods, devices, computer equipment and storage media
By using a pre-trained speech conversion model, combined with a pre-encoder, speaker encoder, and decoder, the problems of slow speech conversion speed and poor quality in real-time Q&A scenarios in financial customer service are solved, achieving fast and accurate speech conversion and improving user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- PING AN TECH (SHENZHEN) CO LTD
- Filing Date
- 2024-08-26
- Publication Date
- 2026-07-17
AI Technical Summary
Existing voice conversion methods are slow in real-time Q&A scenarios for financial customer service, and they ignore specific details of the speaker, resulting in poor voice quality and a poor user experience.
A pre-trained speech conversion model is used to extract speech features through a pre-encoder, combine it with a speaker encoder to parse the Mel spectrum, and use a decoder to generate target speech data, thereby enhancing timbre information and achieving fast and accurate speech conversion.
This enables users to receive dedicated customer service without increasing labor costs, thereby improving the user experience and voice conversion quality of the financial system.
Smart Images
Figure CN119107960B_ABST
Abstract
Description
Technical Field
[0001] This application relates to speech processing technology, applied in the financial field, and particularly to a speech conversion method, apparatus, device, and storage medium. Background Technology
[0002] In the financial sector, when providing financial services, such as credit card user service inquiries, bill checks, and other daily services that require voice phone interaction, providing users with a dedicated customer service representative—meaning that every call is answered by the user's assigned representative—can significantly improve the user experience and satisfaction.
[0003] However, current speech conversion methods require processing the input speech frame by frame through a model, resulting in slow inference speed. In real-time question-and-answer scenarios like financial customer service, high speech conversion speed is crucial. Existing methods often overlook speaker-specific details during real-time speech conversion, leading to relatively poor speech quality and a less than ideal user experience. Summary of the Invention
[0004] This application provides a speech conversion method, apparatus, computer device, and storage medium, aiming to solve the problem that existing methods ignore speaker-specific details during real-time speech conversion, resulting in relatively poor speech quality and an unsatisfactory user experience, due to the high requirements for speech conversion speed in real-time question-and-answer scenarios such as financial customer service.
[0005] Firstly, this application provides a speech conversion method, including:
[0006] Obtain the raw audio data to be converted;
[0007] The raw speech data is input into the pre-encoder of the pre-trained speech conversion model. The pre-encoder extracts features from the raw speech data and outputs the first content information corresponding to the raw speech data.
[0008] Obtain the target Mel map of the target object corresponding to the speech conversion, input the target Mel map into the speaker encoder of the speech conversion model, the speaker encoder parses the target Mel map and outputs the first timbre information corresponding to the target object;
[0009] The first content information and the first timbre information are input into the decoder of the speech conversion model. The decoder parses the first content information and the first timbre information and outputs the converted target speech data.
[0010] In some embodiments, before inputting the original speech data into the pre-encoder of the pre-trained speech conversion model, the method further includes: inputting the original speech data into the pre-encoder of the speech conversion model to be trained to extract features from the original speech data and output second content information; extracting the original Mel spectrogram corresponding to the original speech data and inputting the original Mel spectrogram into the speaker encoder to be trained to parse the original Mel spectrogram and output second timbre information corresponding to the original speech data; inputting the second content information and the second timbre information into the decoder to be trained to parse the second content information and the second timbre information and output converted training speech data; and completing the training of the speech conversion model based on the training speech data and the original speech data.
[0011] For example, training a speech conversion model based on training speech data and original speech data includes: inputting the original speech data and training speech data into a preset discriminator, the discriminator discriminating the original speech data and training speech data and outputting discrimination information; and training the speech conversion model based on the discrimination information.
[0012] In some embodiments, the pre-encoder includes a feature extractor and a content encoder; the pre-encoder that inputs raw speech data into a pre-trained speech conversion model includes: inputting the raw speech data into the feature extractor, the feature extractor extracting features from the raw speech data and outputting a feature vector corresponding to the raw speech data; inputting the feature vector into the content encoder, the content encoder calculating the mean and variance of the feature vector, and outputting a normal distribution matrix corresponding to the raw speech data based on the mean and variance, the normal distribution matrix constituting first content information.
[0013] For example, the pre-encoder further includes a normalization stream module, which, before inputting the first content information and the first timbre information to the decoder of the speech conversion model, further includes: inputting the first content information and the first timbre information to the normalization stream module, wherein the normalization stream module normalizes the first content information and the first timbre information.
[0014] In some embodiments, before inputting the first content information and the first timbre information into the decoder of the speech conversion model, the method further includes: obtaining the target linear spectrum of the target object corresponding to the speech conversion; inputting the first timbre information and the target linear spectrum into the post-encoder of the speech conversion model, wherein the post-encoder enhances the first timbre information according to the target linear spectrum, so as to input the enhanced first timbre information into the decoder.
[0015] For example, the method further includes: extracting target customer service voice data corresponding to the target object from a preset financial customer service voice database; and extracting the target customer service voice data using Mel scale and linear scale respectively to obtain the target Mel spectrum and the target linear spectrum.
[0016] Secondly, this application provides a speech conversion device, including:
[0017] The data acquisition unit is used to acquire the raw speech data to be converted;
[0018] The data input unit is used to input the raw speech data into the pre-encoder of the pre-trained speech conversion model. The pre-encoder extracts features from the raw speech data and outputs the first content information corresponding to the raw speech data.
[0019] The spectrogram acquisition unit is used to acquire the target Mel spectrogram of the target object corresponding to the speech conversion, input the target Mel spectrogram into the speaker encoder of the speech conversion model, and the speaker encoder parses the target Mel spectrogram and outputs the first timbre information corresponding to the target object.
[0020] The speech conversion unit is used to input the first content information and the first timbre information into the decoder of the speech conversion model. The decoder parses the first content information and the first timbre information and outputs the converted target speech data.
[0021] Thirdly, this application also provides a computer device, comprising:
[0022] Memory and processor;
[0023] The memory is used to store computer programs;
[0024] The processor is configured to execute the computer program and, in executing the computer program, implement the steps of the speech conversion method as described in the first aspect above.
[0025] Fourthly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the steps of the speech conversion method described in the first aspect above.
[0026] This application discloses a speech conversion method, apparatus, computer device, and storage medium. First, the original speech data to be converted is acquired. Then, the original speech data is input into the pre-encoder of a pre-trained speech conversion model. The pre-encoder extracts features from the original speech data and outputs first content information corresponding to the original speech data. Next, the target Mel spectrum of the target object corresponding to the speech conversion is acquired and input into the speaker encoder of the speech conversion model. The speaker encoder parses the target Mel spectrum and outputs first timbre information corresponding to the target object. Finally, the first content information and the first timbre information are input into the decoder of the speech conversion model. The decoder parses the first content information and the first timbre information and outputs the converted target speech data. Therefore, the provided method can accurately and quickly convert current original speech data into target speech data for any target user in real time. This enables dedicated customer service to be provided to users in financial systems without increasing personnel, improving the user service experience without increasing labor costs.
[0027] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description
[0028] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0029] Figure 1 This is a schematic flowchart illustrating the steps of a speech conversion method provided in an embodiment of this application;
[0030] Figure 2 This is a schematic diagram of the structure of a speech conversion model provided in an embodiment of this application;
[0031] Figure 3 This is a schematic flowchart illustrating the steps of a speech conversion model training method provided in an embodiment of this application;
[0032] Figure 4 This is a schematic diagram of the structure of a speech conversion device provided in an embodiment of this application;
[0033] Figure 5 This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application.
[0034] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Detailed Implementation
[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0036] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.
[0037] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.
[0038] It should be understood that, in order to clearly describe the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with matching functions and effects. For example, the first timbre information and the second timbre information are only used to distinguish different timbre information and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.
[0039] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0040] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.
[0041] With the development of Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) technologies, high-quality speech conversion (VC) can be achieved by reconstructing waveforms by extracting source content information and target speaker information. However, current methods still need improvement in inference speed. ASR techniques are widely used to extract language content from speech, ignoring speaker-specific details, resulting in relatively poor speech quality. While VC models capable of producing perceptually good speech exist, research on high-quality, fast VC is still lacking. In practical applications, real-time VC requires frame-by-frame processing of input speech, making model inference speed crucial. However, previous research in real-time VC has not yet achieved sufficient naturalness and similarity in synthesized speech.
[0042] Meanwhile, when providing financial services, such as credit card user inquiries, bill checks, and problem feedback that require voice call interaction, we can offer users a dedicated customer service representative. This means that each call will be answered by a designated representative. The representative can be a real person or an AI-powered system. Currently, the voice conversion process suffers from poor quality in cross-gender conversion, with male and female voices having different tones and high levels of distortion.
[0043] To resolve the above issues, please refer to [link / reference]. Figure 1 , Figure 1 This is a schematic flowchart of a speech conversion method provided in one embodiment of this application. The speech conversion method can be implemented by a computer device, which can be deployed on a single server or a server cluster. It can also be deployed on a handheld terminal, laptop, wearable device, or robot, etc.
[0044] It should be noted that the provided method can be applied to the financial field as well as the medical field, and can achieve good results in any application scenario that requires customized customer service for users. The embodiments of this application do not limit the application scenarios of the provided method.
[0045] like Figure 1 As shown, the speech conversion method provided in this embodiment includes steps S101 to S104. Details are as follows:
[0046] S101. Obtain the raw speech data to be converted.
[0047] Specifically, in a financial scenario, the method acquires raw voice data input by any customer service representative as the data to be converted. This raw voice data can then be transformed into target voice data for the corresponding target customer service representative / user, enabling rapid conversion of raw voice data. Furthermore, the provided method can convert any raw voice data into target voice data for any target user of the same or different gender, achieving personalized customization for users within the financial system or any other service system.
[0048] S102. Input the raw speech data into the pre-encoder of the pre-trained speech conversion model. The pre-encoder extracts features from the raw speech data and outputs the first content information corresponding to the raw speech data.
[0049] Specifically, by inputting the raw speech data into a pre-trained speech conversion model, such as... Figure 2 In the pre-encoder shown, the pre-encoder can extract features from the original speech data and extract the content representation corresponding to the original speech data as the first content information.
[0050] S103. Obtain the target Mel map of the target object corresponding to the speech conversion, input the target Mel map into the speaker encoder of the speech conversion model, the speaker encoder parses the target Mel map and outputs the first timbre information corresponding to the target object.
[0051] Specifically, by obtaining the target Mel map corresponding to the target object in this speech conversion, the target Mel map can be input into a system such as... Figure 2 The speaker encoder in the speech conversion model shown can then be used to analyze the target Mel spectrum, confirming that the target Mel spectrum represents the target user's primary timbre information. This timbre information can include arbitrary parameters such as pitch and rhythm. Therefore, the provided method can determine the timbre to be converted, ensuring accurate conversion of the original speech data.
[0052] In some embodiments, the pre-encoder includes a feature extractor and a content encoder; the pre-encoder that inputs raw speech data into a pre-trained speech conversion model includes: inputting the raw speech data into the feature extractor, the feature extractor extracting features from the raw speech data and outputting a feature vector corresponding to the raw speech data; inputting the feature vector into the content encoder, the content encoder calculating the mean and variance of the feature vector, and outputting a normal distribution matrix corresponding to the raw speech data based on the mean and variance, the normal distribution matrix constituting first content information.
[0053] like Figure 2 As shown, the pre-encoder includes at least a feature extractor ( Figure 2The Hubert-Soft is a feature extractor based on Hubert-Base. It takes the original waveform as input and a content encoder, and then extracts the feature vectors corresponding to the original speech data and calculates the mean and variance of its normal posterior distribution to construct the corresponding normal distribution matrix as content information.
[0054] For example, the pre-encoder further includes a normalization stream module, which, before inputting the first content information and the first timbre information to the decoder of the speech conversion model, further includes: inputting the first content information and the first timbre information to the normalization stream module, wherein the normalization stream module normalizes the first content information and the first timbre information.
[0055] The normalized flow module is a set of affine coupling layers. This flow module is used to increase the complexity of the prior distribution (i.e., content information). The condition is that the speaker embedding g (in this process, g refers to...) Figure 2 The normalized flow module generates samples by learning an invertible mapping from a simple random noise distribution to a complex target distribution. This invertible mapping consists of a series of invertible transformations, which can be affine transformations and nonlinear function transformations. Furthermore, the normalized flow module can normalize the pitch and variable information of speakers of different genders and from different speakers, reducing variable factors.
[0056] In some embodiments, the speaker encoder generates an encoded speaker representation from the utterance. The speaker encoder can be implemented using four convolutional layers plus two LSTM (Long Short-Term Memory) layers, or it can be a simple LSTM layer plus a fully connected layer. It is trained from scratch along with the rest of the model. The network structure of the speaker encoder includes one LSTM layer and one fully connected layer. The purpose of speaker encoding is to encode different speakers. For example, a simple speaker, Zhang San, can be represented as a numerical vector 1 or a two-digit number [1 0]. We use the target Mel spectrogram as the input to the speaker encoder. The audio signal represents all the information of the speaker, including whether it is Zhang San or Li Si, but we do not know who they are. However, we can extract features from the audio to represent the speech of the same speaker as the same vector without relying on textual name information, etc. We assume that the output of the pre-encoder does not contain any speaker information. Then, the speech conversion model replaces the missing speaker information according to the input of the speaker encoder to synthesize speech.
[0057] S104. Input the first content information and the first timbre information into the decoder of the speech conversion model. The decoder parses the first content information and the first timbre information and outputs the converted target speech data.
[0058] Specifically, by inputting the first content information and the first timbre information into the decoder of the speech conversion model, the decoder can synthesize target speech data by parsing the first content information and the first timbre information. Therefore, the provided method can accurately and quickly convert the current raw speech data into target speech data for any target user in real time. This enables the provision of dedicated customer service to users in financial systems without increasing personnel, improving the user service experience without increasing labor costs.
[0059] In some embodiments, the decoder is as follows: Figure 2 The MS-iSTFT (MS-Inverse Short-Time Fourier Transform) decoder in the code. The decoder performs the following steps in sequence:
[0060] First, the variational autoencoder (VAE) latent variable Z (composed of first content information and first timbre information) is embedded in the speaker using g (in this process, g refers to...). Figure 2 The first timbre information corresponding to the target Mel spectrum is used as a condition, and then upsampled through an upsampled convolutional residual block sequence. The upsampled Z is then projected onto the amplitude and phase variables of each sub-band signal.
[0061] Then, using amplitude and phase variables, an iSTFT (inverse short-time Fourier transform) operation is performed to generate each sub-band signal.
[0062] Finally, these sub-band signals are upsampled to match the sampling rate of the original signal by inserting zeros between samples, and then integrated into the full-band waveform using a trainable synthesis filter to synthesize the target speech data.
[0063] In some embodiments, please refer to Figure 3 ,like Figure 3 As shown, the provided speech conversion model training method includes the above steps S101 and S105 to S108.
[0064] S105. Input the raw speech data into the pre-encoder of the speech conversion model to be trained to extract features from the raw speech data and output the second content information.
[0065] S106. Extract the original Mel graph corresponding to the original speech data, input the original Mel graph into the speaker encoder to be trained, and parse the original Mel graph to output the second timbre information corresponding to the original speech data.
[0066] S107. Input the second content information and the second timbre information into the decoder to be trained, so as to parse the second content information and the second timbre information and output the converted training speech data.
[0067] S108. Train the speech conversion model based on the training speech data and the original speech data.
[0068] Furthermore, the provided method can be based on the original Mel spectrogram corresponding to the original speech data input by the speaker encoder (i.e., g refers to during training). Figure 2 (The second timbre information of the source Mel spectrum corresponding to the original Mel spectrum). This is used to determine the deviation between the converted training speech data and the original speech data of the corresponding target user, thereby completing the training of the model.
[0069] Among them, such as Figure 2 As shown, the content encoder in the pre-encoder aims to extract information related to content representation from the audio. To achieve the purpose of the content encoder, this application trains the audio using different speech content from the same speaker. Since the two contents are different, the extracted content encoding should be different and complete. Since it is the same speaker, the speaker encoder vector should be the same or consistent. The content encoding vector and the speaker vector are swapped to reconstruct the speech content. If reconstruction is successful, it means that the content encoder has achieved the expected effect. During the training process, the model is continuously trained to eventually realize the content encoder and the speaker encoder.
[0070] For example, training a speech conversion model based on training speech data and original speech data includes: inputting the original speech data and training speech data into a preset discriminator, the discriminator discriminating the original speech data and training speech data and outputting discrimination information; and training the speech conversion model based on the discrimination information.
[0071] Furthermore, through the introduction of a discriminator, such as Figure 2 As shown, by combining the principle of adversarial generation, we can determine the authenticity of the original speech data and the training speech data. Furthermore, adversarial training can enhance the timbre and pitch of the synthesized training speech data to be more similar to the timbre and pitch of the original speech data.
[0072] In some embodiments, before inputting the first content information and the first timbre information into the decoder of the speech conversion model, the method further includes: obtaining the target linear spectrum of the target object corresponding to the speech conversion; inputting the first timbre information and the target linear spectrum into the post-encoder of the speech conversion model, wherein the post-encoder enhances the first timbre information according to the target linear spectrum, so as to input the enhanced first timbre information into the decoder.
[0073] like Figure 2 As shown, the provided method further enhances the first timbre information through a post-encoder. Figure 2 (g) is used to improve the accuracy of the final converted target speech data.
[0074] For example, the method further includes: extracting target customer service voice data corresponding to the target object from a preset financial customer service voice database; and extracting the target customer service voice data using both Mel-scale and linear-scale methods to obtain the target Mel-scale spectrum and the target linear spectrum. Thus, the provided method can obtain the target Mel-scale spectrum and the target linear spectrum corresponding to the target customer service voice data based on Mel-scale and linear-scale methods respectively, ensuring that the original voice data can be converted into target voice data for any target user at any time.
[0075] In some embodiments, such as Figure 2 As shown, the provided speech conversion model based on inverse short-time Fourier transform can be the QuickVC (Quick Vociie Conversion) model. It consists of a speaker encoder, a pre-encoder, a post-encoder, an MS-iSTFT decoder, and a discriminator. The structures of the post-encoder and discriminator follow VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech, a highly expressive speech synthesis model combining variational inference, normalizing flows, and adversarial training). QuickVC inherits its backbone from VITS, employs variational inference, and adds a normalization flow module and an adversarial training process.
[0076] Please see Figure 4 As shown, Figure 4 This is a schematic diagram of the structure of the voice conversion device 200 provided in the embodiments of this application. The voice conversion device 200 is used to perform the steps of the voice conversion method shown in the above embodiments. The voice conversion device 200 can be a single server or a server cluster, or the voice conversion device 200 can be a terminal, such as a handheld terminal, a laptop computer, a wearable device, or a robot.
[0077] like Figure 4 As shown, the voice conversion device 200 includes:
[0078] The data acquisition unit 201 is used to acquire the raw speech data to be converted;
[0079] The data input unit 202 is used to input the original speech data into the pre-encoder of the pre-trained speech conversion model. The pre-encoder performs feature extraction on the original speech data and outputs the first content information corresponding to the original speech data.
[0080] The spectrogram acquisition unit 203 is used to acquire the target Mel spectrogram of the target object corresponding to the speech conversion, input the target Mel spectrogram into the speaker encoder of the speech conversion model, and the speaker encoder parses the target Mel spectrogram and outputs the first timbre information corresponding to the target object.
[0081] The speech conversion unit 204 is used to input the first content information and the first timbre information into the decoder of the speech conversion model. The decoder parses the first content information and the first timbre information and outputs the converted target speech data.
[0082] The aforementioned speech conversion method can be implemented as a computer program, which can be used in various ways, such as... Figure 4 It runs on the device shown.
[0083] Please see Figure 5 , Figure 5 This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application. The computer device includes a processor, a memory, and a network interface connected via a device bus, wherein the memory may include a storage medium and internal memory.
[0084] The storage medium may store operating devices and computer programs. The computer program includes program instructions that, when executed, cause the processor to perform any speech conversion method.
[0085] The processor provides computing and control capabilities, supporting the operation of the entire computer device.
[0086] Internal memory provides an environment for the execution of computer programs stored in non-volatile storage media. When these computer programs are executed by a processor, the processor can perform any speech conversion method.
[0087] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the terminal to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0088] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.
[0089] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps:
[0090] Obtain the raw audio data to be converted;
[0091] The raw speech data is input into the pre-encoder of the pre-trained speech conversion model. The pre-encoder extracts features from the raw speech data and outputs the first content information corresponding to the raw speech data.
[0092] Obtain the target Mel map of the target object corresponding to the speech conversion, input the target Mel map into the speaker encoder of the speech conversion model, the speaker encoder parses the target Mel map and outputs the first timbre information corresponding to the target object;
[0093] The first content information and the first timbre information are input into the decoder of the speech conversion model. The decoder parses the first content information and the first timbre information and outputs the converted target speech data.
[0094] In some embodiments, before inputting the original speech data into the pre-encoder of the pre-trained speech conversion model, the method further includes: inputting the original speech data into the pre-encoder of the speech conversion model to be trained to extract features from the original speech data and output second content information; extracting the original Mel spectrogram corresponding to the original speech data and inputting the original Mel spectrogram into the speaker encoder to be trained to parse the original Mel spectrogram and output second timbre information corresponding to the original speech data; inputting the second content information and the second timbre information into the decoder to be trained to parse the second content information and the second timbre information and output converted training speech data; and completing the training of the speech conversion model based on the training speech data and the original speech data.
[0095] For example, training a speech conversion model based on training speech data and original speech data includes: inputting the original speech data and training speech data into a preset discriminator, the discriminator discriminating the original speech data and training speech data and outputting discrimination information; and training the speech conversion model based on the discrimination information.
[0096] In some embodiments, the pre-encoder includes a feature extractor and a content encoder; the pre-encoder that inputs raw speech data into a pre-trained speech conversion model includes: inputting the raw speech data into the feature extractor, the feature extractor extracting features from the raw speech data and outputting a feature vector corresponding to the raw speech data; inputting the feature vector into the content encoder, the content encoder calculating the mean and variance of the feature vector, and outputting a normal distribution matrix corresponding to the raw speech data based on the mean and variance, the normal distribution matrix constituting first content information.
[0097] For example, the pre-encoder further includes a normalization stream module, which, before inputting the first content information and the first timbre information to the decoder of the speech conversion model, further includes: inputting the first content information and the first timbre information to the normalization stream module, wherein the normalization stream module normalizes the first content information and the first timbre information.
[0098] In some embodiments, before inputting the first content information and the first timbre information into the decoder of the speech conversion model, the method further includes: obtaining the target linear spectrum of the target object corresponding to the speech conversion; inputting the first timbre information and the target linear spectrum into the post-encoder of the speech conversion model, wherein the post-encoder enhances the first timbre information according to the target linear spectrum, so as to input the enhanced first timbre information into the decoder.
[0099] For example, the method further includes: extracting target customer service voice data corresponding to the target object from a preset financial customer service voice database; and extracting the target customer service voice data using Mel scale and linear scale respectively to obtain the target Mel spectrum and the target linear spectrum.
[0100] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement the steps of the speech conversion method provided in the above embodiments of this application.
[0101] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.
[0102] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A speech conversion method, characterized in that, include: Obtain the raw audio data to be converted; The original speech data is input into the pre-encoder of the speech conversion model to be trained to extract features from the original speech data and output second content information; the original Mel spectrogram corresponding to the original speech data is extracted and input into the speaker encoder to be trained to parse the original Mel spectrogram and output second timbre information corresponding to the original speech data. The second content information and the second timbre information are input into the decoder to be trained to parse the second content information and the second timbre information and output the converted training speech data. The speech conversion model is trained based on the training speech data and the original speech data. The pre-encoder includes a feature extractor and a content encoder; The process of inputting raw speech data into the pre-encoder of a pre-trained speech conversion model includes: inputting raw speech data into a feature extractor, the feature extractor extracting features from the raw speech data and outputting feature vectors corresponding to the raw speech data; inputting the feature vectors into a content encoder, the content encoder calculating the mean and variance of the feature vectors, and outputting a normal distribution matrix corresponding to the raw speech data based on the mean and variance, the normal distribution matrix constituting the first content information; The original speech data is input into the pre-encoder of the pre-trained speech conversion model. The pre-encoder extracts features from the original speech data and outputs the first content information corresponding to the original speech data. Obtain the target Mel map of the target object corresponding to the speech conversion, input the target Mel map into the speaker encoder of the speech conversion model, the speaker encoder parses the target Mel map and outputs the first timbre information corresponding to the target object; The first content information and the first timbre information are input into the decoder of the speech conversion model. The decoder parses the first content information and the first timbre information and outputs the converted target speech data.
2. The method according to claim 1, characterized in that, The step of training the speech conversion model based on the training speech data and the original speech data includes: The original speech data and the training speech data are input into a preset discriminator, which discriminates the original speech data and the training speech data and outputs discrimination information. The speech conversion model is trained based on the identification information.
3. The method according to claim 1, characterized in that, The pre-encoder includes a feature extractor and a content encoder; The step of inputting the raw speech data into the pre-encoder of the pre-trained speech conversion model includes: The original speech data is input into the feature extractor, which performs feature extraction on the original speech data and outputs the feature vector corresponding to the original speech data. The feature vector is input to the content encoder, which calculates the mean and variance of the feature vector and outputs the normal distribution matrix corresponding to the original speech data based on the mean and variance. The normal distribution matrix constitutes the first content information.
4. The method according to claim 3, characterized in that, The pre-encoder further includes a normalized stream module, which, before inputting the first content information and the first timbre information to the decoder of the speech conversion model, also includes: The first content information and the first timbre information are input to the normalization stream module, which normalizes the first content information and the first timbre information.
5. The method according to claim 1, characterized in that, Before inputting the first content information and the first timbre information into the decoder of the speech conversion model, the method further includes: Obtain the target linear spectrum of the target object corresponding to the speech conversion; The first timbre information and the target linear spectrum are input to the post-encoder of the speech conversion model. The post-encoder enhances the first timbre information according to the target linear spectrum, so as to input the enhanced first timbre information to the decoder.
6. The method according to claim 5, characterized in that, The method further includes: Extract the target customer service voice data corresponding to the target object from the preset financial customer service voice database; Based on the target customer service voice data, Mel-scale and linear-scale extractions are performed to obtain the target Mel spectrum and target linear spectrum.
7. A voice conversion device, characterized in that, include: The data acquisition unit is used to acquire the raw speech data to be converted; The original speech data is input into the pre-encoder of the speech conversion model to be trained to extract features from the original speech data and output second content information; the original Mel spectrogram corresponding to the original speech data is extracted and input into the speaker encoder to be trained to parse the original Mel spectrogram and output second timbre information corresponding to the original speech data. The second content information and the second timbre information are input into the decoder to be trained to parse the second content information and the second timbre information and output the converted training speech data. The speech conversion model is trained based on the training speech data and the original speech data. The pre-encoder includes a feature extractor and a content encoder; The process of inputting raw speech data into the pre-encoder of a pre-trained speech conversion model includes: inputting raw speech data into a feature extractor, the feature extractor extracting features from the raw speech data and outputting feature vectors corresponding to the raw speech data; inputting the feature vectors into a content encoder, the content encoder calculating the mean and variance of the feature vectors, and outputting a normal distribution matrix corresponding to the raw speech data based on the mean and variance, the normal distribution matrix constituting the first content information; The data input unit is used to input the raw speech data into the pre-encoder of the pre-trained speech conversion model. The pre-encoder extracts features from the raw speech data and outputs the first content information corresponding to the raw speech data. The spectrogram acquisition unit is used to acquire the target Mel spectrogram of the target object corresponding to the speech conversion, input the target Mel spectrogram into the speaker encoder of the speech conversion model, and the speaker encoder parses the target Mel spectrogram and outputs the first timbre information corresponding to the target object. The speech conversion unit is used to input the first content information and the first timbre information into the decoder of the speech conversion model. The decoder parses the first content information and the first timbre information and outputs the converted target speech data.
8. A computer device, characterized in that, The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and, in executing the computer program, implement the speech conversion method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement the speech conversion method as described in any one of claims 1 to 6.