Voice conversion method and device, computer equipment and storage medium

Abstract

The invention discloses a voice conversion method and device, computer equipment and a storage medium. Obtaining original voice data to be converted; inputting the original voice data into a pre-encoder of a pre-trained voice conversion model, performing feature extraction on the original voice data by the pre-encoder, and outputting first content information corresponding to the original voice data; obtaining a target Mel map of a target object corresponding to voice conversion, inputting the target Mel map into a speaker encoder of the voice conversion model, analyzing the target Mel map by the speaker encoder, and outputting first tone information corresponding to the target object; and inputting the first content information and the first timbre information into a decoder of the voice conversion model, analyzing the first content information and the first timbre information by the decoder, and outputting converted target voice data. The current original voice data can be accurately and quickly converted into the target voice data of any target user in real time.

Description

Technical Field

[0001] The present application relates to speech processing technology, which is applied in the financial field, and in particular to a speech conversion method, device, equipment and storage medium. Background Art

[0002] When providing financial services in the financial field, such as credit card user service inquiries, bill checking and other daily service scenarios that require voice telephone interaction, by providing users with exclusive customer service, that is, the voice of each call is the voice of the user's designated customer service, the user experience and satisfaction can be greatly improved.

[0003] However, current speech conversion methods require the model to process the input speech frame by frame, resulting in a slow model inference speed. Financial customer service, a real-time question-and-answer scenario, has high requirements for speech conversion speed. Existing methods ignore specific details of the speaker during real-time speech conversion, resulting in relatively poor quality of generated speech, which leads to a less than ideal user experience. Summary of the invention

[0004] The present application provides a speech conversion method, apparatus, computer equipment and storage medium, aiming to solve the problem that due to the high requirements on speech conversion speed in the real-time question-and-answer scenario of financial customer service, the existing methods ignore the speaker's specific details in the real-time speech conversion process, and the generated speech quality is relatively poor, resulting in a less than ideal user experience.

[0005] In a first aspect, the present application provides a speech conversion method, comprising:

[0006] Obtaining original voice data to be converted;

[0007] Inputting the original speech data into a pre-encoder of a pre-trained speech conversion model, the pre-encoder extracts features from the original speech data, and outputs first content information corresponding to the original speech data;

[0008] Obtaining a target Mel-atlas of a target object corresponding to speech conversion, inputting the target Mel-atlas into a speaker encoder of a speech conversion model, and the speaker encoder parsing the target Mel-atlas to output first timbre information corresponding to the target object;

[0009] The first content information and the first timbre information are input to a decoder of a speech conversion model, and the decoder analyzes the first content information and the first timbre information and outputs converted target speech data.

[0010] In some embodiments, before the original speech data is input into the pre-encoder of the pre-trained speech conversion model, it also includes: inputting the original speech data into the pre-encoder of the speech conversion model to be trained to extract features from the original speech data and output second content information; extracting the original Mel-gram spectrum corresponding to the original speech data, inputting the original Mel-gram spectrum into the speaker encoder to be trained to parse the original Mel-gram spectrum, and outputting the second timbre information corresponding to the original speech data; inputting the second content information and the second timbre information into the decoder to be trained to parse the second content information and the second timbre information, and outputting the converted training speech data; completing the training of the speech conversion model according to the training speech data and the original speech data.

[0011] Exemplarily, the training of the speech conversion model is completed based on the training speech data and the original speech data, including: inputting the original speech data and the training speech data into a preset discriminator, the discriminator discriminates the original speech data and the training speech data, and outputs discrimination information; and completing the training of the speech conversion model based on the discrimination information.

[0012] In some embodiments, the pre-encoder includes a feature extractor and a content encoder; the pre-encoder inputs the original speech data into a pre-trained speech conversion model, including: inputting the original speech data into the feature extractor, the feature extractor extracts features from the original speech data, and outputs a feature vector corresponding to the original speech data; inputting the feature vector into the content encoder, the content encoder calculates the mean and variance of the feature vector, and outputs a normal distribution matrix corresponding to the original speech data based on the mean and variance, and the normal distribution matrix constitutes the first content information.

[0013] Exemplarily, the pre-encoder also includes a normalization stream module, and before the first content information and the first timbre information are input into the decoder of the speech conversion model, it also includes: inputting the first content information and the first timbre information into the normalization stream module, and the normalization stream module normalizes the first content information and the first timbre information.

[0014] In some embodiments, before the first content information and the first timbre information are input into the decoder of the speech conversion model, it also includes: obtaining a target linear spectrum of the target object corresponding to the speech conversion; inputting the first timbre information and the target linear spectrum into the post encoder of the speech conversion model, and the post encoder enhances the first timbre information according to the target linear spectrum to input the enhanced first timbre information into the decoder.

[0015] Exemplarily, the method also includes: extracting target customer service voice data corresponding to the target object from a preset financial customer service voice database; extracting the Mel scale and linear scale respectively according to the target customer service voice data to obtain a target Mel spectrum and a target linear spectrum.

[0016] In a second aspect, the present application provides a speech conversion device, comprising:

[0017] A data acquisition unit, used to acquire original speech data to be converted;

[0018] A data input unit, configured to input the original speech data into a pre-encoder of a pre-trained speech conversion model, wherein the pre-encoder extracts features from the original speech data and outputs first content information corresponding to the original speech data;

[0019] A graph acquisition unit, used for acquiring a target Mel-gram graph of a target object corresponding to speech conversion, inputting the target Mel-gram graph into a speaker encoder of the speech conversion model, wherein the speaker encoder parses the target Mel-gram graph and outputs first timbre information corresponding to the target object;

[0020] The speech conversion unit is used to input the first content information and the first timbre information into the decoder of the speech conversion model, and the decoder parses the first content information and the first timbre information and outputs the converted target speech data.

[0021] In a third aspect, the present application further provides a computer device, including:

[0022] Memory and processor;

[0023] The memory is used to store computer programs;

[0024] The processor is used to execute the computer program and implement the steps of the speech conversion method described in the first aspect when executing the computer program.

[0025] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the steps of the speech conversion method as described in the first aspect above.

[0026] The present application discloses a method, device, computer equipment and storage medium for voice conversion. First, the original voice data to be converted is obtained; then, the original voice data is input into the pre-encoder of the pre-trained voice conversion model, the pre-encoder extracts features from the original voice data, and outputs the first content information corresponding to the original voice data; then, the target Mel-map of the target object corresponding to the voice conversion is obtained, the target Mel-map is input into the speaker encoder of the voice conversion model, the speaker encoder parses the target Mel-map, and outputs the first timbre information corresponding to the target object; finally, the first content information and the first timbre information are input into the decoder of the voice conversion model, the decoder parses the first content information and the first timbre information, and outputs the converted target voice data. The method provided can accurately and quickly convert the current original voice data into the target voice data of any target user in real time. It enables the financial system to provide exclusive customer service to users without adding personnel, and improves the user's service experience without increasing the cost of employment.

[0027] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 is a schematic flow chart of the steps of a voice conversion method provided by an embodiment of the present application;

[0030] Figure 2 is a structural diagram of a speech conversion model provided in an embodiment of the present application;

[0031] Figure 3 is a flowchart showing the steps of a speech conversion model training method provided in one embodiment of the present application;

[0032] Figure 4 is a structural diagram of a speech conversion device provided in one embodiment of the present application;

[0033] Figure 5 It is a schematic block diagram of the structure of a computer device provided in one embodiment of the present application.

[0034] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0037] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0038] It should be understood that, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with matching functions and effects. For example, the first timbre information and the second timbre information are only used to distinguish different timbre information, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0039] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0040] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0041] With the development of Automatic Speech Recognition (ASR) and Text To Speech (TTS) technologies, high-quality voice conversion (VC) can be achieved by extracting source content information and target speaker information to reconstruct waveforms. However, current methods still need to be improved in terms of inference speed. Technologies in the field of ASR are widely used to extract language content from speech, while ignoring speaker-specific details, and the generated speech quality is relatively poor. Although there are VC models that can produce speech with good perceptual quality, research on high-quality and fast VC is relatively lacking. In practical applications, if real-time VC is to be achieved, the input speech needs to be processed frame by frame, and the inference speed of the model is important for real-time VC. However, previous research in real-time VC has not done well enough in terms of the naturalness and similarity of the synthesized speech.

[0042] At the same time, when providing financial services, such as credit card user service inquiries, bill checking, and problem feedback, which require voice and telephone interaction, we can provide users with exclusive customer service services, that is, the voice of each call is the voice of the designated customer service. The customer service can be a real customer service or an intelligent customer service. At present, the quality of cross-gender conversion is poor in the voice conversion process, the pitch of male and female voices is different, and the cross-gender distortion is high.

[0043] To solve the above problems, please refer to Figure 1 , Figure 1 1 is a schematic flow chart of a voice conversion method provided by an embodiment of the present application. The voice conversion method can be implemented by a computer device, which can be deployed on a single server or a server cluster. It can also be deployed on a handheld terminal, a laptop computer, a wearable device or a robot, etc.

[0044] It should be noted that the provided method can be applied to the financial field as well as the medical field, and can achieve good results in any application scenario that requires, such as developing user-exclusive customer service. The embodiments of this application do not limit the application scenarios of the provided method.

[0045] like Figure 1 As shown, the voice conversion method provided in this embodiment includes steps S101 to S104. The details are as follows:

[0046] S101. Obtain original speech data to be converted.

[0047] Specifically, by obtaining the original voice data input by any customer service as the data to be converted in a financial scenario, the original voice data can be converted into the target voice data of the corresponding target customer service / user, and the conversion of the original voice data can be quickly realized. The provided method can convert any original voice data into the target voice data of any target user with the same or different gender, and realize the personalized customization of the financial system or any other service system for users.

[0048] S102. Input the original speech data into a pre-encoder of a pre-trained speech conversion model, and the pre-encoder extracts features from the original speech data and outputs first content information corresponding to the original speech data.

[0049] Specifically, by inputting the original speech data into the pre-trained speech conversion model such as Figure 2 In the pre-encoder shown, the pre-encoder can further perform feature extraction on the original speech data, and extract the content representation corresponding to the original speech data as the first content information.

[0050] S103. Obtain a target mel-atlas of a target object corresponding to speech conversion, input the target mel-atlas to a speaker encoder of a speech conversion model, and the speaker encoder parses the target mel-atlas to output first timbre information corresponding to the target object.

[0051] Specifically, by obtaining the target Mel-map corresponding to the target object of this speech conversion, the target Mel-map can be input into Figure 2 The speaker encoder of the voice conversion model shown in the figure can then parse the target Mel-gram spectrogram through the speaker encoder to confirm that the target Mel-gram spectrogram is the first timbre information of the target user. The timbre information can include any parameters such as pitch and rhythm. The provided method can then determine the timbre to be converted, ensuring that the original voice data can be accurately converted.

[0052] In some embodiments, the pre-encoder includes a feature extractor and a content encoder; the pre-encoder inputs the original speech data into a pre-trained speech conversion model, including: inputting the original speech data into the feature extractor, the feature extractor extracts features from the original speech data, and outputs a feature vector corresponding to the original speech data; inputting the feature vector into the content encoder, the content encoder calculates the mean and variance of the feature vector, and outputs a normal distribution matrix corresponding to the original speech data based on the mean and variance, and the normal distribution matrix constitutes the first content information.

[0053] like Figure 2 As shown, the pre-encoder includes at least a feature extractor ( Figure 2HuBERT-Soft in HuBERT-Soft is a feature extractor based on HuBERT-Base, which takes the original waveform as input) and content encoder, and can then extract the feature vector corresponding to the original speech data and calculate the mean and variance of its normal posterior distribution to construct the corresponding normal distribution matrix as content information.

[0054] Exemplarily, the pre-encoder also includes a normalization stream module, and before the first content information and the first timbre information are input into the decoder of the speech conversion model, it also includes: inputting the first content information and the first timbre information into the normalization stream module, and the normalization stream module normalizes the first content information and the first timbre information.

[0055] The normalization flow module is a stack of affine coupling layers. This flow module is used to increase the complexity of the prior distribution (i.e., content information). The condition is that the speaker embedding g (in this process, g refers to Figure 2 The first timbre information corresponding to the target Mel spectrum in the normalized stream can be generated by learning a reversible mapping from a simple random noise distribution to a complex target distribution. This reversible mapping is composed of a series of reversible transformations, and can use affine transformations and nonlinear function transformations. The normalized stream module can then normalize the pitch information and variable information of different genders and speakers to reduce the variable factor.

[0056] In some embodiments, the speaker encoder generates an encoded speaker representation from the utterance. The speaker encoder can be implemented as a four-layer convolutional layer plus a two-sided LSTM (Long Short-Term Memory) network, or as a simple one-layer LSTM layer plus a fully connected layer. It is trained from scratch together with the rest of the model. The network structure of the speaker encoder includes a layer of LSTM structure and a layer of fully connected layer. The purpose of the speaker encoder is to encode different speakers. For example, a simple speaker Zhang San can be represented as a numerical vector 1 or a two-digit number [1 0]. We use the target Mel spectrogram as the input of the speaker encoder. The audio signal represents all the information of the speaker, including whether he is Zhang San or Li Si, but we don’t know who it is. However, we can extract features from the audio to represent the voice of the same speaker as the same vector without the help of text name information. We assume that the output of the pre-encoder does not contain any speaker information. Then, the voice conversion model replaces the missing speaker information according to the input of the speaker encoder to synthesize speech.

[0057] S104. Input the first content information and the first timbre information to the decoder of the speech conversion model. The decoder parses the first content information and the first timbre information and outputs the converted target speech data.

[0058] Specifically, by inputting the first content information and the first timbre information into the decoder of the voice conversion model, the decoder can parse the first content information and the first timbre information to synthesize the target voice data. The provided method can accurately and quickly convert the current original voice data into the target voice data of any target user in real time. This enables the financial system to provide exclusive customer service to users without adding personnel, thereby improving the user's service experience without increasing the labor cost.

[0059] In some embodiments, the decoder is as follows Figure 2 The MS-iSTFT (MS-Inverse Short-Time Fourier Transform) decoder in the decoder performs the following steps in sequence:

[0060] First, the variational autoencoder (VAE) latent variable Z (consisting of the first content information and the first timbre information) is embedded in the speaker g (in this process, g refers to Figure 2 The first timbre information corresponding to the target Mel spectrum in the image is conditioned and then upsampled by a sequence of upsampled convolutional residual blocks. The upsampled Z is then projected to the amplitude and phase variables of each subband signal.

[0061] Then, using the amplitude and phase variables, an iSTFT (Inverse Short Time Fourier Transform) operation is performed to generate each sub-band signal.

[0062] Finally, these subband signals are upsampled by inserting zeros between samples to match the sampling rate of the original signal and then integrated into the full-band waveform using a trainable synthesis filter to synthesize the target speech data.

[0063] In some embodiments, please refer to Figure 3 ,like Figure 3 As shown, the provided speech conversion model training method includes the above-mentioned step S101 and steps S105 to S108.

[0064] S105. Input the original speech data into the pre-encoder of the speech conversion model to be trained to extract features of the original speech data and output the second content information.

[0065] S106. Extracting the original Mel-atlas corresponding to the original speech data, inputting the original Mel-atlas into the speaker encoder to be trained to parse the original Mel-atlas, and outputting the second timbre information corresponding to the original speech data.

[0066] S107. Input the second content information and the second timbre information to the decoder to be trained to parse the second content information and the second timbre information, and output the converted training speech data.

[0067] S108. Complete the training of the speech conversion model based on the training speech data and the original speech data.

[0068] The method provided can be based on the original Mel spectrogram corresponding to the original speech data input by the speaker encoder (i.e., g refers to Figure 2 The second timbre information of the source mel atlas corresponding to the original mel atlas in the image is obtained). The deviation between the converted training speech data and the original speech data of the corresponding target user is determined to complete the training of the model.

[0069] Among them, Figure 2 As shown, the role of the content encoder in the pre-encoder is to extract information related to content representation in the audio. In order to achieve the purpose of the content encoder, this application will use different speech contents of the same speaker for training during the training process. Since the two contents are different, the extracted content encodings should be different and should be complete. Since they are the same speaker, the vectors of the speaker encoder should be the same or consistent. The content encoding vector and the speaker vector are exchanged to reconstruct the speech content. If reconstruction is possible, it means that the content encoder has achieved the expected effect. During the training process, the model is constantly learned to finally realize the content encoder and the speaker encoder.

[0070] Exemplarily, the training of the speech conversion model is completed based on the training speech data and the original speech data, including: inputting the original speech data and the training speech data into a preset discriminator, the discriminator discriminates the original speech data and the training speech data, and outputs discrimination information; and completing the training of the speech conversion model based on the discrimination information.

[0071] Then, by introducing the discriminator, such as Figure 2 As shown, combined with the adversarial generation principle, the true / false identification information of the original speech data and the training speech data can be judged, and then the adversarial training can be used to enhance the timbre and pitch of the synthesized training speech data to be more similar to the timbre and pitch of the original speech data.

[0072] In some embodiments, before the first content information and the first timbre information are input into the decoder of the speech conversion model, it also includes: obtaining a target linear spectrum of the target object corresponding to the speech conversion; inputting the first timbre information and the target linear spectrum into the post encoder of the speech conversion model, and the post encoder enhances the first timbre information according to the target linear spectrum to input the enhanced first timbre information into the decoder.

[0073] like Figure 2 As shown, the provided method can enhance the first timbre information through the post encoder ( Figure 2 g) in order to improve the accuracy of the final converted target speech data.

[0074] Exemplarily, the method further includes: extracting target customer service voice data corresponding to the target object from a preset financial customer service voice database; extracting the Mel scale and linear scale respectively according to the target customer service voice data to obtain a target Mel spectrum and a target linear spectrum. The provided method can obtain a target Mel spectrum and a target linear spectrum corresponding to the target customer service voice data based on the Mel scale and the linear scale respectively. This ensures that the original voice data can be converted into the target voice data of any target user at any time.

[0075] In some embodiments, Figure 2 As shown, the provided inverse short-time Fourier transform-based voice conversion model can be a QuickVC (Quick Vocie Conversion) model. It consists of a speaker encoder, a pre-encoder, a post-encoder, an MS-iSTFT decoder and a discriminator, wherein the structures of the post-encoder and the discriminator follow VITS (VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech) is a highly expressive speech synthesis model that combines variational inference, normalizing flows and adversarial training). The backbone of QuickVC is inherited from VITS, using variational inference, and adding a normalizing flow module and adversarial training process.

[0076] See also Figure 4 As shown, Figure 4 1 is a schematic diagram of the structure of a speech conversion device 200 provided in an embodiment of the present application. The speech conversion device 200 is used to execute the steps of the speech conversion method shown in the above embodiments. The speech conversion device 200 can be a single server or a server cluster, or the speech conversion device 200 can be a terminal, which can be a handheld terminal, a laptop computer, a wearable device or a robot, etc.

[0077] like Figure 4 As shown, the speech conversion device 200 includes:

[0078] The data acquisition unit 201 is used to acquire the original speech data to be converted;

[0079] The data input unit 202 is used to input the original speech data into a pre-encoder of a pre-trained speech conversion model, and the pre-encoder extracts features from the original speech data and outputs first content information corresponding to the original speech data;

[0080] A graph acquisition unit 203 is used to acquire a target mel-gram graph of a target object corresponding to speech conversion, and input the target mel-gram graph into a speaker encoder of the speech conversion model. The speaker encoder parses the target mel-gram graph and outputs first timbre information corresponding to the target object.

[0081] The speech conversion unit 204 is used to input the first content information and the first timbre information into the decoder of the speech conversion model, and the decoder parses the first content information and the first timbre information and outputs the converted target speech data.

[0082] The above-mentioned voice conversion method can be implemented in the form of a computer program. Figure 4 Run on the device shown.

[0083] See also Figure 5 , Figure 5 1 is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application. The computer device includes a processor, a memory and a network interface connected via a device bus, wherein the memory may include a storage medium and an internal memory.

[0084] The storage medium can store an operating device and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any one of the speech conversion methods.

[0085] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0086] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any speech conversion method.

[0087] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present application, and does not constitute a limitation on the terminal to which the scheme of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0088] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0089] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0090] Obtaining original voice data to be converted;

[0091] Inputting the original speech data into a pre-encoder of a pre-trained speech conversion model, the pre-encoder extracts features from the original speech data, and outputs first content information corresponding to the original speech data;

[0092] Obtaining a target Mel-atlas of a target object corresponding to speech conversion, inputting the target Mel-atlas into a speaker encoder of a speech conversion model, and the speaker encoder parsing the target Mel-atlas to output first timbre information corresponding to the target object;

[0093] The first content information and the first timbre information are input to a decoder of a speech conversion model, and the decoder analyzes the first content information and the first timbre information and outputs converted target speech data.

[0094] In some embodiments, before the original speech data is input into the pre-encoder of the pre-trained speech conversion model, it also includes: inputting the original speech data into the pre-encoder of the speech conversion model to be trained to extract features from the original speech data and output second content information; extracting the original Mel-gram spectrum corresponding to the original speech data, inputting the original Mel-gram spectrum into the speaker encoder to be trained to parse the original Mel-gram spectrum, and outputting the second timbre information corresponding to the original speech data; inputting the second content information and the second timbre information into the decoder to be trained to parse the second content information and the second timbre information, and outputting the converted training speech data; completing the training of the speech conversion model according to the training speech data and the original speech data.

[0095] Exemplarily, the training of the speech conversion model is completed based on the training speech data and the original speech data, including: inputting the original speech data and the training speech data into a preset discriminator, the discriminator discriminates the original speech data and the training speech data, and outputs discrimination information; and completing the training of the speech conversion model based on the discrimination information.

[0096] In some embodiments, the pre-encoder includes a feature extractor and a content encoder; the pre-encoder inputs the original speech data into a pre-trained speech conversion model, including: inputting the original speech data into the feature extractor, the feature extractor extracts features from the original speech data, and outputs a feature vector corresponding to the original speech data; inputting the feature vector into the content encoder, the content encoder calculates the mean and variance of the feature vector, and outputs a normal distribution matrix corresponding to the original speech data based on the mean and variance, and the normal distribution matrix constitutes the first content information.

[0097] Exemplarily, the pre-encoder also includes a normalization stream module, and before the first content information and the first timbre information are input into the decoder of the speech conversion model, it also includes: inputting the first content information and the first timbre information into the normalization stream module, and the normalization stream module normalizes the first content information and the first timbre information.

[0098] In some embodiments, before the first content information and the first timbre information are input into the decoder of the speech conversion model, it also includes: obtaining a target linear spectrum of the target object corresponding to the speech conversion; inputting the first timbre information and the target linear spectrum into the post encoder of the speech conversion model, and the post encoder enhances the first timbre information according to the target linear spectrum to input the enhanced first timbre information into the decoder.

[0099] Exemplarily, the method also includes: extracting target customer service voice data corresponding to the target object from a preset financial customer service voice database; extracting the Mel scale and linear scale respectively according to the target customer service voice data to obtain a target Mel spectrum and a target linear spectrum.

[0100] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a computer program. The computer program includes program instructions. The processor executes the program instructions to implement the steps of the speech conversion method provided in the above embodiments of the present application.

[0101] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0102] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A speech conversion method, characterized in that: include: Obtaining original voice data to be converted; Inputting the original speech data into a pre-encoder of a pre-trained speech conversion model, wherein the pre-encoder extracts features from the original speech data and outputs first content information corresponding to the original speech data; Obtaining a target mel-atlas of a target object corresponding to speech conversion, inputting the target mel-atlas into a speaker encoder of the speech conversion model, wherein the speaker encoder parses the target mel-atlas and outputs first timbre information corresponding to the target object; The first content information and the first timbre information are input into a decoder of the speech conversion model, and the decoder parses the first content information and the first timbre information and outputs converted target speech data.

2. The method according to claim 1, characterized in that Before inputting the original speech data into the pre-encoder of the pre-trained speech conversion model, the method further includes: Inputting the original speech data into a pre-encoder of a speech conversion model to be trained to extract features from the original speech data and output second content information; Extracting an original Mel-atrogram corresponding to the original speech data, inputting the original Mel-atrogram into the speaker encoder to be trained to parse the original Mel-atrogram, and outputting second timbre information corresponding to the original speech data; Inputting the second content information and the second timbre information into the decoder to be trained to parse the second content information and the second timbre information, and outputting converted training speech data; The speech conversion model is trained according to the training speech data and the original speech data.

3. The method according to claim 2, characterized in that The completing the training of the speech conversion model according to the training speech data and the original speech data comprises: Inputting the original voice data and the training voice data into a preset discriminator, the discriminator discriminates the original voice data and the training voice data, and outputs discrimination information; The training of the speech conversion model is completed according to the identification information.

4. The method according to claim 1, characterized in that: The pre-encoder includes a feature extractor and a content encoder; The method of inputting the original speech data into a pre-encoder of a pre-trained speech conversion model comprises: Inputting the original speech data into the feature extractor, the feature extractor extracts features from the original speech data and outputs a feature vector corresponding to the original speech data; The feature vector is input to the content encoder, and the content encoder calculates the mean and variance of the feature vector, and outputs a normal distribution matrix corresponding to the original speech data according to the mean and variance, and the normal distribution matrix constitutes the first content information.

5. The method according to claim 4, characterized in that The pre-encoder further includes a normalized stream module, and before the first content information and the first timbre information are input to the decoder of the speech conversion model, further includes: The first content information and the first timbre information are input into the normalized stream module, and the normalized stream module normalizes the first content information and the first timbre information.

6. The method according to claim 1, characterized in that Before inputting the first content information and the first timbre information into the decoder of the speech conversion model, the method further includes: Obtain a target linear spectrum of a target object corresponding to speech conversion; The first timbre information and the target linear spectrum are input into a post-encoder of the speech conversion model, and the post-encoder enhances the first timbre information according to the target linear spectrum, so as to input the enhanced first timbre information into the decoder.

7. The method according to claim 6, characterized in that The method further comprises: Extracting target customer service voice data corresponding to the target object from a preset financial customer service voice database; Mel scale and linear scale are extracted respectively according to the target customer service voice data to obtain the target Mel spectrum and target linear spectrum.

8. A speech conversion device, characterized in that: include: A data acquisition unit, used to acquire original speech data to be converted; A data input unit, configured to input the original speech data into a pre-encoder of a pre-trained speech conversion model, wherein the pre-encoder extracts features from the original speech data and outputs first content information corresponding to the original speech data; A graph acquisition unit, used for acquiring a target Mel-gram graph of a target object corresponding to speech conversion, inputting the target Mel-gram graph into a speaker encoder of the speech conversion model, wherein the speaker encoder parses the target Mel-gram graph and outputs first timbre information corresponding to the target object; The speech conversion unit is used to input the first content information and the first timbre information into the decoder of the speech conversion model, and the decoder parses the first content information and the first timbre information and outputs the converted target speech data.

9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the speech conversion method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the speech conversion method according to any one of claims 1 to 7.

Images