A voice conversion method, device, equipment and medium

Abstract

The present application relates to artificial intelligence technology and provides a speech conversion method, device, equipment and medium. The method includes: obtaining source speech information from a source speaker; extracting content features of the source speech information through a trained speech conversion model to obtain content features of the source speech information; extracting speaker features of the source speech information to obtain speaker features of the source speaker; processing content features, speaker features of the source speaker and target fundamental frequency information to obtain converted target speech information; outputting the target speech information to improve the naturalness and rhythmic coherence of the target speech after speech conversion.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a speech conversion method, device, equipment and medium. Background Art

[0002] Speech conversion refers to converting the timbre of a source speaker's speech into the timbre of another person's speech while keeping the semantic content of the source speaker's speech unchanged. There are many neural network models that implement speech conversion, such as the AutoVC model. The AutoVC model is a zero-shot audio conversion based on audio coding loss and a many-to-many non-parallel audio conversion framework. However, research has found that the speech obtained by speech conversion through AutoVC still has a very obvious robot-like feeling. Therefore, how to improve the naturalness of the speech after speech conversion is a technical problem that needs to be solved urgently. Summary of the invention

[0003] The embodiments of the present application provide a speech conversion method, apparatus, device and medium, which can improve the naturalness and rhythmic coherence characteristics of the target speech after speech conversion.

[0004] On the one hand, an embodiment of the present application provides a voice conversion method, the method comprising:

[0005] Acquire source speech information from a source speaker;

[0006] Extracting content features of source speech information through the trained speech conversion model to obtain content features of the source speech information;

[0007] Perform speaker feature extraction on the source voice information to obtain the speaker features of the source speaker;

[0008] Processing the content features, the speaker features of the source speaker and the target fundamental frequency information to obtain the converted target speech information;

[0009] Output the target voice information.

[0010] In one embodiment, the trained speech conversion model includes a conditional autoencoder, the conditional autoencoder including an encoder and a decoder;

[0011] The specific implementation process of processing the content features, the speaker features of the source speaker and the target fundamental frequency information to obtain the converted target speech information is as follows:

[0012] The speaker feature and the content feature of the source speaker are concatenated by an encoder to obtain a first feature, and the first feature is encoded to obtain an encoded feature;

[0013] The encoded feature is concatenated with the target base frequency information to obtain the second feature;

[0014] The second feature is decoded by a decoder to obtain target voice information.

[0015] In one embodiment, before the content features, the speaker features of the source speaker and the target fundamental frequency information are processed to obtain the converted target speech information, the following process may be performed:

[0016] Display the style setting interface, which includes multiple style options;

[0017] In response to a selection operation of a target style option among the plurality of style options, determining a target style option, the target style option being any style option among the plurality of style options;

[0018] According to the correspondence between the style tag and the baseband information, the baseband information corresponding to the style tag indicated by the target style option is searched in the multiple baseband information, and the searched baseband information is used as the target baseband information.

[0019] In one embodiment, before the content features, the speaker features of the source speaker and the target fundamental frequency information are processed to obtain the converted target speech information, the following process may be performed:

[0020] Perform timbre analysis on the source voice information to determine the gender of the source speaker;

[0021] According to the correspondence between gender and fundamental frequency information, fundamental frequency information corresponding to the determined gender is searched in multiple fundamental frequency information, and the found fundamental frequency information is used as target fundamental frequency information.

[0022] In one embodiment, before the content features, the speaker features of the source speaker and the target fundamental frequency information are processed to obtain the converted target speech information, the following process may be performed:

[0023] Extracting fundamental frequency information of source speech information to obtain fundamental frequency information of the source speaker;

[0024] When an adjustment operation on the fundamental frequency information of the source speaker is detected, the fundamental frequency information of the source speaker is adjusted to obtain target fundamental frequency information.

[0025] In one embodiment, the specific implementation process of the training method of the speech conversion model is:

[0026] Acquire a training sample, where the training sample includes a first voice message submitted by a first user for the target text message and a second voice message submitted by a second user for the target text message;

[0027] Extracting content features of the first voice information through a voice conversion model to obtain content features of the first voice information;

[0028] Extracting speaker features from the first voice information to obtain speaker features of the first user;

[0029] Performing speaker feature extraction on the second voice information to obtain speaker features of the second user;

[0030] Extracting fundamental frequency information from the second voice information to obtain fundamental frequency information of the second user;

[0031] Processing the content features, the speaker features of the first user, the fundamental frequency information, and the speaker features of the second user to obtain predicted speech information;

[0032] The speech conversion model is trained according to the predicted speech information and the second speech information to obtain a trained speech conversion model.

[0033] In one embodiment, the speech conversion model is trained according to the predicted speech information and the second speech information to obtain the trained speech conversion model in the following specific implementation process:

[0034] Acquire a first spectrum of predicted speech information and a second spectrum of second speech information;

[0035] Comparing the first spectrum with the second spectrum to obtain a loss value;

[0036] Based on the loss value, the speech conversion model is trained to obtain a trained speech conversion model.

[0037] On the other hand, an embodiment of the present application provides a voice conversion device, the voice conversion device comprising:

[0038] An acquisition unit, used for acquiring source speech information from a source speaker;

[0039] A processing unit, configured to extract content features of the source voice information through the trained voice conversion model to obtain content features of the source voice information;

[0040] The processing unit is further used to extract speaker features from the source voice information to obtain speaker features of the source speaker;

[0041] The processing unit is further used to process the content features, the speaker features of the source speaker and the target fundamental frequency information to obtain the converted target speech information;

[0042] The output unit is used to output the target voice information.

[0043] On the other hand, an embodiment of the present application provides an electronic device, including a processor, a memory and a communication interface, wherein the processor, the memory and the communication interface are interconnected, wherein the memory is used to store a computer program that supports a terminal to execute the above method, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the following steps: obtaining source voice information from a source speaker; extracting content features of the source voice information through a trained voice conversion model to obtain content features of the source voice information; extracting speaker features of the source voice information to obtain speaker features of the source speaker; processing the content features, the speaker features of the source speaker and the target fundamental frequency information to obtain converted target voice information; and outputting the target voice information.

[0044] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the above-mentioned speech conversion method.

[0045] In the embodiment of the present application, after obtaining the content features of the source voice information and the speaker features of the source speaker, the content features, the speaker features of the source speaker and the target fundamental frequency information are processed to decouple the three information of fundamental frequency, content and speaker. This can not only control the fundamental frequency in the target voice information, but also ensure the consistency of the fundamental frequency in the target voice information, eliminate the disturbance related to the fundamental frequency in the target voice information, and thus improve the naturalness and rhythmic coherence characteristics of the target voice after voice conversion. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 It is a flowchart of a voice conversion method provided by an embodiment of the present application;

[0048] Figure 2 It is a flowchart of a method for training a speech conversion model provided in an embodiment of the present application;

[0049] Figure 3 It is a schematic diagram of the architecture of a training system for a speech conversion model provided in an embodiment of the present application;

[0050] Figure 4 is a structural schematic diagram of a speech conversion device provided in an embodiment of the present application;

[0051] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The speech conversion method in the embodiment of the present application can decouple the three information of fundamental frequency (F0), content, and speaker, which can not only control the fundamental frequency in the target speech information, but also ensure the consistency of the fundamental frequency in the target speech information, thereby eliminating the disturbance related to the fundamental frequency in the target speech information, thereby improving the naturalness and rhythmic coherence characteristics of the target speech after speech conversion.

[0053] Among them, the fundamental frequency refers to the frequency of the fundamental tone, which determines the pitch of the entire sound.

[0054] The voice conversion method in the embodiment of the present application can be applied in a first electronic device, wherein the first electronic device can be any one or more of a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart vehicle-mounted device, and a smart wearable device.

[0055] For example, the first electronic device runs a voice-changing client. After the user inputs a voice through the voice-changing client, the voice-changing client can perform voice conversion on the voice, obtain the target voice information, and play the target voice information. For another example, in the scenario of identifying people by listening to voice, the first electronic device can be a voice-changing microphone, etc., and the user inputs a voice through the voice-changing microphone. The voice-changing microphone can perform voice conversion on the voice, obtain the target voice information, and send the target voice information to the sound amplification device, which can play the target voice information. For another example, in the scenario of identifying people by listening to voice, the first electronic device can be a voice-changing sound amplification device, etc., and the user inputs a voice through the microphone, and the microphone can send the voice to the voice-changing sound amplification device, and the voice-changing sound amplification device can perform voice conversion on the voice, obtain the target voice information, and play the target voice information. For another example, after the user inputs video data (such as cartoons or animations, etc.) into the first electronic device, the first electronic device can obtain the voice information of the same performer from the video data, and different performers correspond to different preset fundamental frequency information. The first electronic device can extract content features from the voice information of the same performer to obtain the content features of the performer, extract speaker features from the voice information of the performer to obtain the speaker features of the performer, process the content features, the speaker features of the performer, and the preset fundamental frequency information corresponding to the performer to obtain the converted target voice information, and then the first electronic device can replace the voice information of the performer in the video data with the target voice information. The timbre of the target voice information is different from the timbre of the user, but the semantic content of the target voice information is the same as the semantic content of the voice input by the user.

[0056] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0057] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0058] See also Figure 1 , Figure 1 is a flow chart of a voice conversion method provided by an embodiment of the present application; Figure 1 The voice conversion method shown can be performed by a first electronic device, and the scheme includes but is not limited to steps S101 to S105, wherein:

[0059] S101, obtaining source speech information from a source speaker.

[0060] In one example, the first electronic device can collect external sounds through a microphone configured in the first electronic device. For example, if the user has a need to convert the voice information of a specified object into voice, the user can move the first electronic device to a position closer to the source speaker so that the first electronic device can collect the source voice information from the source speaker through the microphone. For another example, if the user has a need to convert the specified voice information into voice, the user can send a voice information collection instruction to the first electronic device when the source speaker outputs the source voice information or before the source speaker outputs the source voice information, and the first electronic device can respond to the voice information collection instruction to collect the source voice information from the source speaker.

[0061] Optionally, after collecting the voice information, the first electronic device may display a voice information confirmation interface, which may include the collected voice information, a play button, and a duration editing control for the voice information, and the duration editing control may include a first end and a second end. The user may click the play button, and after the first electronic device detects the user's click operation, it may play the collected voice information. If the user finds that there are voice segments to be voice-converted and other voice segments in the played voice information, the user may edit the voice information collected by the first electronic device through the duration editing control, so that the edited voice information only contains the voice segments to be voice-converted, and then the first electronic device may use the edited voice information as the source voice information from the source speaker.

[0062] For example, if the starting time point of the voice segment to be voice-converted in the voice information collected by the first electronic device is after the starting time point of the collected voice information, and the ending time point of the voice segment to be voice-converted is the same as the ending time point of the collected voice information, then the user can drag the first end of the duration editing control until the position of the first end moves to the starting time point of the voice segment to be voice-converted. The first electronic device can respond to the user's dragging operation and determine the edited voice information, wherein the starting time point of the edited voice information is the time point indicated by the first end of the duration editing control, and the ending time point of the edited voice information is the time point indicated by the second end of the duration editing control (i.e., the ending time point of the collected voice information).

[0063] For another example, if the starting time point of the voice segment to be voice-converted in the voice information collected by the first electronic device is the same as the starting time point of the collected voice information, and the ending time point of the voice segment to be voice-converted is before the ending time point of the collected voice information, then the user can drag the second end of the duration editing control until the position of the second end moves to the ending time point of the voice segment to be voice-converted. The first electronic device can respond to the user's dragging operation and determine the edited voice information, wherein the starting time point of the edited voice information is the time point indicated by the first end of the duration editing control (i.e., the starting time point of the collected voice information), and the ending time point of the edited voice information is the time point indicated by the second end of the duration editing control.

[0064] For another example, if the starting time point of the voice segment to be voice-converted in the voice information collected by the first electronic device is after the starting time point of the collected voice information, and the ending time point of the voice segment to be voice-converted is before the ending time point of the collected voice information, then the user can drag the first end of the duration editing control until the first end is moved to the starting time point of the voice segment to be voice-converted, and drag the second end of the duration editing control until the second end is moved to the ending time point of the voice segment to be voice-converted. The first electronic device can respond to the user's dragging operation and determine the edited voice information, wherein the starting time point of the edited voice information is the time point indicated by the first end of the duration editing control, and the ending time point of the edited voice information is the time point indicated by the second end of the duration editing control.

[0065] In another example, the first electronic device may be the source voice information from the source speaker obtained from the local storage of the first electronic device, or the first electronic device may be the source voice information from the source speaker received from other electronic devices, or the first electronic device may be the source voice information from the source speaker downloaded from the Internet. For example, the first electronic device runs an instant messaging client, and if the user has a need to convert a certain voice information in a conversation window in the instant messaging client, the user can long press the voice information, and the first electronic device can respond to the user's long press operation and display a voice information control interface, and the voice information control interface may include a voice conversion option. The user can click on the voice conversion option, and the first electronic device can respond to the user's click operation, obtain the voice information, and use the voice information as the source voice information from the source speaker. For another example, the first electronic device runs a voice conversion plug-in, and if the user has a need to convert a certain voice information into voice, the user can upload the voice information to the voice conversion plug-in, and after the voice conversion plug-in obtains the voice information, it can use the voice information as the source voice information from the source speaker.

[0066] S102, extracting content features of the source voice information through the trained voice conversion model to obtain content features of the source voice information.

[0067] The first electronic device can convert the source voice information from time domain to frequency domain through the trained voice conversion model. That is, the first electronic device can convert the source voice information from time domain to frequency domain, and then extract content features of the source voice information to obtain content features (i.e., acoustic features) of the source voice information. The content features can be mel spectrum.

[0068] S103, extracting speaker features from the source voice information to obtain speaker features of the source speaker.

[0069] The trained speech conversion model may include encoders such as one-hot encoding, d-vector or x-vector. The first electronic device may use encoders such as one-hot encoding, d-vector or x-vector to extract speaker features from the source speech information to obtain speaker features of the source speaker.

[0070] Among them, One-Hot encoding, also known as one-bit effective encoding, mainly uses an N-bit state register to encode N states. Each state has an independent register bit, and only one bit is valid at any time.

[0071] The d-vector is obtained through DNN training. After the deep neural network (DNN) is trained, the filter bank energy feature of each frame of speech is extracted as the DNN input, and the hidden parameters (Activations) are extracted from the last hidden layer (Last Hidden Layer), L2 regularized, and then accumulated. The resulting vector is called d-vector. If a person records multiple voices, then each voice can get a d-vector, and the average of all these d-vectors is the representation of this person.

[0072] Since x-vector sets an embedding layer to extract features, plda scoring is performed directly based on the features, so there is no need for a training set in a specific language. Specifically, the first 5 layers of x-vector are at the frame level, and then two layers of segment-level embedding are inserted after pooling. The segment6 layer is used to extract xvector features, which can be used as ivert for plda scoring. The last layer is the softmax layer, which can obtain all speaker targets (i.e. speaker features) in the training set.

[0073] S104, processing the content features, the speaker features of the source speaker and the target fundamental frequency information to obtain converted target speech information.

[0074] The embodiment of the present application adds the speaker characteristics of the source speaker during the speech conversion process, enhances the representation of the speaker's timbre information in addition to the content characteristics in the latent variables, and can improve the naturalness and rhythmic coherence of the target speech after speech conversion.

[0075] In a feasible implementation, the trained speech conversion model may further include a conditional autoencoder, which includes an encoder and a decoder. On this basis, the first electronic device may concatenate the speaker feature and content feature of the source speaker through the encoder to obtain a first feature, encode the first feature to obtain an encoded feature, concatenate the encoded feature with the target fundamental frequency information to obtain a second feature, and decode the second feature through the decoder to obtain the target speech information.

[0076] Specifically, the encoder can be composed of 3 layers of 5*1 convolutional layers and ReLU activation function, with 512 channels in the convolutional layer. Each layer is followed by a batch normalization layer, and finally two layers of bidirectional long short-term memory (LSTM) layers. The decoder consists of three LSTM layers, followed by a post-processing network to optimize the output mel spectrum. The post-processing network contains 5 layers of 5*1 convolutional network layers, and each convolutional layer is followed by a batch normalization layer.

[0077] The encoder in the conditional autoencoder concatenates the speaker features and content features of the source speaker, and then encodes the concatenated features to obtain the encoded features (i.e., latent variables), which are concatenated with the preset fundamental frequency information (i.e., pitch sequence), and then decoded as the input of the decoder to obtain the converted acoustic features (conversion features), and finally the acoustic features are input into the vocoder to obtain the converted target audio waveform (i.e., target audio information). The latent variables obtained after the encoder are generally considered to contain all information mainly based on the content. This scheme concatenates the speaker features and content features of the source speaker and inputs them into the encoder, which enhances the representation of the speaker's timbre information in addition to the content in the latent variables.

[0078] In a feasible implementation, the first electronic device processes the content features, the speaker features of the source speaker and the target fundamental frequency information, and before obtaining the converted target voice information, a style setting interface may be displayed, the style setting interface including multiple style options, and in response to a selection operation of a target style option among the multiple style options, the target style option is determined, the target style option is any style option among the multiple style options, and based on the correspondence between the style label and the fundamental frequency information, the fundamental frequency information corresponding to the style label indicated by the target style option is searched in the multiple fundamental frequency information, and the found fundamental frequency information is used as the target fundamental frequency information.

[0079] For example, the first electronic device stores preset fundamental frequency information corresponding to different style options (such as vicissitudes and rich style, loli playful style), and the style setting interface can be displayed in the first electronic device. The style setting interface includes different style options. If the user selects a certain style option, the first electronic device can obtain the preset fundamental frequency information corresponding to the style option selected by the user. After the user inputs the source voice information through the first electronic device, the first electronic device can extract the content features of the source voice information through the trained voice conversion model to obtain the content features of the source voice information, extract the speaker features of the source voice information to obtain the speaker features of the source speaker (i.e., the above-mentioned user), and process the preset fundamental frequency information corresponding to the content features, the speaker features of the source speaker, and the style options selected by the user to obtain the converted target voice information, wherein the style of the target voice information matches the style selected by the user. The embodiment of the present application can set the target fundamental frequency information based on different effect requirements, and the target voice information obtained by voice conversion is more in line with the user's wishes.

[0080] In a feasible implementation, the first electronic device processes the content features, the speaker features of the source speaker, and the target fundamental frequency information, and before obtaining the converted target voice information, the source voice information can be subjected to timbre analysis to determine the gender of the source speaker, and based on the correspondence between the gender and the fundamental frequency information, the fundamental frequency information corresponding to the determined gender can be searched in multiple fundamental frequency information, and the fundamental frequency information found can be used as the target fundamental frequency information. In the embodiment of the present application, the target fundamental frequency information can be set based on the gender of the source speaker, so that the target voice information obtained by voice conversion is more matched with the source speaker.

[0081] For example, the first electronic device stores preset fundamental frequency information corresponding to different genders. After acquiring the source voice information of the source speaker, the first electronic device can perform timbre analysis on the source voice information to determine the gender of the source speaker, and then acquire the preset fundamental frequency information corresponding to the gender of the source speaker. After the user inputs the source voice information through the first electronic device, the first electronic device can extract content features of the source voice information through the trained voice conversion model to obtain the content features of the source voice information, extract speaker features of the source voice information to obtain the speaker features of the source speaker, process the content features, the speaker features of the source speaker, and the preset fundamental frequency information corresponding to the gender of the source speaker to obtain the converted target voice information, wherein the target voice information matches the gender of the source speaker.

[0082] In a feasible implementation, the first electronic device processes the content features, the speaker features of the source speaker, and the target fundamental frequency information, and before obtaining the converted target voice information, the source voice information can be subjected to timbre analysis to determine that the gender of the source speaker is the first gender, and based on the correspondence between gender and fundamental frequency information, the fundamental frequency information corresponding to the second gender is searched in multiple fundamental frequency information, and the fundamental frequency information found is used as the target fundamental frequency information, wherein the second gender refers to a gender different from the first gender, for example, if the first gender is female, the second gender is male; if the first gender is male, the second gender is female. In the embodiment of the present application, the target fundamental frequency information can be set based on the gender of the source speaker, and the target voice information obtained by voice conversion has a greater contrast with the source speaker.

[0083] In a feasible implementation, the first electronic device processes the content features, the speaker features of the source speaker, and the target fundamental frequency information, and before obtaining the converted target voice information, the fundamental frequency information of the source voice information can be extracted to obtain the fundamental frequency information of the source speaker, and when an adjustment operation on the fundamental frequency information of the source speaker is detected, the fundamental frequency information of the source speaker is adjusted to obtain the target fundamental frequency information. In the embodiment of the present application, the user can adjust the fundamental frequency information according to his own needs to ensure that the target voice information obtained by voice conversion is more in line with the user's wishes.

[0084] In the prediction stage, since there is no target speaker's voice and the target speaker's pitch information cannot be obtained, the source speaker's pitch can be used in the prediction stage to make the pitch information as consistent as possible with the pronunciation tone in the source voice information, and the pitch information can be controlled and fine-tuned at the same time.

[0085] S105, outputting target voice information.

[0086] Exemplarily, the first electronic device may play the target voice information. Alternatively, the first electronic device may send the target voice information to other electronic devices so that the other electronic devices play the target voice information. Alternatively, the first electronic device may display an identifier of the target voice information (e.g., a voice icon showing the duration of the target voice information or the name of the target voice information, etc.), and after the user clicks the identifier or the play button, the first electronic device may respond to the user's click operation to play the target voice information.

[0087] In an embodiment of the present application, source speech information from a source speaker is obtained, content features of the source speech information are extracted through a trained speech conversion model to obtain content features of the source speech information, speaker features of the source speaker are extracted to obtain speaker features of the source speaker, the content features, the speaker features of the source speaker and the target fundamental frequency information are processed to obtain converted target speech information, and the target speech information is output, which can not only control the fundamental frequency in the target speech information, but also ensure the consistency of the fundamental frequency in the target speech information, eliminate disturbances related to the fundamental frequency in the target speech information, and thus improve the naturalness and rhythmic coherence characteristics of the target speech after speech conversion.

[0088] See also Figure 2 , Figure 2 is a flow chart of a method for training a speech conversion model provided in an embodiment of the present application; Figure 2 The training method of the voice conversion model shown can be performed by the second electronic device, and the scheme includes but is not limited to steps S201 to S207, wherein:

[0089] S201, obtaining a training sample, where the training sample includes first voice information submitted by a first user for target text information and second voice information submitted by a second user for target text information.

[0090] Among them, the second electronic device can be any one or more of a smart phone, a tablet computer, a laptop computer, a desktop computer, an intelligent vehicle-mounted device, and an intelligent wearable device. Optionally, the second electronic device can also be a server, which can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. In other words, the server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. The second electronic device and the first electronic device can be the same device or different devices.

[0091] S202: extract content features of the first voice information through a voice conversion model to obtain content features of the first voice information.

[0092] S203: extract speaker features from the first voice information to obtain speaker features of the first user.

[0093] S204: extract speaker features from the second voice information to obtain speaker features of the second user.

[0094] S205: Extract fundamental frequency information from the second voice information to obtain fundamental frequency information of the second user.

[0095] The fundamental frequency information (ie, pitch sequence) used in the training phase is the pitch value extracted from the second voice information. This is done in order to make the rhythm of the converted target voice information more similar to that of the second user.

[0096] The specific method of obtaining the pitch sequence is to extract it through a python extraction tool corresponding to the pYin algorithm. The input is an audio file, and the output is the pitch sequence value corresponding to the audio file, which is a one-dimensional vector with the horizontal axis in time frames.

[0097] S206, processing the content features, the speaker features of the first user, the fundamental frequency information and the speaker features of the second user to obtain predicted speech information.

[0098] by Figure 3 Taking the architecture diagram of the training system of the speech conversion model shown as an example, the second electronic device can perform spectrum extraction on the first speech information to obtain content features (i.e., source features), concatenate the content features and the speaker features of the first user (i.e., source speaker feature vector) and input them into the encoder to obtain a latent variable Z, which is then decoded. The fundamental frequency information extracted from the second speech information and the speaker features of the second user (i.e., target speaker feature vector) are added during decoding. The converted features output by the decoder, i.e., the target spectrum, are compared with the real spectrum extracted from the corresponding speaker's speech to calculate the loss value, such as the mean square error (MSEloss), thereby completing the mapping from the source speaker's speech to the target speaker's speech.

[0099] S207: Train a speech conversion model according to the predicted speech information and the second speech information to obtain a trained speech conversion model.

[0100] In a feasible implementation, the second electronic device can obtain a first spectrum of the predicted voice information and a second spectrum of the second voice information, compare the first spectrum with the second spectrum to obtain a loss value, and train a voice conversion model based on the loss value to obtain a trained voice conversion model.

[0101] Through training, the encoder in the conditional autoencoder can learn a mapping conversion relationship from the source speaker feature vector to the target speaker feature vector, thereby enhancing the parsing ability of speech conversion.

[0102] During the training phase, the parameters of the speech synthesis model are continuously adjusted and optimized to continuously reduce the loss value between the spectral features of the model output and the actual target spectral features, ultimately achieving accurate conversion between input and output.

[0103] In an embodiment of the present application, a training sample is obtained, and the training sample includes a first voice information submitted by a first user for a target text information and a second voice information submitted by a second user for the target text information. Content features are extracted from the first voice information through a voice conversion model to obtain content features of the first voice information. Speaker features are extracted from the first voice information to obtain speaker features of the first user. Speaker features are extracted from the second voice information to obtain speaker features of the second user. Fundamental frequency information is extracted from the second voice information to obtain fundamental frequency information of the second user. The content features, speaker features of the first user, fundamental frequency information and speaker features of the second user are processed to obtain predicted voice information. According to the predicted voice information and the second voice information, a voice conversion model is trained to obtain a trained voice conversion model, which can ensure that the three information of fundamental frequency, content and speaker are decoupled through the trained voice conversion model, thereby ensuring that the naturalness and rhythmic coherence characteristics of the target voice obtained by voice conversion through the trained voice conversion model are high.

[0104] An embodiment of the present application further provides a computer storage medium, in which program instructions are stored. When the program instructions are executed, they are used to implement the corresponding methods described in the above embodiments.

[0105] See also Figure 4 , Figure 4 It is a structural schematic diagram of a speech conversion device provided in an embodiment of the present application.

[0106] In one implementation of the device of the embodiment of the present application, the device includes the following structure.

[0107] An acquisition unit 401 is used to acquire source speech information from a source speaker;

[0108] The processing unit 402 is used to extract content features of the source voice information through the trained voice conversion model to obtain content features of the source voice information;

[0109] The processing unit 402 is further used to extract speaker features from the source voice information to obtain speaker features of the source speaker;

[0110] The processing unit 402 is further used to process the content feature, the speaker feature of the source speaker and the target fundamental frequency information to obtain the converted target speech information;

[0111] The output unit 403 is used to output the target voice information.

[0112] In one embodiment, the trained speech conversion model includes a conditional autoencoder, the conditional autoencoder including an encoder and a decoder;

[0113] The processing unit 402 processes the content features, the speaker features of the source speaker and the target fundamental frequency information to obtain the converted target speech information, including:

[0114] The speaker feature and the content feature of the source speaker are concatenated by an encoder to obtain a first feature, and the first feature is encoded to obtain an encoded feature;

[0115] The encoded feature is concatenated with the target base frequency information to obtain the second feature;

[0116] The second feature is decoded by a decoder to obtain target voice information.

[0117] In one embodiment, the output unit 403 is further configured to display a style setting interface before the processing unit 402 processes the content features, the speaker features of the source speaker, and the target fundamental frequency information to obtain the converted target speech information, wherein the style setting interface includes a plurality of style options;

[0118] The processing unit 402 is further configured to determine a target style option in response to a selection operation on a target style option among the multiple style options, where the target style option is any style option among the multiple style options;

[0119] The processing unit 402 is further configured to search for the fundamental frequency information corresponding to the style tag indicated by the target style option in the multiple fundamental frequency information according to the correspondence between the style tag and the fundamental frequency information, and use the searched fundamental frequency information as the target fundamental frequency information.

[0120] In one embodiment, the processing unit 402 is further used to perform timbre analysis on the source voice information to determine the gender of the source speaker before processing the content features, the speaker features of the source speaker and the target fundamental frequency information to obtain the converted target voice information; and search for fundamental frequency information corresponding to the determined gender in multiple fundamental frequency information according to the corresponding relationship between gender and fundamental frequency information, and use the found fundamental frequency information as the target fundamental frequency information.

[0121] In one embodiment, the processing unit 402 is further configured to extract the fundamental frequency information of the source voice information to obtain the fundamental frequency information of the source speaker before processing the content features, the speaker features of the source speaker and the target fundamental frequency information to obtain the converted target voice information; and to adjust the fundamental frequency information of the source speaker to obtain the target fundamental frequency information when an adjustment operation on the fundamental frequency information of the source speaker is detected.

[0122] In one embodiment, the acquisition unit 401 is further used to acquire a training sample, where the training sample includes a first voice message submitted by a first user for the target text message and a second voice message submitted by a second user for the target text message;

[0123] The processing unit 402 is further configured to extract content features of the first voice information through a voice conversion model to obtain content features of the first voice information;

[0124] The processing unit 402 is further configured to extract speaker features from the first voice information to obtain speaker features of the first user;

[0125] The processing unit 402 is further configured to extract speaker features from the second voice information to obtain speaker features of the second user;

[0126] The processing unit 402 is further configured to extract fundamental frequency information from the second voice information to obtain fundamental frequency information of the second user;

[0127] The processing unit 402 is further configured to process the content feature, the speaker feature of the first user, the fundamental frequency information, and the speaker feature of the second user to obtain predicted speech information;

[0128] The processing unit 402 is further configured to train the speech conversion model according to the predicted speech information and the second speech information to obtain a trained speech conversion model.

[0129] In one embodiment, the processing unit 402 trains the speech conversion model according to the predicted speech information and the second speech information to obtain the trained speech conversion model, including:

[0130] Acquire a first spectrum of predicted speech information and a second spectrum of second speech information;

[0131] Comparing the first spectrum with the second spectrum to obtain a loss value;

[0132] Based on the loss value, the speech conversion model is trained to obtain a trained speech conversion model.

[0133] In an embodiment of the present application, source speech information from a source speaker is obtained, content features of the source speech information are extracted through a trained speech conversion model to obtain content features of the source speech information, speaker features of the source speaker are extracted to obtain speaker features of the source speaker, the content features, the speaker features of the source speaker and the target fundamental frequency information are processed to obtain converted target speech information, and the target speech information is output, which can not only control the fundamental frequency in the target speech information, but also ensure the consistency of the fundamental frequency in the target speech information, eliminate disturbances related to the fundamental frequency in the target speech information, and thus improve the naturalness and rhythmic coherence characteristics of the target speech after speech conversion.

[0134] See also Figure 5 , Figure 51 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device in the embodiment of the present application includes a power supply module and other structures, and includes a processor 501, a memory 502, and a communication interface 503. The processor 501, the memory 502, and the communication interface 503 can exchange data, and the processor 501 implements the corresponding data processing solution.

[0135] The memory 502 may include a volatile memory, such as a random-access memory (RAM); the memory 502 may also include a non-volatile memory, such as a flash memory, a solid-state drive (SSD), etc.; the memory 502 may also include a combination of the above-mentioned types of memory.

[0136] The processor 501 may be a central processing unit (CPU) 501. The processor 501 may also be a combination of a CPU and a GPU. In an electronic device, multiple CPUs and GPUs may be included as needed to perform corresponding data processing. In one embodiment, the memory 502 is used to store program instructions. The processor 501 may call program instructions to implement various methods involved in the embodiments of the present application.

[0137] In a first possible implementation, the processor 501 of the electronic device calls the program instructions stored in the memory 502 to perform the following operations:

[0138] Acquire source speech information from a source speaker;

[0139] Extracting content features of source speech information through the trained speech conversion model to obtain content features of the source speech information;

[0140] Perform speaker feature extraction on the source voice information to obtain the speaker features of the source speaker;

[0141] Processing the content features, the speaker features of the source speaker and the target fundamental frequency information to obtain the converted target speech information;

[0142] The target voice information is output through the communication interface 503 .

[0143] In one embodiment, the trained speech conversion model includes a conditional autoencoder, the conditional autoencoder including an encoder and a decoder;

[0144] When the processor 501 processes the content feature, the speaker feature of the source speaker and the target fundamental frequency information to obtain the converted target speech information, the following process is specifically implemented:

[0145] The speaker feature and the content feature of the source speaker are concatenated by an encoder to obtain a first feature, and the first feature is encoded to obtain an encoded feature;

[0146] The encoded feature is concatenated with the target base frequency information to obtain the second feature;

[0147] The second feature is decoded by a decoder to obtain target voice information.

[0148] In one embodiment, the processor 501 may further implement the following process before processing the content feature, the speaker feature of the source speaker and the target fundamental frequency information to obtain the converted target speech information:

[0149] Displaying a style setting interface via the communication interface 503, the style setting interface including a plurality of style options;

[0150] In response to a selection operation of a target style option among the plurality of style options, determining a target style option, the target style option being any style option among the plurality of style options;

[0151] According to the correspondence between the style tag and the baseband information, the baseband information corresponding to the style tag indicated by the target style option is searched in the multiple baseband information, and the searched baseband information is used as the target baseband information.

[0152] In one embodiment, the processor 501 may further implement the following process before processing the content feature, the speaker feature of the source speaker and the target fundamental frequency information to obtain the converted target speech information:

[0153] Perform timbre analysis on the source voice information to determine the gender of the source speaker;

[0154] According to the correspondence between gender and fundamental frequency information, fundamental frequency information corresponding to the determined gender is searched in multiple fundamental frequency information, and the found fundamental frequency information is used as target fundamental frequency information.

[0155] In one embodiment, the processor 501 may further implement the following process before processing the content feature, the speaker feature of the source speaker and the target fundamental frequency information to obtain the converted target speech information:

[0156] Extracting fundamental frequency information of source speech information to obtain fundamental frequency information of the source speaker;

[0157] When an adjustment operation on the fundamental frequency information of the source speaker is detected, the fundamental frequency information of the source speaker is adjusted to obtain target fundamental frequency information.

[0158] In one embodiment, the processor 501 may further implement the following process before extracting content features of the source voice information through the trained voice conversion model to obtain the content features of the source voice information:

[0159] Acquire a training sample, where the training sample includes a first voice message submitted by a first user for the target text message and a second voice message submitted by a second user for the target text message;

[0160] Extracting content features of the first voice information through a voice conversion model to obtain content features of the first voice information;

[0161] Extracting speaker features from the first voice information to obtain speaker features of the first user;

[0162] Performing speaker feature extraction on the second voice information to obtain speaker features of the second user;

[0163] Extracting fundamental frequency information from the second voice information to obtain fundamental frequency information of the second user;

[0164] Processing the content features, the speaker features of the first user, the fundamental frequency information, and the speaker features of the second user to obtain predicted speech information;

[0165] The speech conversion model is trained according to the predicted speech information and the second speech information to obtain a trained speech conversion model.

[0166] In one embodiment, when the processor 501 trains the speech conversion model according to the predicted speech information and the second speech information to obtain the trained speech conversion model, the processor 501 specifically implements the following process:

[0167] Acquire a first spectrum of predicted speech information and a second spectrum of second speech information;

[0168] Comparing the first spectrum with the second spectrum to obtain a loss value;

[0169] Based on the loss value, the speech conversion model is trained to obtain a trained speech conversion model.

[0170] In an embodiment of the present application, source speech information from a source speaker is obtained, content features of the source speech information are extracted through a trained speech conversion model to obtain content features of the source speech information, speaker features of the source speaker are extracted to obtain speaker features of the source speaker, the content features, the speaker features of the source speaker and the target fundamental frequency information are processed to obtain converted target speech information, and the target speech information is output, which can not only control the fundamental frequency in the target speech information, but also ensure the consistency of the fundamental frequency in the target speech information, eliminate disturbances related to the fundamental frequency in the target speech information, and thus improve the naturalness and rhythmic coherence characteristics of the target speech after speech conversion.

[0171] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM). The computer-readable storage medium can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of blockchain nodes, etc.

[0172] Among them, the blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0173] The above disclosure is only part of the embodiments of the present application, which certainly cannot be used to limit the scope of rights of the present application. Ordinary technicians in this field can understand that all or part of the processes of implementing the above embodiments and making equivalent changes according to the claims of this application are still within the scope of the invention.

Claims

1. A speech conversion method, characterized in that: include: Acquire source speech information from a source speaker; Extracting content features of the source voice information through a trained voice conversion model to obtain content features of the source voice information, wherein the trained voice conversion model includes a conditional autoencoder, and the conditional autoencoder includes an encoder and a decoder; Performing speaker feature extraction on the source voice information to obtain speaker features of the source speaker; The encoder concatenates the speaker feature and the content feature of the source speaker to obtain a first feature, and encodes the first feature to obtain an encoded feature; Concatenating the encoded feature with the target baseband information to obtain a second feature; Decoding the second feature by the decoder to obtain converted target speech information; The target voice information is output.

2. The method according to claim 1, characterized in that Before the encoder splices the speaker features and content features of the source speaker, the method further includes: Display a style setting interface, wherein the style setting interface includes a plurality of style options; In response to a selection operation on a target style option among the plurality of style options, determining a target style option, wherein the target style option is any style option among the plurality of style options; According to the correspondence between the style tag and the baseband information, the baseband information corresponding to the style tag indicated by the target style option is searched in multiple baseband information, and the searched baseband information is used as the target baseband information.

3. The method according to claim 1, characterized in that Before the encoder splices the speaker features and content features of the source speaker, the method further includes: Performing timbre analysis on the source voice information to determine the gender of the source speaker; According to the correspondence between gender and fundamental frequency information, fundamental frequency information corresponding to the determined gender is searched in multiple fundamental frequency information, and the searched fundamental frequency information is used as the target fundamental frequency information.

4. The method according to claim 1, characterized in that Before the encoder splices the speaker features and content features of the source speaker, the method further includes: Extracting fundamental frequency information from the source voice information to obtain fundamental frequency information of the source speaker; When an adjustment operation on the fundamental frequency information of the source speaker is detected, the fundamental frequency information of the source speaker is adjusted to obtain the target fundamental frequency information.

5. A speech conversion device, characterized in that: The device comprises: An acquisition unit, used for acquiring source speech information from a source speaker; A processing unit, configured to extract content features of the source voice information through a trained voice conversion model to obtain content features of the source voice information, wherein the trained voice conversion model comprises a conditional autoencoder, and the conditional autoencoder comprises an encoder and a decoder; The processing unit is further used to extract speaker features from the source voice information to obtain speaker features of the source speaker; The processing unit is further configured to concatenate the speaker feature and the content feature of the source speaker through the encoder to obtain a first feature, encode the first feature to obtain an encoded feature; concatenate the encoded feature with the target fundamental frequency information to obtain a second feature; and decode the second feature through the decoder to obtain the converted target speech information; An output unit is used to output the target voice information.

6. An electronic device, characterized in that: The invention comprises a processor, a memory and a communication interface, wherein the processor, the memory and the communication interface are connected to each other, wherein the memory is used to store computer program instructions, and the processor is configured to execute the program instructions to implement the speech conversion method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, they are used to execute the speech conversion method according to any one of claims 1 to 4.

Images