Voice conversion method and device, computer equipment and storage medium

Abstract

The invention provides a voice conversion method and device, computer equipment and a storage medium. The method comprises the steps of determining a target voice in response to a skin selection operation; when the target voice is determined, extracting a target timbre feature corresponding to the target voice; and after the user inputs the original voice stream, obtaining a target voice stream according to the original voice stream and the target timbre feature. According to the method, feature extraction is carried out by using the idle time from execution of the selection operation by the user to completion of inputting of the original voice stream, so that the time for waiting for voice conversion by the user is greatly shortened, conversion can be completed under a real-time or near-real-time condition, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of audio generation technology, and in particular to a speech conversion method, apparatus, computer equipment and storage medium. Background Art

[0002] Voice conversion is to make someone's words sound like another person's without changing the content of the words. It has great application value in many fields such as driving navigation, video production, game production, voice customer service and voice social networking. Traditional voice conversion technology needs to select a speaker in the sound library as the target and perform voice conversion by extracting the features of the speaker's voice. However, this method has the disadvantages of low conversion efficiency and long waiting time for users. Summary of the invention

[0003] The purpose of this application is to solve at least one of the above-mentioned technical defects, especially the defects of low efficiency and long time of speech conversion in the prior art.

[0004] In a first aspect, the present application provides a speech conversion method, comprising:

[0005] In response to the skin selection operation, determining a target voice;

[0006] When the target speech is determined, a target timbre feature corresponding to the target speech is extracted;

[0007] After the user inputs the original voice stream, the target voice stream is obtained according to the original voice stream and the target timbre characteristics.

[0008] In one embodiment, obtaining a target speech stream according to an original speech stream and a target timbre feature includes:

[0009] Inputting the original speech stream into the first feature extraction model to obtain original speech features; the original speech features include shallow timbre features and semantic features of the original speech stream;

[0010] The target speech stream is obtained according to the original speech features and the target timbre features.

[0011] In one embodiment, the process of constructing the first feature extraction model includes:

[0012] Constructing an initial autoencoder model; the autoencoder model includes a first encoder and a first decoder, the first encoder is used to extract shallow timbre features and semantic features of the input speech and output them to the first decoder, and the first decoder is used to obtain a reconstruction result of the input speech according to the received features;

[0013] With the goal of narrowing the gap between the reconstruction result and the samples of the first speech training set, the initial autoencoder model is trained using the first speech training set;

[0014] The first encoder in the trained autoencoder model is used as the first feature extraction model.

[0015] In one embodiment, the target timbre feature includes a target global timbre feature, and extracting the target timbre feature corresponding to the target speech includes:

[0016] The target speech is input into the second feature extraction model to obtain the target global timbre feature; the dimension of the target global timbre feature is higher than the shallow timbre feature.

[0017] In one embodiment, the construction process of the second feature extraction module includes:

[0018] Constructing an initial classification and recognition model; the classification and recognition model includes a feature extraction part and a classification part, the feature extraction part is used to extract global timbre features from the input speech and output them to the classification part, and the classification part is used to obtain the classification result of the input speech according to the received features;

[0019] With the goal of narrowing the gap between the classification result and the classification label of the second speech training set, the initial classification recognition model is trained using the second speech training set;

[0020] The feature extraction part of the trained classification recognition model is used as the second feature extraction model.

[0021] In one embodiment, obtaining a target speech stream according to original speech features and target timbre features includes:

[0022] The original speech feature is used as input and the target global timbre feature is used as conditional input, and is input into a first feature generation model to obtain a first target speech feature; the first target speech feature includes a semantic feature of the original speech stream and a target global timbre feature;

[0023] A target speech stream is obtained according to the first target speech feature.

[0024] In one embodiment, the process of constructing the first feature generation model includes:

[0025] Constructing an initial first conditional autoencoder model; the first conditional autoencoder model includes a second encoder and a second decoder, the second encoder is used to perform feature fusion on the input and the condition to obtain a first speech feature, the first speech feature includes a semantic feature in the input and a global timbre feature in the condition, and the second decoder is used to restore the masked first speech feature to the first speech feature before masking;

[0026] With the goal of narrowing the gap between the restoration result of the second decoder and the feature fusion result of the second encoder on the first training set of speech features and the global timbre feature training set, the first conditional autoencoder model is trained using the first training set of speech features and the global timbre feature training set;

[0027] The second encoder in the trained first conditional autoencoder model is used as the first feature generation model.

[0028] In one embodiment, with the goal of narrowing the gap between the restoration result of the second decoder and the feature fusion result of the second encoder on the first training set of speech features and the global timbre feature training set, the first conditional autoencoder model is trained using the first training set of speech features and the global timbre feature training set, further comprising:

[0029] The first training set of speech features is subjected to timbre perturbation, and the first conditional autoencoder model is trained with the goal of reducing the difference between feature fusion results of the same speech feature sample and the same global feature training sample before and after the timbre perturbation of the second encoder.

[0030] In one embodiment, the original speech feature is used as input and the target global timbre feature is used as a conditional input, and inputted into a first feature generation model to obtain a first target speech feature, including:

[0031] The original speech features are subjected to timbre perturbation, and the original speech features after timbre perturbation are used as input and the target global timbre features are used as conditional input, and are input into the first feature generation model to obtain the first target speech features.

[0032] In one embodiment, the target timbre feature also includes a target complete timbre feature, and extracting the target timbre feature corresponding to the target speech further includes:

[0033] Feature extraction is performed on the target speech to obtain a target complete timbre feature; the dimension of the target complete timbre feature is higher than the target global timbre feature.

[0034] In one embodiment, extracting features from the target speech includes:

[0035] According to the target speech, generate the corresponding Mel spectrum matrix.

[0036] In one embodiment, obtaining a target speech stream according to the first target speech feature includes:

[0037] The first target speech feature is used as input and the target complete timbre feature is used as conditional input, and inputted into the second feature generation model to obtain the second target speech feature; the second target speech feature includes the semantic feature of the original speech stream and the target complete timbre feature;

[0038] A target speech stream is obtained according to the second target speech feature.

[0039] In one embodiment, the process of constructing the second feature generation model includes:

[0040] Constructing an initial second conditional autoencoder model; the second conditional autoencoder model includes a third encoder and a third decoder, the third encoder is used to fuse the input and the condition to obtain a second speech feature, the second speech feature includes the semantic feature in the input and the complete timbre feature in the condition, and the third decoder is used to restore the masked second speech feature to the second speech feature before masking;

[0041] With the goal of narrowing the gap between the restoration result of the third decoder and the feature fusion result of the third encoder on the second training set of speech features and the complete timbre feature training set, the second conditional autoencoder model is trained using the second training set of speech features and the complete timbre feature training set;

[0042] The third encoder in the trained second conditional autoencoder model is used as the second feature generation model.

[0043] In one embodiment, obtaining a target speech stream according to the second target speech feature includes:

[0044] The second target speech feature is input into the speech generation model to obtain a target speech stream.

[0045] In a second aspect, the present application provides a speech conversion device, comprising:

[0046] a target voice determination module, for determining a target voice in response to a skin selection operation;

[0047] A target timbre feature extraction module is used to extract the target timbre feature corresponding to the target voice when the target voice is determined;

[0048] The voice conversion module is used to obtain a target voice stream according to the original voice stream and the target timbre characteristics after the user inputs the original voice stream.

[0049] In a third aspect, the present application provides a computer device comprising one or more processors and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the steps of the speech conversion method in any of the above embodiments are performed.

[0050] In a fourth aspect, the present application provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the speech conversion method in any of the above embodiments.

[0051] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0052] Based on the voice conversion method in the embodiment of the present application, first respond to the user's skin selection operation, determine the target voice, and at the same time as the target voice is determined, start to perform feature extraction on the target voice to obtain the target timbre features corresponding to the target voice. Finally, when the user inputs the original voice stream, the target voice stream is generated according to the original voice stream and the pre-extracted target timbre features. This method can provide a highly personalized user experience, allowing users to customize voice output according to their preferences. When it is necessary to add a new optional voice skin, it is only necessary to add the corresponding reference voice to the sound library, thereby improving the scalability of the timbre. In addition, it utilizes the idle time between the user performing the selection operation and the input of the original voice stream for feature extraction, which greatly shortens the time the user waits for voice conversion, and can complete the conversion under real-time or near real-time conditions, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0054] Figure 1 A flowchart of a voice conversion method provided by an embodiment of the present application;

[0055] Figure 2 A schematic diagram of a process for generating a target voice stream in one embodiment of the present application;

[0056] Figure 3 A schematic diagram of a process for constructing a first feature extraction model in one embodiment of the present application;

[0057] Figure 4 A schematic diagram of a process for constructing a second feature extraction model in one embodiment of the present application;

[0058] Figure 5 A schematic diagram of a process for generating a target voice stream in another embodiment of the present application;

[0059] Figure 6 A schematic diagram of a process for constructing a first feature generation model in one embodiment of the present application;

[0060] Figure 7 A schematic diagram of a process for generating a target voice stream in another embodiment of the present application;

[0061] Figure 8 A schematic diagram of a process for constructing a second feature generation model in one embodiment of the present application;

[0062] Fig. 9 A flowchart of a voice conversion method provided by another embodiment of the present application;

[0063] Fig.10 An internal structural diagram of a computer device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0065] This application provides a method for voice conversion. Figure 1 , including steps S102 to S106.

[0066] S102, in response to the skin selection operation, determining a target voice.

[0067] It can be understood that the skin selection operation is an operation when the user selects the voice skin he wants to use from multiple optional voice skins on the interactive interface. The target voice is the reference voice corresponding to the voice skin selected by the user, and the voice input by the user will be converted into a target voice stream with similar or identical timbre but unchanged semantics as the target voice. Each optional voice skin has a one-to-one corresponding reference voice, and a mapping relationship can be established between the optional voice skin and the corresponding reference voice. For example, a unique ID is configured for each optional voice skin, and a mapping relationship is established between the unique ID and the reference voice. When the user selects the target voice skin using the skin selection operation, the reference voice corresponding to the unique ID of the target voice skin can be determined as the target voice. In the interactive interface, the audition operation can provide users with a timbre audition of each optional voice skin, making it convenient for users to make choices.

[0068] S104, when the target speech is determined, extracting the target timbre feature corresponding to the target speech.

[0069] It can be understood that the original voice stream is the voice stream to be converted input by the user after selecting the target voice skin. In order to convert the original voice stream into the target voice stream, it is necessary to extract the timbre features related to the timbre (i.e., the target timbre features) from the target voice, so that the original voice stream can be converted with reference to the features. In the conventional technology, the extraction of the target timbre features is started only after the user inputs the original voice stream, and the time the user needs to wait is the sum of the feature extraction time and the timbre conversion time. For feature extraction that needs to rely on a neural network model, the time spent on feature extraction cannot be ignored, which will cause the user to wait too long when using the voice conversion function. Therefore, in this application, the idle time from the time when the user selects the voice skin to the time when the original voice stream is input is selected to perform feature extraction, so that the feature extraction time in the time the user needs to wait is reduced (the idle time is less than the feature extraction time) or completely eliminated (the idle time is greater than or equal to the feature extraction time), thereby shortening the user's waiting time. Of course, if the storage space is large enough, the timbre features of all reference voices can also be extracted and saved in advance, and can be directly read when needed, so that the feature extraction time can be completely eliminated.

[0070] S106, after the user inputs the original voice stream, a target voice stream is obtained according to the original voice stream and the target timbre feature.

[0071] It can be understood that after the target timbre characteristics have been obtained, it is only necessary to wait for the user to input the original voice stream, and then start converting the original voice stream according to the target timbre characteristics, so that the original voice stream is converted into a new voice stream (i.e., the target voice stream) with a timbre similar to or the same as the target voice.

[0072] Based on the voice conversion method in this embodiment, first respond to the user's skin selection operation, determine the target voice, and at the same time as the target voice is determined, start to perform feature extraction on the target voice to obtain the target timbre features corresponding to the target voice. Finally, when the user inputs the original voice stream, the target voice stream is generated according to the original voice stream and the pre-extracted target timbre features. This method can provide a highly personalized user experience, allowing users to customize voice output according to their preferences. When it is necessary to add a new optional voice skin, it is only necessary to add the corresponding reference voice to the sound library, thereby improving the scalability of the timbre. In addition, it utilizes the idle time between the user performing the selection operation and the input of the original voice stream for feature extraction, which greatly shortens the time the user waits for voice conversion, and can complete the conversion under real-time or near real-time conditions, thereby improving the user experience.

[0073] In one embodiment, see Figure 2 , according to the original speech stream and the target timbre characteristics, a target speech stream is obtained, including steps S202 and S204.

[0074] S202, inputting the original speech stream into a first feature extraction model to obtain original speech features. The original speech features include shallow timbre features and semantic features of the original speech stream.

[0075] It can be understood that the first feature extraction model is a neural network model used to extract shallow timbre features and semantic features from the original speech stream. Speech features refer to features that contain both semantic features and timbre features, and the original speech features represent speech features from the original speech stream. Semantic features refer to features related to the speech content in the original speech stream, while shallow timbre features are basic timbre features in the original speech stream. The first feature extraction model compresses the timbre features in the original speech stream to a great extent, only capturing the basic timbre of the speech and ignoring the details.

[0076] The purpose of extracting speech features using the first feature extraction model is to quickly extract the basic features of the original speech, and the semantic features can help regenerate a semantically consistent speech stream. Although the ultimate goal of speech conversion is to convert the timbre of the original speech to the target timbre, some solutions in traditional technologies directly use semantic information to generate speech with the target timbre. However, retaining the shallow timbre features of the original speech can help the system maintain the continuity of the speech during the conversion process. This can make the converted speech sound more natural and reduce the sense of abruptness. In addition, timbre features can sometimes assist in the understanding and expression of semantics. For example, information such as the tone of speech, stress, and emotional changes may be contained in the shallow timbre features, which are important for accurately conveying semantics. At the same time, retaining shallow timbre features can also help improve the quality of conversion. By comparing the original shallow timbre features with the target timbre features, you can better understand how to adjust the timbre in subsequent processing. This before-and-after comparison helps to generate more accurate conversion results.

[0077] S204, obtaining a target speech stream according to the original speech features and the target timbre features.

[0078] It can be understood that the original voice feature retains the semantics and basic timbre of the user's speech, and the user's basic timbre needs to be adjusted, wherein the basis for the adjustment is the target timbre feature.

[0079] In one embodiment, see Figure 3 , the construction process of the first feature extraction model includes steps S302 to S306.

[0080] S302, constructing an initial autoencoder model. The autoencoder model includes a first encoder and a first decoder, wherein the first encoder is used to extract shallow timbre features and semantic features from the input speech and output them to the first decoder, and the first decoder is used to obtain a reconstruction result of the input speech according to the received features.

[0081] S304, with the goal of narrowing the gap between the reconstruction result and the samples of the first speech training set, the initial autoencoder model is trained using the first speech training set.

[0082] S306: Use the first encoder in the trained autoencoder model as a first feature extraction model.

[0083] It can be understood that after a speech is input into the autoencoder model, the first encoder performs feature extraction, and the first decoder restores the speech itself input into the autoencoder model based on the features extracted by the first encoder. Before training, the feature extraction capability of the first encoder and the reconstruction capability of the first decoder do not meet the requirements. Therefore, it is necessary to use the first speech training set for training. The first speech training set includes many speech samples, each of which will be input into the autoencoder model and obtain the corresponding reconstruction result. By adjusting the parameters of the first encoder and the first decoder, the difference between the reconstruction result and the corresponding speech sample can be gradually reduced, and the feature extraction capability of the first encoder and the reconstruction capability of the first decoder can be improved.

[0084] During the training process, since the output dimension of the first encoder is smaller than the input dimension, it is essentially compressing the information carried by the input speech (for example, compressing it into a vector form with only one dimension). The first encoder must extract the most important information for reconstructing the speech from the input, that is, it must contain both timbre information (making the reconstructed speech sound like the original speaker) and semantic information (making the reconstructed speech content correct). Among them, the timbre information has a low dimension due to compression, and will correspond to the shallow timbre features, while the semantic information corresponds to the semantic features. Therefore, the first encoder in the trained autoencoder model has the ability to extract semantic features and shallow timbre features from the input speech, and the first encoder in the autoencoder model will be selected as the first feature extraction model. Here, the first decoder part can be stripped out, and the separate first encoder can be used as the first feature extraction model. The output of the first encoder can also be directly connected to the input end of the subsequent processing flow model. In addition, it is worth mentioning that the first feature extraction model is generally pre-trained before the entire voice conversion function is launched, embedded in the system, and will not be temporarily trained.

[0085] In one embodiment, the target timbre feature includes a target global timbre feature, and extracting the target timbre feature corresponding to the target speech includes: inputting the target speech into a second feature extraction model to obtain the target global timbre feature. The dimension of the target global timbre feature is higher than that of the shallow timbre feature.

[0086] It can be understood that the target global timbre feature is obtained through the second feature extraction model, which represents the overall timbre feature of the speech and has a higher dimension than the shallow timbre feature. However, the second feature extraction model will still compress the timbre feature to a certain extent, which means that it captures the main timbre features of the speech but ignores some of the details. As a result, the speech generated with reference to it can be clearly distinguished as coming from the speaker of the selected voice skin, but lacks some details.

[0087] In one embodiment, see Figure 4 , the construction process of the second feature extraction module includes steps S402 to S406.

[0088] S402, constructing an initial classification recognition model. The classification recognition model includes a feature extraction part and a classification part. The feature extraction part is used to extract global timbre features from the input speech and output them to the classification part. The classification part is used to obtain the classification result of the input speech according to the received features.

[0089] S404, with the goal of narrowing the gap between the classification result and the classification label of the second speech training set, the initial classification recognition model is trained using the second speech training set.

[0090] S406, using the feature extraction part of the trained classification recognition model as the second feature extraction model.

[0091] It can be understood that the global timbre feature refers to a feature set that can comprehensively summarize the timbre of the speech, which includes the overall characteristics of the speaker's voice, such as pitch range, timbre warmth, and voice thickness. These features can better represent the voice characteristics of a speaker, but do not contain subtle timbre details. The classification part determines which category the speech belongs to based on these global timbre features, for example, which speaker it belongs to. Before training, the feature extraction ability of the feature extraction part and the classification ability of the classification part do not meet the requirements. Therefore, it is necessary to use the first speech training set for training. The first speech training set includes many speech samples, each of which is marked with a corresponding classification label (for example, which speaker it belongs to). Each speech sample will be input into the classification recognition model and the corresponding classification result will be obtained. By adjusting the parameters of the classification recognition model, the difference between the classification result and the classification label of the corresponding speech sample can be gradually reduced, and the feature extraction ability of the feature extraction part and the classification ability of the classification part can be improved.

[0092] During the training process, since the output dimension of the feature extraction part is smaller than the input dimension, it is essentially compressing the information carried by the input speech. The feature extraction part must extract information about the classification of the input speech (such as distinguishing who the speaker is) from the input, that is, it must contain both timbre information (making the reconstructed speech sound like the original speaker). Among them, the timbre information is compressed less than the first encoder, so the feature dimension obtained is higher than the shallow timbre feature, which will correspond to the global timbre feature. Therefore, the feature extraction part in the trained classification recognition model has the ability to extract global timbre features from the input speech, and the feature extraction part in the classification recognition model will be selected as the second feature extraction model. Here, the feature extraction part in the classification recognition model can be stripped out and used as the second feature extraction model alone. It is also possible to directly connect the output of the feature extraction part in the classification recognition model to the input end of the subsequent processing flow model. In addition, it is worth mentioning that the second feature extraction model is generally pre-trained before the entire voice conversion function is launched, embedded in the system, and will not be temporarily trained.

[0093] In one embodiment, see Figure 5 , according to the original speech features and the target timbre features, a target speech stream is obtained, including steps S502 to S504.

[0094] S502: Input the original speech feature as input and the target global timbre feature as conditional input to a first feature generation model to obtain a first target speech feature. The first target speech feature includes the semantic feature of the original speech stream and the target global timbre feature.

[0095] S504: Obtain a target speech stream according to the first target speech feature.

[0096] It can be understood that the first feature generation model is a trained neural network, and its main function is to refer to the global timbre features of the conditional input and fuse them with the input voice features, so that the semantic features in the voice features can be retained, but the timbre features are fused with the global timbre features. Therefore, by taking the original voice features as input and the target global timbre features as conditional input, the first target voice features including the semantic features of the original voice stream and the target global timbre features can be obtained. The shallow timbre features from the user in the first target voice features have been replaced by the global timbre features of the target voice. Therefore, the first target voice features can be directly processed using a voice generation model (such as HiFi-GAN) to obtain the target voice stream. This embodiment can realize cross-language, multi-language, and zero-sample voice conversion. Since some details are lost in the global timbre features, in order to improve the conversion effect, the details of the target voice can also be taken into consideration. The specific scheme will be introduced later.

[0097] In one embodiment, see Figure 6 , the construction process of the first feature generation model includes steps S602 to S606.

[0098] S602, construct an initial first conditional autoencoder model. The first conditional autoencoder model includes a second encoder and a second decoder, the second encoder is used to perform feature fusion on the input and the condition to obtain a first speech feature, the first speech feature includes a semantic feature in the input and a global timbre feature in the condition, and the second decoder is used to restore the masked first speech feature to the first speech feature before masking.

[0099] S604, with the goal of narrowing the gap between the restoration result of the second decoder and the feature fusion result of the second encoder on the first training set of speech features and the global timbre feature training set, the first conditional autoencoder model is trained using the first training set of speech features and the global timbre feature training set.

[0100] S606: Use the second encoder in the trained first conditional autoencoder model as a first feature generation model.

[0101] It can be understood that the conditional autoencoder is a variant of the autoencoder, which not only accepts input data, but also accepts an additional conditional input. In this scenario, the first conditional autoencoder model consists of a second encoder and a second decoder. The role of the second encoder is to fuse the input and the condition to generate the first speech feature. The first speech feature is a composite feature that contains the semantic features in the input and the global timbre features in the condition. The second decoder is responsible for restoring the masked first speech feature to the state before masking. The "masking" operation here refers to randomly covering or modifying certain parts of the feature.

[0102] The structural design of the conditional autoencoder requires the second encoder to learn how to effectively combine two different types of input (speech features and global timbre features). The first speech feature training set is a set of speech features of some speech processed by the first feature extraction model, which includes the semantic features and shallow timbre features of these speech. The global timbre feature training set is a set of speech features of other speech processed by the second feature extraction model, which includes the global timbre features of these speech.

[0103] During the training process, by continuously adjusting the parameters of the second encoder and the second decoder, the entire second encoder gradually learns how to optimally fuse the two types of information. The setting of the training goal forces the second encoder to learn to generate feature representations that can still be restored when some information is missing (masked). This means that the generated first speech feature must contain enough information to express both the semantic content of the input and the global timbre characteristics of the condition. Therefore, the second encoder in the trained first conditional autoencoder has the ability to fuse the input and conditions of the entire model to obtain the first speech feature. Finally, the second encoder in the first conditional autoencoder will be selected as the first feature generation model. Here, the second encoder in the first conditional autoencoder can be stripped out and used as the first feature generation model alone. The output of the second encoder in the first conditional autoencoder can also be directly connected to the input end of the subsequent processing flow model. In addition, it is worth mentioning that the first feature generation model is generally pre-trained before the entire speech conversion function is launched, embedded in the system, and will not be temporarily trained.

[0104] In one of the embodiments, the first conditional autoencoder model is trained using the first training set of speech features and the global timbre feature training set with the goal of narrowing the gap between the restoration result of the second decoder and the feature fusion result of the second encoder on the first training set of speech features and the global timbre feature training set, and also includes: timbre perturbation of the first training set of speech features, and training the first conditional autoencoder model with the goal of narrowing the difference between the feature fusion results of the same speech feature sample and the same global feature training sample before and after the timbre perturbation of the second encoder.

[0105] It can be understood that in the process of training, in order to improve the training effect, the first training set of speech features can also be perturbed in timbre, that is, the shallow timbre features of each sample can be adjusted (such as changing the fundamental frequency). In addition, since the shallow timbre features in the input are the parts that need to be replaced and modified, perturbing them adds additional training targets (that is, the second encoder obtains increasingly similar results for the same fusion object before and after the perturbation), so that the second encoder learns that the perturbed part needs to be modified and replaced according to the features in the conditional input during feature fusion, while the unperturbed part needs to be retained. In this way, no matter who the user is speaking, the second encoder can well modify and replace the shallow timbre features in the original voice stream with the global timbre features of the target voice.

[0106] Based on this, before taking the original speech features as input and the target global timbre features as conditional input and inputting them into the first feature generation model to obtain the first target speech features, the original speech features can be timbre perturbed, and finally the original speech features after timbre perturbation are used as input.

[0107] In one embodiment, the target timbre feature also includes a target complete timbre feature, and extracting the target timbre feature corresponding to the target speech further includes: extracting features of the target speech to obtain the target complete timbre feature. The dimension of the target complete timbre feature is higher than the target global timbre feature.

[0108] It can be understood that the target complete timbre feature represents the complete timbre feature of the speech, has a higher dimension than the global timbre feature (in matrix form), and can more comprehensively describe the timbre feature of the speech, including the details and changes lost by the global timbre feature. Specifically, the corresponding Mel spectrum matrix can be generated based on the target speech. The Mel spectrum matrix can capture more detailed features of the speech, especially those related to timbre. It takes into account the auditory characteristics of the human ear and has a higher resolution for the low-frequency part, which is consistent with the way humans perceive speech.

[0109] In one embodiment, see Figure 7 , according to the first target speech feature, obtaining a target speech stream, including step S702 and step S704.

[0110] S702: input the first target speech feature as input and the target complete timbre feature as conditional input to a second feature generation model to obtain a second target speech feature. The second target speech feature includes the semantic feature of the original speech stream and the target complete timbre feature.

[0111] S704: Obtain a target speech stream according to the second target speech feature.

[0112] It can be understood that the second feature generation model is a trained neural network, and its main function is to refer to the complete timbre features of the conditional input and fuse them with the input speech features, so that the semantic features in the speech features can be retained, but the timbre features refer to the complete timbre features to complete the details. Therefore, by taking the first target speech feature as input and the target complete timbre feature as the conditional input, the second target speech feature including the semantic features of the original speech stream and the target complete timbre features can be obtained. The target global timbre features in the second target speech feature have been completed by the global timbre features of the target speech, and the previously missing details are supplemented. Therefore, the second target speech feature can be processed using a speech generation model (such as HiFi-GAN) to obtain the target speech stream. This embodiment adopts a hierarchical and progressive timbre feature fusion generation, from the reference global timbre feature to the reference complete timbre feature to achieve feature fusion from coarse to fine, and gradually replaces the shallow timbre features in the original speech with complete timbre features with complete timbre details of the target speech. Compared with directly using the complete timbre features, the two-step process can enhance the robustness of the system. If the high-dimensional complete timbre features are used directly, unstable results may occur due to noise or irrelevant information in the features. Using low-dimensional global features first can filter out some potential interference and produce more natural and stable conversion results.

[0113] In one embodiment, see Figure 8 , the construction process of the second feature generation model includes step S802 and step S806.

[0114] S802, construct an initial second conditional autoencoder model. The second conditional autoencoder model includes a third encoder and a third decoder, the third encoder is used to fuse the input and the condition to obtain a second speech feature, the second speech feature includes a semantic feature in the input and a complete timbre feature in the condition, and the third decoder is used to restore the masked second speech feature to the second speech feature before masking.

[0115] S804, with the goal of narrowing the gap between the restoration result of the third decoder and the feature fusion result of the third encoder on the second training set of speech features and the complete timbre feature training set, the second conditional autoencoder model is trained using the second training set of speech features and the complete timbre feature training set.

[0116] S806, using the third encoder in the trained second conditional autoencoder model as a second feature generation model.

[0117] It can be understood that the conditional autoencoder is a variant of the autoencoder, which not only accepts input data, but also accepts an additional conditional input. In this scenario, the second conditional autoencoder model consists of a third encoder and a third decoder. The role of the third encoder is to fuse the input and the condition to generate the second speech feature. The second speech feature is a composite feature that contains the semantic features in the input and the complete timbre features in the condition. The third decoder is responsible for restoring the masked second speech feature to the state before masking. The "masking" operation here refers to randomly covering or modifying certain parts of the feature.

[0118] The structural design of the conditional autoencoder requires the third encoder to learn how to effectively combine two different types of input (speech features and complete timbre features). The second training set of speech features is a set of speech features obtained by fusing the features extracted by the first feature extraction model with the features extracted by the second feature extraction model through the first feature generation model. Each sample contains the semantic features extracted by the first feature extraction model and the global timbre features extracted by the second feature extraction model. The complete timbre feature training set is a set of speech features after some speech has been processed by the third feature extraction model, which contains the complete timbre features of these speech.

[0119] During the training process, by continuously adjusting the parameters of the third encoder and the third decoder, the entire third encoder gradually learns how to find the global timbre features from the complete timbre features. The setting of the training goal forces the third encoder to learn to generate feature representations that can still be restored when some information is missing (masked). This means that the generated second speech features must contain enough information to express both the semantic content of the input and the complete timbre features of the condition. Therefore, the third encoder in the trained second conditional autoencoder has the ability to fuse the input and conditions of the entire model to obtain the second speech features. Finally, the third encoder in the second conditional autoencoder will be selected as the second feature generation model. Here, the third encoder in the second conditional autoencoder can be stripped out and used as a separate second feature generation model. The output of the third encoder in the second conditional autoencoder can also be directly connected to the input of the subsequent processing flow model. In addition, it is worth mentioning that the second feature generation model is generally pre-trained before the entire voice conversion function is launched, embedded in the system, and will not be temporarily trained.

[0120] In one of the embodiments, obtaining a target speech stream according to the second target speech feature includes: inputting the second target speech feature into a speech generation model to obtain the target speech stream.

[0121] It can be understood that the second target speech feature is input into the speech generation model. The model receives this feature as input, and then converts this abstract feature representation into an actual audio waveform, that is, the target speech stream, through its internal complex neural network structure. Restoring the audio waveform based on speech features is a relatively mature technology in the field of speech generation. Taking HiFi-GAN as an example, it is a high-fidelity generative adversarial network that is specifically used to convert abstract audio features into actual audio waveforms. The principle of HiFi-GAN is based on the framework of generative adversarial networks (GANs), but is specially optimized for audio generation tasks. In this model, the generator is responsible for converting the input features into audio waveforms, while the discriminator is responsible for distinguishing whether the generated audio is real. Through this adversarial training, the model is able to generate high-quality, natural speech.

[0122] In one embodiment, see Fig. 9 The speech conversion method using the target global timbre feature and the target complete timbre feature includes steps S902 to S910.

[0123] S902, in response to the skin selection operation, determine the target voice. When the target voice is determined, extract the target global timbre feature and the target complete timbre feature corresponding to the target voice; the dimension of the target global timbre feature is lower than the target complete timbre feature, and the target global timbre feature target voice is input into the second feature extraction model to obtain the complete acoustic feature of the target complete timbre feature target voice.

[0124] S904, after the user inputs the original voice stream, the original voice stream is input into the first feature extraction model to obtain the original voice features; the original voice features include the shallow timbre features and semantic features of the original voice stream, and the dimension of the shallow timbre features is lower than the target global timbre features.

[0125] S906, input the original speech feature as input and the target global timbre feature as conditional input to the first feature generation model to obtain the first target speech feature; the first target speech feature includes the semantic feature of the original speech stream and the target global timbre feature.

[0126] S908, taking the first target speech feature as input and the target complete timbre feature as conditional input, input them into the second feature generation model to obtain the second target speech feature; the second target speech feature includes the semantic feature of the original speech stream and the target complete timbre feature.

[0127] S910, input the second target speech feature into the speech generation model to obtain a target speech stream.

[0128] The present application provides a speech conversion device, including a target speech determination module, a target timbre feature extraction module and a speech conversion module. The target speech determination module is used to determine the target speech in response to a skin selection operation. The target timbre feature extraction module is used to extract the target timbre feature corresponding to the target speech when the target speech is determined. The speech conversion module is used to obtain the target speech stream according to the original speech stream and the target timbre feature after the user inputs the original speech stream.

[0129] For the specific definition of the voice conversion device, please refer to the definition of the voice conversion method above, which will not be repeated here. The various modules in the above-mentioned voice conversion device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0130] The present application provides a computer device, including one or more processors and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the speech conversion method in any of the above embodiments are executed.

[0131] Indicatively, Fig.10 As shown, Fig.10 A schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. Fig.10 The computer device 1000 includes a processing component 1002, which further includes one or more processors, and a memory resource represented by a memory 1001, for storing instructions executable by the processing component 1002, such as an application. The application stored in the memory 1001 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1002 is configured to execute instructions to perform the steps of the speech conversion method of any of the above embodiments.

[0132] The computer device 1000 may further include a power supply component 1003 configured to perform power management of the computer device 1000 , a wired or wireless model interface 1004 configured to connect the computer device 1000 to the model, and an input / output (I / O) interface 1005 .

[0133] The present application provides a storage medium, in which computer-readable instructions are stored. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the speech conversion method in any of the above embodiments.

[0134] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0135] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0136] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech conversion method, characterized in that: include: In response to the skin selection operation, determining a target voice; When the target voice is determined, extracting a target timbre feature corresponding to the target voice; After the user inputs an original voice stream, a target voice stream is obtained according to the original voice stream and the target timbre feature.

2. The voice conversion method according to claim 1, characterized in that: The step of obtaining a target voice stream according to the original voice stream and the target timbre feature comprises: Inputting the original speech stream into a first feature extraction model to obtain original speech features; the original speech features include shallow timbre features and semantic features of the original speech stream; The target speech stream is obtained according to the original speech feature and the target timbre feature.

3. The voice conversion method according to claim 2, characterized in that: The construction process of the first feature extraction model includes: Constructing an initial autoencoder model; the autoencoder model includes a first encoder and a first decoder, the first encoder is used to extract the shallow timbre features and the semantic features from the input speech, and output them to the first decoder, and the first decoder is used to obtain a reconstruction result of the input speech according to the received features; With the goal of narrowing the gap between the reconstruction result and the samples of the first speech training set, the initial autoencoder model is trained using the first speech training set; The first encoder in the trained autoencoder model is used as a first feature extraction model.

4. The voice conversion method according to claim 2, characterized in that: The target timbre feature includes a target global timbre feature, and the step of extracting the target timbre feature corresponding to the target speech includes: The target speech is input into a second feature extraction model to obtain the target global timbre feature; the dimension of the target global timbre feature is higher than that of the shallow timbre feature.

5. The voice conversion method according to claim 4, characterized in that: The construction process of the second feature extraction model includes: Constructing an initial classification recognition model; the classification recognition model includes a feature extraction part and a classification part, the feature extraction part is used to extract global timbre features from the input speech and output them to the classification part, and the classification part is used to obtain a classification result of the input speech according to the received features; With the goal of narrowing the gap between the classification result and the classification label of the second speech training set, the initial classification recognition model is trained using the second speech training set; The feature extraction part of the trained classification recognition model is used as the second feature extraction model.

6. The method for voice conversion according to claim 4, characterized in that: The step of obtaining the target speech stream according to the original speech feature and the target timbre feature comprises: The original speech feature is used as input and the target global timbre feature is used as conditional input, and inputted into a first feature generation model to obtain a first target speech feature; the first target speech feature includes the semantic feature of the original speech stream and the target global timbre feature; The target speech stream is obtained according to the first target speech feature.

7. The voice conversion method according to claim 6, characterized in that: The construction process of the first feature generation model includes: Constructing an initial first conditional autoencoder model; the first conditional autoencoder model includes a second encoder and a second decoder, the second encoder is used to perform feature fusion on the input and the condition to obtain a first speech feature, the first speech feature includes a semantic feature in the input and a global timbre feature in the condition, and the second decoder is used to restore the masked first speech feature to the first speech feature before masking; The first conditional autoencoder model is trained using the first speech feature training set and the global timbre feature training set with the goal of narrowing the gap between the restoration result of the second decoder and the feature fusion result of the second encoder on the first speech feature training set and the global timbre feature training set; The second encoder in the first conditional autoencoder model that has been trained is used as a first feature generation model.

8. The method for voice conversion according to claim 7, characterized in that: The training of the first conditional autoencoder model using the first speech feature training set and the global timbre feature training set also includes: The first training set of speech features is subjected to timbre perturbation, and the first conditional autoencoder model is trained with the goal of reducing the difference between feature fusion results of the same speech feature sample and the same global feature training sample before and after the timbre perturbation of the second encoder.

9. The method for voice conversion according to claim 8, characterized in that: The method of taking the original speech feature as input and the target global timbre feature as conditional input and inputting them into a first feature generation model to obtain a first target speech feature includes: The timbre perturbation is performed on the original speech feature, and the original speech feature after the timbre perturbation is used as input and the target global timbre feature is used as a conditional input, and input into the first feature generation model to obtain the first target speech feature.

10. The voice conversion method according to claim 6, characterized in that: The target timbre feature also includes a target complete timbre feature, and the step of extracting the target timbre feature corresponding to the target speech further includes: Feature extraction is performed on the target speech to obtain the target complete timbre feature; the dimension of the target complete timbre feature is higher than the target global timbre feature.

11. The method for voice conversion according to claim 10, characterized in that: The feature extraction of the target speech comprises: According to the target speech, a corresponding mel spectrum matrix is ​​generated.

12. The method for voice conversion according to claim 10, characterized in that: The step of obtaining the target speech stream according to the first target speech feature includes: The first target speech feature is used as input and the target complete timbre feature is used as conditional input, and inputted into a second feature generation model to obtain a second target speech feature; the second target speech feature includes the semantic feature of the original speech stream and the target complete timbre feature; The target speech stream is obtained according to the second target speech feature.

13. The voice conversion method according to claim 12, characterized in that: The construction process of the second feature generation model includes: Constructing an initial second conditional autoencoder model; the second conditional autoencoder model includes a third encoder and a third decoder, the third encoder is used to fuse the input and the condition to obtain a second speech feature, the second speech feature includes a semantic feature in the input and a complete timbre feature in the condition, and the third decoder is used to restore the masked second speech feature to the second speech feature before masking; The second conditional autoencoder model is trained using the second speech feature training set and the complete timbre feature training set with the goal of narrowing the gap between the restoration result of the third decoder and the feature fusion result of the third encoder on the second speech feature training set and the complete timbre feature training set; The third encoder in the trained second conditional autoencoder model is used as a second feature generation model.

14. The method for voice conversion according to claim 12, characterized in that: The step of obtaining the target speech stream according to the second target speech feature includes: The second target speech feature is input into a speech generation model to obtain the target speech stream.

15. A speech conversion device, characterized in that: include: a target voice determination module, for determining a target voice in response to a skin selection operation; A target timbre feature extraction module, used to extract the target timbre feature corresponding to the target voice when the target voice is determined; The voice conversion module is used to obtain a target voice stream according to the original voice stream and the target timbre feature after the user inputs the original voice stream.

16. A computer device, characterized in that: The method comprises one or more processors and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the speech conversion method according to any one of claims 1 to 14 are executed.

17. A storage medium, characterized in that: The storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the speech conversion method according to any one of claims 1 to 14.

Images