A voice conversion method, device, computer equipment and storage medium

Abstract

The present application provides a method, apparatus, computer device and storage medium for voice conversion. The method includes: determining a target voice in response to a skin selection operation; extracting a target timbre feature corresponding to the target voice when the target voice is determined; and obtaining a target voice stream according to the original voice stream and the target timbre feature after the user inputs the original voice stream. The method utilizes the idle time between the user performing the selection operation and the input of the original voice stream for feature extraction, which greatly shortens the time the user waits for voice conversion, and can complete the conversion in real time or near real time, thereby improving the user experience.

Description

Technical Field

[0001] The present application relates to the field of audio generation technology, and in particular to a voice conversion method, apparatus, computer equipment, and storage medium. Background Art

[0002] Speech conversion allows one person to sound like another without changing the content of the speech. It has demonstrated significant application value in a variety of fields, including driving navigation, video production, game development, voice customer service, and voice-based social networking. Traditional speech conversion technology requires selecting a specific speaker from a database and extracting features from that speaker's voice to perform speech conversion. However, this approach suffers from low conversion efficiency and long user wait times. Summary of the Invention

[0003] The purpose of this application is to solve at least one of the above-mentioned technical deficiencies, especially the deficiencies of low efficiency and long time required for voice conversion in the prior art.

[0004] In a first aspect, the present application provides a speech conversion method, comprising:

[0005] In response to the skin selection operation, determining a target voice;

[0006] When the target speech is determined, the target timbre features corresponding to the target speech are extracted;

[0007] After the user inputs the original voice stream, the target voice stream is obtained according to the original voice stream and the target timbre characteristics.

[0008] In one embodiment, obtaining a target speech stream according to an original speech stream and target timbre features includes:

[0009] Inputting the original speech stream into the first feature extraction model to obtain original speech features; the original speech features include shallow timbre features and semantic features of the original speech stream;

[0010] The target speech stream is obtained according to the original speech features and the target timbre features.

[0011] In one embodiment, the process of constructing the first feature extraction model includes:

[0012] Constructing an initial autoencoder model; the autoencoder model includes a first encoder and a first decoder, wherein the first encoder is used to extract shallow timbre features and semantic features from the input speech and output them to the first decoder, and the first decoder is used to obtain a reconstruction result of the input speech based on the received features;

[0013] With the goal of narrowing the gap between the reconstruction results and the samples in the first speech training set, the initial autoencoder model is trained using the first speech training set;

[0014] The first encoder in the trained autoencoder model is used as the first feature extraction model.

[0015] In one embodiment, the target timbre feature includes a target global timbre feature, and extracting the target timbre feature corresponding to the target speech includes:

[0016] The target speech is input into the second feature extraction model to obtain the target global timbre feature; the dimension of the target global timbre feature is higher than the shallow timbre feature.

[0017] In one embodiment, the construction process of the second feature extraction module includes:

[0018] Constructing an initial classification and recognition model; the classification and recognition model includes a feature extraction part and a classification part. The feature extraction part is used to extract global timbre features from the input speech and output them to the classification part. The classification part is used to obtain a classification result for the input speech based on the received features.

[0019] With the goal of narrowing the gap between the classification results and the classification labels of the second speech training set, the initial classification and recognition model is trained using the second speech training set;

[0020] The feature extraction part of the trained classification recognition model is used as the second feature extraction model.

[0021] In one embodiment, obtaining a target speech stream based on original speech features and target timbre features includes:

[0022] The original speech feature is used as input and the target global timbre feature is used as a conditional input, and inputted into a first feature generation model to obtain a first target speech feature; the first target speech feature includes the semantic feature of the original speech stream and the target global timbre feature;

[0023] A target speech stream is obtained according to the first target speech feature.

[0024] In one embodiment, the process of constructing the first feature generation model includes:

[0025] Constructing an initial first conditional autoencoder model; the first conditional autoencoder model includes a second encoder and a second decoder, the second encoder is used to fuse the input and the condition to obtain a first speech feature, the first speech feature including the semantic feature in the input and the global timbre feature in the condition, and the second decoder is used to restore the masked first speech feature to the first speech feature before masking;

[0026] With the goal of narrowing the gap between the restoration result of the second decoder and the feature fusion result of the second encoder on the first training set of speech features and the global timbre feature training set, the first conditional autoencoder model is trained using the first training set of speech features and the global timbre feature training set;

[0027] The second encoder in the trained first conditional autoencoder model is used as the first feature generation model.

[0028] In one embodiment, with the goal of narrowing the gap between the restoration result of the second decoder and the feature fusion result of the second encoder on the first training set of speech features and the global timbre feature training set, training the first conditional autoencoder model using the first training set of speech features and the global timbre feature training set further includes:

[0029] The first training set of speech features is subjected to timbre perturbation, and the first conditional autoencoder model is trained with the goal of reducing the difference between the feature fusion results of the same speech feature sample and the same global feature training sample before and after the timbre perturbation of the second encoder.

[0030] In one embodiment, the original speech feature is used as input and the target global timbre feature is used as a conditional input, and inputted into a first feature generation model to obtain the first target speech feature, including:

[0031] The original speech features are subjected to timbre perturbation, and the original speech features after timbre perturbation are used as input and the target global timbre features are used as conditional input, which are input into the first feature generation model to obtain the first target speech features.

[0032] In one embodiment, the target timbre feature also includes a target complete timbre feature, and extracting the target timbre feature corresponding to the target speech further includes:

[0033] Feature extraction is performed on the target speech to obtain a target complete timbre feature; the dimension of the target complete timbre feature is higher than the target global timbre feature.

[0034] In one embodiment, extracting features from the target speech includes:

[0035] According to the target speech, generate the corresponding Mel spectrum matrix.

[0036] In one embodiment, obtaining a target speech stream according to the first target speech feature includes:

[0037] The first target speech feature is used as input and the target complete timbre feature is used as a conditional input, and inputted into the second feature generation model to obtain the second target speech feature; the second target speech feature includes the semantic feature of the original speech stream and the target complete timbre feature;

[0038] A target speech stream is obtained according to the second target speech feature.

[0039] In one embodiment, the process of constructing the second feature generation model includes:

[0040] Constructing an initial second conditional autoencoder model; the second conditional autoencoder model includes a third encoder and a third decoder. The third encoder is used to fuse the input and the condition to obtain a second speech feature. The second speech feature includes the semantic features in the input and the complete timbre features in the condition. The third decoder is used to restore the masked second speech feature to the second speech feature before masking.

[0041] With the goal of narrowing the gap between the restoration result of the third decoder and the feature fusion result of the third encoder on the second training set of speech features and the complete timbre feature training set, the second conditional autoencoder model is trained using the second training set of speech features and the complete timbre feature training set;

[0042] The third encoder in the trained second conditional autoencoder model is used as the second feature generation model.

[0043] In one embodiment, obtaining a target speech stream according to the second target speech feature includes:

[0044] The second target speech feature is input into the speech generation model to obtain a target speech stream.

[0045] In a second aspect, the present application provides a speech conversion device, comprising:

[0046] a target voice determination module, configured to determine a target voice in response to a skin selection operation;

[0047] A target timbre feature extraction module is used to extract the target timbre feature corresponding to the target voice when the target voice is determined;

[0048] The voice conversion module is used to obtain a target voice stream according to the original voice stream and the target timbre characteristics after the user inputs the original voice stream.

[0049] In a third aspect, the present application provides a computer device comprising one or more processors and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the steps of the speech conversion method in any of the above embodiments are performed.

[0050] In a fourth aspect, the present application provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the speech conversion method in any of the above embodiments.

[0051] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0052] Based on the voice conversion method in the embodiment of the present application, the user's skin selection operation is first responded to and the target voice is determined. When the target voice is determined, feature extraction is started on the target voice to obtain the target timbre features corresponding to the target voice. Finally, after the user inputs the original voice stream, the target voice stream is generated based on the original voice stream and the pre-extracted target timbre features. This method can provide a highly personalized user experience, allowing users to customize voice output according to their preferences. When a new optional voice skin needs to be added, it is only necessary to add the corresponding reference voice to the sound library, thereby improving the scalability of the timbre. In addition, it utilizes the idle time between the user performing the selection operation and the input of the original voice stream for feature extraction, which greatly shortens the time the user waits for voice conversion and can complete the conversion under real-time or near real-time conditions, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0054] Figure 1 A flowchart of a voice conversion method provided in one embodiment of the present application;

[0055] Figure 2 A schematic diagram of a process for generating a target voice stream in one embodiment of the present application;

[0056] Figure 3 A schematic diagram of a process for constructing a first feature extraction model in one embodiment of the present application;

[0057] Figure 4 A schematic diagram of a process for constructing a second feature extraction model in one embodiment of the present application;

[0058] Figure 5 This is a flow chart of generating a target voice stream in another embodiment of the present application;

[0059] Figure 6 A schematic diagram of a process for constructing a first feature generation model in one embodiment of the present application;

[0060] Figure 7 This is a flow chart of generating a target voice stream in another embodiment of the present application;

[0061] Figure 8 A schematic diagram of a process for constructing a second feature generation model in one embodiment of the present application;

[0062] Figure 9 A flowchart of a voice conversion method provided in another embodiment of the present application;

[0063] Figure 10 A diagram of the internal structure of a computer device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0065] This application provides a method for voice conversion. Figure 1 , including steps S102 to S106.

[0066] S102: In response to the skin selection operation, determine the target voice.

[0067] It can be understood that the skin selection operation is an operation when the user selects the voice skin he wants to use from multiple optional voice skins on the interactive interface. The target voice is the reference voice corresponding to the voice skin selected by the user, and the voice input by the user will be converted into a target voice stream with similar or identical timbre but unchanged semantics as the target voice. Each optional voice skin has a one-to-one corresponding reference voice, and a mapping relationship can be established between the optional voice skin and the corresponding reference voice. For example, a unique ID is configured for each optional voice skin, and a mapping relationship is established between the unique ID and the reference voice. When the user selects the target voice skin using the skin selection operation, the reference voice corresponding to the unique ID of the target voice skin can be determined as the target voice. In the interactive interface, the audition operation can provide users with a timbre audition of each optional voice skin, making it easier for users to make a choice.

[0068] S104: When the target speech is determined, extract the target timbre features corresponding to the target speech.

[0069] It can be understood that the original voice stream is the voice stream to be converted, which is input by the user after selecting a target voice skin. To convert the original voice stream into the target voice stream, timbre-related timbre features (i.e., target timbre features) must be extracted from the target voice stream. This timbre feature can then be used to convert the original voice stream. Conventional technology only extracts target timbre features after the user inputs the original voice stream, requiring the user to wait for the sum of the feature extraction time and the timbre conversion time. For feature extraction that relies on a neural network model, the time required for feature extraction is significant, resulting in excessively long wait times for users when using the voice conversion function. Therefore, in this application, feature extraction is performed during the idle time between the user selecting a voice skin and the completion of the original voice stream input. This reduces the feature extraction time (the idle time is less than the feature extraction time) or eliminates it entirely (the idle time is greater than or equal to the feature extraction time), thereby shortening user wait time. Of course, if storage space is sufficient, the timbre features of all reference voices can also be extracted and stored in advance, allowing them to be directly accessed when needed, thus completely eliminating feature extraction time.

[0070] S106: After the user inputs the original voice stream, a target voice stream is obtained according to the original voice stream and the target timbre feature.

[0071] It can be understood that after the target timbre characteristics have been obtained, it is only necessary to wait for the user to input the original voice stream, and then start converting the original voice stream according to the target timbre characteristics, so that the original voice stream is converted into a new voice stream (i.e., the target voice stream) with a timbre similar to or the same as the target voice.

[0072] Based on the voice conversion method in this embodiment, the user's skin selection operation is first responded to and the target voice is determined. When the target voice is determined, feature extraction is started on the target voice to obtain the target timbre features corresponding to the target voice. Finally, after the user inputs the original voice stream, the target voice stream is generated based on the original voice stream and the pre-extracted target timbre features. This method can provide a highly personalized user experience, allowing users to customize voice output according to their preferences. When a new optional voice skin needs to be added, it only needs to add the corresponding reference voice to the sound library, thereby improving the scalability of the timbre. In addition, it utilizes the idle time between the user performing the selection operation and the input of the original voice stream for feature extraction, which greatly shortens the time the user waits for voice conversion and can complete the conversion under real-time or near real-time conditions, thereby improving the user experience.

[0073] In one embodiment, see Figure 2 , according to the original speech stream and the target timbre characteristics, a target speech stream is obtained, including steps S202 and S204.

[0074] S202: Input the original speech stream into a first feature extraction model to obtain original speech features. The original speech features include shallow timbre features and semantic features of the original speech stream.

[0075] It can be understood that the first feature extraction model is a neural network model used to extract shallow timbre features and semantic features from the original speech stream. Speech features refer to features that contain both semantic features and timbre features, while raw speech features represent speech features from the original speech stream. Semantic features refer to features related to the speech content in the original speech stream, while shallow timbre features are the basic timbre features in the original speech stream. The first feature extraction model significantly compresses the timbre features in the original speech stream, capturing only the basic timbre of the speech while ignoring the details.

[0076] The goal of extracting speech features using the first feature extraction model is to quickly extract the essential features of the original speech. Semantic features can help regenerate a semantically consistent speech stream. While the ultimate goal of speech conversion is to convert the timbre of the original speech to the target timbre, some conventional approaches directly utilize semantic information to generate speech with the target timbre. However, preserving the shallow timbre features of the original speech can help the system maintain speech continuity during the conversion process. This makes the converted speech sound more natural and less abrupt. Furthermore, timbre features can sometimes assist in understanding and expressing semantic meaning. For example, information such as tone, emphasis, and emotional inflection in speech may be contained in shallow timbre features, which are crucial for accurately conveying semantic meaning. Preserving shallow timbre features can also help improve conversion quality. By comparing the original shallow timbre features with the target timbre features, subsequent processing can better understand the necessary timbre adjustments. This comparison helps produce more accurate conversion results.

[0077] S204: Obtain a target speech stream based on the original speech features and the target timbre features.

[0078] It is understandable that the original speech features retain the semantics and basic timbre of the user's speech, and the user's basic timbre needs to be adjusted. The basis for the adjustment is the target timbre features.

[0079] In one embodiment, see Figure 3 The construction process of the first feature extraction model includes steps S302 to S306.

[0080] S302: Build an initial autoencoder model. The autoencoder model includes a first encoder and a first decoder. The first encoder is used to extract shallow timbre and semantic features from the input speech and output them to the first decoder. The first decoder is used to reconstruct the input speech based on the received features.

[0081] S304 , with the goal of narrowing the gap between the reconstruction result and the samples in the first speech training set, the initial autoencoder model is trained using the first speech training set.

[0082] S306: Use the first encoder in the trained autoencoder model as a first feature extraction model.

[0083] It can be understood that after a speech is input into the autoencoder model, the first encoder performs feature extraction, and the first decoder restores the speech input into the autoencoder model based on the features extracted by the first encoder. Before training, the feature extraction capabilities of the first encoder and the reconstruction capabilities of the first decoder do not meet the requirements. Therefore, training is required using the first speech training set. The first speech training set includes many speech samples, each of which will be input into the autoencoder model and a corresponding reconstruction result will be obtained. By adjusting the parameters of the first encoder and the first decoder, the difference between the reconstruction result and the corresponding speech sample can be gradually reduced, thereby improving the feature extraction capabilities of the first encoder and the reconstruction capabilities of the first decoder.

[0084] During training, since the output dimension of the first encoder is smaller than the input dimension, it essentially compresses the information carried by the input speech (for example, compressing it into a single-dimensional vector). The first encoder must extract the most important information from the input for reconstructing the speech: both timbre information (ensuring that the reconstructed speech sounds like the original speaker) and semantic information (ensuring that the reconstructed speech content is correct). The timbre information, due to its compression, has a lower dimensionality and corresponds to shallow timbre features, while the semantic information corresponds to semantic features. Therefore, the first encoder in the trained autoencoder model is capable of extracting both semantic and shallow timbre features from the input speech, and is therefore selected as the first feature extraction model. The first decoder can be separated, with the standalone first encoder serving as the first feature extraction model. Alternatively, the output of the first encoder can be directly connected to the input of a subsequent processing model. It is also worth noting that the first feature extraction model is typically pre-trained before the entire speech conversion function is launched, embedded in the system, and not retrained on an ad hoc basis.

[0085] In one embodiment, the target timbre feature includes a target global timbre feature, and extracting the target timbre feature corresponding to the target speech includes: inputting the target speech into a second feature extraction model to obtain the target global timbre feature. The target global timbre feature has a higher dimension than the shallow timbre feature.

[0086] It can be understood that the target global timbre feature is obtained through the second feature extraction model. It represents the overall timbre feature of the speech and has a higher dimension than the shallow timbre feature. However, the second feature extraction model will still compress the timbre feature to a certain extent, which means that it captures the main timbre features of the speech but ignores some of the details. As a result, the speech generated with reference to it can be clearly distinguished as coming from the speaker of the selected speech skin, but lacks some details.

[0087] In one embodiment, see Figure 4 The construction process of the second feature extraction module includes steps S402 to S406.

[0088] S402: Build an initial classification and recognition model. The classification and recognition model includes a feature extraction part and a classification part. The feature extraction part is used to extract global timbre features from the input speech and output them to the classification part. The classification part is used to obtain a classification result for the input speech based on the received features.

[0089] S404 , with the goal of narrowing the gap between the classification result and the classification label of the second speech training set, the initial classification recognition model is trained using the second speech training set.

[0090] S406: Using the feature extraction part of the trained classification recognition model as the second feature extraction model.

[0091] It can be understood that global timbre features refer to a set of features that comprehensively summarize the timbre of speech. They encompass the overall characteristics of a speaker's voice, such as pitch range, timbre warmth, and voice roughness. These features can well represent the characteristics of a speaker's voice, but do not capture subtle timbre details. The classification component uses these global timbre features to determine the speech's category, for example, the speaker. Before training, the feature extraction capabilities of the feature extraction component and the classification capabilities of the classification component did not meet the requirements. Therefore, training was performed using the first speech training set. The first speech training set includes many speech samples, each labeled with a corresponding classification label (for example, speaker identity). Each speech sample is input into the classification recognition model, generating a corresponding classification result. By adjusting the parameters of the classification recognition model, the discrepancy between the classification result and the corresponding speech sample's classification label can be gradually narrowed, thereby improving the feature extraction and classification capabilities of the feature extraction and classification components.

[0092] During the training process, since the output dimension of the feature extraction component is smaller than the input dimension, it essentially compresses the information carried by the input speech. Therefore, the feature extraction component must extract information from the input to classify the input speech (e.g., distinguish the speaker), which means it must also include timbre information (to ensure that the reconstructed speech sounds like the original speaker). Because timbre information undergoes less compression than the first encoder, the resulting feature dimension is higher than the shallow timbre features and corresponds to the global timbre features. Therefore, the feature extraction component of the trained classification and recognition model is capable of extracting global timbre features from the input speech. This feature extraction component of the classification and recognition model is selected as the second feature extraction model. The feature extraction component of the classification and recognition model can be separated and used as a separate second feature extraction model. Alternatively, the output of the feature extraction component of the classification and recognition model can be directly connected to the input of the subsequent processing flow model. It is also worth noting that the second feature extraction model is generally pre-trained before the entire speech conversion function is launched, embedded in the system, and not temporarily trained.

[0093] In one embodiment, see Figure 5 , obtaining a target speech stream according to the original speech features and the target timbre features, including steps S502 to S504.

[0094] S502: Input the original speech features as input and the target global timbre features as conditional input to a first feature generation model to obtain first target speech features. The first target speech features include the semantic features of the original speech stream and the target global timbre features.

[0095] S504: Obtain a target speech stream according to the first target speech feature.

[0096] It can be understood that the first feature generation model is a trained neural network whose main function is to reference the global timbre features of the conditional input and fuse them with the input speech features, so that the semantic features in the speech features are retained but the timbre features are fused with the global timbre features. Therefore, by using the original speech features as input and the target global timbre features as the conditional input, a first target speech feature can be obtained that includes the semantic features of the original speech stream and the target global timbre features. The shallow timbre features from the user in the first target speech feature have been replaced by the global timbre features of the target speech. Therefore, the first target speech feature can be directly processed using a speech generation model (such as HiFi-GAN) to obtain the target speech stream. This embodiment can achieve cross-language, multilingual, zero-shot speech conversion. Because some details are lost in the global timbre features, in order to improve the conversion effect, the details of the target speech can also be taken into consideration. The specific solution will be described later.

[0097] In one embodiment, see Figure 6 The construction process of the first feature generation model includes steps S602 to S606.

[0098] S602: Construct an initial first conditional autoencoder model. The first conditional autoencoder model includes a second encoder and a second decoder. The second encoder is used to perform feature fusion on the input and the condition to obtain a first speech feature. The first speech feature includes semantic features in the input and global timbre features in the condition. The second decoder is used to restore the masked first speech feature to the unmasked first speech feature.

[0099] S604, with the goal of narrowing the gap between the restoration result of the second decoder and the feature fusion result of the second encoder on the first training set of speech features and the global timbre feature training set, the first conditional autoencoder model is trained using the first training set of speech features and the global timbre feature training set.

[0100] S606: Use the second encoder in the trained first conditional autoencoder model as a first feature generation model.

[0101] As you can understand, a conditional autoencoder is a variant of an autoencoder that accepts not only input data but also an additional conditional input. In this scenario, the first conditional autoencoder model consists of a second encoder and a second decoder. The second encoder fuses the input and conditional features to generate the first speech feature. The first speech feature is a composite feature that combines the semantic features of the input and the global timbre features of the conditional. The second decoder is responsible for restoring the masked first speech feature to its pre-masking state. The "masking" operation here refers to randomly masking or modifying certain parts of the feature.

[0102] The structural design of the conditional autoencoder requires the second encoder to learn how to effectively combine two different types of input (speech features and global timbre features). The first speech feature training set is a collection of speech features processed by the first feature extraction model, including the semantic features and shallow timbre features of these speech sounds. The global timbre feature training set is a collection of speech features processed by the second feature extraction model, including the global timbre features of these speech sounds.

[0103] During training, by continuously adjusting the parameters of the second encoder and decoder, the entire second encoder gradually learns how to optimally integrate these two types of information. The training objective forces the second encoder to learn to generate feature representations that can be restored even when some information is missing (masked). This means that the generated first speech features must contain sufficient information to both express the semantic content of the input and reflect the global timbre characteristics of the condition. Therefore, the second encoder in the trained first conditional autoencoder is capable of integrating the input and condition of the entire model to generate the first speech features. Ultimately, the second encoder in the first conditional autoencoder is selected as the first feature generation model. The second encoder in the first conditional autoencoder can be separated and used as a separate first feature generation model. Alternatively, the output of the second encoder in the first conditional autoencoder can be directly connected to the input of the subsequent processing flow model. It is also worth noting that the first feature generation model is generally pre-trained before the entire speech conversion function is launched and is embedded in the system, without any temporary training.

[0104] In one embodiment, the first conditional autoencoder model is trained using the first speech feature training set and the global timbre feature training set with the goal of narrowing the gap between the restoration result of the second decoder and the feature fusion result of the second encoder on the first speech feature training set and the global timbre feature training set. The method also includes: performing timbre perturbation on the first speech feature training set, and training the first conditional autoencoder model with the goal of narrowing the difference between the feature fusion results of the same speech feature sample and the same global feature training sample before and after the timbre perturbation of the second encoder.

[0105] It is understandable that during the training process, in order to improve the training effect, the first training set of speech features can also be subjected to timbre perturbations, that is, the shallow timbre features of each sample can be adjusted (such as changing the fundamental frequency). Moreover, since the shallow timbre features in the input are the parts that need to be replaced and modified, perturbing them adds an additional training goal (that is, the second encoder obtains increasingly similar results for the same fusion object before and after the perturbation), so that the second encoder learns that the perturbed part needs to be modified and replaced according to the features in the conditional input during feature fusion, while the unperturbed part needs to be retained. In this way, regardless of the user who is speaking, the second encoder can effectively modify and replace the shallow timbre features in the original speech stream with the global timbre features of the target speech.

[0106] Based on this, before the original speech features are taken as input and the target global timbre features are taken as conditional input and input into the first feature generation model to obtain the first target speech features, the original speech features can be timbre disturbed, and finally the original speech features after timbre disturbance are taken as input.

[0107] In one embodiment, the target timbre feature also includes a target complete timbre feature, and extracting the target timbre feature corresponding to the target speech further includes: extracting features of the target speech to obtain the target complete timbre feature. The dimension of the target complete timbre feature is higher than that of the target global timbre feature.

[0108] As you can understand, the target complete timbre feature represents the complete timbre characteristics of the speech. It has a higher dimensionality (in matrix form) than the global timbre feature and can more comprehensively describe the timbre characteristics of the speech, including details and variations lost by the global timbre feature. Specifically, a corresponding mel-spectrogram matrix can be generated based on the target speech. The mel-spectrogram matrix can capture more detailed features of the speech, especially those related to timbre. It takes into account the auditory characteristics of the human ear and has higher resolution for low frequencies, which is consistent with how humans perceive speech.

[0109] In one embodiment, see Figure 7 , obtaining a target speech stream according to the first target speech feature, including step S702 and step S704.

[0110] S702: Input the first target speech feature as input and the target complete timbre feature as conditional input to a second feature generation model to obtain a second target speech feature. The second target speech feature includes the semantic feature of the original speech stream and the target complete timbre feature.

[0111] S704: Obtain a target speech stream according to the second target speech feature.

[0112] It can be understood that the second feature generation model is a trained neural network whose primary function is to reference the complete timbre features of the conditional input and fuse them with the input speech features. This preserves the semantic features of the speech features, while the timbre features are supplemented with details by referencing the complete timbre features. Therefore, by using the first target speech features as input and the target complete timbre features as conditional input, a second target speech feature can be obtained that includes the semantic features of the original speech stream and the target complete timbre features. The target global timbre features in the second target speech feature have been supplemented by the global timbre features of the target speech, filling in the previously missing details. Therefore, a speech generation model (such as HiFi-GAN) can be used to process the second target speech feature to obtain the target speech stream. This embodiment employs a hierarchical and progressive timbre feature fusion generation approach, achieving coarse-to-fine feature fusion from reference to the global timbre features to reference to the complete timbre features. This two-step process enhances system robustness compared to directly using the complete timbre features. Directly using high-dimensional complete timbre features can lead to unstable results due to noise or irrelevant information in the features. Using low-dimensional global features first can filter out some potential interference and produce more natural and stable conversion results.

[0113] In one embodiment, see Figure 8 The construction process of the second feature generation model includes step S802 and step S806.

[0114] S802: Construct an initial second conditional autoencoder model. The second conditional autoencoder model includes a third encoder and a third decoder. The third encoder is configured to fuse the input and the condition to obtain a second speech feature. The second speech feature includes semantic features from the input and complete timbre features from the condition. The third decoder is configured to restore the masked second speech feature to the pre-masked second speech feature.

[0115] S804, with the goal of narrowing the gap between the restoration result of the third decoder and the feature fusion result of the third encoder on the second training set of speech features and the complete timbre feature training set, the second conditional autoencoder model is trained using the second training set of speech features and the complete timbre feature training set.

[0116] S806: Use the third encoder in the trained second conditional autoencoder model as a second feature generation model.

[0117] As you can understand, a conditional autoencoder is a variant of an autoencoder that accepts not only input data but also an additional conditional input. In this scenario, the second conditional autoencoder model consists of a third encoder and a third decoder. The third encoder fuses the input and conditional features to generate a second speech feature. This second speech feature is a composite feature that combines the semantic features of the input and the complete timbre features of the conditional. The third decoder is responsible for restoring the masked second speech feature to its pre-masking state. "Masking" here refers to randomly masking or modifying certain parts of the feature.

[0118] The structural design of the conditional autoencoder requires the third encoder to learn how to effectively combine two different types of input (speech features and complete timbre features). The second speech feature training set is a collection of speech features obtained by fusing the features extracted by the first feature extraction model with the features extracted by the second feature extraction model through the first feature generation model. Each sample contains the semantic features extracted by the first feature extraction model and the global timbre features extracted by the second feature extraction model. The complete timbre feature training set is a collection of speech features processed by the third feature extraction model, including the complete timbre features of these speech sounds.

[0119] During training, by continuously adjusting the parameters of the third encoder and decoder, the entire third encoder gradually learns how to extract global timbre features from complete timbre features. The training objective forces the third encoder to learn to generate feature representations that can be restored even when some information is missing (masked). This means that the generated second speech features must contain sufficient information to both express the semantic content of the input and reflect the complete timbre characteristics of the condition. Therefore, the trained third encoder in the second conditional autoencoder is capable of integrating the input and condition of the entire model to obtain the second speech features. Ultimately, the third encoder in the second conditional autoencoder is selected as the second feature generation model. The third encoder in the second conditional autoencoder can be separated and used as a separate second feature generation model. Alternatively, the output of the third encoder in the second conditional autoencoder can be directly connected to the input of the subsequent processing flow model. It is also worth noting that the second feature generation model is generally pre-trained before the entire speech conversion function is launched and is embedded in the system, without any temporary training.

[0120] In one embodiment, obtaining a target speech stream according to the second target speech feature includes: inputting the second target speech feature into a speech generation model to obtain the target speech stream.

[0121] As you can understand, the second target speech feature is input into the speech generation model. The model receives this feature as input and then, through its complex internal neural network structure, converts this abstract feature representation into an actual audio waveform, the target speech stream. Recovering audio waveforms based on speech features is a relatively mature technology in the field of speech generation. For example, HiFi-GAN is a high-fidelity generative adversarial network specifically designed to convert abstract audio features into actual audio waveforms. HiFi-GAN is based on the Generative Adversarial Network (GAN) framework, but is specifically optimized for audio generation tasks. In this model, the generator is responsible for converting the input features into an audio waveform, while the discriminator is responsible for distinguishing whether the generated audio is authentic. Through this adversarial training, the model is able to generate high-quality, natural speech.

[0122] In one embodiment, see Figure 9 The speech conversion method using the target global timbre feature and the target complete timbre feature includes steps S902 to S910.

[0123] S902: In response to the skin selection operation, a target voice is determined. When the target voice is determined, a target global timbre feature and a target complete timbre feature corresponding to the target voice are extracted; the target global timbre feature has a lower dimension than the target complete timbre feature, and the target global timbre feature is input into a second feature extraction model to obtain the complete acoustic features of the target voice.

[0124] S904, after the user inputs the original voice stream, the original voice stream is input into the first feature extraction model to obtain the original voice features; the original voice features include the shallow timbre features and semantic features of the original voice stream, and the dimension of the shallow timbre features is lower than the target global timbre features.

[0125] S906, input the original speech feature as input and the target global timbre feature as conditional input, and input them into the first feature generation model to obtain the first target speech feature; the first target speech feature includes the semantic feature of the original speech stream and the target global timbre feature.

[0126] S908, input the first target speech feature as input and the target complete timbre feature as conditional input to the second feature generation model to obtain the second target speech feature; the second target speech feature includes the semantic feature of the original speech stream and the target complete timbre feature.

[0127] S910: Input the second target speech feature into the speech generation model to obtain a target speech stream.

[0128] The present application provides a speech conversion device, comprising a target speech determination module, a target timbre feature extraction module, and a speech conversion module. The target speech determination module is configured to determine a target speech in response to a skin selection operation. The target timbre feature extraction module is configured to extract target timbre features corresponding to the target speech upon determining the target speech. The speech conversion module is configured to obtain a target speech stream based on the original speech stream and the target timbre features after a user inputs an original speech stream.

[0129] For the specific definition of the voice conversion device, please refer to the definition of the voice conversion method above, which will not be repeated here. The various modules in the above-mentioned voice conversion device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0130] The present application provides a computer device comprising one or more processors and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the one or more processors, the steps of the speech conversion method in any of the above embodiments are performed.

[0131] Schematically, as Figure 10 As shown, Figure 10 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. Figure 10 Computer device 1000 includes a processing component 1002, which further includes one or more processors, and a memory resource represented by memory 1001 for storing instructions executable by processing component 1002, such as an application. The application stored in memory 1001 may include one or more modules, each corresponding to a set of instructions. In addition, processing component 1002 is configured to execute the instructions to perform the steps of the speech conversion method of any of the above-mentioned embodiments.

[0132] The computer device 1000 may further include a power supply component 1003 configured to perform power management of the computer device 1000 , a wired or wireless model interface 1004 configured to connect the computer device 1000 to the model, and an input / output (I / O) interface 1005 .

[0133] The present application provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the speech conversion method in any of the above embodiments.

[0134] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0135] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referenced to each other.

[0136] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech conversion method, characterized in that: include: In response to the skin selection operation, determining a target voice; When the target voice is determined, a target timbre feature corresponding to the target voice is extracted, wherein the target timbre feature includes a target global timbre feature and a target complete timbre feature; Specifically, it includes: inputting the target speech into a second feature extraction model to obtain a target global timbre feature; performing feature extraction on the target speech to obtain the target complete timbre feature; the dimension of the target complete timbre feature is higher than the target global timbre feature; After the user inputs an original voice stream, a target voice stream is obtained according to the original voice stream and the target timbre feature; specifically comprising: inputting the original voice stream into a first feature extraction model to obtain original voice features; the original voice features include shallow timbre features and semantic features of the original voice stream; the dimension of the shallow timbre feature is lower than the target global timbre feature; The original speech feature is used as input and the target global timbre feature is used as conditional input, and inputted into a first feature generation model to obtain a first target speech feature; the first target speech feature includes the semantic feature of the original speech stream and the target global timbre feature; The first target speech feature is used as input and the target complete timbre feature is used as conditional input, and input into a second feature generation model to obtain a second target speech feature; the second target speech feature includes the semantic feature of the original speech stream and the target complete timbre feature.

2. The voice conversion method according to claim 1, characterized in that: The construction process of the first feature extraction model includes: Constructing an initial autoencoder model; the autoencoder model includes a first encoder and a first decoder, the first encoder is used to extract the shallow timbre features and the semantic features from the input speech, and output them to the first decoder, and the first decoder is used to obtain a reconstruction result of the input speech according to the received features; With the goal of narrowing the gap between the reconstruction result and the samples of the first speech training set, the initial autoencoder model is trained using the first speech training set; The first encoder in the trained autoencoder model is used as a first feature extraction model.

3. The voice conversion method according to claim 1, characterized in that: The construction process of the second feature extraction model includes: Constructing an initial classification recognition model; the classification recognition model includes a feature extraction part and a classification part, the feature extraction part is used to extract global timbre features from the input speech and output them to the classification part, and the classification part is used to obtain a classification result of the input speech according to the received features; With the goal of narrowing the gap between the classification result and the classification label of the second speech training set, the initial classification recognition model is trained using the second speech training set; The feature extraction part of the trained classification recognition model is used as the second feature extraction model.

4. The method for voice conversion according to claim 1, characterized in that: The construction process of the first feature generation model includes: Constructing an initial first conditional autoencoder model; the first conditional autoencoder model includes a second encoder and a second decoder, the second encoder is used to perform feature fusion on the input and the condition to obtain a first speech feature, the first speech feature includes a semantic feature in the input and a global timbre feature in the condition, and the second decoder is used to restore the masked first speech feature to the first speech feature before masking; The first conditional autoencoder model is trained using the first speech feature training set and the global timbre feature training set with the goal of narrowing the gap between the restoration result of the second decoder and the feature fusion result of the second encoder on the first speech feature training set and the global timbre feature training set; The second encoder in the first conditional autoencoder model that has been trained is used as a first feature generation model.

5. The voice conversion method according to claim 4, characterized in that: The training of the first conditional autoencoder model using the first speech feature training set and the global timbre feature training set also includes: The first training set of speech features is subjected to timbre perturbation, and the first conditional autoencoder model is trained with the goal of reducing the difference between feature fusion results of the same speech feature sample and the same global feature training sample before and after the timbre perturbation of the second encoder.

6. The method for voice conversion according to claim 5, characterized in that: The method of taking the original speech feature as input and the target global timbre feature as conditional input and inputting them into a first feature generation model to obtain a first target speech feature includes: The timbre perturbation is performed on the original speech feature, and the original speech feature after the timbre perturbation is used as input and the target global timbre feature is used as a conditional input, and input into the first feature generation model to obtain the first target speech feature.

7. The voice conversion method according to claim 1, characterized in that: The feature extraction of the target speech comprises: According to the target speech, a corresponding mel spectrum matrix is ​​generated.

8. The method for voice conversion according to claim 1, characterized in that: The construction process of the second feature generation model includes: Constructing an initial second conditional autoencoder model; the second conditional autoencoder model includes a third encoder and a third decoder, the third encoder is used to fuse the input and the condition to obtain a second speech feature, the second speech feature includes a semantic feature in the input and a complete timbre feature in the condition, and the third decoder is used to restore the masked second speech feature to the second speech feature before masking; The second conditional autoencoder model is trained using the second speech feature training set and the complete timbre feature training set with the goal of narrowing the gap between the restoration result of the third decoder and the feature fusion result of the third encoder on the second speech feature training set and the complete timbre feature training set; The third encoder in the trained second conditional autoencoder model is used as a second feature generation model.

9. The method for voice conversion according to claim 1, characterized in that: The step of obtaining the target speech stream according to the second target speech feature includes: The second target speech feature is input into a speech generation model to obtain the target speech stream.

10. A speech conversion device, characterized in that: include: a target voice determination module, for determining a target voice in response to a skin selection operation; A target timbre feature extraction module is used to extract a target timbre feature corresponding to the target voice when the target voice is determined, wherein the target timbre feature includes a target global timbre feature and a target complete timbre feature; Specifically, it includes: inputting the target speech into a second feature extraction model to obtain a target global timbre feature; performing feature extraction on the target speech to obtain the target complete timbre feature; the dimension of the target complete timbre feature is higher than the target global timbre feature; The speech conversion module is used to obtain a target speech stream according to the original speech stream and the target timbre feature after the user inputs the original speech stream; specifically comprising: inputting the original speech stream into a first feature extraction model to obtain the original speech feature; the original speech feature includes the shallow timbre feature and semantic feature of the original speech stream; the dimension of the shallow timbre feature is lower than the target global timbre feature; The original speech feature is used as input and the target global timbre feature is used as conditional input, and inputted into a first feature generation model to obtain a first target speech feature; the first target speech feature includes the semantic feature of the original speech stream and the target global timbre feature; The first target speech feature is used as input and the target complete timbre feature is used as conditional input, and input into a second feature generation model to obtain a second target speech feature; the second target speech feature includes the semantic feature of the original speech stream and the target complete timbre feature.

11. A computer device, characterized in that: The method comprises one or more processors and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the speech conversion method according to any one of claims 1 to 9 are executed.

12. A storage medium, characterized in that: The storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the speech conversion method according to any one of claims 1 to 9.

Images