Voice conversion method, voice conversion model training method and electronic equipment
Feature extraction through audio language model and vocoder, combined with audio generation adversarial network, the problem of poor tone conversion effect in the existing technology is solved, high-fidelity voice conversion is achieved, and the robustness and naturalness of the model is improved. .
Patent Information
- Application Number
- CN202410977468.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-11-08
AI Technical Summary
In the existing voice conversion technology, the tone conversion effect is poor, mainly because Melsong is not suitable for the tone representation of the speaker, resulting in poor tone conversion effect. It is difficult for the existing technology to effectively decouple content features and tone characteristics, affecting the conversion effect. .
The audio language model is used to extract the tone characteristics of the target speaker, combine the vocoder to extract the content characteristics of the source audio, and combine the two through the audio generation and adversarial network to generate high-quality converted voices, reduce the interference of the tone characteristics on the content characteristics, and improve the The fidelity of the tone and content.
It significantly improves the fidelity of tone conversion and the fidelity of content. The generated voice is highly matched in tone and content, and is naturally smooth, enhancing the robustness and generalization ability of the model, and reducing dependence on text information.
Smart Images

Figure CN118918910A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of speech processing technology, and in particular to a speech conversion method, a speech conversion model training method and an electronic device. Background Art
[0002] With the continuous development of speech technology and artificial intelligence technology, a series of technologies for realizing voice conversion (VC) have emerged, aiming to convert the speech of an input speaker (source speaker) into speech with the same content but timbre of a different speaker (target speaker).
[0003] At present, some scholars have proposed some advanced speech conversion technologies, which use discrete speech features based on self-supervised learning (SSL) to perform timbre conversion, so that this conversion does not require text information. At the same time, the self-supervised features can better preserve the pronunciation characteristics to achieve high-fidelity timbre conversion.
[0004] In these SSL speech conversion technologies, a vocoder with discrete speech features as input is usually required, which can be called CTX-vec2wav technology, so that in addition to receiving discrete units as input, it also needs to use an additional Mel spectrum input to provide timbre information, so that this vocoder has timbre controllability, thereby achieving the goal of timbre conversion.
[0005] However, in these SSL speech conversion technologies, the source of the speaker's timbre is only the Mel-spectrogram, but this representation is not necessarily the most suitable information source for representing the speaker's timbre, resulting in poor timbre conversion results.
[0006] In response to the above problems, the industry has not yet proposed a better technical solution. Summary of the invention
[0007] The present application provides a speech conversion method, a speech conversion model training method, a storage medium and an electronic device, which are used to at least solve the problem of poor timbre conversion effect in the current related technology.
[0008] In a first aspect, an embodiment of the present application provides a speech conversion method, comprising: extracting a target speaker's timbre features corresponding to the target speaker's speech based on an audio language model; extracting content features corresponding to the source audio based on a vocoder; and processing the target speaker's timbre features and the content features with a generator based on an audio generative adversarial network to generate a corresponding target converted speech.
[0009] In a second aspect, an embodiment of the present application provides a training method for a speech conversion model, the speech conversion model including an audio language model, a vocoder and an audio generative adversarial network, the training method including: for each speech sample in a speech sample set, extracting a first audio segment and a second audio segment from an audio file in the speech sample; each audio sample includes an audio file and a corresponding timbre label; extracting a first speaker timbre feature corresponding to the first audio segment based on the audio language model; extracting a first content feature corresponding to the second audio segment based on the vocoder; training the audio generative adversarial network and the audio language model based on the first content feature, the first speaker timbre feature and the speech sample; wherein the loss function of the audio language model is determined based on the timbre label and the first speaker timbre feature.
[0010] According to a third aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the speech conversion method or speech conversion model training method of any embodiment of the present application.
[0011] In a fourth aspect, an embodiment of the present application provides a storage medium on which a computer program is stored, characterized in that when the program is executed by a processor, the steps of the speech conversion method or the speech conversion model training method of any embodiment of the present application are implemented.
[0012] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the speech conversion method or speech conversion model training method of any embodiment of the present application.
[0013] The beneficial effects of the embodiments of the present application are:
[0014] Through the embodiments of the present application, by introducing an audio language model to extract the timbre features of the target speaker, the timbre information of the target speaker can be captured and characterized more accurately, and the timbre features can be extracted more robustly and accurately, which can significantly improve the fidelity of the timbre conversion. By extracting the content features of the source audio through a vocoder, the integrity and clarity of the speech content can be better preserved, while avoiding the interference of timbre features on content features, and achieving better performance in content fidelity. Combined with the generator of the audio generative adversarial network, the timbre features of the target speaker and the content features of the source audio can be effectively fused, and the generated speech is not only highly matched in timbre and content, but also natural and fluent. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0016] Figure 1 A flowchart of an example of a voice conversion method according to an embodiment of the present application is shown;
[0017] Figure 2 An operation flow chart showing an example of a method for training a speech conversion model according to an embodiment of the present application is shown;
[0018] Figure 3 A schematic diagram of the architecture of a system for implementing a voice conversion method according to an embodiment of the present application is shown;
[0019] Figure 4 A schematic diagram of the structural connection of an example of a BigVGAN generator with an adaptive activation function according to an embodiment of the present application is shown;
[0020] Figure 5 It is a schematic structural diagram of an embodiment of an electronic device of the present application. DETAILED DESCRIPTION
[0021] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0022] It should be noted that some related technologies have proposed some vocoders based on discrete speech features. Specifically, first of all, discrete speech features usually come from self-supervised models, such as vq-wav2vec models. The CTX-vec2wav structure first inputs the output vector of its quantizer into a conformer network, and at the same time, the mel spectrum from the target speaker is linearly transformed and inserted into the conformer network in a cross-attention manner to achieve the injection of timbre information. The output of the conformer is then input into a HifiGAN network, and the model is optimized using the training criteria of HifiGAN. In order to make the model converge faster, CTX-vec2wav also designs auxiliary tasks such as pitch and mel spectrum prediction. The input of CTX-vec2wav is replaced with the clustering result of the HuBERT self-supervised model, and it is found that the timbre conversion performance exceeds many well-known public models. Both S3PRL-VC and UUVC use discrete self-supervised features such as HuBERT as content information, and assist additional inputs such as pitch and speaker embedding representation to achieve speech reconstruction and timbre conversion.
[0023] The timbre conversion based on CTX-vec2wav has achieved promising results, but the subjective and objective performance of the timbre conversion is often not good enough in more test sets; the input features of S3PRL-VC, UUVC, Vec-Tok-VC++ and other work are diverse and complicated, and there is still room for improvement in performance, and it is not simple enough.
[0024] Specifically, in the timbre conversion work based on the CTX-vec2wav vocoder, the source of the speaker's timbre is only the mel spectrum, which is not necessarily the most suitable information source for speaker representation; and under the current network structure, the speaker information is only injected into the conformer, and the HifiGAN model after the conformer still has many parameters, which may cause the model to forget, which is not conducive to the effective conversion of timbre. For the rest of the existing timbre conversion work, in most cases, it is necessary to decouple the timbre of content-related features, which is a very challenging task. The timbre conversion effect also depends greatly on the decoupling effect, resulting in the unsatisfactory results of most timbre conversion models.
[0025] In view of this, Figure 1 A flowchart of an example of a voice conversion method according to an embodiment of the present application is shown.
[0026] Regarding the executor of the method of the embodiment of the present application, it can be any controller or processor with computing or processing capabilities, which executes the speech conversion method. By introducing the audio language model and generative adversarial network technology, the accuracy of timbre representation, the fidelity of content features, and the naturalness and controllability of generated speech are effectively improved, while reducing the dependence on text information and improving the robustness and generalization ability of the model.
[0027] In some examples, it may be integrated and configured in an electronic device or terminal by means of software, hardware, or a combination of software and hardware, and the type of the terminal or electronic device may be diverse, such as a mobile phone, a tablet computer, or a desktop computer, etc.
[0028] like Figure 1 As shown, in step S110, the timbre features of the target speaker corresponding to the speech of the target speaker are extracted based on the audio language model.
[0029] In some embodiments, the speech of the target speaker is input into the audio language model, and the model extracts features of the input speech to generate a series of feature vectors. From the generated feature vectors, specific dimensions are extracted or timbre-related features are extracted through further processing, and they can be processed through clustering, dimensionality reduction and other technical means to ensure that the extracted timbre features accurately reflect the timbre characteristics of the target speaker.
[0030] Here, a pre-trained audio language model (e.g., Wav2Vec 2.0, HuBERT, etc.) is selected. These models have been self-supervised on a large amount of unlabeled audio data and can extract rich speech features. Preferably, the audio language model can adopt the WavLM-Large self-supervised model, and its advantages in speaker verification have been widely verified. As a result, the extracted timbre features can fully reflect the timbre characteristics of the target speaker and improve the accuracy of timbre conversion.
[0031] In step S120, content features corresponding to the source audio are extracted based on the vocoder.
[0032] In some embodiments, a suitable vocoder (such as Tacotron, WaveNet, etc.) is selected to effectively encode and decode the input speech. The source audio is input into the vocoder, and the vocoder converts the speech signal into content feature vectors through the encoder part. These vectors mainly contain the content information of the speech, while minimizing the interference of the timbre information. Exemplarily, the content features include at least one of the following: content information, prosody information, and speaker timbre leakage information. Therefore, by extracting content features through the vocoder, the timbre information can be effectively isolated, the content features of the speech can be accurately extracted, and the integrity and accuracy of the content during the speech conversion process can be ensured.
[0033] In step S130, the generator based on the audio generative adversarial network processes the timbre features and content features of the target speaker to generate the corresponding target converted speech.
[0034] In some embodiments, the audio generation adversarial network includes a generator and a discriminator. The generator is responsible for combining the timbre features and content features to generate a new speech signal, and the discriminator is responsible for judging the authenticity of the generated speech. More specifically, by fusing the timbre features of the target speaker with the content features, it is ensured that the generator can make full use of both types of feature information. Here, the audio generation adversarial network can generate high-quality, realistic target conversion speech through an adversarial training mechanism, retaining the timbre characteristics of the target speaker and the content information of the source speech.
[0035] Regarding the implementation details of the above step S130, in an example of an embodiment of the present application, conformer encoding is performed for the timbre features and content features of the target speaker, and a corresponding cross-attention calculation without positional encoding is performed to obtain the corresponding attention encoding features. Then, the generator based on the audio generative adversarial network processes the attention encoding features to generate the corresponding target converted speech. Therefore, in the process of cross-attention calculation, positional encoding is not applied, so that the model cannot know the speaking order of the timbre source, and can only know the timbre information without being disturbed by factors such as language and content.
[0036] In another example of the embodiment of the present application, the activation function of the audio generation adversarial network is updated according to the timbre characteristics of the target speaker; the timbre characteristics and content characteristics of the target speaker are processed based on the generator of the updated audio generation adversarial network to generate the corresponding target conversion speech. As a result, the audio generation adversarial network can accept an additional speaker embedding as input and perform layer-level adaptive activation. By adding an adaptive activation function, the model can very effectively learn timbre information from the target speaker's voice, which can significantly improve the timbre conversion capability.
[0037] Figure 2 The operation flow chart of an example of a training method for a speech conversion model according to an embodiment of the present application is shown. Here, the speech conversion model includes an audio language model, a vocoder, and an audio generative adversarial network.
[0038] like Figure 2 As shown, in step S210, for each speech sample in the speech sample set, a first audio segment and a second audio segment are extracted from an audio file in the speech sample, and each audio sample includes an audio file and a corresponding timbre label.
[0039] In some embodiments, the speech audio file of the target speaker can be cut from any position at the beginning and end of the sentence as the timbre source of the target speaker, and the hidden layer representation sequence is extracted using the self-supervised pre-trained WavLM model. In addition, the remaining content of the speech audio file is used as a self-supervised speech representation, which contains content information, prosody information and a small amount of speaker timbre leakage, for example, which can be extracted by a self-supervised pre-trained vq-wav2vec model. Here, the speech of the target speaker can be cut from any position at the beginning and end of the sentence, thereby improving the training efficiency.
[0040] In step S220, a timbre feature of a first speaker corresponding to the first audio segment is extracted based on an audio language model.
[0041] In step S230, a first content feature corresponding to the second audio segment is extracted based on a vocoder.
[0042] In step S240, the audio generative adversarial network and the audio language model are trained based on the first content feature, the first speaker timbre feature and the speech sample.
[0043] Here, the loss function of the audio language model is determined based on the timbre label and the first speaker timbre features. Exemplarily, the auxiliary mel prediction loss is the L1 loss between the actual mel spectrogram and the predicted mel spectrogram after linear projection of the consistency front end, thereby warming up the entire model. In addition, the audio generative adversarial network can adopt all other GAN losses.
[0044] In the embodiment of the present application, the two tasks of discrete speech unit vocoder and timbre conversion are regarded as the same task, that is, the same model can generate good quality speech from discrete speech units and perform very accurate timbre conversion. In addition, there is no need for labeled data for training, it can be easily expanded to massive data, and can naturally support multi-language timbre conversion, even if the model is only trained on monolingual data. In addition, it has a more far-reaching significance for speech synthesis in the era of large models, that is, the large model itself only needs to be responsible for modeling semantic and prosodic information, and the timbre can be completely controlled by the vocoder.
[0045] Figure 3 A schematic diagram of the architecture of an example of a system for implementing a voice conversion method according to an embodiment of the present application is shown.
[0046] It should be noted that the core problem of timbre conversion is to construct a content representation with good decoupling ability and a synthesis model with strong fusion ability to effectively combine the timbre of the target speaker with the original content representation. There are many possible ways to try decoupling, but they either require parallel data or more complex training indicators, leading to instability and other problems; for the fusion of the target speaker's timbre, the common approach is to use a certain speaker representation extraction method to extract a representation representing the timbre from the target speaker, and then use some adaptive transformation technology to fuse the information.
[0047] In some examples of the embodiments of the present application, a discrete speech unit vocoder that can efficiently fuse the timbre of the target speaker is designed based on the CTX-vec2wav timbre conversion framework. The audio generative adversarial network adopts the BigVGAN model, and the generator of the BigVGAN model includes multiple anti-aliased multiperiodicity composition (AMP) blocks and activation layers. Each AMP block is used to receive and process the timbre features of the target speaker to update the activation function of the activation layer.
[0048] It should be noted that the Snake activation function was proposed to model periodic inductive bias and then adopted in the BigVGAN vocoder to achieve state-of-the-art performance. This activation function can be expressed as f θ (x) = x + 1 / βsin2(αx). The learnable parameters θ = {α, β} are designed to control the frequency and amplitude, respectively, f θ Each input channel can be operated independently, ie, each input channel has a different θ.
[0049] Since this Snake activation can cleverly capture the periodic patterns in the speech signal, this application proposes to inject more information from the timbre of the target speaker. Let s∈R d are some representative speaker embeddings extracted from the target speaker. Specifically, the adaptive Snake activation, where both the frequency and amplitude of the sine function are affected by s.
[0050] In each layer of the vocoder, the latent timbre representation of the target speaker is averaged and then sent to the activation function. Taking the original sinusoidal activation function in BigVGAN as an example, the activation function of the activation layer uses the adaptive Snake activation function:
[0051] T(s) = tanh(Ws+b), formula (1)
[0052]
[0053] In the formula, s represents the average hidden layer representation of the target speaker's timbre characteristics, and x represents the input of the activation function. T is the linear transformation following the tanh activation function. The operations in the formula are element-by-element. T(s) represents the speaker's timbre characteristics after the tanh activation function. W and b represent the weight matrix and bias term corresponding to the activation function, respectively. They are completed by element-by-element operations such as addition, multiplication and division. θ (x, s) represents the final output function; θ = {α, β, W, b} represents the set of learnable parameters of the Snake activation function.
[0054] To ensure numerical stability, T(s) is discounted by 1 / 2 in the amplitude part. In order to save parameters, this application applies the same T transform to amplitude and frequency. Therefore, the learnable parameters of each adaptive Snake are θ = {α, β, W, b}, and the target timbre information can be effectively injected into each layer of the vocoder through adaptive activation, thereby greatly enhancing the controllability of the timbre. Through adaptive Snake activation, vec2wav 2.0 is designed to be a prompt discrete marker vocoder.
[0055] It should be noted that there are many ways to select layer-level adaptive activation. The main idea is to insert the timbre representation information of the target speaker into the activation function of each layer of the neural network, so that the timbre directly affects the parameters in the activation function, that is, to achieve timbre manipulation from a more essential level of the signal, and to change the scope of timbre control from a single-layer network to each layer in the entire model, thereby greatly enhancing the timbre control effect.
[0056] In some examples of the embodiments of the present application, the WavLM-Large self-supervised model with strong speaker verification capability is used as the source of timbre representation, and a convolutional module-based neural network is designed to preprocess this part of the timbre representation before it is fed to the conformer with cross attention. In addition, the prosody auxiliary task designed in CTX-vec2wav is cancelled, making the data preparation process simpler.
[0057] Through the embodiments of the present application, a new speech discrete unit vocoder vec2wav 2.0 is provided, which realizes universal timbre conversion. Specifically, discrete units from a speech self-supervision model can be used as content features of the source speech. In order to make up for the speaker timbre features lost in the content unit, vec2wav 2.0 uses the WavLM model to provide powerful timbre-related representations, which control the resynthesized speech through a position-independent cross-attention mechanism and a novel adaptive activation mechanism. In this way, although the content features do not achieve perfect speaker decoupling, vec2wav 2.0 is still able to appropriately change the speaker timbre according to different reference prompt voices. vec2wav 2.0 can easily achieve any-to-any and cross-language timbre conversion.
[0058] like Figure 3 As shown in the figure, for each speech waveform in the data set, a part can be randomly cut from the head or tail as the timbre source of the target speaker, and the self-supervised pre-trained WavLM model is used to extract the hidden layer representation sequence. For the remaining content part, the self-supervised pre-trained vq-wav2vec model is used to extract discrete self-supervised speech representations, which contain content information, prosody information and a small part of the speaker's timbre leakage. Then, a convolutional neural network is used to perform preliminary processing on the hidden layer representation sequence extracted by WavLM, and then the output sequence and the content representation are subjected to cross-attention calculation after being encoded by the conformer. In the cross-attention calculation here, position encoding is not applied, so that the model cannot know the speaking order of the timbre source, and can only know the timbre information without being disturbed by factors such as language and content. In the BigVGAN part after the conformer, the hidden layer representation sequence extracted by WavLM is averaged and applied to each layer activation function of BigVGAN, and finally a waveform is generated, and the loss of BigVGAN is calculated. In addition, similar to CTX-vec2wav, the mel spectrum auxiliary prediction task after the conformer frontend is still retained to help the model converge effectively.
[0059] Through adaptive Snake activation, this application designs vec2wav 2.0 to become a prompt discrete marker vocoder, such as Figure 3 The overall design inherits the front-end-generator framework in CTX-vec2wav, but with several key improvements.
[0060] For the details of the content features, the off-the-shelf vq-wav2vec SSL model is used to extract discrete content representations for resynthesis. These discrete tags are extracted from the quantizer output before the feature aggregator, which is a sequence of two sets of integer indices. This representation is used in the embodiment of the present application because a lot of speaker timbre information is removed due to the information bottleneck, while most of the speech pronunciation and prosody are retained. In addition, compared with the HuBERT-style Transformer SSL model, vq-wav2vec is fully convolutional and has a certain receptive field. This produces a representation that is agnostic to the total sequence length, maintaining consistent results for a given window. Despite the measurable speaker timbre leakage in the discrete tags, it will be verified that the proposed vec2wav 2.0 vocoder exhibits strong timbre controllability, so good voice conversion can be achieved.
[0061] Regarding the details of the prompt features, following CTX-vec2wav, the reference prompt segment is randomly cut from the current utterance to keep the same speaker identity without labeling data. This application uses a pre-trained WavLM model as a timbre feature extractor, instead of using mel spectrograms to provide timbre information for reference prompts, because WavLM's advantages in speaker verification have been widely verified. Freeze the WavLM model during training and use only the output features of a certain position of its Transformer block.
[0062] Regarding the details of the front-end module, a CNN-based cue pre-network is designed to process the hidden embeddings before the encoded reference cue segments are input to the consistency front-end. The consistency front-end still consists of two consistency decoder blocks, each of which receives the encoded cue embeddings through a position-independent cross-attention mechanism. This cross-attention does not apply position encoding to the query sequence, which simulates the shuffling of the query sequence and inherently breaks the local patterns in the reference cues, such as language and prosodic features. This makes the learning of the target timbre more accurate because it is a global information and also facilitates cross-lingual voice conversion.
[0063] Figure 4 A schematic diagram of the structural connection of an example of a BigVGAN generator with adaptive activation function according to an embodiment of the present application is shown.
[0064] Specifically, after the timbre information in the reference cues is integrated into the network through the consistency front-end, a BigVGAN-based generator is used to perform the actual vocoding operation, while also being cued to embed adaptively. Here, the cued embeddings are first pooled to form a single vector that averages out the language details and preserves the global timbre, and then inserted into each adaptive activation layer of the BigVGAN generator. Figure 4As shown, the input hidden state is iteratively upsampled by transposed convolution and transformed by AMP blocks. Each AMP block receives an additional cue embedding that is fed to the adaptive Snake activation layer for timbre control. A low-pass (LP) filter is applied after each upsampling and downsampling operation to prevent aliasing. The hidden state is restored to the sample point after the final adaptive activation and convolution block.
[0065] Regarding the details of the BigVGAN discriminator, it can inherit the multi-scale discriminator (MSD) and multi-period discriminator (MPD) in HifiGAN. These discriminators are trained together with the generator to distinguish true and false signals at multiple scales and periods. The generator is adversarially trained to deceive the discriminator, achieving high-fidelity speech resynthesis and speech conversion results.
[0066] Training criteria for speech conversion models, which can include auxiliary mel prediction loss and all other GAN losses of HifiGAN. Auxiliary mel prediction loss is the L1 loss between the actual mel spectrogram and the predicted mel spectrogram after linear projection of the consistency front end, which is used to warm up the whole model. This loss is added with a certain coefficient, following vec2wav and CTX-vec2wav, and cancels it after warming up.
[0067] It should be noted that, although not directly optimized for speech conversion, vec2wav 2.0 still has strong conversion capabilities due to its effectiveness in incorporating the target speaker's timbre. The content features retain most of the phonetic and prosodic information while losing much of the speaker identity, while the speaker timbre is controlled by the reference cue. Therefore, speech conversion can be achieved by using the reference speech of the target speaker as the cue input. This approach naturally supports any-to-any speech conversion because both the content features and the cue features are obtained by an SSL model trained on a large amount of speech data with sufficient speaker variation.
[0068] Furthermore, since both the cross-attention mechanism and adaptive Snake activation are position-independent, the order of cue features plays little role in timbre control. This allows cross-lingual speech transfer, since almost all language-dependent patterns are broken by these position-independent operations. As long as the global features are prominent enough in the WavLM features, the speaker timbre can be successfully transferred.
[0069] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of actions combined, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application. In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0070] In some embodiments, an embodiment of the present application provides a non-volatile computer-readable storage medium, which stores one or more programs including execution instructions, and the execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute the steps of any of the above-mentioned speech conversion methods or speech conversion model training methods of the present application.
[0071] In some embodiments, the embodiments of the present application also provide a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer performs the steps of any one of the above-mentioned speech conversion methods or speech conversion model training methods.
[0072] In some embodiments, an embodiment of the present application also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the speech conversion method or the speech conversion model training method.
[0073] Figure 5 is a schematic diagram of the hardware structure of an electronic device for executing a speech conversion method or a speech conversion model training method provided by another embodiment of the present application, such as Figure 5 As shown, the device includes:
[0074] One or more processors 510 and memory 520, Figure 5 A processor 510 is taken as an example.
[0075] The device for executing the speech conversion method or the speech conversion model training method may further include: an input device 530 and an output device 540 .
[0076] The processor 510, the memory 520, the input device 530 and the output device 540 may be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.
[0077] The memory 520, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the speech conversion method or speech conversion model training method in the embodiments of the present application. The processor 510 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 520, that is, implementing the speech conversion method or speech conversion model training method in the above method embodiment.
[0078] The memory 520 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 520 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 520 may optionally include a memory remotely arranged relative to the processor 510, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0079] The input device 530 may receive input digital or character information and generate signals related to user settings and function control of the electronic device. The output device 540 may include a display device such as a display screen.
[0080] The one or more modules are stored in the memory 520, and when executed by the one or more processors 510, the speech conversion method or speech conversion model training method in any of the above method embodiments is executed.
[0081] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.
[0082] The electronic devices of the embodiments of the present application exist in various forms, including but not limited to:
[0083] (1) Mobile communication equipment: This type of equipment is characterized by having mobile communication functions and its main purpose is to provide voice and data communications. This type of terminal includes: smart phones, multimedia phones, functional phones, and low-end phones.
[0084] (2) Ultra-mobile personal computer devices: These devices belong to the category of personal computers, have computing and processing functions, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, etc.
[0085] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.
[0086] (4) Other onboard electronic devices with data interaction functions, such as on-board devices installed in vehicles.
[0087] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0088] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A voice conversion method, comprising: Extracting the target speaker's timbre features corresponding to the target speaker's voice based on the audio language model; Extracting content features corresponding to the source audio based on the vocoder; The generator based on the audio generative adversarial network processes the target speaker timbre features and the content features to generate corresponding target converted speech.
2. The method according to claim 1, wherein: The generator based on the audio generative adversarial network processes the target speaker timbre feature and the content feature to generate a corresponding target converted speech, including: Performing conformer encoding on the target speaker's timbre features and the content features, and performing cross-attention calculation corresponding to the position-free encoding to obtain corresponding attention encoding features; A generator based on the audio generative adversarial network processes the attention encoded features to generate corresponding target converted speech.
3. The method according to claim 1, wherein: The generator based on the audio generative adversarial network processes the target speaker timbre feature and the content feature to generate a corresponding target converted speech, including: According to the timbre characteristics of the target speaker, updating the activation function of the audio generation adversarial network; A generator based on the updated audio generative adversarial network processes the target speaker timbre features and the content features to generate corresponding target converted speech.
4. The method according to claim 3, wherein: The audio generative adversarial network adopts the BigVGAN model, and the generator of the BigVGAN model includes multiple anti-aliased multi-cycle AMP blocks and an activation layer; each AMP block is used to receive and process the timbre characteristics of the target speaker to update the activation function of the activation layer.
5. The method according to claim 4, wherein: The activation function of the activation layer adopts the adaptive Snake activation function: T(s)=tanh(Ws+b) Where s represents the average hidden layer representation of the target speaker's timbre characteristics, x represents the input of the activation function, T(s) represents the speaker's timbre characteristics after the tanh activation function, W and b represent the weight matrix and bias term corresponding to the activation function respectively; f θ (x, s) represents the final output function; θ = {α, β, W, b} represents the set of learnable parameters of the Snake activation function.
6. The method according to any one of claims 1 to 5, wherein: The content feature includes at least one of the following: content information, prosody information, and speaker timbre leakage information.
7. A method for training a speech conversion model, the speech conversion model comprising an audio language model, a vocoder, and an audio generative adversarial network, the training method comprising: For each speech sample in the speech sample set, extracting a first audio segment and a second audio segment from an audio file in the speech sample; Each audio sample contains an audio file and a corresponding timbre label; Extracting a timbre feature of a first speaker corresponding to the first audio segment based on an audio language model; extracting a first content feature corresponding to the second audio segment based on the vocoder; Training the audio generative adversarial network and the audio language model based on the first content feature, the first speaker timbre feature and the speech sample; The loss function of the audio language model is determined based on the timbre label and the timbre characteristics of the first speaker.
8. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.
9. An electronic device, comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method described in any one of claims 1 to 7.
10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 can be performed.
