Vocoder training method, audio synthesis method, medium, apparatus, and computing device

Abstract

Embodiments of the present disclosure provide a vocoder training method. The vocoder training method comprises: obtaining a first fundamental frequency sequence of audio in an audio corpus; performing fundamental frequency perturbation processing on the first fundamental frequency sequence to obtain a second fundamental frequency sequence; performing mapping processing on the second fundamental frequency sequence to obtain a target tensor; and training the target tensor, an acoustic feature sequence corresponding to the audio, and the audio to obtain a vocoder used for audio synthesis. The method of the present disclosure improves the robustness of the vocoder to fundamental frequency prediction errors in actual application by introducing fundamental frequency perturbation, thereby significantly improving the accuracy and quality of audio synthesis and providing a better experience for users. In addition, embodiments of the present disclosure provide an audio synthesis method, a medium, a device and a computing device.

Description

Technical Field

[0001] The embodiments of this disclosure relate to the field of computer technology, and more specifically, to vocoder training methods, audio synthesis methods, media, apparatus, and computing devices. Background Technology

[0002] This section is intended to provide background or context for embodiments of this disclosure. The description herein is not intended to imply that it is prior art simply because it is included in this section.

[0003] Audio synthesis refers to the artificial synthesis of audio by converting input text into corresponding speech based on artificial intelligence (AI) models. This can be used in human-computer interaction scenarios such as voice broadcasting. The basic framework of audio synthesis can include a front-end module, an acoustic model, and a vocoder. A vocoder is a speech signal encoder and decoder that uses only model parameters during transmission, and utilizes model parameter estimation and speech synthesis techniques during encoding and decoding. It is a type of encoder and decoder that analyzes and synthesizes speech; it is also called a speech analysis and synthesis system or a speech audioband compression system.

[0004] In related technologies, when synthesizing audio using a vocoder, the fundamental frequency (F0) is usually extracted from real audio corpus. The fundamental frequency is then introduced into the vocoder for training. A well-trained vocoder can optimize the glitches in the synthesized dry audio and improve the quality of the synthesized dry audio.

[0005] However, vocoders trained using the above techniques have poor robustness. Summary of the Invention

[0006] This disclosure provides a vocoder training method, an audio synthesis method, a medium, an apparatus, and a computing device to improve the robustness of the vocoder to fundamental frequency prediction errors during application, thereby significantly improving the accuracy and quality of audio synthesis and bringing a better user experience.

[0007] In a first aspect of the present disclosure, a vocoder training method is provided, comprising: acquiring a first fundamental frequency sequence of audio from an audio corpus; performing fundamental frequency perturbation processing on the first fundamental frequency sequence to obtain a second fundamental frequency sequence; performing mapping processing on the second fundamental frequency sequence to obtain a target tensor; and training the target tensor, the acoustic feature sequence corresponding to the audio, and the audio to obtain a vocoder for audio synthesis.

[0008] In one embodiment of this disclosure, a second fundamental frequency sequence is obtained by performing fundamental frequency perturbation processing on a first fundamental frequency sequence, including: performing the following fundamental frequency perturbation processing on each fundamental frequency in the first fundamental frequency sequence to obtain the second fundamental frequency sequence: obtaining a first logarithm of the fundamental frequency; performing maximum / minimum truncation processing on the first logarithm to obtain a second logarithm; performing quantization processing on the second logarithm to obtain a fundamental frequency category label corresponding to the fundamental frequency; and performing inverse quantization processing on the fundamental frequency category label.

[0009] In another embodiment of this disclosure, a second fundamental frequency sequence is obtained by performing fundamental frequency perturbation processing on a first fundamental frequency sequence, including: performing the following fundamental frequency perturbation processing on each fundamental frequency in the first fundamental frequency sequence to obtain the second fundamental frequency sequence: randomly sampling the fundamental frequency within a preset sampling range of the fundamental frequency to obtain perturbation parameters corresponding to the fundamental frequency; and summing the perturbation parameters and the fundamental frequency.

[0010] In another embodiment of this disclosure, the mapping process includes excitation mapping and embedding mapping. The mapping process on the second fundamental frequency sequence to obtain the target tensor includes: performing excitation mapping on the second fundamental frequency sequence to obtain a first tensor; performing embedding mapping on the second fundamental frequency sequence to obtain a second tensor; and combining the first tensor and the second tensor to obtain the target tensor.

[0011] In another embodiment of this disclosure, embedding mapping is performed on the second fundamental frequency sequence to obtain a second tensor, including: in response to the second fundamental frequency sequence being a floating-point value in a continuous domain, upsampling interpolation is used to upsample the second fundamental frequency sequence to obtain a first sampled fundamental frequency sequence; embedding mapping is performed on the first sampled fundamental frequency sequence to obtain a second tensor.

[0012] In another embodiment of this disclosure, embedding mapping is performed on the second fundamental frequency sequence to obtain a second tensor, including: in response to the second fundamental frequency sequence being a quantized fundamental frequency category label, upsampling processing is performed on the second fundamental frequency sequence using nearest neighbor interpolation to obtain a second sampled fundamental frequency sequence; embedding mapping is performed on the second sampled fundamental frequency sequence to obtain a second tensor.

[0013] In another embodiment of this disclosure, combining the first tensor and the second tensor to obtain the target tensor includes: concatenating the first tensor and the second tensor in response to their different sizes to obtain the target tensor; and summing the first tensor and the second tensor in response to their identical sizes to obtain the target tensor.

[0014] In a second aspect of the present disclosure, an audio synthesis method is provided, comprising: acquiring target text; extracting an acoustic feature sequence from the target text; inputting the acoustic feature sequence into a fundamental frequency prediction model to perform fundamental frequency prediction, thereby obtaining a fundamental frequency sequence corresponding to the acoustic feature sequence; and inputting the fundamental frequency sequence and the acoustic feature sequence into a vocoder to perform audio synthesis, thereby obtaining audio corresponding to the target text, wherein the vocoder is trained using the vocoder training method of the first aspect.

[0015] In one embodiment of this disclosure, the fundamental frequency prediction model includes a convolutional layer, a nonlinear layer, a normalization layer, a random deactivation layer, a linear layer, and a classification layer. The acoustic feature sequence is input into the fundamental frequency prediction model to predict the fundamental frequency, obtaining the fundamental frequency sequence corresponding to the acoustic feature sequence. This includes: inputting the acoustic feature sequence into the convolutional layer for convolution processing to obtain a first feature; inputting the first feature into the nonlinear layer for activation processing to obtain a second feature; inputting the second feature into the normalization layer for normalization processing to obtain a third feature; inputting the third feature into the random deactivation layer for deactivation processing to obtain a fourth feature; inputting the fourth feature into the linear layer for linear processing to obtain a fifth feature; and inputting the fifth feature into the classification layer for classification processing to obtain the fundamental frequency sequence.

[0016] In another embodiment of this disclosure, the fundamental frequency sequence and the acoustic feature sequence are input into a vocoder for audio synthesis to obtain the audio corresponding to the target text, including: mapping the fundamental frequency sequence to obtain a target tensor; and inputting the target tensor and the acoustic feature sequence into a vocoder for audio synthesis to obtain the audio.

[0017] In another embodiment of this disclosure, the mapping process includes excitation mapping and embedding mapping. The mapping process on the fundamental frequency sequence to obtain the target tensor includes: performing excitation mapping on the fundamental frequency sequence to obtain a first tensor; performing embedding mapping on the fundamental frequency sequence to obtain a second tensor; and combining the first tensor and the second tensor to obtain the target tensor.

[0018] In another embodiment of this disclosure, embedding mapping is performed on the fundamental frequency sequence to obtain a second tensor, including: in response to the fundamental frequency sequence being a continuous domain floating-point value, upsampling the fundamental frequency sequence using linear interpolation to obtain a first sampled fundamental frequency sequence; embedding mapping is performed on the first sampled fundamental frequency sequence to obtain a second tensor; in response to the fundamental frequency sequence being a quantized fundamental frequency category label, upsampling the fundamental frequency sequence using nearest neighbor interpolation to obtain a second sampled fundamental frequency sequence; and embedding mapping is performed on the second sampled fundamental frequency sequence to obtain a second tensor.

[0019] In another embodiment of this disclosure, combining the first tensor and the second tensor to obtain the target tensor includes: concatenating the first tensor and the second tensor in response to their different sizes to obtain the target tensor; and summing the first tensor and the second tensor in response to their identical sizes to obtain the target tensor.

[0020] In another embodiment of this disclosure, the method further includes: training a fundamental frequency prediction model using an audio corpus and acoustic feature sequences.

[0021] In another embodiment of this disclosure, the target text is the text corresponding to the singing voice, and the target text includes at least one of the phoneme type of the singing voice, the start and end positions of the phoneme boundaries in the singing voice, and the pitch of the singing voice.

[0022] In a third aspect of the present disclosure, a medium is provided that stores computer execution instructions, which, when executed by a processor, are used to implement a vocoder training method as in the first aspect or an audio synthesis method as in the second aspect.

[0023] In a fourth aspect of the present disclosure, a vocoder training apparatus is provided, comprising: an acquisition module for acquiring a first fundamental frequency sequence of audio from an audio corpus; a first processing module for performing fundamental frequency perturbation processing on the first fundamental frequency sequence to obtain a second fundamental frequency sequence; a second processing module for performing mapping processing on the second fundamental frequency sequence to obtain a target tensor; and a training module for training a vocoder for audio synthesis using the target tensor and acoustic feature sequences of the text corresponding to the audio.

[0024] In a fifth aspect of the present disclosure, an audio synthesis apparatus is provided, comprising: an acquisition module for acquiring target text; an extraction module for extracting an acoustic feature sequence of the target text; a prediction module for inputting the acoustic feature sequence into a fundamental frequency prediction model to perform fundamental frequency prediction, thereby obtaining a fundamental frequency sequence corresponding to the acoustic feature sequence; and a synthesis module for inputting the fundamental frequency sequence and the acoustic feature sequence into a vocoder to perform audio synthesis, thereby obtaining audio corresponding to the target text, wherein the vocoder is trained using the vocoder training method of the first aspect.

[0025] In a sixth aspect of the present disclosure, a computing device is provided, comprising: a processor and a memory connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement a vocoder training method as described in the first aspect or an audio synthesis method as described in the second aspect.

[0026] According to the vocoder training method, audio synthesis method, medium, apparatus, and computing device of this disclosure, a fundamental frequency perturbation can be introduced during vocoder training, thereby improving the robustness of the trained vocoder to fundamental frequency prediction errors in practical applications. Furthermore, during audio synthesis, a fundamental frequency prediction model can be introduced to predict the fundamental frequency of the acoustic feature sequence of the target text, and then the acoustic feature sequence and the predicted fundamental frequency sequence can be input into the trained vocoder for audio synthesis, thereby significantly improving the accuracy and quality of audio synthesis and providing a better user experience. Attached Figure Description

[0027] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which:

[0028] Figure 1 A schematic diagram illustrating an application scenario of the audio synthesis system provided in this disclosure.

[0029] Figure 2 A flowchart illustrating the vocoder training method provided in this embodiment of the disclosure;

[0030] Figure 3 A schematic flowchart of the audio synthesis method provided in this disclosure embodiment;

[0031] Figure 4 A schematic diagram of the fundamental frequency prediction model provided in this embodiment of the disclosure;

[0032] Figure 5 This is a schematic diagram of the structure of the storage medium according to an embodiment of the present disclosure;

[0033] Figure 6 A schematic diagram of the structure of the vocoder training device provided in the embodiments of this disclosure;

[0034] Figure 7 This is a schematic diagram of the structure of the audio synthesis apparatus according to an embodiment of the present disclosure;

[0035] Figure 8 This is a schematic diagram of the structure of a computing device according to an embodiment of the present disclosure.

[0036] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0037] The principles and spirit of this disclosure will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0038] Those skilled in the art will recognize that embodiments of this disclosure can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0039] According to embodiments of this disclosure, a vocoder training method, an audio synthesis method, a medium, an apparatus, and a computing device are proposed. In this document, it should be understood that the terms used refer to the following:

[0040] Text-to-speech (TTS): Based on artificial intelligence (AI) models, input text is converted into corresponding audio, realizing the artificial synthesis of speech, which can be used in human-computer interaction scenarios such as voice broadcasting.

[0041] Singing voice synthesis (SVS): Similar to speech synthesis, it uses AI models to convert input text information into corresponding dry audio recordings of a performance. The input text consists of lyrics and melody information. The lyrics are plain text, while the melody is typically represented by a sequence of notes, describing the duration and pitch of each word. Like speech synthesis, the output of singing voice synthesis is dry audio recordings of a performance, which can be used in song creation, virtual idols, and other similar applications. Generally, singing voice synthesis uses a similar technical framework to speech synthesis, with the main difference being customization in the modeling of melody pitch.

[0042] The basic framework of audio synthesis mainly consists of the following parts: a front-end module, an acoustic model, and a vocoder. The front-end module primarily preprocesses the input text, converting it into a phoneme sequence that the audio synthesis model can understand. For example, if the text to be synthesized is "speech," the front-end model analyzes and processes it, converting it into a phoneme sequence like "sp iych," which describes how the word should be pronounced. After processing by the front-end module, the phoneme sequence to be synthesized can be fed into the acoustic model. The acoustic model maps the phoneme sequence to be synthesized into an acoustic feature sequence composed of frames of feature vectors. Currently, mainstream TTS / SVS systems typically use mel-spectrograms as the acoustic feature sequence. Finally, the acoustic feature sequence is fed into the vocoder, which reconstructs the time-domain signal based on the acoustic feature sequence. The quality of the acoustic model affects the pronunciation accuracy and rhythm of the final synthesized audio, while the vocoder has a greater impact on the sound quality of the synthesized audio (such as whether there is an electronic feel, background noise, signal breakage, etc.).

[0043] Corpus: A library composed of "audio-annotation" pairs, organized by sentence, used for training the synthesis model. The audio consists of recorded dry audio of a predetermined target speaker (whose timbre will be used in the synthesized audio). The annotations are the annotation information obtained by labeling the recorded audio, including but not limited to the phoneme types, the start and end positions of phoneme boundaries, pitch, etc.

[0044] A vocoder is a speech signal encoder and decoder that utilizes only model parameters during transmission, but employs model parameter estimation and speech synthesis techniques during encoding and decoding. It is a type of encoder and decoder that analyzes and synthesizes speech, also known as a speech analysis and synthesis system or a speech audio band compression system. Typically, vocoders are based on neural networks. The vocoder in this disclosure can apply (multiple) upsampling and neural network mapping operations to the input acoustic feature sequence (usually in frames) to gradually refine the temporal resolution from "frames" (typically one frame corresponds to tens of milliseconds) to "samples" (one sample corresponds to a time of 1 / sampling rate; for example, 16000Hz audio means 16000 samples per second, and one sample corresponds to a time of 1 / 16000 = 0.0625 milliseconds). Finally, through neural network mapping, the temporal domain audio signal is reconstructed.

[0045] Fundamental frequency (F0): Sound originates from the vibration of a sound-producing body. A sound signal can be decomposed into a linear superposition of many sine waves (the essence of Fourier transform). The lowest frequency sine wave is the fundamental tone, and its corresponding frequency is the fundamental frequency (measured in Hz, representing the number of vibrations per second). The fundamental frequency is used to determine the pitch characteristics of the entire sound. For example, in a singing context, the pitch of a melody is determined by the frequency of the singer's vocal cord vibrations.

[0046] Pitch: In music, pitch is often used to describe the highness or lowness of a sound, but it is essentially interchangeable with fundamental frequency (F0). Pitch and fundamental frequency (F0) can be converted to each other using the following formula:

[0047]

[0048] f0 = 440 × 2 (p-69) / 12

[0049] Where p represents pitch and f0 represents fundamental frequency.

[0050] Embedding: A commonly used concept in deep learning, referring to the mapping of input data into a fixed-dimensional vector through a network. Specifically, in the input layer of a neural network, embedding layers are frequently used to map class labels into fixed-dimensional vectors before feeding them into the model.

[0051] Tensor: A commonly used concept in the field of deep learning. Essentially, it is a multidimensional array used to record input and output data or network parameters.

[0052] Furthermore, the number of any elements in the accompanying drawings is for illustrative purposes only and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0053] In addition, the data involved in this disclosure may be data authorized by the user or fully authorized by all parties. The collection, dissemination and use of the data shall comply with the requirements of relevant national laws and regulations. The implementation methods / executives of this disclosure may be combined with each other.

[0054] The principles and spirit of this disclosure will be explained in detail below with reference to several representative embodiments. Invention Overview

[0056] The inventors have discovered that in singing voice synthesis tasks, the naturalness, continuity, and accuracy of the synthesized audio in terms of pitch and melody directly affect the user's listening experience. Therefore, many related technologies currently propose modeling the fundamental frequency in the vocoder. Specifically, in addition to introducing a frame-level Mel spectrum sequence as input, the vocoder also introduces a frame-level fundamental frequency F0 sequence (in Hz) as input. During vocoder training, the frame-level fundamental frequency F0 sequence is extracted from the audio corpus, and then sampled to obtain the fundamental frequency F0 corresponding to each sample point. A sinusoidal excitation signal for each fundamental frequency F0 is then generated, and a tensor is constructed using this two-dimensional array of sinusoidal excitation signals. This tensor is then mapped through the vocoder's network and combined with the Mel spectrum before participating in the training of the neural network vocoder.

[0057] In the training process, the fundamental frequency F0 sequence introduced by the relevant technology is the real and accurate fundamental frequency F0. This will cause the trained vocoder to rely too much on the fundamental frequency F0. In the actual use stage, the input fundamental frequency F0 sequence is predicted. Once a prediction error occurs, even a slight error may cause the synthesized audio to be out of tune, thus making the robustness of the trained vocoder poor.

[0058] Therefore, this disclosure applies a certain degree of noise perturbation to the fundamental frequency F0 sequence extracted from real audio in an audio corpus before feeding it into the vocoder for training. Thus, the vocoder uses the perturbated fundamental frequency F0 throughout the training process, which allows the vocoder to tolerate a certain degree of fundamental frequency deviation, thereby improving the robustness of the trained vocoder.

[0059] After the vocoder is trained, acoustic feature sequences are extracted from the acquired target text. Then, a fundamental frequency prediction model is introduced to predict the acoustic feature sequences, resulting in a predicted fundamental frequency sequence. The predicted fundamental frequency sequence and the acoustic feature sequence are then input into the trained vocoder for audio synthesis. Because fundamental frequency perturbation is introduced during vocoder training, and excitation mapping and embedding mapping are performed on the perturbated fundamental frequency sequence, the resulting tensors are combined before vocoder training. Therefore, the trained vocoder is robust. Thus, even if the fundamental frequency prediction model makes a fundamental frequency prediction error when synthesizing audio through the vocoder, it will not affect the sound quality and pitch accuracy of the synthesized audio.

[0060] After introducing the basic principles of this disclosure, various non-limiting embodiments of this disclosure will be described in detail below.

[0061] Application Scenarios Overview

[0062] First refer to Figure 1 , Figure 1This diagram illustrates an application scenario of the audio synthesis system provided in this embodiment. The audio synthesis system may include an audio corpus, a fundamental frequency extraction module, a fundamental frequency perturbation module, a fundamental frequency prediction model, a fundamental frequency mapping module, and a vocoder. The audio corpus, fundamental frequency extraction module, fundamental frequency perturbation module, fundamental frequency mapping module, and vocoder are used to train the vocoder during the training phase; the fundamental frequency prediction model, fundamental frequency mapping module, and the trained vocoder are used for inference during the inference phase of audio synthesis. During the training phase, the fundamental frequency prediction model does not participate; during the inference phase, the fundamental frequency extraction module and the fundamental frequency perturbation module do not participate.

[0063] During the training phase, the fundamental frequency extraction module extracts the fundamental frequency from the audio corpus, obtaining the fundamental frequency sequence F0. gt and the fundamental frequency sequence F0 gt Output to the baseband perturbation module; the baseband perturbation module processes the received baseband sequence F0. gt The fundamental frequency is perturbed to obtain the perturbed fundamental frequency sequence F0. gt ', and the perturbed fundamental frequency sequence F0 gt The output is sent to the baseband mapping module; the baseband mapping module uses excitation mapping and embedding mapping to process the received perturbed baseband sequence F0. gt The mapping process is performed to obtain a tensor, and the obtained tensor is output to the vocoder. The vocoder is trained on the tensor, the audio in the audio corpus and the corresponding acoustic feature sequence of the audio to obtain a trained vocoder.

[0064] During the inference phase, the acoustic feature sequence corresponding to the target text can be input into the fundamental frequency prediction model to predict the fundamental frequency, resulting in the predicted fundamental frequency sequence F0. pred and the predicted fundamental frequency sequence F0 pred The output is sent to the baseband mapping module; the baseband mapping module uses excitation mapping and embedding mapping to predict the received baseband sequence F0. pred The mapping process is performed to obtain a tensor, and the obtained tensor is output to the trained vocoder. The vocoder synthesizes the tensor and the acoustic feature sequence corresponding to the target text to obtain the audio corresponding to the target text.

[0065] Specifically, introducing fundamental frequency perturbation during the training phase can improve the robustness of the trained vocoder. Thus, even if the fundamental frequency prediction model has certain prediction biases during the inference phase, the vocoder can still guarantee the accuracy and quality of the synthesized audio output. Furthermore, by using a combination of excitation mapping and embedding mapping during fundamental frequency mapping, the vocoder can be trained better, thereby improving the accuracy and quality of the trained vocoder in audio synthesis.

[0066] Exemplary methods

[0067] The following is combined Figure 1 Application scenarios, refer to Figure 2 To describe a vocoder training method according to exemplary embodiments of the present disclosure, and to refer to Figure 3 This document describes an audio synthesis method according to exemplary embodiments of the present disclosure. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in any way. Rather, the embodiments of the present disclosure can be applied to any applicable scenario.

[0068] refer to Figure 2 , Figure 2 A schematic flowchart illustrating the vocoder training method provided in this embodiment. The vocoder training method includes the following steps:

[0069] S201: Obtain the first fundamental frequency sequence of audio from the audio corpus.

[0070] Specifically, most commonly used speech synthesis and / or singing synthesis currently employs deep learning modeling. Specifically, an audio corpus for vocoder training can be constructed first, then a suitable vocoder model structure can be designed, and the audio corpus can be fed into commonly used deep learning training frameworks (such as PyTorch, TensorFlow, etc.) to train the corresponding vocoder, thereby achieving speech synthesis and / or singing synthesis.

[0071] Specifically, when constructing an audio corpus, the target speaker can be identified first (the synthesized audio will be the speaker's timbre); then, the speaker's speech or singing dry audio can be recorded in a recording studio or similar environment; then, the recorded audio can be annotated, and the annotation information includes, but is not limited to, the phoneme types contained in the audio, the start and end positions of the phonemes in the audio, pitch, etc.; finally, several "audio-annotation" pairs can be obtained in units of sentences, and these pairs can be used to form an audio corpus for vocoder training.

[0072] Specifically, after the audio corpus is built, the vocoder can be trained. That is, real audio from the audio corpus can be obtained, and the first fundamental frequency sequence in the audio can be extracted. This first fundamental frequency sequence is used for subsequent training of the vocoder.

[0073] S202: Perform fundamental frequency perturbation processing on the first fundamental frequency sequence to obtain the second fundamental frequency sequence.

[0074] Specifically, the first fundamental frequency sequence is extracted from real audio data in an audio corpus. If the vocoder is trained using the first fundamental frequency sequence, it will become overly reliant on real audio, resulting in poor robustness. Therefore, the first fundamental frequency sequence can be perturbed to a certain extent (the degree of perturbation can be controlled). The resulting second fundamental frequency sequence is the perturbed sequence. Using the second fundamental frequency sequence to train the vocoder can improve its robustness.

[0075] S203: Map the second fundamental frequency sequence to obtain the target tensor.

[0076] Specifically, after obtaining the fundamental frequency sequence, the fundamental frequency sequence can be converted into data suitable for training the vocoder. Therefore, the second fundamental frequency sequence can be mapped to obtain the target tensor, which can then be used to train the vocoder.

[0077] S204: Train the target tensor, the acoustic feature sequence corresponding to the audio, and the audio to obtain a vocoder for audio synthesis.

[0078] Specifically, after obtaining the target tensor, the target tensor, the acoustic feature sequence corresponding to the audio in the audio corpus, and the audio can be trained to obtain a vocoder.

[0079] Specifically, the target tensor and the acoustic feature sequence corresponding to the audio can be input into a pre-trained model, such as a neural network model. The audio and the output of the pre-trained model are then compared, and the pre-trained model is debugged and optimized to finally obtain the vocoder.

[0080] The vocoder training method disclosed herein applies a certain degree of noise perturbation to the fundamental frequency F0 sequence extracted from real audio data in an audio corpus before feeding it into the vocoder for training. Therefore, the vocoder uses the perturbed fundamental frequency F0 throughout the training process, which allows the vocoder to tolerate a certain degree of fundamental frequency deviation, thereby improving the robustness of the trained vocoder.

[0081] In one embodiment of this disclosure, a second fundamental frequency sequence is obtained by performing fundamental frequency perturbation processing on a first fundamental frequency sequence, including: performing the following fundamental frequency perturbation processing on each fundamental frequency in the first fundamental frequency sequence to obtain the second fundamental frequency sequence: obtaining a first logarithm of the fundamental frequency; performing maximum / minimum truncation processing on the first logarithm to obtain a second logarithm; performing quantization processing on the second logarithm to obtain a fundamental frequency category label corresponding to the fundamental frequency; and performing inverse quantization processing on the fundamental frequency category label.

[0082] Specifically, during the training phase of the vocoder, it is necessary to extract the real fundamental frequency sequence, or first fundamental frequency sequence, from the real audio in the audio corpus. This first fundamental frequency sequence is then introduced into the vocoder for training. However, directly introducing the first fundamental frequency sequence into the vocoder for training makes the resulting vocoder overly reliant on the real audio in the audio corpus, resulting in poor robustness. Therefore, before introducing the extracted first fundamental frequency sequence into the vocoder, it can be perturbed to obtain a perturbed fundamental frequency sequence, or second fundamental frequency sequence, thereby improving the robustness of the vocoder.

[0083] Specifically, when performing fundamental frequency perturbation processing on the first fundamental frequency sequence to obtain the second fundamental frequency sequence, for the first fundamental frequency sequence F0 gt For each fundamental frequency f in the equation, the fundamental frequency f can first be transformed into the logarithmic domain using the following formula:

[0084] f log =ln(f+1)

[0085] Among them, f log This represents the fundamental frequency in the logarithmic domain, also known as the first logarithm. When converting the fundamental frequency f to the logarithmic domain, adding 1 to the fundamental frequency f avoids the problem of not being able to take the logarithm when the fundamental frequency is 0Hz.

[0086] Specifically, by transforming the fundamental frequency f into the logarithmic domain, we obtain f log Then, in the logarithmic field, for f log Perform extremum truncation, that is, truncate the maximum and minimum values ​​to obtain the second logarithm for subsequent quantization operations. Extremum truncation can be achieved using the following formula:

[0087]

[0088] Among them, f log ' represents the second logarithm, f log_min f log The minimum value of f in the logarithmic field. log_max f log The maximum value of f in the logarithmic field log_min and f log_max These are all preset values, which can be set as needed.

[0089] Specifically, after obtaining the second logarithm, a quantization operation can be performed on the second logarithm to obtain multiple possible candidate values. Each candidate value can serve as a corresponding fundamental frequency category label. The quantization of the second logarithm can be achieved using the following formula:

[0090]

[0091] f log_q= (label + 0.5) × step + f log_min

[0092] Where label represents the fundamental frequency category label, f log_q This represents the quantized fundamental frequency, and step represents the step size of the quantization operation, as follows:

[0093]

[0094] Where M represents the number of candidate values, that is, the number of baseband category labels.

[0095] Specifically, f log_q The quantized fundamental frequency is in the logarithmic domain; therefore, f can be expressed by the following formula. log_q Dequantization back to the Hz domain:

[0096] f′=exp(f log_q )-1

[0097] Where f' represents the dequantized fundamental frequency, which is also the fundamental frequency in the second fundamental frequency sequence.

[0098] Specifically, f' introduces a quantization error perturbation compared to the true fundamental frequency f, thus improving the robustness of the trained vocoder.

[0099] In another embodiment of this disclosure, a second fundamental frequency sequence is obtained by performing fundamental frequency perturbation processing on a first fundamental frequency sequence, including: performing the following fundamental frequency perturbation processing on each fundamental frequency in the first fundamental frequency sequence to obtain a second fundamental frequency sequence: randomly sampling the fundamental frequency within a preset sampling range of the fundamental frequency to obtain perturbation parameters corresponding to the fundamental frequency; and summing the perturbation parameters and the fundamental frequency.

[0100] Specifically, when performing fundamental frequency perturbation processing on the first fundamental frequency sequence to obtain the second fundamental frequency sequence, in addition to the aforementioned quantization processing, for each fundamental frequency in the first fundamental frequency sequence, random sampling can also be performed directly within the preset sampling range of the fundamental frequency. After obtaining the perturbation parameters, the perturbation parameters are directly added to the fundamental frequency to obtain the perturbed fundamental frequency, which is the fundamental frequency in the second fundamental frequency sequence.

[0101] Specifically, for each fundamental frequency f in the first fundamental frequency sequence, the perturbation parameter is first obtained by random sampling within a preset sampling range of the fundamental frequency, and then the fundamental frequency f and the perturbation parameter are summed using the following formula:

[0102] f′=f+Δf

[0103] Where f' represents the perturbated fundamental frequency, which is also the fundamental frequency in the second fundamental frequency sequence; Δf represents the perturbation parameter, which can be randomly sampled from a Gaussian distribution, i.e., Δf ~ N(0,σ).2 ), where σ represents the standard deviation, used to control the degree of disturbance, and σ is a controllable parameter that can be adjusted as needed.

[0104] Optionally, the preset sampling range of the fundamental frequency can be other preset sampling ranges besides Gaussian distribution, such as Markov chain Monte Carlo (MCMC), Bayesian network, imbalanced sample set, etc.

[0105] Specifically, f' introduces perturbation parameters obtained by random sampling compared to the true fundamental frequency f, thus improving the robustness of the trained vocoder.

[0106] In another embodiment of this disclosure, the mapping process includes excitation mapping and embedding mapping. The mapping process on the second fundamental frequency sequence to obtain the target tensor includes: performing excitation mapping on the second fundamental frequency sequence to obtain a first tensor; performing embedding mapping on the second fundamental frequency sequence to obtain a second tensor; and combining the first tensor and the second tensor to obtain the target tensor.

[0107] Specifically, the excitation mapping can be done using source excitation, and the embedding mapping can be done using an embedding with the fundamental frequency F0. When mapping the second fundamental frequency sequence to obtain the target tensor, the second fundamental frequency sequence can first be excited to obtain the first tensor. That is, the second fundamental frequency sequence is converted into a sinusoidal excitation signal using source excitation, and the first tensor is constructed using the obtained sinusoidal excitation signal. Then, the second fundamental frequency sequence is embedded to obtain the second tensor. That is, the second fundamental frequency sequence is mapped using an embedding with the fundamental frequency F0. Finally, the first and second tensors are combined, and the resulting target tensor can achieve better training results when used for subsequent vocoder training, thus improving the audio synthesis effect of the trained vocoder.

[0108] Specifically, when converting the second fundamental frequency sequence into a sinusoidal excitation signal using source excitation, and constructing the first tensor using the obtained sinusoidal excitation signal, the following method can be used:

[0109] First, the second fundamental frequency sequence is upsampled using linear interpolation to refine its temporal granularity from "frames" to "samples". For example, assuming the frame shift (hop_size) used in the second fundamental frequency sequence is 256, 255 F0 values ​​will be interpolated between two adjacent F0 values ​​in the second fundamental frequency sequence using linear interpolation, resulting in the fundamental frequency F0 corresponding to each sample. Then, a sinusoidal excitation signal is generated according to the following formula:

[0110]

[0111] Among them, e ij The signal represents a sinusoidal excitation signal; i represents the i-th harmonic signal (i ranges from [1, K], there are K harmonic signals in total, and K is a settable integer parameter); j represents the j-th sample point; f j Let represent the fundamental frequency of the j-th sample point; Sr represent the sampling rate; φ represent the initial phase of the i-th harmonic signal, which can be randomly generated within the range [-π, π]; Znoise represent Gaussian noise (i.e., the sample point with a fundamental frequency of 0, whose value is randomly sampled from a Gaussian distribution); T represent the total number of samples. From the above formula, it can be seen that the i-th harmonic signal e i Essentially, it is a sinusoidal excitation signal.

[0112] After obtaining the sinusoidal excitation signal e ij After that, e can be used ij This two-dimensional array constructs a first tensor of size (K,T), where the K harmonic signals correspond to the K channels of the first tensor, and the T samples correspond to the time axis of the first tensor.

[0113] In another embodiment of this disclosure, embedding mapping is performed on the second fundamental frequency sequence to obtain a second tensor, including: in response to the second fundamental frequency sequence being a floating-point value in a continuous domain, upsampling interpolation is used to upsample the second fundamental frequency sequence to obtain a first sampled fundamental frequency sequence; embedding mapping is performed on the first sampled fundamental frequency sequence to obtain a second tensor.

[0114] Specifically, when embedding and mapping the second fundamental frequency sequence to obtain the second tensor—that is, mapping the second fundamental frequency sequence using an embedding method that introduces the fundamental frequency F0—there are two possibilities: one is that the second fundamental frequency sequence is a continuous-domain floating-point value, and the other is that the second fundamental frequency sequence is a quantized fundamental frequency category label. Different embedding and mapping methods correspond to different types of second fundamental frequency sequences, thus ensuring the accuracy of the obtained second tensor and improving the quality of the vocoder obtained through subsequent training. The following section describes the second fundamental frequency sequence as a continuous-domain floating-point value.

[0115] Specifically, if the second fundamental frequency sequence is a floating-point value in the continuous domain, for example, a frequency value f in the Hz domain, or f in the log domain. logThen, an upsampling interpolation method can be used to upsample the second fundamental frequency sequence. For example, linear interpolation can be used to upsample the second fundamental frequency sequence (other upsampling interpolation methods) to refine the temporal granularity of the second fundamental frequency sequence from "frames" to "samples", resulting in a floating-point value with dimension 1, which is the first sampled fundamental frequency sequence. Then, the first sampled fundamental frequency sequence is embedded and mapped into a vector of dimension L (L is a configurable parameter) using a linear layer of a neural network. This allows the second fundamental frequency sequence to be converted into a second tensor at the sample level. The size of this second tensor can be (L, T), where T can be used to represent the number of samples.

[0116] In another embodiment of this disclosure, embedding mapping is performed on the second fundamental frequency sequence to obtain a second tensor, including: in response to the second fundamental frequency sequence being a quantized fundamental frequency category label, upsampling processing is performed on the second fundamental frequency sequence using nearest neighbor interpolation to obtain a second sampled fundamental frequency sequence; embedding mapping is performed on the second sampled fundamental frequency sequence to obtain a second tensor.

[0117] Specifically, the following describes the second fundamental frequency sequence as a quantized fundamental frequency category label. If the second fundamental frequency sequence is a quantized fundamental frequency category label, nearest neighbor interpolation can be used to upsample the second fundamental frequency sequence, refining its temporal granularity from "frames" to "samples" to obtain a second sampled fundamental frequency sequence. At this point, each F0 in the second sampled fundamental frequency sequence is still a fundamental frequency category label. The second sampled fundamental frequency sequence can be embedded and mapped into a vector of dimension L (L is a configurable parameter) using the embedding layer of a neural network. This converts the second fundamental frequency sequence into a sample-level second tensor, whose size can be (L, T), where T represents the number of samples. This can improve the quality of the vocoder obtained through subsequent training.

[0118] In another embodiment of this disclosure, combining the first tensor and the second tensor to obtain the target tensor includes: concatenating the first tensor and the second tensor in response to their different sizes to obtain the target tensor; and summing the first tensor and the second tensor in response to their identical sizes to obtain the target tensor.

[0119] Specifically, after performing excitation mapping on the second fundamental frequency sequence to obtain the first tensor and embedding mapping on the second fundamental frequency sequence to obtain the second tensor, in order to improve the quality of the trained vocoder, the first tensor and the second tensor can be combined.

[0120] Specifically, if the first tensor and the second tensor have different dimensions, where the first tensor has a dimension of (K,T) and the second tensor has a dimension of (L,T), then the first tensor and the second tensor can be concatenated along the channel axis to obtain a target tensor with a dimension of (K+L,T).

[0121] Specifically, when embedding and mapping the second fundamental frequency sequence, the dimension can be set from L to K in advance, so that the size of the first tensor and the second tensor are the same, both (K,T). Then, the first tensor and the second tensor with the same size can be directly added together to obtain the target tensor.

[0122] refer to Figure 3 , Figure 3 This is a schematic flowchart illustrating an audio synthesis method provided in this disclosure. The audio synthesis method includes the following steps:

[0123] S301: Obtain the target text.

[0124] Specifically, the target text can be the text corresponding to the audio to be synthesized. For example, if the audio to be synthesized is a song, the target text can include the lyrics and melody of the song, where the lyrics can be ordinary text and the melody can be represented by a sequence of notes.

[0125] S302: Extract the acoustic feature sequence of the target text.

[0126] Specifically, after obtaining the target text, it can be preprocessed to convert it into a phoneme sequence, which can be used to describe the pronunciation of the target text. Then, the phoneme sequence is mapped to obtain an acoustic feature sequence composed of frame-by-frame feature vectors.

[0127] Optionally, the acoustic feature sequence may include a frame-level Mel spectrum sequence, or other similar feature sequences, such as a vector quantized feature vector sequence, a latent variable sequence of a variational autoencoder, or a combination of these sequences (e.g., splicing, addition, etc.).

[0128] S303: Input the acoustic feature sequence into the fundamental frequency prediction model to predict the fundamental frequency and obtain the fundamental frequency sequence corresponding to the acoustic feature sequence.

[0129] Specifically, the fundamental frequency prediction model can take the acoustic feature sequence of the target text as input to obtain the predicted fundamental frequency F0 sequence.

[0130] Specifically, the fundamental frequency prediction model can directly predict the fundamental frequency sequence in the output Hz domain, or it can predict the fundamental frequency sequence in other domains, such as predicting the fundamental frequency floating-point value in the output log domain, or the quantized fundamental frequency category label, as long as it can be converted back to the Hz domain later.

[0131] S304: Input the fundamental frequency sequence and acoustic feature sequence into the vocoder to synthesize audio and obtain the audio corresponding to the target text.

[0132] Specifically, the vocoder is trained using the vocoder training method described above.

[0133] Specifically, after obtaining the predicted fundamental frequency sequence, the fundamental frequency sequence and the acoustic feature sequence can be input together into a pre-trained vocoder for audio synthesis, thereby obtaining the audio corresponding to the target text.

[0134] The audio synthesis method disclosed herein extracts acoustic feature sequences from the acquired target text, then introduces a fundamental frequency prediction model to predict the acoustic feature sequences, thereby obtaining a predicted fundamental frequency sequence. The predicted fundamental frequency sequence and the acoustic feature sequence are then input into a pre-trained vocoder for audio synthesis. Since fundamental frequency perturbation is introduced during vocoder training, and excitation mapping and embedding mapping are performed on the perturbated fundamental frequency sequence, the resulting tensors are combined before vocoder training. Therefore, the trained vocoder has strong robustness. Thus, even if the fundamental frequency prediction model makes a fundamental frequency prediction error when synthesizing audio through the vocoder, it will not affect the sound quality and pitch accuracy of the synthesized audio.

[0135] In one embodiment of this disclosure, the fundamental frequency prediction model includes a convolutional layer, a nonlinear layer, a normalization layer, a random deactivation layer, a linear layer, and a classification layer. The acoustic feature sequence is input into the fundamental frequency prediction model to predict the fundamental frequency, obtaining the fundamental frequency sequence corresponding to the acoustic feature sequence. This includes: inputting the acoustic feature sequence into the convolutional layer for convolution processing to obtain a first feature; inputting the first feature into the nonlinear layer for activation processing to obtain a second feature; inputting the second feature into the normalization layer for normalization processing to obtain a third feature; inputting the third feature into the random deactivation layer for deactivation processing to obtain a fourth feature; inputting the fourth feature into the linear layer for linear processing to obtain a fifth feature; and inputting the fifth feature into the classification layer for classification processing to obtain the fundamental frequency sequence.

[0136] Specifically, the fundamental frequency prediction model can be modeled using common neural networks. The structure of the fundamental frequency prediction model can be as follows: Figure 4 As shown. Figure 4This is a schematic diagram of the fundamental frequency prediction model provided in this embodiment. The fundamental frequency prediction model may include convolutional layers, nonlinear layers, normalization layers, random deactivation layers, linear layers, and classification layers, wherein the convolutional layers can be one-dimensional convolutions. The acoustic feature sequence can first pass through the convolutional layer, nonlinear layer, normalization layer, and random deactivation layer N times (N can be determined by the user), and then pass through the linear layer and classification layer to obtain the predicted fundamental frequency sequence, such as a fundamental frequency category label or a fundamental frequency value.

[0137] Specifically, the fundamental frequency prediction model can perform fundamental frequency prediction processing on the acoustic feature sequence of the target text to obtain the predicted fundamental frequency sequence, which can be used as the input of the vocoder to achieve audio synthesis. Since the trained vocoder is robust, even if the fundamental frequency prediction model makes a prediction error when performing fundamental frequency prediction processing on the acoustic feature sequence, it will not affect the quality of the synthesized audio.

[0138] Optionally, the structure of the fundamental frequency prediction model, in addition to the following... Figure 4 Besides the structure shown, it can also be any other model structure that can achieve fundamental frequency prediction.

[0139] In another embodiment of this disclosure, the fundamental frequency sequence and the acoustic feature sequence are input into a vocoder for audio synthesis to obtain the audio corresponding to the target text, including: mapping the fundamental frequency sequence to obtain a target tensor; and inputting the target tensor and the acoustic feature sequence into a vocoder for audio synthesis to obtain the audio.

[0140] Specifically, after obtaining the predicted fundamental frequency sequence, the predicted fundamental frequency sequence can be converted into data suitable for vocoder processing. Therefore, the predicted fundamental frequency sequence can be mapped to obtain the target tensor. The target tensor and acoustic feature sequence can then be input into the vocoder for audio synthesis to obtain the audio corresponding to the target text. This can achieve high-quality synthesized audio and accurate pitch.

[0141] In another embodiment of this disclosure, the mapping process includes excitation mapping and embedding mapping. The mapping process on the fundamental frequency sequence to obtain the target tensor includes: performing excitation mapping on the fundamental frequency sequence to obtain a first tensor; performing embedding mapping on the fundamental frequency sequence to obtain a second tensor; and combining the first tensor and the second tensor to obtain the target tensor.

[0142] Specifically, excitation mapping can be performed using source excitation, and embedding mapping can be performed using an embedding method that incorporates the fundamental frequency F0. When mapping the predicted fundamental frequency sequence to obtain the target tensor, the predicted fundamental frequency sequence can first be excited to obtain the first tensor. That is, the predicted fundamental frequency sequence is converted into a sinusoidal excitation signal using source excitation, and the first tensor is constructed using the obtained sinusoidal excitation signal. Then, the predicted fundamental frequency sequence is embedded to obtain the second tensor. That is, the predicted fundamental frequency sequence is mapped using an embedding method that incorporates the fundamental frequency F0 to obtain the second tensor. Finally, the first and second tensors are combined, and the resulting target tensor can achieve better synthesis results and improve the synthesized audio quality and pitch accuracy when used in subsequent audio synthesis.

[0143] Specifically, when using source excitation to convert the predicted fundamental frequency sequence into a sinusoidal excitation signal, and then using the obtained sinusoidal excitation signal to construct the first tensor, the following method can be used:

[0144] First, the predicted fundamental frequency sequence is upsampled using linear interpolation, refining the temporal granularity from "frames" to "samples". For example, assuming the frame shift (hop_size) used in the predicted fundamental frequency sequence is 256, 255 F0 values ​​will be interpolated between two adjacent F0 values ​​in the predicted fundamental frequency sequence using linear interpolation, obtaining the fundamental frequency F0 corresponding to each sample. Then, a sinusoidal excitation signal is generated according to the following formula:

[0145]

[0146] Among them, e ij The signal represents a sinusoidal excitation signal; i represents the i-th harmonic signal (i ranges from [1, K], there are K harmonic signals in total, and K is a settable integer parameter); j represents the j-th sample point; f j Let Sr represent the fundamental frequency of the j-th sample point; Sr can be used to represent the sampling rate; φ represents the initial phase of the i-th harmonic signal, which can be randomly generated within the range [-π, π]; Znoise represents Gaussian noise (i.e., the sample point with a fundamental frequency of 0, whose value is randomly sampled from a Gaussian distribution); T can be used to represent the total number of samples. From the above formula, it can be seen that the i-th harmonic signal e i Essentially, it is a sinusoidal excitation signal.

[0147] After obtaining the sinusoidal excitation signal e ij After that, e can be used ijThis two-dimensional array constructs a first tensor of size (K,T), where the K harmonic signals correspond to the K channels of the first tensor, and the T samples correspond to the time axis of the first tensor.

[0148] In another embodiment of this disclosure, embedding mapping is performed on the fundamental frequency sequence to obtain a second tensor, including: in response to the fundamental frequency sequence being a continuous domain floating-point value, upsampling the fundamental frequency sequence using linear interpolation to obtain a first sampled fundamental frequency sequence; embedding mapping is performed on the first sampled fundamental frequency sequence to obtain a second tensor; in response to the fundamental frequency sequence being a quantized fundamental frequency category label, upsampling the fundamental frequency sequence using nearest neighbor interpolation to obtain a second sampled fundamental frequency sequence; and embedding mapping is performed on the second sampled fundamental frequency sequence to obtain a second tensor.

[0149] Specifically, when embedding the predicted fundamental frequency sequence to obtain the second tensor—that is, mapping the predicted fundamental frequency sequence using an embedding method that introduces the fundamental frequency F0—two cases can be included: one where the predicted fundamental frequency sequence is a continuous-domain floating-point value, and the other where the predicted fundamental frequency sequence is a quantized fundamental frequency category label. Different embedding mapping methods correspond to different types of predicted fundamental frequency sequences, thus ensuring the accuracy of the obtained second tensor, achieving better synthesis results, and improving the synthesized audio quality and pitch accuracy.

[0150] Specifically, if the predicted fundamental frequency sequence is a floating-point value in the continuous domain, for example, a frequency value f in the Hz domain, or f in the log domain... log Then, an upsampling interpolation method can be used to upsample the predicted fundamental frequency sequence. For example, linear interpolation can be used to upsample the predicted fundamental frequency sequence (other upsampling interpolation methods) to refine the temporal granularity of the predicted fundamental frequency sequence from "frames" to "samples", resulting in a floating-point value with dimension 1, which is the first sampled fundamental frequency sequence. Then, the first sampled fundamental frequency sequence is embedded and mapped into a vector of dimension L (L is a configurable parameter) using a linear layer of a neural network. This allows the predicted fundamental frequency sequence to be converted into a second tensor at the sample level. The size of this second tensor can be (L, T), where T can be used to represent the number of samples.

[0151] Specifically, if the predicted fundamental frequency sequence is a quantized fundamental frequency category label, then nearest neighbor interpolation can be used to upsample the predicted fundamental frequency sequence, refining the temporal granularity of the predicted fundamental frequency sequence from "frames" to "samples" to obtain a second sampled fundamental frequency sequence. At this time, each F0 in the second sampled fundamental frequency sequence is still a fundamental frequency category label. The second sampled fundamental frequency sequence can be embedded and mapped into a vector of dimension L (L is a configurable parameter) using the embedding layer of a neural network. Thus, the predicted fundamental frequency sequence can be converted into a second tensor at the sample level. The size of the second tensor can be (L, T), where T can be used to represent the number of samples.

[0152] In another embodiment of this disclosure, combining the first tensor and the second tensor to obtain the target tensor includes: concatenating the first tensor and the second tensor in response to their different sizes to obtain the target tensor; and summing the first tensor and the second tensor in response to their identical sizes to obtain the target tensor.

[0153] Specifically, after performing excitation mapping on the predicted fundamental frequency sequence to obtain the first tensor and embedding mapping on the predicted fundamental frequency sequence to obtain the second tensor, the first tensor and the second tensor can be combined to improve the synthesized audio quality and pitch accuracy.

[0154] Specifically, if the first tensor and the second tensor have different dimensions, where the first tensor has a dimension of (K,T) and the second tensor has a dimension of (L,T), then the first tensor and the second tensor can be concatenated along the channel axis to obtain a target tensor with a dimension of (K+L,T).

[0155] Specifically, when embedding and mapping the second fundamental frequency sequence, the dimension can be set from L to K in advance, so that the size of the first tensor and the second tensor are the same, both (K,T). Then, the first tensor and the second tensor with the same size can be directly added together to obtain the target tensor.

[0156] In another embodiment of this disclosure, the method further includes: training a fundamental frequency prediction model using an audio corpus and acoustic feature sequences.

[0157] Specifically, when training the fundamental frequency prediction model, the true fundamental frequency F0 extracted from the audio corpus can be quantized first to obtain the target fundamental frequency. Then, the quantized target fundamental frequency is used as the target output of the fundamental frequency prediction model, and training can be performed using the cross-entropy loss function. During the inference phase, after the target fundamental frequency is predicted by the fundamental frequency prediction model, the target fundamental frequency is dequantized to convert it back to the fundamental frequency in the Hz domain.

[0158] For example, when training a fundamental frequency prediction model, the target fundamental frequency can be a floating-point value in the log domain. The minimum mean square error (MSE) can be used as the loss function to train the model, resulting in a fundamental frequency prediction in the log domain. The log domain fundamental frequency can then be converted back to the Hz domain fundamental frequency by performing an exp operation. As another example, when training the fundamental frequency prediction model, the target fundamental frequency can be quantized (e.g., by directly performing uniform quantization on the Hz domain fundamental frequency, or by taking the logarithm and then performing uniform quantization), resulting in M ​​possible fundamental frequency category labels. The model is then trained using a cross-entropy loss function. During the inference phase, the fundamental frequency category labels predicted by the model can be converted back to the Hz domain fundamental frequency by performing a dequantization operation.

[0159] Optionally, when training the vocoder, if the first fundamental frequency sequence is perturbed to obtain a fundamental frequency category label as the second fundamental frequency sequence, this label, being a quantized fundamental frequency category label, can also be used as the target output when training the fundamental frequency prediction model, achieving two goals at once. The predicted fundamental frequency sequence obtained by the fundamental frequency prediction model can be used as input to the vocoder for audio synthesis. Because the well-trained vocoder is robust, even if a prediction error occurs when the fundamental frequency prediction model performs fundamental frequency prediction processing on the acoustic feature sequence, it will not affect the quality of the synthesized audio.

[0160] In another embodiment of this disclosure, the target text is the text corresponding to the singing voice, and the target text includes at least one of the phoneme type of the singing voice, the start and end positions of the phoneme boundaries in the singing voice, and the pitch of the singing voice.

[0161] Specifically, the target text can be the text corresponding to a song or other audio, such as speech or vocal recordings. If the target text is the text corresponding to a song, it can include the lyrics, the melody information of the song (represented by notes), and annotation information of the song. The annotation information can include, but is not limited to, at least one of the following: the phoneme types contained in the song, the start and end positions of the phonemes in the song, and the pitch of the song. By obtaining the target text, acoustic feature sequences can be extracted. These acoustic feature sequences are then input into a fundamental frequency prediction model for fundamental frequency prediction. The resulting predicted fundamental frequency sequence can then be used as input to a trained vocoder, thereby enabling audio synthesis by the vocoder and improving the audio quality and pitch accuracy of the synthesized audio.

[0162] Exemplary media

[0163] After introducing the methods of exemplary embodiments of this disclosure, the following references are made. Figure 5 The storage medium of the exemplary embodiments of this disclosure will be described.

[0164] refer to Figure 5 As shown, the storage medium 50 stores a program product for implementing the above-described methods according to embodiments of the present disclosure. This program product may be a portable compact disc read-only memory (CD-ROM) and includes computer-executable instructions for causing a computing device to execute the vocoder training method or audio synthesis method provided in this disclosure. However, the program product of this disclosure is not limited thereto.

[0165] The program product may employ any combination of one or more readable media. A readable media may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0166] A readable signal medium may include data signals propagated in baseband or as part of a carrier wave, carrying computer-executed instructions. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium.

[0167] Computer-executable instructions for performing the operations disclosed herein can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The computer-executable instructions can be executed entirely on the user's computing device, partially on the user's computing device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN).

[0168] Exemplary device

[0169] Having introduced the medium of exemplary embodiments of this disclosure, the following references are made to... Figure 6 A vocoder training apparatus according to an exemplary embodiment of this disclosure will be described, which is used to implement the vocoder training method in any of the above method embodiments, and referenced from Figure 7The audio synthesis apparatus of the exemplary embodiments of this disclosure will be described to implement the audio synthesis method in any of the above method embodiments. The implementation principle and technical effect are similar, and will not be repeated here.

[0170] Figure 6 This is a schematic diagram of the structure of a vocoder training device provided in this embodiment of the present disclosure. The vocoder training device 600 includes:

[0171] The acquisition module 601 is used to acquire the first fundamental frequency sequence of audio in the audio corpus;

[0172] The first processing module 602 is used to perform fundamental frequency perturbation processing on the first fundamental frequency sequence to obtain the second fundamental frequency sequence;

[0173] The second processing module 603 is used to map the second fundamental frequency sequence to obtain the target tensor;

[0174] Training module 604 is used to train a vocoder for audio synthesis using the target tensor and the acoustic feature sequence of the text corresponding to the audio.

[0175] Optionally, the first processing module 602 performs fundamental frequency perturbation processing on the first fundamental frequency sequence to obtain the second fundamental frequency sequence through the following steps: For each fundamental frequency in the first fundamental frequency sequence, the following fundamental frequency perturbation processing is performed to obtain the second fundamental frequency sequence: obtaining the first logarithm of the fundamental frequency; performing maximum / minimum truncation processing on the first logarithm to obtain the second logarithm; performing quantization processing on the second logarithm to obtain the fundamental frequency category label corresponding to the fundamental frequency; and performing inverse quantization processing on the fundamental frequency category label.

[0176] Optionally, the first processing module 602 performs fundamental frequency perturbation processing on the first fundamental frequency sequence to obtain the second fundamental frequency sequence through the following steps: performing the following fundamental frequency perturbation processing on each fundamental frequency in the first fundamental frequency sequence to obtain the second fundamental frequency sequence: randomly sampling the fundamental frequency within the preset sampling range of the fundamental frequency to obtain the perturbation parameter corresponding to the fundamental frequency; and summing the perturbation parameter and the fundamental frequency.

[0177] Optionally, the mapping process includes excitation mapping and embedding mapping. The second processing module 603 performs mapping processing on the second fundamental frequency sequence to obtain the target tensor through the following steps: performing excitation mapping on the second fundamental frequency sequence to obtain the first tensor; performing embedding mapping on the second fundamental frequency sequence to obtain the second tensor; and combining the first tensor and the second tensor to obtain the target tensor.

[0178] Optionally, the second processing module 603 performs embedding mapping on the second fundamental frequency sequence to obtain the second tensor through the following steps: in response to the second fundamental frequency sequence being a floating-point value in a continuous domain, upsampling interpolation is used to perform upsampling interpolation on the second fundamental frequency sequence to obtain the first sampled fundamental frequency sequence; embedding mapping is performed on the first sampled fundamental frequency sequence to obtain the second tensor.

[0179] Optionally, the second processing module 603 performs embedding mapping on the second fundamental frequency sequence to obtain the second tensor through the following steps: in response to the second fundamental frequency sequence being a quantized fundamental frequency category label, the second fundamental frequency sequence is upsampled using nearest neighbor interpolation to obtain the second sampled fundamental frequency sequence; the second sampled fundamental frequency sequence is then embedded and mapped to obtain the second tensor.

[0180] Optionally, the second processing module 603 performs the following steps to combine the first tensor and the second tensor to obtain the target tensor: in response to the different sizes of the first tensor and the second tensor, the first tensor and the second tensor are concatenated to obtain the target tensor; in response to the same size of the first tensor and the second tensor, the first tensor and the second tensor are summed to obtain the target tensor.

[0181] Figure 7 This is a schematic diagram of the structure of an audio synthesis apparatus according to an embodiment of the present disclosure. The audio synthesis apparatus 700 includes:

[0182] Module 701 is used to acquire the target text;

[0183] Extraction module 702 is used to extract the acoustic feature sequence of the target text;

[0184] Prediction module 703 is used to input the acoustic feature sequence into the fundamental frequency prediction model to predict the fundamental frequency and obtain the fundamental frequency sequence corresponding to the acoustic feature sequence;

[0185] The synthesis module 704 is used to input the fundamental frequency sequence and acoustic feature sequence into the vocoder for audio synthesis to obtain the audio corresponding to the target text. The vocoder is trained by the vocoder training method described above.

[0186] Optionally, the fundamental frequency prediction model includes a convolutional layer, a nonlinear layer, a normalization layer, a random deactivation layer, a linear layer, and a classification layer. The prediction module 703 performs the following steps to input the acoustic feature sequence into the fundamental frequency prediction model for fundamental frequency prediction, thereby obtaining the fundamental frequency sequence corresponding to the acoustic feature sequence: inputting the acoustic feature sequence into the convolutional layer for convolution processing to obtain the first feature; inputting the first feature into the nonlinear layer for activation processing to obtain the second feature; inputting the second feature into the normalization layer for normalization processing to obtain the third feature; inputting the third feature into the random deactivation layer for deactivation processing to obtain the fourth feature; inputting the fourth feature into the linear layer for linear processing to obtain the fifth feature; and inputting the fifth feature into the classification layer for classification processing to obtain the fundamental frequency sequence.

[0187] Optionally, the synthesis module 704 performs the following steps to input the fundamental frequency sequence and acoustic feature sequence into the vocoder for audio synthesis to obtain the audio corresponding to the target text: mapping the fundamental frequency sequence to obtain the target tensor; inputting the target tensor and acoustic feature sequence into the vocoder for audio synthesis to obtain the audio.

[0188] Optionally, the mapping process includes excitation mapping and embedding mapping. The synthesis module 704 performs mapping processing on the fundamental frequency sequence to obtain the target tensor through the following steps: performing excitation mapping on the fundamental frequency sequence to obtain a first tensor; performing embedding mapping on the fundamental frequency sequence to obtain a second tensor; and combining the first tensor and the second tensor to obtain the target tensor.

[0189] Optionally, the synthesis module 704 performs embedding mapping on the fundamental frequency sequence to obtain a second tensor through the following steps: in response to the fundamental frequency sequence being a continuous domain floating-point value, linear interpolation is used to upsample the fundamental frequency sequence to obtain a first sampled fundamental frequency sequence; embedding mapping is performed on the first sampled fundamental frequency sequence to obtain a second tensor; in response to the fundamental frequency sequence being a quantized fundamental frequency category label, nearest neighbor interpolation is used to upsample the fundamental frequency sequence to obtain a second sampled fundamental frequency sequence; embedding mapping is performed on the second sampled fundamental frequency sequence to obtain a second tensor.

[0190] Optionally, the synthesis module 704 performs the following steps to combine the first tensor and the second tensor to obtain the target tensor: in response to the different sizes of the first tensor and the second tensor, the first tensor and the second tensor are concatenated to obtain the target tensor; in response to the same size of the first tensor and the second tensor, the first tensor and the second tensor are summed to obtain the target tensor.

[0191] Optionally, it also includes a processing module for training a fundamental frequency prediction model using an audio corpus and acoustic feature sequences.

[0192] Optionally, the target text is the text corresponding to the singing, and the target text includes at least one of the phoneme type of the singing, the start and end positions of the phonemes in the singing, and the pitch of the singing.

[0193] Exemplary computing device

[0194] Having described the methods, media, and apparatus of exemplary embodiments of this disclosure, the following references... Figure 8 A computing device according to an exemplary embodiment of the present disclosure will be described.

[0195] Figure 8 The computing device 80 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0196] like Figure 8 As shown, the computing device 80 is presented in the form of a general-purpose computing device. The components of the computing device 80 may include, but are not limited to: at least one processing unit 801, at least one storage unit 802, and a bus 803 connecting different system components (including the processing unit 801 and the storage unit 802). The at least one storage unit 802 stores computer-executable instructions; the at least one processing unit 801 includes a processor that executes the computer-executable instructions to implement the methods described above.

[0197] The 803 bus includes a data bus, a control bus, and an address bus.

[0198] Storage unit 802 may include readable media in the form of volatile memory, such as random access memory (RAM) 8021 and / or cache memory 8022, and may further include readable media in the form of non-volatile memory, such as read-only memory (ROM) 8023.

[0199] Storage unit 802 may also include a program / utility 8025 having a set (at least one) of program modules 8024, such program modules 8024 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0200] The computing device 80 can also communicate with one or more external devices 804 (e.g., keyboard, pointing device, etc.). This communication can be performed via input / output (I / O) interface 805. Furthermore, the computing device 80 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 806. Figure 8As shown, network adapter 806 communicates with other modules of computing device 80 via bus 803. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with computing device 80, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0201] It should be noted that although several units / modules or sub-units / modules of the vocoder training apparatus and audio synthesis apparatus have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0202] Furthermore, although the operations of the methods disclosed herein are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0203] While the spirit and principles of this disclosure have been described with reference to several specific embodiments, it should be understood that this disclosure is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for convenience of expression. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A vocoder training method, comprising: Obtain the first fundamental frequency sequence of audio from the audio corpus; The following fundamental frequency perturbation process is applied to each fundamental frequency in the first fundamental frequency sequence to obtain the second fundamental frequency sequence: Obtain the first logarithm of the fundamental frequency; The first logarithm is truncated to its maximum or minimum value to obtain the second logarithm; The second logarithm is quantized to obtain the baseband category label corresponding to the baseband frequency; The fundamental frequency category label is dequantized; The second fundamental frequency sequence is mapped to obtain the target tensor; The mapping process includes activation mapping and embedding mapping; The target tensor, the acoustic feature sequence corresponding to the audio, and the audio are trained to obtain a vocoder for audio synthesis.

2. The vocoder training method according to claim 1, wherein mapping the second fundamental frequency sequence to obtain the target tensor includes: The second fundamental frequency sequence is subjected to excitation mapping to obtain the first tensor; The second fundamental frequency sequence is embedded and mapped to obtain the second tensor; The first tensor and the second tensor are combined to obtain the target tensor.

3. The vocoder training method according to claim 2, wherein the embedding mapping of the second fundamental frequency sequence to obtain the second tensor includes: In response to the fact that the second fundamental frequency sequence is a floating-point value in the continuous domain, an upsampling interpolation method is used to perform upsampling interpolation on the second fundamental frequency sequence to obtain a first sampled fundamental frequency sequence; The first sampled fundamental frequency sequence is embedded and mapped to obtain the second tensor.

4. The vocoder training method according to claim 2, wherein the embedding mapping of the second fundamental frequency sequence to obtain the second tensor includes: In response to the fact that the second fundamental frequency sequence is a quantized fundamental frequency category label, the second fundamental frequency sequence is upsampled using nearest neighbor interpolation to obtain the second sampled fundamental frequency sequence; The second sampled fundamental frequency sequence is embedded and mapped to obtain the second tensor.

5. The vocoder training method according to claim 2, wherein combining the first tensor and the second tensor to obtain the target tensor comprises: In response to the different sizes of the first tensor and the second tensor, the first tensor and the second tensor are concatenated to obtain the target tensor; In response to the fact that the first tensor and the second tensor have the same size, the first tensor and the second tensor are summed to obtain the target tensor.

6. An audio synthesis method, comprising: A vocoder is obtained by training the vocoder training method as described in any one of claims 1 to 5; Obtain the target text; Extract the acoustic feature sequence of the target text; The acoustic feature sequence is input into the fundamental frequency prediction model to predict the fundamental frequency, thereby obtaining the fundamental frequency sequence corresponding to the acoustic feature sequence; The fundamental frequency sequence is mapped to obtain the target tensor; the mapping process includes excitation mapping and embedding mapping. The target tensor and the acoustic feature sequence are input into the vocoder for audio synthesis to obtain the audio.

7. The audio synthesis method according to claim 6, wherein the fundamental frequency prediction model comprises a convolutional layer, a nonlinear layer, a normalization layer, a random deactivation layer, a linear layer, and a classification layer, and the step of inputting the acoustic feature sequence into the fundamental frequency prediction model to predict the fundamental frequency, thereby obtaining the fundamental frequency sequence corresponding to the acoustic feature sequence, comprises: The acoustic feature sequence is input into the convolutional layer for convolution processing to obtain the first feature; The first feature is input into the nonlinear layer for activation processing to obtain the second feature; The second feature is input into the standardization layer for standardization processing to obtain the third feature; The third feature is input into the random deactivation layer for deactivation processing to obtain the fourth feature; The fourth feature is input into the linear layer for linear processing to obtain the fifth feature; The fifth feature is input into the classification layer for classification processing to obtain the fundamental frequency sequence.

8. The audio synthesis method according to claim 6, wherein mapping the fundamental frequency sequence to obtain the target tensor comprises: The fundamental frequency sequence is subjected to excitation mapping to obtain the first tensor; The fundamental frequency sequence is embedded and mapped to obtain a second tensor; The first tensor and the second tensor are combined to obtain the target tensor.

9. The audio synthesis method according to claim 8, wherein the embedding mapping of the fundamental frequency sequence to obtain the second tensor comprises: In response to the fact that the fundamental frequency sequence is a floating-point value in the continuous domain, the fundamental frequency sequence is upsampled using linear interpolation to obtain a first sampled fundamental frequency sequence; The first sampled fundamental frequency sequence is embedded and mapped to obtain the second tensor; In response to the fact that the fundamental frequency sequence is a quantized fundamental frequency category label, the fundamental frequency sequence is upsampled using nearest neighbor interpolation to obtain a second sampled fundamental frequency sequence; the second sampled fundamental frequency sequence is then embedded and mapped to obtain the second tensor.

10. The audio synthesis method according to claim 8, wherein combining the first tensor and the second tensor to obtain the target tensor comprises: In response to the different sizes of the first tensor and the second tensor, the first tensor and the second tensor are concatenated to obtain the target tensor; In response to the fact that the first tensor and the second tensor have the same size, the first tensor and the second tensor are summed to obtain the target tensor.

11. The audio synthesis method according to claim 6 or 7, further comprising: The fundamental frequency prediction model is obtained by training with an audio corpus and the acoustic feature sequences.

12. The audio synthesis method according to claim 6 or 7, wherein the target text is the text corresponding to the singing voice, and the target text includes at least one of the phoneme type of the singing voice, the boundary start and end positions of the phonemes in the singing voice, and the pitch of the singing voice.

13. A medium storing computer-executable instructions, which, when executed by a processor, are used to implement the vocoder training method as claimed in any one of claims 1 to 5 or the audio synthesis method as claimed in any one of claims 6 to 12.

14. A vocoder training device, comprising: The acquisition module is used to acquire the first fundamental frequency sequence of audio in the audio corpus; The first processing module is used to perform fundamental frequency perturbation processing on the first fundamental frequency sequence to obtain the second fundamental frequency sequence; The second processing module is used to perform mapping processing on the second fundamental frequency sequence to obtain a target tensor; the mapping processing includes excitation mapping and embedding mapping. The training module is used to train a vocoder for audio synthesis using the target tensor and the acoustic feature sequence of the text corresponding to the audio. The first processing module performs fundamental frequency perturbation processing on the first fundamental frequency sequence through the following steps to obtain the second fundamental frequency sequence: The following fundamental frequency perturbation process is applied to each fundamental frequency in the first fundamental frequency sequence to obtain the second fundamental frequency sequence: Obtain the first logarithm of the fundamental frequency; The first logarithm is truncated to its maximum or minimum value to obtain the second logarithm; The second logarithm is quantized to obtain the baseband category label corresponding to the baseband frequency; The fundamental frequency category label is dequantized.

15. The vocoder training apparatus according to claim 14, wherein the second processing module performs mapping processing on the second fundamental frequency sequence to obtain the target tensor through the following steps: The second fundamental frequency sequence is subjected to excitation mapping to obtain the first tensor; The second fundamental frequency sequence is embedded and mapped to obtain the second tensor; The first tensor and the second tensor are combined to obtain the target tensor.

16. The vocoder training apparatus according to claim 15, wherein the second processing module performs embedding mapping on the second fundamental frequency sequence through the following steps to obtain the second tensor: In response to the fact that the second fundamental frequency sequence is a floating-point value in the continuous domain, an upsampling interpolation method is used to perform upsampling interpolation on the second fundamental frequency sequence to obtain the first sampled fundamental frequency sequence; The first sampled fundamental frequency sequence is embedded and mapped to obtain the second tensor.

17. The vocoder training apparatus according to claim 15, wherein the second processing module performs embedding mapping on the second fundamental frequency sequence through the following steps to obtain a second tensor: In response to the fact that the second fundamental frequency sequence is a quantized fundamental frequency category label, the second fundamental frequency sequence is upsampled using nearest neighbor interpolation to obtain the second sampled fundamental frequency sequence; The second sampled fundamental frequency sequence is embedded and mapped to obtain the second tensor.

18. The vocoder training apparatus according to claim 15, wherein the second processing module performs combined processing on the first tensor and the second tensor through the following steps to obtain the target tensor: In response to the different sizes of the first tensor and the second tensor, the first tensor and the second tensor are concatenated to obtain the target tensor; In response to the fact that the first tensor and the second tensor have the same size, the first tensor and the second tensor are summed to obtain the target tensor.

19. An audio synthesis apparatus, comprising: The acquisition module is configured to train a vocoder using the vocoder training method as described in any one of claims 1 to 5, and to acquire the target text; An extraction module is used to extract the acoustic feature sequence of the target text; The prediction module is used to input the acoustic feature sequence into the fundamental frequency prediction model to predict the fundamental frequency, thereby obtaining the fundamental frequency sequence corresponding to the acoustic feature sequence; A synthesis module is used to perform mapping processing on the fundamental frequency sequence to obtain a target tensor, the mapping processing including excitation mapping and embedding mapping; and to input the target tensor and the acoustic feature sequence into the vocoder for audio synthesis to obtain the audio.

20. The audio synthesis apparatus according to claim 19, wherein the fundamental frequency prediction model comprises a convolutional layer, a nonlinear layer, a normalization layer, a random deactivation layer, a linear layer, and a classification layer, and the prediction module performs fundamental frequency prediction by inputting the acoustic feature sequence into the fundamental frequency prediction model through the following steps to obtain the fundamental frequency sequence corresponding to the acoustic feature sequence: The acoustic feature sequence is input into the convolutional layer for convolution processing to obtain the first feature; The first feature is input into the nonlinear layer for activation processing to obtain the second feature; The second feature is input into the standardization layer for standardization processing to obtain the third feature; The third feature is input into the random deactivation layer for deactivation processing to obtain the fourth feature; The fourth feature is input into the linear layer for linear processing to obtain the fifth feature; The fifth feature is input into the classification layer for classification processing to obtain the fundamental frequency sequence.

21. The audio synthesis apparatus according to claim 19, wherein the synthesis module performs mapping processing on the fundamental frequency sequence to obtain the target tensor through the following steps: The fundamental frequency sequence is subjected to excitation mapping to obtain the first tensor; The fundamental frequency sequence is embedded and mapped to obtain a second tensor; The first tensor and the second tensor are combined to obtain the target tensor.

22. The audio synthesis apparatus according to claim 21, wherein the synthesis module performs an embedding mapping on the fundamental frequency sequence to obtain a second tensor through the following steps: In response to the fact that the fundamental frequency sequence is a floating-point value in a continuous domain, the fundamental frequency sequence is upsampled using linear interpolation to obtain a first sampled fundamental frequency sequence; the first sampled fundamental frequency sequence is then embedded and mapped to obtain the second tensor. In response to the fact that the fundamental frequency sequence is a quantized fundamental frequency category label, the fundamental frequency sequence is upsampled using nearest neighbor interpolation to obtain a second sampled fundamental frequency sequence; the second sampled fundamental frequency sequence is then embedded and mapped to obtain the second tensor.

23. The audio synthesis apparatus according to claim 21, wherein the synthesis module performs the following steps to combine the first tensor and the second tensor to obtain the target tensor: In response to the different sizes of the first tensor and the second tensor, the first tensor and the second tensor are concatenated to obtain the target tensor; In response to the fact that the first tensor and the second tensor have the same size, the first tensor and the second tensor are summed to obtain the target tensor.

24. The audio synthesis apparatus according to claim 19 or 20, further comprising: The processing module is used to train the fundamental frequency prediction model using an audio corpus and the acoustic feature sequence.

25. The audio synthesis apparatus according to claim 19 or 20, wherein the target text is text corresponding to a singing voice, and the target text includes at least one of the phoneme type of the singing voice, the boundary start and end positions of the phonemes in the singing voice, and the pitch of the singing voice.

26. A computing device, comprising: A processor, and a memory connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the vocoder training method as described in any one of claims 1 to 5 or the audio synthesis method as described in any one of claims 6 to 12.

Images