Speech synthesis method and device, computer device and storage medium

Abstract

The application provides a speech synthesis method and device, computer equipment and a storage medium. The method comprises obtaining a target score to be processed, and extracting score features in the target score; determining acoustic features corresponding to the score features according to the score features and a pre-trained acoustic model; generating an initial waveform with a harmonic structure based on fundamental frequency information in the acoustic features; and generating a target audio corresponding to the target score according to the acoustic features and the initial waveform. After the acoustic model is used to obtain acoustic features including a fundamental frequency, energy, a mel-frequency spectrum and the like, an initial waveform with a harmonic structure is generated based on the fundamental frequency information in the acoustic features, which can effectively avoid possible fundamental frequency jitter in the subsequently synthesized audio, guarantee the pitch continuity and pronunciation stability of the target audio generated based on the acoustic features and the initial waveform, and thus improve the speech synthesis effect.

Description

Technical Field

[0001] This application relates to the field of speech synthesis technology, specifically to a speech synthesis generation method, apparatus, computer device, and storage medium (computer-readable storage medium). Background Technology

[0002] Singing synthesis aims to enable machines to sing with emotion, much like humans. It can be seen as an extension of text-to-speech technology into music. Compared to speech synthesis, singing synthesis audio not only needs to reflect the lyrics presented in text, but also the melody and emotion of the song. Therefore, it has broad research and application value in the field of entertainment and interaction, and more and more researchers are beginning to study how to improve the naturalness and sound quality of singing synthesis.

[0003] Current methods for synthesizing vocals mainly consist of two steps: acoustic model processing and vocoder processing. The acoustic model is primarily used to extract acoustic features for synthesizing audio from musical scores containing lyrics in text form and symbolic notes. These features include fundamental frequency, energy, Mel spectrum, or other latent variables. The vocoder then converts these acoustic features into audio waveforms. However, vocals synthesized using this method still suffer from problems such as discontinuous pitch and unstable pronunciation, affecting the overall quality of speech synthesis. Summary of the Invention

[0004] Therefore, it is necessary to provide a speech synthesis method, apparatus, computer equipment, and storage medium to address the aforementioned technical problems, thereby resolving the issues of discontinuous pitch perception and unstable pronunciation that still exist in existing singing voice synthesis technologies.

[0005] Firstly, this application provides a speech synthesis method, including:

[0006] Obtain the target musical score to be processed, and extract the musical score features from the target musical score;

[0007] Based on the musical score features and the pre-trained acoustic model, the acoustic features corresponding to the musical score features are determined;

[0008] An initial waveform with a harmonic structure is generated based on the fundamental frequency information in the acoustic features.

[0009] The target audio corresponding to the target musical score is generated based on the acoustic features and the initial waveform.

[0010] As a feasible embodiment of this application, the step of generating an initial waveform with a harmonic structure based on the fundamental frequency information in the acoustic features includes:

[0011] The fundamental frequency information is upsampled, and the corresponding fundamental frequency signal at the waveform point level is output;

[0012] A sinusoidal excitation signal is generated based on the fundamental frequency signal and the multiple harmonics corresponding to the fundamental frequency signal.

[0013] The sinusoidal excitation signal is linearly transformed to obtain an initial waveform with a harmonic structure.

[0014] As a feasible embodiment of this application, the linear transformation of the sinusoidal excitation signal to obtain an initial waveform with a harmonic structure includes:

[0015] Obtain a post-processing network model pre-trained based on the training sample sheet music and training sample audio;

[0016] The sinusoidal excitation signal is input into the post-processing network model to obtain an initial waveform with a harmonic structure.

[0017] As a feasible embodiment of this application, the step of generating the target audio corresponding to the target musical score based on the acoustic features and the initial waveform includes:

[0018] The acoustic features and the initial waveform are sampled and processed respectively to obtain a first feature sequence and a second feature sequence with the same resolution;

[0019] The first feature sequence and the second feature sequence are added together to obtain the fused feature sequence;

[0020] The target audio corresponding to the target musical score is generated based on the fusion feature sequence.

[0021] As a feasible embodiment of this application, the step of generating the target audio corresponding to the target musical score based on the fused feature sequence includes:

[0022] Obtain a vocoder that has been pre-trained based on the musical scores and audio samples of the training samples;

[0023] The fused feature sequence is input into the vocoder, and the target audio corresponding to the target musical score is output.

[0024] As a possible embodiment of this application, the acoustic model includes a vibrato predictor;

[0025] The step of determining the acoustic features corresponding to the musical score features based on the musical score features and the pre-trained acoustic model includes:

[0026] Determine the fundamental frequency information corresponding to the musical score features based on the musical score features;

[0027] The logarithmic fundamental frequency corresponding to the fundamental frequency information is input into the vibrato predictor, and the vibrato distribution feature corresponding to the logarithmic fundamental frequency is output.

[0028] The trill distribution characteristics are determined as the acoustic characteristics corresponding to the musical score characteristics.

[0029] As a feasible embodiment of this application, the vibrato predictor is a generative model based on a flow model.

[0030] As a feasible embodiment of this application, before determining the acoustic features corresponding to the musical score features based on the musical score features and the pre-trained acoustic model, the method further includes:

[0031] Obtain the training sample sheet music and the corresponding training sample audio, and extract the training sheet music features and the training acoustic features corresponding to the training sample audio.

[0032] The trained musical score features are input into a preset initial acoustic model, and the predicted acoustic features are output.

[0033] Based on the difference between the predicted acoustic features and the trained acoustic features, and the number of training iterations corresponding to the initial acoustic model, the parameters in the initial acoustic model are updated to obtain the updated acoustic model; the parameters in the initial acoustic model include at least the residual connection coefficients in the prior network model.

[0034] The current acoustic model is determined as the pre-trained acoustic model only when the predicted acoustic features obtained by inputting the training musical score features into the updated acoustic model meet the preset requirements.

[0035] As a feasible embodiment of this application, the extraction of musical score features from the target musical score includes:

[0036] Based on the text lyrics information in the target musical score, generate text encoding features;

[0037] The musical score features in the target score are determined based on the note durations corresponding to the note information in the target score and the text encoding features.

[0038] Secondly, this application provides a speech synthesis device, comprising:

[0039] An extraction module is used to acquire the target musical score to be processed and extract the musical score features from the target musical score;

[0040] The determination module is used to determine the acoustic features corresponding to the musical score features based on the musical score features and the pre-trained acoustic model.

[0041] The first generation module is used to generate an initial waveform with a harmonic structure based on the fundamental frequency information in the acoustic features;

[0042] The second generation module is used to generate the target audio corresponding to the target musical score based on the acoustic features and the initial waveform.

[0043] Thirdly, this application also provides a computer device, characterized in that the computer device comprises:

[0044] One or more processors;

[0045] Memory; and

[0046] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the speech synthesis method provided by any of the above.

[0047] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to perform the steps of the speech synthesis method provided in any of the above claims.

[0048] Fifthly, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the speech synthesis method provided above.

[0049] The speech synthesis method provided in this application extracts the spectral features of the target musical score and obtains acoustic features including fundamental frequency, energy, and Mel spectrum using an acoustic model. Then, it generates an initial waveform with a harmonic structure based on the fundamental frequency information in the acoustic features. This effectively avoids fundamental frequency jitter that may exist in the subsequent synthesized audio, and ensures the pitch continuity and pronunciation stability of the target audio generated based on the acoustic features and the initial waveform, thereby improving the effect of speech synthesis. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 A schematic diagram illustrating an end-to-end model-based speech synthesis method for existing technologies;

[0052] Figure 2 This is a flowchart illustrating the steps of a speech synthesis method provided in an embodiment of this application.

[0053] Figure 3 This application provides a schematic flowchart illustrating the steps for generating an initial waveform with a harmonic structure based on fundamental frequency information in an embodiment of the present application.

[0054] Figure 3 (a) A schematic diagram of an excitation module provided in an embodiment of this application;

[0055] Figure 4 This application provides a schematic flowchart of a step for linearly transforming a sinusoidal excitation signal, as illustrated in an embodiment of the present application.

[0056] Figure 5 This application provides a schematic flowchart illustrating the steps for generating target audio by fusing acoustic features and an initial waveform, as illustrated in an embodiment of the present application.

[0057] Figure 6 A schematic diagram of the steps for extracting vibrato distribution features as acoustic features for audio synthesis, as provided in an embodiment of this application;

[0058] Figure 6 (a) is a schematic diagram of a vibrato predictor provided in an embodiment of this application;

[0059] Figure 7 This application provides a schematic flowchart illustrating the steps involved in training an acoustic model based on weighted residual connections.

[0060] Figure 7 (a) A schematic diagram of the structure of a priori network model provided in an embodiment of this application;

[0061] Figure 8 This application provides a schematic diagram of a process for extracting musical score features based on text and musical note dimensions in an embodiment of the present application.

[0062] Figure 9 A schematic diagram illustrating speech synthesis based on an end-to-end model, provided as an embodiment of this application;

[0063] Figure 10 This is a schematic diagram of the structure of a speech synthesis device provided in an embodiment of this application;

[0064] Figure 11 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0065] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0066] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0067] In the description of this application, the word "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0068] To facilitate understanding of the complete implementation scheme of the speech synthesis method provided in the embodiments of this application, the implementation scenarios of the speech synthesis method are first described in detail below.

[0069] The speech synthesis method provided in this application is mainly used in the field of singing voice synthesis. Compared with conventional speech synthesis technology, singing voice synthesis not only needs to reflect the lyrics presented as text, but also the melody and emotion of the song. Currently, mainstream singing voice synthesis is based on an end-to-end model, mainly including two models: an acoustic model and a vocoder. The former (acoustic model) extracts acoustic features related to acoustics, such as fundamental frequency, energy, Mel spectrum, or other high-dimensional latent variables, from the musical score containing lyrics in text form and symbolic notes. The latter (vocoder) is mainly used to convert the aforementioned extracted acoustic features into audio waveforms, thereby outputting the audio corresponding to the musical score. For ease of understanding, Figure 1 A schematic diagram of speech synthesis based on an end-to-end model is provided for existing technologies, and is described in detail below.

[0070] like Figure 1 As shown, end-to-end model-based speech synthesis methods typically involve model training and inference. Specifically, the acoustic model and vocoder contain several parameters for acoustic feature extraction and audio conversion. Model training involves pre-training the model parameters using corresponding training samples, namely, training sample scores and their corresponding training audio. Specifically, the waveform y in the training audio is processed by a linear spectrum extractor and a posterior encoder network to obtain coded training acoustic features. The training sample score c, after being processed by a text encoder to obtain text-coded features related to the lyrics, is combined with the note information and corresponding duration d from the score c. N The initial acoustic features are further generated, and the (logarithmic) fundamental frequency and (logarithmic) energy corresponding to the musical score are predicted based on the pitch predictor and energy predictor, respectively, and added to the initial acoustic features. Finally, predicted acoustic features are obtained based on a prior network with residual structure and a linear transform layer. On this basis, the differences between the trained acoustic features and the predicted acoustic features are compared, and the parameters in the acoustic model and vocoder are updated using the idea of ​​backpropagation until training is complete. The training stage of the neural network model is not described in detail in this embodiment. Furthermore, after training, based on the existing acoustic model and vocoder, the musical score information to be processed is input into the acoustic model to obtain acoustic features in coded form. These acoustic features are then input into the vocoder to output the target waveform corresponding to the musical score, thereby generating the target audio.

[0071] However, during the above implementation process, it was found that the lack of explicit constraint on the fundamental frequency in the audio leads to problems such as discontinuous pitch and unstable pronunciation in the final synthesized audio. To address this issue, this application provides a speech synthesis method that further processes the fundamental frequency in the acoustic features within the vocoder to avoid fundamental frequency jitter in the synthesized audio, thereby improving the speech synthesis effect. This speech synthesis method is installed as a computer program in a speech synthesis device, which is deployed as a processor in a computer device. The speech synthesis device in the computer device executes the computer program corresponding to the speech synthesis method, thereby executing the speech synthesis method provided in this application. Specifically, as... Figure 2 As shown, Figure 2 This application provides a flowchart illustrating the steps of a speech synthesis method, specifically including steps S210 to S240:

[0072] S210, Obtain the target musical score to be processed, and extract the musical score features from the target musical score.

[0073] In this embodiment of the application, the target musical score to be processed usually refers to the musical score information that needs to be synthesized into a song to be converted into corresponding audio. The target musical score usually includes lyrics in text form and musical notes in symbol form.

[0074] In this embodiment, the musical score features extracted from the target musical score mainly involve features related to phoneme categories, note pitch, note duration, phoneme duration, and text information related to lyrics. This embodiment will not elaborate on the specific implementation scheme for extracting the musical score features. Details will follow later. Figure 8 As shown, Figure 8 This paper presents an implementation scheme for extracting musical score features based on two dimensions: text and symbols.

[0075] S220, Based on the musical score features and the pre-trained acoustic model, determine the acoustic features corresponding to the musical score features.

[0076] In this embodiment of the application, in conjunction with the foregoing Figure 1 The diagram shown illustrates speech synthesis based on an end-to-end model. The acoustic model here can be similar to the one described above. Figure 1 Similar to the acoustic model shown, the musical features extracted from the musical score are input into the trained acoustic model to obtain the acoustic features corresponding to the musical score. Specifically, the acoustic features here include, but are not limited to, fundamental frequency, energy, Mel spectrum, or other acoustically related high-dimensional latent variables, such as non-periodic harmonic components, voiced and unvoiced tones, etc., which will not be elaborated further in the embodiments of this application.

[0077] To further improve the speech synthesis effect achievable by the singing voice synthesis method provided in this application embodiment, in addition to the aforementioned acoustic features such as fundamental frequency and energy, as another optional embodiment of this application, the acoustic features may also include vibrato information used to simulate the actual singing process of a singer. Specifically, vibrato information is a distributed sound signal that can be superimposed on pitch, which can be trained using vibrato information in training sample audio during the training process. That is to say, it can be obtained by... Figure 1 The provided acoustic model includes an additional vibrato predictor to predict vibrato information corresponding to the musical score, which is then used as an acoustic feature for audio conversion in the subsequent vocoder. For details on the implementation scheme, please refer to the subsequent sections. Figure 6 And its explanations and descriptions.

[0078] Furthermore, considering that the models used in the aforementioned end-to-end speech synthesis methods are all complex network models with many parameters, their iterative update efficiency during training is relatively low. Therefore, to further improve the training efficiency of the model and avoid gradient vanishing / exploding anomalies during model training, as an optional embodiment of this application, a scheme is proposed to introduce weighted residual connections into the prior network model of the acoustic model to accelerate model convergence. Specific implementation details can be found in subsequent sections. Figure 7 And its explanations and descriptions.

[0079] S230, an initial waveform with a harmonic structure is generated based on the fundamental frequency information in the acoustic features.

[0080] In this embodiment, to improve the stability of the fundamental frequency in the subsequently generated target audio, the speech synthesis device first extracts the fundamental frequency information from the acoustic features separately. The fundamental frequency is the reciprocal of the fundamental period, corresponding to the frequency of vocal cord vibration, and represents the pitch of the sound. The faster the vocal cords vibrate, the higher the fundamental frequency. Specifically, in conjunction with the foregoing... Figure 1 As shown in the diagram, fundamental frequency information can be obtained based on the pitch predictor in the acoustic model. This is because human perception of fundamental frequency follows the logarithmic law; for example, the difference between 100Hz and 200Hz perceived by humans is the same as the difference between 200Hz and 400Hz. Therefore, pitch can usually be represented by the logarithm of the fundamental frequency. In other words, the pitch obtained based on the pitch predictor (i.e., the logarithmic fundamental frequency) can be regarded as fundamental frequency information related to acoustic characteristics and the fundamental frequency.

[0081] Building upon the foregoing, by processing the fundamental frequency information in the acoustic features, an initial waveform with a harmonic structure can be obtained. This initial waveform will serve as the foundation waveform for further generation of the target audio in subsequent steps. For a detailed implementation scheme of generating the initial waveform with a harmonic structure based on the fundamental frequency information, please refer to the following sections. Figure 3 And its explanations and descriptions.

[0082] Furthermore, it should be noted that the step of generating an initial waveform with a harmonic structure based on fundamental frequency information in acoustic features provided in the embodiments of this application can be deployed in the vocoder in a modular implementation manner using a network model. For example, by fusing an excitation module in the vocoder and processing the aforementioned fundamental frequency information based on the excitation module, an initial waveform for vocoder processing can be generated. A schematic diagram of this excitation module will be provided later.

[0083] S240, Generate the target audio corresponding to the target musical score based on the acoustic features and the initial waveform.

[0084] In this embodiment of the application, after generating an initial waveform with a harmonic structure based on the fundamental frequency information in the acoustic features, the initial waveform is used as the basis to effectively avoid the fundamental frequency jitter abnormality in the subsequently generated audio. Thus, by integrating the other acoustic features extracted from the musical score, a target audio with continuous pitch and stable fundamental frequency can be generated.

[0085] Specifically, in the process of generating target audio based on the initial waveform and acoustic features, considering the different dimensional information of the initial waveform and acoustic features, it is necessary to downsample both the acoustic features and the initial waveform during the fusion process. For details on the implementation scheme, please refer to the subsequent sections. Figure 5 And its explanations and descriptions.

[0086] The speech synthesis method provided in this application extracts the spectral features of the target musical score and obtains acoustic features including fundamental frequency, energy, and Mel spectrum using an acoustic model. Then, it generates an initial waveform with a harmonic structure based on the fundamental frequency information in the acoustic features. This effectively avoids fundamental frequency jitter that may exist in the subsequent synthesized audio, and ensures the pitch continuity and pronunciation stability of the target audio generated based on the acoustic features and the initial waveform, thereby improving the effect of speech synthesis.

[0087] like Figure 3 As shown, Figure 3 This application provides a flowchart illustrating the steps for generating an initial waveform with a harmonic structure based on fundamental frequency information, specifically including steps S310 to S330:

[0088] S310, the base frequency information is upsampled and the corresponding waveform point level base frequency signal is output.

[0089] In this embodiment of the application, as can be seen from the foregoing description, the fundamental frequency information, that is, the fundamental frequency, exists in the form of a serialized numerical value. Therefore, before generating the initial waveform with a harmonic structure, the speech synthesis device will upsample the fundamental frequency to obtain the fundamental frequency signal at the waveform point level, that is, the discretized fundamental frequency signal.

[0090] S320, generate a sinusoidal excitation signal based on the fundamental frequency signal and the multiple harmonics corresponding to the fundamental frequency signal.

[0091] In this embodiment of the application, the speech synthesis device generates a sinusoidal excitation signal based on the multiple harmonics corresponding to the base frequency signal.

[0092] Of course, it should be noted that, as another optional embodiment of this application, a normally distributed noise signal may be further superimposed on the sinusoidal excitation signal, thereby forming a voiced segment that is a superposition of a sine wave and noise, and a voiceless segment that is an excitation signal of noise.

[0093] Specifically, using the base frequency f 1:T For example, suppose the instantaneous frequency of the t-th frame is... For unvoiced frames f t =0, at this time the sinusoidal excitation signal is These correspond to harmonics from 0 to H, respectively. At this point, the excitation signal corresponding to the h-th harmonic... Satisfy the following formula:

[0094]

[0095] in, That is, n t The noise is Gaussian noise following a normal distribution with standard deviation σ, and φ is a randomly generated initial phase, where φ∈[-π,π], f s The sampling rate of the waveform.

[0096] S330, the sinusoidal excitation signal is linearly transformed to obtain an initial waveform with a harmonic structure.

[0097] In this embodiment of the application, after obtaining the sinusoidal excitation signal corresponding to the fundamental frequency and H harmonic frequencies, the sinusoidal excitation signal is further linearly transformed to obtain the initial waveform with harmonic structure.

[0098] Specifically, to facilitate understanding of the implementation scheme for generating an initial waveform with a harmonic structure based on fundamental frequency information provided in the above embodiments, the following... Figure 3 (a) A further schematic diagram is shown, specifically, as implemented by the excitation module. Figure 3 As shown in (a), this is a schematic diagram of the structure of an excitation module provided in an embodiment of this application. The fundamental frequency information, that is, the fundamental frequency signal at the corresponding waveform point level, is first generated into a sinusoidal excitation signal by a sine generator. Then, after introducing a normally distributed noise signal, the voiced segment is generated based on multiple harmonics. The voiceless segment is the superposition of the sine and the noise, and the unvoiced segment is the excitation signal of the noise. After being linearly transformed by a subsequent linear transformer, the linearly transformed excitation signal is obtained, that is, the initial waveform with a harmonic structure, so that the target audio can be generated based on the segment of the signal.

[0099] Furthermore, as an optional embodiment of this application, the linear transformation of the sinusoidal excitation signal can be implemented based on the post-processing network model trained in the vocoder. Specific implementation schemes can be found in the subsequent sections. Figure 4 And its explanations and descriptions.

[0100] like Figure 4 As shown, Figure 4 This application provides a flowchart illustrating the steps for linearly transforming a sinusoidal excitation signal, specifically including steps S410 to S420:

[0101] S410, Obtain the post-processing network model pre-trained based on the training sample score and the training sample audio.

[0102] In this embodiment, the post-processing network model can be a neural network model, such as, but not limited to, deep neural network models, recurrent neural network models, convolutional neural network models, etc. The model parameters in the post-processing network model are pre-trained based on training samples, namely, training sample musical scores and training sample audio, which will not be elaborated further in this embodiment.

[0103] S420, the sinusoidal excitation signal is input to the post-processing network model to obtain an initial waveform with a harmonic structure.

[0104] In this embodiment of the application, the sinusoidal excitation signal obtained above is input into the trained post-processing network model for processing, thereby generating an initial waveform with a harmonic structure.

[0105] like Figure 5 As shown, Figure 5 This application provides a flowchart illustrating the steps for generating target audio by fusing acoustic features and an initial waveform, specifically including steps S510 to S530:

[0106] S510, the acoustic features and the initial waveform are sampled and processed respectively to obtain a first feature sequence and a second feature sequence with the same resolution.

[0107] In this embodiment, to predict the final synthesized audio, the speech synthesis device needs to fuse acoustic features and an initial waveform obtained based on fundamental frequency information. However, considering the differences in feature dimensions between acoustic features and the initial waveform—for example, the feature dimension, or resolution, of acoustic features is typically at the frame level (sequence length is typically N / 256), while the initial waveform, being obtained based on fundamental frequency signal processing at the corresponding waveform point level, typically has a feature dimension at the waveform point level (sequence length is N)—the speech synthesis device first performs downsampling processing on the acoustic features and the initial waveform to different degrees to obtain the corresponding feature dimensions, namely, the first feature sequence and the second feature sequence with the same resolution.

[0108] Specifically, as a feasible embodiment of this application, by processing the frame-level acoustic features sequentially according to an upsampling factor of [8,8,2,2], a first feature sequence of length [ / 32,N / 4,N / 2,N] can be obtained. Similarly, by processing the initial waveform at the waveform point level according to a downsampling factor of [1 / 32,1 / 4,1 / 2], a second feature sequence of length [N / 32,N / 4,N / 2,N] can be obtained. It can be seen that, at this time, the first feature sequence and the second feature sequence are two sets of feature sequences with the same corresponding resolution.

[0109] S520, the first feature sequence and the second feature sequence are added together to obtain the fused feature sequence.

[0110] In this embodiment of the application, the fused feature sequence can be obtained by adding the sequences with the same resolution in the first feature sequence and the second feature sequence. It can be seen that the length of the fused feature sequence itself is the same as that of the first feature sequence and the second feature sequence, which is [N / 32, N / 4, N / 2, N].

[0111] S530, Based on the fusion feature sequence, generate the target audio corresponding to the target musical score.

[0112] In this embodiment, after obtaining the fused feature sequence, the target audio corresponding to the target musical score can be generated by converting the sequence features to audio. Typically, this step can be performed by the aforementioned... Figure 1 The vocoder part shown is used to implement this, meaning that the vocoder can be pre-trained based on the training sample musical score and the training sample audio. Then, the fusion feature sequence is input into the trained vocoder to output the final synthesized target audio. The specific implementation scheme will not be described in detail in this embodiment of the application.

[0113] like Figure 6 As shown, Figure 6 This is a flowchart illustrating the steps for extracting vibrato distribution features as acoustic features for audio synthesis, as provided in an embodiment of this application. Specifically, it includes steps S610 to S630:

[0114] S610, determine the fundamental frequency information corresponding to the musical score feature based on the musical score feature.

[0115] In this embodiment, considering that vibrato information is usually a distributed signal superimposed on pitch, it is necessary to determine the pitch corresponding to the musical score features before determining the vibrato distribution characteristics. As described above, pitch is usually in the logarithmic form of the fundamental frequency, i.e., the logarithmic fundamental frequency. Therefore, the fundamental frequency information corresponding to the musical score features can be determined based on the musical score features. Specifically, in conjunction with the aforementioned... Figure 1As shown in the schematic diagram, the fundamental frequency information can be obtained by a pitch predictor based on an acoustic model, which will not be described in detail in the embodiments of this application.

[0116] S620, the logarithmic fundamental frequency corresponding to the fundamental frequency information is input to the vibrato predictor, and the vibrato distribution feature corresponding to the logarithmic fundamental frequency is output.

[0117] In this embodiment, to extract vibrato features, the aforementioned acoustic model may further include a vibrato predictor trained pre-trained using vibrato samples. Specifically, considering the quasi-periodicity and stable amplitude of vibrato, vibrato features can be viewed as a distribution. Therefore, the vibrato predictor uses a flow-based generative model (Flow) and is trained according to the maximum likelihood criterion. To address the low-dimensional bottleneck in the flow model, in addition to using vibrato as input, variational data augmentation is employed to introduce random variables to form a high-dimensional representation. Specifically, for ease of understanding, as follows... Figure 6 As shown in (a). Figure 6 (a) is a schematic diagram of a vibrato predictor provided in an embodiment of this application. During the training phase, after extracting the logarithmic fundamental frequency from the training sample audio, vibrato information can be extracted from the training sample audio through low-pass filtering. Then, the pitch information corresponding to the training sample musical score is fused together. After handling normally distributed noise, a Flow model can be trained, which exists in the form of an invertible transformation function f and can map the input vibrato x to random variables ∈. That is, the Flow model can establish the input vibrato distribution logp(x; θ) and the Gaussian distribution logp according to the variable transformation theorem. ∈ The relationship between parentheses () is as follows:

[0118]

[0119] The last term is the Jacobian determinant of the invertible transformation function f.

[0120] After training the generative model Flow in the aforementioned manner, during the inference process, when it is necessary to output the vibrato distribution features, it is only necessary to input the logarithmic fundamental frequency into the vibrato predictor, that is, first calculate the vibrato based on the Gaussian distribution p. ∈ After obtaining the random variable ∈ by sampling the logarithmic fundamental frequency, the vibrato distribution feature x is obtained by the inverse operation of the invertible transformation function f.

[0121] S630, the vibrato distribution feature is determined as the acoustic feature corresponding to the musical score feature.

[0122] In this embodiment of the application, the vibrato distribution feature is a sound signal that can be superimposed on pitch. Therefore, the vibrato distribution feature can also be used as an acoustic feature corresponding to the musical score feature, so that in the subsequent process of generating audio through a vocoder, the vibrato effect can also be superimposed on the generated target audio based on the vibrato distribution feature, thereby reducing the mechanical feel of the synthesized singing voice.

[0123] like Figure 7 As shown, Figure 7 This application provides a flowchart illustrating the steps for training an acoustic model based on weighted residual connections, specifically including steps S710 to S740:

[0124] S710, acquire the training sample sheet music and the training sample audio corresponding to the training sample sheet music, and extract the training sheet music features corresponding to the training sample sheet music and the training acoustic features corresponding to the training sample audio.

[0125] In this embodiment, the training sample musical scores and training sample audios refer to a set of training samples that have a pre-defined correspondence and are manually labeled. The implementation schemes for extracting the training musical score features corresponding to the training sample musical scores and the training acoustic features corresponding to the training sample audios can be found in the foregoing. Figure 1 The relevant explanations and descriptions are not repeated here in the embodiments of this application.

[0126] S720, the trained musical score features are input into a preset initial acoustic model, and the predicted acoustic features are output.

[0127] In this embodiment, after constructing the initial acoustic model, inputting the training musical score features into the preset initial acoustic model will output the corresponding predicted acoustic features. It should be noted that since the parameters in the initial acoustic model are usually randomly initialized, the output predicted acoustic features usually differ significantly from the actual acoustic features corresponding to the training sample audio, i.e., the training acoustic features.

[0128] S730, based on the difference between the predicted acoustic features and the trained acoustic features, and the number of training iterations corresponding to the initial acoustic model, the parameters in the initial acoustic model are updated to obtain the updated acoustic model.

[0129] Based on the aforementioned determination of the differences between the predicted acoustic features and the training acoustic features, and using the concept of backpropagation, the parameters in the initial acoustic model are updated to obtain the updated acoustic model. Specifically, as described above... Figure 1Taking the provided diagram as an example, acoustic models typically include frame-level prior network models, and therefore, the model parameters in the prior network model are updated synchronously. However, as a feasible embodiment of this application, in order to improve model training speed and reduce the problem of unsatisfactory model training results caused by gradient vanishing / exploding, residual connection coefficients can also be introduced into the prior network model. Specifically, such as... Figure 7 As shown in (a), this is a schematic diagram of a priori network model provided in an embodiment of this application. The priori network model is composed of several residual structures. The output y of each residual structure is determined by its input x and the residual connection F(x), i.e., y = +αF(x), where α is the weight of the residual connection F(x), or the residual connection coefficient. During training, as the number of training iterations increases, the weight α of the residual connection F(x) gradually increases. That is, α is initially all zero and is dynamically adjusted to a suitable value during subsequent training. This allows for the use of deeper residual networks and also facilitates signal propagation within the priori network model, making the training of the priori network model more stable.

[0130] Of course, it should be noted that the embodiments in this application are only illustrated using the training process of a single acoustic model as an example. In fact, during the actual training process of the parameters in the acoustic model, considering that the acoustic model is usually trained together with the vocoder, the loss values ​​required by the speech synthesis device can include many more types, such as, but not limited to, the following loss values:

[0131] 1) The mean absolute error L between the Mel spectrum of the predicted audio output by the vocoder and the Mel spectrum extracted from the real audio. mel ;

[0132] 2) The root mean square error L between the logarithmic duration of the phonemes predicted by the duration predictor and the actual logarithmic duration of the phonemes dur ;

[0133] 3) The root mean square error L between the fundamental frequency predicted by the energy predictor and the energy extracted from the actual audio signal. energy ;

[0134] 4) The root mean square error L between the logarithmic fundamental frequency predicted by the pitch predictor and the logarithmic fundamental frequency extracted from the actual audio. LF0 ;

[0135] 5) The binary cross-entropy loss L between the voiced and unvoiced tones predicted by the pitch predictor and the voiced and unvoiced tones extracted from the actual audio. UV ;

[0136] 6) A vibrato predictor based on a flow model, with the negative log-likelihood loss L for the true vibrato. vib ;

[0137] 7) The KL (Kullback-Leibler) loss between the prior and posterior distributions. KL ;

[0138] 8) The loss L corresponding to the generated audio waveform being identified as true in a generative adversarial network-based discriminator. adv () and feature matching loss L fm (G);

[0139] The updating of model parameters in the acoustic model and vocoder can be achieved by using a weighted combination of any of the aforementioned loss values. In this embodiment of the application, the steps for updating model parameters in the neural network model will not be described in detail.

[0140] S740, until the predicted acoustic features obtained by inputting the training musical score features into the updated acoustic model meet the preset requirements with the training acoustic features, the current acoustic model is determined as the pre-trained acoustic model.

[0141] In this embodiment of the application, after several iterations of updating the acoustic model, if the predicted acoustic features obtained by inputting the training musical score features into the updated acoustic model meet the preset requirements, such as the difference between the predicted acoustic features and the training acoustic features being consistently less than a preset threshold for several consecutive times, then the training can be considered to have ended. At this time, the acoustic model obtained is the pre-trained acoustic model, which can be used to extract acoustic features from the musical score.

[0142] like Figure 8 As shown, Figure 8 This application provides a flowchart illustrating the steps for extracting musical score features based on text and musical note dimensions, specifically including steps S810 to S820:

[0143] S810, Based on the text lyrics information in the target musical score, generate text encoding features.

[0144] In this embodiment of the application, by inputting the text lyrics information in the target musical score into a trained text encoder, text encoding features for describing the lyrics in the musical score can be generated.

[0145] S820, determine the musical score features in the target musical score based on the note duration corresponding to the note information in the target musical score and the text encoding features.

[0146] In this embodiment of the application, by extracting the note information from the target musical score and combining the note duration corresponding to each note information with the text encoding features, a musical score feature existing in the form of an encoding can be generated. This musical score feature simultaneously includes lyric text features and note features.

[0147] To facilitate understanding of the differences between the speech synthesis method provided in this application and existing speech synthesis methods, such as... Figure 9 As shown, Figure 9 This application provides a schematic diagram of speech synthesis based on an end-to-end model, compared to... Figure 1 The diagram illustrating speech synthesis based on an end-to-end model shown has at least the following differences:

[0148] 1) By adding an excitation module to the vocoder, after the fundamental frequency information is extracted through the acoustic model, the excitation module processes this fundamental frequency information and normally distributed noise information to generate an excitation signal with an initial waveform containing harmonic structures. This effectively avoids fundamental frequency jitter that may occur in the synthesized audio and improves the pitch continuity and pronunciation stability of the synthesized audio. The implementation process of the excitation module for processing fundamental frequency information can be found in the aforementioned... Figure 3 , Figure 4 and its explanations;

[0149] 2) By adding a vibrato predictor to the acoustic model to achieve vibrato prediction, it is possible to predict the vibrato distribution features that can be superimposed on the pitch after obtaining the musical score information, thereby ensuring that the generated audio can show vibrato effect. On the basis of ensuring pitch continuity and pronunciation stability, by adding vibrato features, the mechanical feel of the synthesized audio can be effectively reduced.

[0150] 3) By introducing weighted residual connection coefficients into the prior network model in the acoustic model, the aforementioned problems of low model training efficiency and gradient vanishing / exploding caused by the addition of new model structures can be further solved. While ensuring model training efficiency, deeper network models can be used to improve model training effect and training stability.

[0151] To better implement the speech synthesis method provided in the embodiments of this application, based on the speech synthesis method provided in the embodiments of this application, this application also provides a speech synthesis device, such as... Figure 10 As shown, the speech synthesis device 1000 includes:

[0152] Extraction module 1010 is used to acquire the target musical score to be processed and extract the musical score features from the target musical score;

[0153] The determining module 1020 is used to determine the acoustic features corresponding to the musical score features based on the musical score features and the pre-trained acoustic model.

[0154] The first generation module 1030 is used to generate an initial waveform with a harmonic structure based on the fundamental frequency information in the acoustic features;

[0155] The second generation module 1040 is used to generate the target audio corresponding to the target musical score based on the acoustic features and the initial waveform.

[0156] In some embodiments of this application, the first generation module 1030 is further configured to upsample the fundamental frequency information and output a fundamental frequency signal at the corresponding waveform point level; generate a sinusoidal excitation signal based on the fundamental frequency signal and the multiple harmonics corresponding to the fundamental frequency signal; and perform a linear transformation on the sinusoidal excitation signal to obtain an initial waveform with a harmonic structure.

[0157] In some embodiments of this application, the first generation module 1030 is further configured to obtain a post-processing network model pre-trained based on training sample scores and training sample audio; and input the sinusoidal excitation signal into the post-processing network model to obtain an initial waveform with a harmonic structure.

[0158] In some embodiments of this application, the second generation module 1040 is further configured to sample the acoustic features and the initial waveform respectively to obtain a first feature sequence and a second feature sequence with the same resolution; add the first feature sequence and the second feature sequence to obtain a fused feature sequence; and generate the target audio corresponding to the target musical score based on the fused feature sequence.

[0159] In some embodiments of this application, the second generation module 1040 is further configured to obtain a vocoder pre-trained based on training sample scores and training sample audio; input the fused feature sequence into the vocoder; and output the target audio corresponding to the target score.

[0160] In some embodiments of this application, the determining module 1020 is further configured to determine the fundamental frequency information corresponding to the musical score feature based on the musical score feature; input the logarithmic fundamental frequency corresponding to the fundamental frequency information to the vibrato predictor, and output the vibrato distribution feature corresponding to the logarithmic fundamental frequency; and determine the vibrato distribution feature as the acoustic feature corresponding to the musical score feature.

[0161] In some embodiments of this application, before determining the acoustic features corresponding to the musical score features based on the musical score features and the pre-trained acoustic model, the determining module 1020 is further configured to acquire training sample musical scores and training sample audio corresponding to the training sample musical scores, and extract the training musical score features corresponding to the training sample musical scores and the training acoustic features corresponding to the training sample audio; input the training musical score features into a preset initial acoustic model, and output predicted acoustic features; update the parameters in the initial acoustic model according to the difference between the predicted acoustic features and the training acoustic features, and the number of training iterations corresponding to the initial acoustic model, to obtain an updated acoustic model; the parameters in the initial acoustic model include at least the residual connection coefficients in the prior network model; until the predicted acoustic features obtained by inputting the training musical score features into the updated acoustic model and the training acoustic features meet preset requirements, the current acoustic model is determined as the pre-trained acoustic model.

[0162] In some embodiments of this application, the extraction module 1010 is further configured to generate text encoding features based on the text lyrics information in the target musical score; and to determine the musical score features in the target musical score based on the note durations corresponding to the note information in the target musical score and the text encoding features.

[0163] For specific limitations regarding the speech synthesis device, please refer to the limitations on the speech synthesis method above, which will not be repeated here. Each module in the aforementioned speech synthesis device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.

[0164] In some embodiments of this application, the speech synthesis device 1000 can be implemented as a computer program, which can be implemented in, for example... Figure 11 The computer device shown is used for operation. The computer device's memory can store the various program modules that make up the speech synthesis device 1000, for example, Figure 10 The extraction module 1010, determination module 1020, first generation module 1030, and second generation module 1040 are shown. The computer program comprised of these modules causes the processor to execute the steps of the speech synthesis methods in the various embodiments of this application described in this specification.

[0165] For example, Figure 11 The computer equipment shown can be used as follows Figure 10The extraction module 1010 in the speech synthesis apparatus 1000 shown executes step S210. The computer device can execute step S220 via the determination module 1020. The computer device can execute step S230 via the first generation module 1030. The computer device can execute step S240 via the second generation module 1040. The computer device includes a processor, memory, and a network interface connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used to communicate with external computer devices via a network connection. When the computer program is executed by the processor, it implements a speech synthesis method.

[0166] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0167] In some embodiments of this application, a computer device is provided, including one or more processors; memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to perform the following steps:

[0168] Obtain the target musical score to be processed, and extract the musical score features from the target musical score;

[0169] Based on the musical score features and the pre-trained acoustic model, the acoustic features corresponding to the musical score features are determined;

[0170] An initial waveform with a harmonic structure is generated based on the fundamental frequency information in the acoustic features.

[0171] The target audio corresponding to the target musical score is generated based on the acoustic features and the initial waveform.

[0172] In some embodiments of this application, when the processor executes the computer program, it further implements the following steps: upsampling the fundamental frequency information and outputting a fundamental frequency signal at the corresponding waveform point level; generating a sinusoidal excitation signal based on the fundamental frequency signal and the multiple harmonics corresponding to the fundamental frequency signal; and performing a linear transformation on the sinusoidal excitation signal to obtain an initial waveform with a harmonic structure.

[0173] In some embodiments of this application, when the processor executes the computer program, it further performs the following steps: sampling the acoustic features and the initial waveform respectively to obtain a first feature sequence and a second feature sequence with the same corresponding resolution; adding the first feature sequence and the second feature sequence to obtain a fused feature sequence; and generating the target audio corresponding to the target musical score based on the fused feature sequence.

[0174] In some embodiments of this application, when the processor executes the computer program, it also performs the following steps: obtaining a vocoder pre-trained based on training sample scores and training sample audio; inputting the fused feature sequence into the vocoder, and outputting the target audio corresponding to the target score.

[0175] In some embodiments of this application, when the processor executes the computer program, it further implements the following steps: determining the fundamental frequency information corresponding to the musical score feature based on the musical score feature; inputting the logarithmic fundamental frequency corresponding to the fundamental frequency information to the vibrato predictor, and outputting the vibrato distribution feature corresponding to the logarithmic fundamental frequency; and determining the vibrato distribution feature as the acoustic feature corresponding to the musical score feature.

[0176] In some embodiments of this application, when the processor executes the computer program, it further performs the following steps: acquiring training sample scores and corresponding training sample audio, and extracting training score features and training acoustic features corresponding to the training sample audio; inputting the training score features into a preset initial acoustic model and outputting predicted acoustic features; updating the parameters in the initial acoustic model based on the difference between the predicted acoustic features and the training acoustic features, and the number of training iterations corresponding to the initial acoustic model, to obtain an updated acoustic model; the parameters in the initial acoustic model include at least the residual connection coefficients in the prior network model; until the predicted acoustic features obtained by inputting the training score features into the updated acoustic model meet preset requirements, the current acoustic model is determined as the pre-trained acoustic model.

[0177] In some embodiments of this application, when the processor executes the computer program, it further implements the following steps: generating text encoding features based on the text lyrics information in the target musical score; determining the musical score features in the target musical score based on the note durations corresponding to the note information in the target musical score and the text encoding features.

[0178] In some embodiments of this application, a computer-readable storage medium is provided, storing a computer program that is loaded by a processor, causing the processor to perform the following steps:

[0179] Obtain the target musical score to be processed, and extract the musical score features from the target musical score;

[0180] Based on the musical score features and the pre-trained acoustic model, the acoustic features corresponding to the musical score features are determined;

[0181] An initial waveform with a harmonic structure is generated based on the fundamental frequency information in the acoustic features.

[0182] The target audio corresponding to the target musical score is generated based on the acoustic features and the initial waveform.

[0183] In some embodiments of this application, when the processor executes the computer program, it further implements the following steps: upsampling the fundamental frequency information and outputting a fundamental frequency signal at the corresponding waveform point level; generating a sinusoidal excitation signal based on the fundamental frequency signal and the multiple harmonics corresponding to the fundamental frequency signal; and performing a linear transformation on the sinusoidal excitation signal to obtain an initial waveform with a harmonic structure.

[0184] In some embodiments of this application, when the processor executes the computer program, it further performs the following steps: sampling the acoustic features and the initial waveform respectively to obtain a first feature sequence and a second feature sequence with the same corresponding resolution; adding the first feature sequence and the second feature sequence to obtain a fused feature sequence; and generating the target audio corresponding to the target musical score based on the fused feature sequence.

[0185] In some embodiments of this application, when the processor executes the computer program, it also performs the following steps: obtaining a vocoder pre-trained based on training sample scores and training sample audio; inputting the fused feature sequence into the vocoder, and outputting the target audio corresponding to the target score.

[0186] In some embodiments of this application, when the processor executes the computer program, it further implements the following steps: determining the fundamental frequency information corresponding to the musical score feature based on the musical score feature; inputting the logarithmic fundamental frequency corresponding to the fundamental frequency information to the vibrato predictor, and outputting the vibrato distribution feature corresponding to the logarithmic fundamental frequency; and determining the vibrato distribution feature as the acoustic feature corresponding to the musical score feature.

[0187] In some embodiments of this application, when the processor executes the computer program, it further performs the following steps: acquiring training sample scores and corresponding training sample audio, and extracting training score features and training acoustic features corresponding to the training sample audio; inputting the training score features into a preset initial acoustic model and outputting predicted acoustic features; updating the parameters in the initial acoustic model based on the difference between the predicted acoustic features and the training acoustic features, and the number of training iterations corresponding to the initial acoustic model, to obtain an updated acoustic model; the parameters in the initial acoustic model include at least the residual connection coefficients in the prior network model; until the predicted acoustic features obtained by inputting the training score features into the updated acoustic model meet preset requirements, the current acoustic model is determined as the pre-trained acoustic model.

[0188] In some embodiments of this application, when the processor executes the computer program, it further implements the following steps: generating text encoding features based on the text lyrics information in the target musical score; determining the musical score features in the target musical score based on the note durations corresponding to the note information in the target musical score and the text encoding features.

[0189] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0190] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0191] The above provides a detailed description of a speech synthesis method, apparatus, computer device, and storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A speech synthesis method, characterized in that, include: Obtain the target musical score to be processed, and extract the musical score features from the target musical score; Based on the musical score features and the pre-trained acoustic model, the acoustic features corresponding to the musical score features are determined; An initial waveform with a harmonic structure is generated based on the fundamental frequency information in the acoustic features. Generate the target audio corresponding to the target musical score based on the acoustic features and the initial waveform; The acoustic model includes a vibrato predictor; The step of determining the acoustic features corresponding to the musical score features based on the musical score features and the pre-trained acoustic model includes: Determine the fundamental frequency information corresponding to the musical score features based on the musical score features; The logarithmic fundamental frequency corresponding to the fundamental frequency information is input into the vibrato predictor, and the vibrato distribution feature corresponding to the logarithmic fundamental frequency is output. The vibrato distribution characteristics are determined as the acoustic characteristics corresponding to the musical score characteristics; The vibrato predictor is a generative model based on a flow model. The vibrato predictor is used to describe the relationship between vibrato distribution and Gaussian distribution. The vibrato distribution feature is a sound signal superimposed on pitch, so that the generated target audio has the vibrato effect corresponding to the vibrato distribution feature superimposed on it.

2. The speech synthesis method according to claim 1, characterized in that, The generation of an initial waveform with a harmonic structure based on the fundamental frequency information in the acoustic features includes: The fundamental frequency information is upsampled, and the corresponding fundamental frequency signal at the waveform point level is output; A sinusoidal excitation signal is generated based on the fundamental frequency signal and the multiple harmonics corresponding to the fundamental frequency signal. The sinusoidal excitation signal is linearly transformed to obtain an initial waveform with a harmonic structure.

3. The speech synthesis method according to claim 2, characterized in that, The linear transformation of the sinusoidal excitation signal to obtain an initial waveform with a harmonic structure includes: Obtain a post-processing network model pre-trained based on the training sample sheet music and training sample audio; The sinusoidal excitation signal is input into the post-processing network model to obtain an initial waveform with a harmonic structure.

4. The speech synthesis method according to claim 1, characterized in that, The step of generating the target audio corresponding to the target musical score based on the acoustic features and the initial waveform includes: The acoustic features and the initial waveform are sampled and processed respectively to obtain a first feature sequence and a second feature sequence with the same resolution; The first feature sequence and the second feature sequence are added together to obtain the fused feature sequence; The target audio corresponding to the target musical score is generated based on the fusion feature sequence.

5. The speech synthesis method according to claim 4, characterized in that, The step of generating the target audio corresponding to the target musical score based on the fused feature sequence includes: Obtain a vocoder that has been pre-trained based on the musical scores and audio samples of the training samples; The fused feature sequence is input into the vocoder, and the target audio corresponding to the target musical score is output.

6. The speech synthesis method according to claim 1, characterized in that, Before determining the acoustic features corresponding to the musical score features based on the musical score features and the pre-trained acoustic model, the method further includes: Obtain the training sample sheet music and the corresponding training sample audio, and extract the training sheet music features and the training acoustic features corresponding to the training sample audio. The trained musical score features are input into a preset initial acoustic model, and the predicted acoustic features are output. Based on the difference between the predicted acoustic features and the trained acoustic features, and the number of training iterations corresponding to the initial acoustic model, the parameters in the initial acoustic model are updated to obtain the updated acoustic model; the parameters in the initial acoustic model include at least the residual connection coefficients in the prior network model. The current acoustic model is determined as the pre-trained acoustic model only when the predicted acoustic features obtained by inputting the training musical score features into the updated acoustic model meet the preset requirements.

7. The speech synthesis method according to any one of claims 1 to 6, characterized in that, The extraction of musical features from the target musical score includes: Based on the text lyrics information in the target musical score, generate text encoding features; The musical score features in the target score are determined based on the note durations corresponding to the note information in the target score and the text encoding features.

8. A speech synthesis device, characterized in that, include: An extraction module is used to acquire the target musical score to be processed and extract the musical score features from the target musical score; The determination module is used to determine the acoustic features corresponding to the musical score features based on the musical score features and the pre-trained acoustic model. The first generation module is used to generate an initial waveform with a harmonic structure based on the fundamental frequency information in the acoustic features; The second generation module is used to generate the target audio corresponding to the target musical score based on the acoustic features and the initial waveform; The acoustic model includes a vibrato predictor; The step of determining the acoustic features corresponding to the musical score features based on the musical score features and the pre-trained acoustic model includes: Determine the fundamental frequency information corresponding to the musical score features based on the musical score features; The logarithmic fundamental frequency corresponding to the fundamental frequency information is input into the vibrato predictor, and the vibrato distribution feature corresponding to the logarithmic fundamental frequency is output. The vibrato distribution characteristics are determined as the acoustic characteristics corresponding to the musical score characteristics; The vibrato predictor is a generative model based on a flow model. The vibrato predictor is used to describe the relationship between vibrato distribution and Gaussian distribution. The vibrato distribution feature is a sound signal superimposed on pitch, so that the generated target audio has the vibrato effect corresponding to the vibrato distribution feature superimposed on it.

9. A computer device, characterized in that, The computer device includes: One or more processors; Memory; and One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the steps of the speech synthesis method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program, which is loaded by a processor to perform the steps of the speech synthesis method according to any one of claims 1 to 7.

Images