An audio synthesis method, device, electronic device and storage medium

Abstract

The present disclosure relates to an audio synthesis method, device, electronic device, and storage medium. The method comprises: obtaining a fundamental frequency to be synthesized and a text to be synthesized; inputting the text to be synthesized into a pre-trained spectrum prediction model to obtain spectrum envelope information; inputting the spectrum envelope information and the fundamental frequency to be synthesized into a pre-trained Mel spectrum prediction model to obtain a predicted Mel spectrum; and obtaining a target synthesized audio based on the predicted Mel spectrum. The present disclosure separates the fundamental frequency to be synthesized from the spectrum envelope, thereby allowing the predicted Mel spectrum to be adjusted by precisely controlling the fundamental frequency to be synthesized when predicting the Mel spectrum, ultimately achieving the purpose of precisely controlling the synthesized audio. This is beneficial for ensuring the pitch accuracy of the synthesized audio, greatly improving the flexibility of audio synthesis, and is very suitable for audio synthesis scenarios such as singing synthesis where pitch accuracy is limited.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an audio synthesis method, device, electronic device, and storage medium. Background Art

[0002] Audio synthesis technology can convert text into fluent speech output. Related technologies achieve this by directly mapping the text to be synthesized to a mel-spectrogram to generate the synthesized audio. This approach cannot adjust the pitch accuracy of the synthesized audio, resulting in limited flexibility and unsuitable for scenarios like singing synthesis where pitch accuracy is a concern. Summary of the Invention

[0003] The present disclosure provides an audio synthesis method, apparatus, electronic device, and storage medium to at least address the problem in related art that audio synthesis is inflexible and unsuitable for audio synthesis scenarios with limited pitch accuracy. The technical solutions of the present disclosure are as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an audio method, including:

[0005] Obtain the fundamental frequency and text to be synthesized;

[0006] Inputting the text to be synthesized into a pre-trained spectrum prediction model to obtain spectrum envelope information;

[0007] Inputting the spectrum envelope information and the fundamental frequency to be synthesized into a pre-trained Mel spectrum prediction model to obtain a predicted Mel spectrum;

[0008] A target synthesized audio is obtained according to the predicted Mel spectrum.

[0009] In an exemplary embodiment, obtaining the base frequency to be synthesized includes:

[0010] Get music score data;

[0011] Determining an original fundamental frequency corresponding to the music score data;

[0012] Adjusting the original fundamental frequency to obtain an adjusted fundamental frequency corresponding to the music score data;

[0013] The adjusted fundamental frequency is used as the fundamental frequency to be synthesized.

[0014] In an exemplary embodiment, adjusting the original fundamental frequency to obtain the adjusted fundamental frequency corresponding to the music score data includes:

[0015] Inputting the music score data into a pre-trained fundamental frequency residual prediction model to obtain the fundamental frequency residual corresponding to the music score data;

[0016] Adjusting the fundamental frequency residual to obtain an adjusted fundamental frequency residual;

[0017] The sum of the original fundamental frequency and the residual of the adjusted fundamental frequency is calculated, and the sum is used as the adjusted fundamental frequency corresponding to the music score data.

[0018] In an exemplary embodiment, adjusting the fundamental frequency residual to obtain the adjusted fundamental frequency residual includes:

[0019] determining a semitone fundamental frequency of the music score data according to an original fundamental frequency of the music score data;

[0020] Determining whether the fundamental frequency residual exceeds the semitone fundamental frequency;

[0021] If the fundamental frequency residual exceeds the semitone fundamental frequency, the fundamental frequency residual is adjusted to obtain an adjusted fundamental frequency residual; the adjusted fundamental frequency residual does not exceed the semitone fundamental frequency.

[0022] In an exemplary embodiment, obtaining the base frequency to be synthesized includes:

[0023] Get user voice audio;

[0024] Determining the voice fundamental frequency of the user voice audio;

[0025] The speech fundamental frequency is used as the fundamental frequency to be synthesized.

[0026] In an exemplary embodiment, the method further includes the step of training an audio synthesis model; the training of the audio synthesis model includes:

[0027] Constructing an audio synthesis model to be trained, wherein the audio synthesis model to be trained includes a spectrum prediction model to be trained, a fundamental frequency residual prediction model to be trained, and a mel spectrum prediction model to be trained;

[0028] Acquire training data, where the training data includes sample data and reference data, where the sample data includes sample text and sample music score data, and the reference data includes a reference Mel spectrum and a reference fundamental frequency residual;

[0029] Inputting the sample text and the sample music score data into the spectrum prediction model to be trained and the fundamental frequency residual prediction model to be trained, respectively, to obtain training spectrum envelope information and training fundamental frequency residual;

[0030] Inputting the training spectrum envelope information and the training fundamental frequency residual into the Mel spectrum prediction model to be trained to obtain a training Mel spectrum;

[0031] The audio synthesis model to be trained is trained according to the training fundamental frequency residual, the training Mel spectrum, the reference fundamental frequency residual and the reference Mel spectrum to obtain the audio synthesis model at the end of training; the audio synthesis model at the end of training includes the spectrum prediction model, the fundamental frequency residual prediction model and the Mel spectrum prediction model.

[0032] In an exemplary embodiment, the training the audio synthesis model to be trained according to the training fundamental frequency residual, the training mel spectrum, the reference fundamental frequency residual, and the reference mel spectrum to obtain the audio synthesis model at the end of the training includes:

[0033] Determining a first loss value according to the training fundamental frequency residual and the reference fundamental frequency residual;

[0034] Determining a second loss value according to the training Mel spectrum and the reference Mel spectrum;

[0035] determining a target loss value according to the first loss value and the second loss value;

[0036] According to the target loss value, the model parameters of the spectrum prediction model to be trained, the fundamental frequency residual prediction model to be trained, and the Mel spectrum prediction model to be trained are adjusted respectively until the preset training end conditions are met to obtain the audio synthesis model.

[0037] According to a second aspect of an embodiment of the present disclosure, there is provided an audio synthesis device, comprising:

[0038] An acquiring unit, configured to acquire a base frequency to be synthesized and a text to be synthesized;

[0039] a spectrum envelope prediction unit configured to input the to-be-synthesized text into a pre-trained spectrum prediction model to obtain spectrum envelope information;

[0040] a mel spectrum prediction unit configured to input the spectrum envelope information and the base frequency to be synthesized into a pre-trained mel spectrum prediction model to obtain a predicted mel spectrum;

[0041] The audio synthesis unit is configured to obtain target synthesized audio according to the predicted Mel spectrum.

[0042] In an exemplary embodiment, the acquiring unit includes:

[0043] a music score acquiring unit, configured to acquire music score data;

[0044] A first determining unit is configured to determine an original fundamental frequency corresponding to the music score data;

[0045] The first adjustment unit is configured to adjust the original fundamental frequency to obtain an adjusted fundamental frequency corresponding to the music score data; and use the adjusted fundamental frequency as the fundamental frequency to be synthesized.

[0046] In an exemplary embodiment, the first adjustment unit includes:

[0047] a fundamental frequency residual prediction unit configured to input the music score data into a pre-trained fundamental frequency residual prediction model to obtain a fundamental frequency residual corresponding to the music score data;

[0048] A second adjustment unit is configured to adjust the baseband residual to obtain an adjusted baseband residual;

[0049] The first calculation unit is configured to calculate a sum of the original fundamental frequency and the residual of the adjusted fundamental frequency, and use the sum as the adjusted fundamental frequency corresponding to the music score data.

[0050] In an exemplary embodiment, the second adjustment unit includes:

[0051] a second determining unit configured to determine a semitone fundamental frequency of the music score data according to an original fundamental frequency of the music score data;

[0052] A judging unit is configured to judge whether the fundamental frequency residual exceeds the semitone fundamental frequency;

[0053] The adjustment subunit is configured to adjust the fundamental frequency residual to obtain an adjusted fundamental frequency residual when the fundamental frequency residual exceeds the semitone fundamental frequency; the adjusted fundamental frequency residual does not exceed the semitone fundamental frequency.

[0054] In an exemplary embodiment, the acquiring unit includes:

[0055] A user voice audio acquisition unit is configured to acquire user voice audio;

[0056] The third determining unit is configured to determine the voice fundamental frequency of the user voice audio; and use the voice fundamental frequency as the fundamental frequency to be synthesized.

[0057] In an exemplary embodiment, the apparatus further comprises a training unit for training the audio synthesis model, the training unit comprising:

[0058] A model construction unit is configured to execute construction of an audio synthesis model to be trained, wherein the audio synthesis model to be trained includes a spectrum prediction model to be trained, a fundamental frequency residual prediction model to be trained, and a mel spectrum prediction model to be trained;

[0059] A training data acquisition unit is configured to acquire training data, wherein the training data includes sample data and reference data, wherein the sample data includes sample text and sample music score data, and the reference data includes a reference Mel spectrum and a reference fundamental frequency residual;

[0060] A first input unit is configured to input the sample text and the sample music score data into the spectrum prediction model to be trained and the fundamental frequency residual prediction model to be trained, respectively, to obtain training spectrum envelope information and training fundamental frequency residual;

[0061] A second input unit is configured to input the training spectrum envelope information and the training fundamental frequency residual into the Mel spectrum prediction model to be trained to obtain a training Mel spectrum;

[0062] The training subunit is configured to train the audio synthesis model to be trained based on the training fundamental frequency residual, the training Mel spectrum, the reference fundamental frequency residual and the reference Mel spectrum to obtain the audio synthesis model at the end of the training; the audio synthesis model at the end of the training includes the spectrum prediction model, the fundamental frequency residual prediction model and the Mel spectrum prediction model.

[0063] In an exemplary embodiment, the training subunit includes:

[0064] A first loss determining unit is configured to determine a first loss value according to the training fundamental frequency residual and the reference fundamental frequency residual;

[0065] A second loss determining unit is configured to determine a second loss value according to the training Mel spectrum and the reference Mel spectrum;

[0066] a target loss determining unit, configured to determine a target loss value according to the first loss value and the second loss value;

[0067] The parameter adjustment unit is configured to adjust the model parameters of the spectrum prediction model to be trained, the fundamental frequency residual prediction model to be trained and the Mel spectrum prediction model to be trained according to the target loss value until the preset training end condition is met to obtain the audio synthesis model.

[0068] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0069] processor;

[0070] a memory for storing instructions executable by the processor;

[0071] The processor is configured to execute the instructions to implement the audio synthesis method as described in any of the above embodiments.

[0072] According to a fourth aspect of an embodiment of the present disclosure, a computer storage medium is provided. When instructions in the computer storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the audio synthesis method described in any of the above embodiments.

[0073] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the audio synthesis method provided in any of the above-described embodiments.

[0074] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

[0075] By obtaining the fundamental frequency to be synthesized and the text to be synthesized, the text to be synthesized is input into a pre-trained spectral prediction model to obtain spectral envelope information, and then the spectral envelope information and the fundamental frequency to be synthesized are input into a pre-trained Mel spectrum prediction model to obtain a predicted Mel spectrum, and the target synthesized audio is obtained based on the predicted Mel spectrum. It can be seen that the above technical solution separates the fundamental frequency to be synthesized from the spectral envelope, thereby allowing the predicted Mel spectrum to be adjusted by precise control of the fundamental frequency to be synthesized when predicting the Mel spectrum, and ultimately achieving the purpose of precise control of the synthesized audio, which is conducive to ensuring the pitch accuracy of the synthesized audio, greatly improving the flexibility of audio synthesis, and is very suitable for audio synthesis scenarios such as singing synthesis that have limitations on pitch accuracy.

[0076] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0078] Figure 1 is a diagram showing an application environment of an audio synthesis method according to an exemplary embodiment;

[0079] Figure 2 is a flow chart of an audio synthesis method according to an exemplary embodiment;

[0080] Figure 3 is a flowchart of another audio synthesis method according to an exemplary embodiment;

[0081] Figure 4is a flowchart of another audio synthesis method according to an exemplary embodiment;

[0082] Figure 5 is a schematic diagram of an optional process of adjusting a baseband residual to obtain an adjusted baseband residual according to an exemplary embodiment;

[0083] Figure 6 is a flowchart of another audio synthesis method according to an exemplary embodiment;

[0084] Figure 7 is a schematic diagram of a process for training an audio synthesis model according to an exemplary embodiment;

[0085] Figure 8 is a block diagram of an audio synthesis device according to an exemplary embodiment;

[0086] Figure 9 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0087] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0088] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0089] Mel frequency represents the average human ear's sensitivity to frequency. The unit of frequency is Hertz (Hz). The frequency range that the human ear can hear is 20-20000Hz, but the human ear does not have a linear perception relationship with the scale unit of Hz. For example, if we are adapted to a 1000Hz tone, if the tone frequency is increased to 2000Hz, our ears can only perceive that the frequency has increased a little bit, and we cannot perceive that the frequency has doubled at all. If the ordinary frequency scale is converted to the Mel frequency scale, the human ear's perception of frequency becomes a linear relationship. In other words, under the Mel frequency scale, if the Mel frequencies of two speech segments differ by twice, the pitch that the human ear can perceive will also differ by about twice. The relationship between Mel frequency and Hertz frequency is: F mel=1125ln(1+f / 700), where F mel Mel frequency, f is the Hertz frequency, and the nonlinear spectrum obtained by adding the Mel filter function to the linear spectrum is called the Mel spectrum.

[0090] Mel-spectrograms contain both the spectrum envelope and fundamental frequency information. The fundamental frequency (f0) refers to a set of sine waves that make up the original signal. The lowest frequency sine wave is the fundamental frequency, and the others are overtones. The fundamental frequency and pitch have the following corresponding relationship: Among them, N is the pitch and P is the fundamental frequency.

[0091] The spectral envelope (SP), commonly known as timbre, refers to the envelope line obtained by connecting the highest amplitude points of different frequencies through a smooth curve.

[0092] In the related art, audio is synthesized by directly mapping the text to be synthesized to the Mel spectrum. However, the spectral envelope and fundamental frequency information in the Mel spectrum are combined together, which makes it impossible for the audio synthesis method in the related art to accurately control the fundamental frequency. The flexibility of audio synthesis is poor, and it is impossible to adjust the situation where the synthesized audio is "out of tune". Therefore, it is not suitable for audio synthesis scenarios such as singing synthesis that have restrictions on pitch accuracy.

[0093] In view of this, an embodiment of the present disclosure provides an audio synthesis method, which separates the fundamental frequency to be synthesized from the spectral envelope, thereby allowing the predicted Mel spectrum to be adjusted by precisely controlling the fundamental frequency to be synthesized when predicting the Mel spectrum, ultimately achieving the purpose of precisely controlling the synthesized audio, which is conducive to ensuring the pitch accuracy of the synthesized audio, greatly improving the flexibility of audio synthesis, and is very suitable for audio synthesis scenarios such as singing synthesis that have limitations on pitch accuracy.

[0094] See also Figure 1 , which shows an application environment diagram of an audio synthesis method according to an exemplary embodiment. The application environment may include a terminal 110 and a server 120. The terminal 110 and the server 120 may be connected via a wired network or a wireless network.

[0095] Terminal 110 may be, but is not limited to, a smartphone, tablet computer, laptop computer, or desktop computer. Client software, such as an application (App), that provides audio synthesis functionality may be installed on terminal 110. This application may be a dedicated audio synthesis application or another application with audio synthesis functionality, such as a live broadcast application with audio synthesis functionality. Users of terminal 110 may log in to the application using pre-registered user information, which may include an account number and password.

[0096] The server 120 may be a server that provides background services for the application in the terminal 110, or may be another server that is connected to and communicates with the background server of the application. It may be a single server or a server cluster composed of multiple servers. The server 120 may store an audio synthesis model, which may include a spectrum prediction model, a fundamental frequency residual prediction model, and a mel spectrum prediction model. The server 120 may train and update the audio synthesis model according to a predetermined period. When the terminal 110 needs to perform audio synthesis, it may send the corresponding data to the server 120, and the server 120 may call the audio synthesis model to perform audio synthesis based on the data and return the synthesized audio to the terminal 110.

[0097] It is understandable that the terminal 110 may also download the audio synthesis model from the server 120 and store it locally. When an audio synthesis operation is required, the terminal 110 may directly call the locally stored audio synthesis model to perform audio synthesis.

[0098] It can be seen that the audio synthesis method of the embodiment of the present disclosure can be executed by an electronic device, which can be a terminal or a server. It can be executed by the terminal or the server alone, or by the terminal and the server in cooperation with each other.

[0099] Figure 2 is a flow chart of an audio synthesis method according to an exemplary embodiment. Figure 2 As shown, the audio synthesis method is applied to an electronic device for illustration, and includes the following steps.

[0100] In step S210, the fundamental frequency to be synthesized and the text to be synthesized are obtained.

[0101] The text to be synthesized may be a lyric text, and the fundamental frequency to be synthesized is a fundamental frequency sequence used to synthesize the target synthesized audio.

[0102] In step S220, the text to be synthesized is input into a pre-trained spectrum prediction model to obtain spectrum envelope information.

[0103] Among them, the spectrum prediction model can be a CBHG (Convolution Bank + Highway network + bidirectional Gated Recurrent Unit, convolution layer + highway network + bidirectional recurrent neural network) model, that is, the spectrum prediction model consists of a 1-D convolution filter, a highway network and a bidirectional recurrent neural network. The text to be synthesized is input into the spectrum prediction model, and the spectrum envelope information can be obtained through the hidden layer output of the spectrum prediction model. The spectrum envelope information includes the spectrum envelope and text content.

[0104] The specific training process of the spectrum prediction model will be introduced in detail in the subsequent parts of the embodiments of this disclosure.

[0105] In step S230, the spectrum envelope information and the fundamental frequency to be synthesized are input into a pre-trained mel spectrum prediction model to obtain a predicted mel spectrum.

[0106] The mel spectrum prediction model may be a recursive neural network such as a recurrent neural network (RNN), which can predict the mel spectrum based on input spectrum envelope information and the fundamental frequency to be synthesized.

[0107] The specific training process of the mel spectrum prediction model will be introduced in detail in the subsequent parts of the embodiments of this disclosure.

[0108] In step S240 , a target synthesized audio is obtained according to the predicted Mel-spectrogram.

[0109] Specifically, a vocoder can be used to convert the predicted Mel-spectrogram into audio to obtain the target synthesized audio. The vocoder can be Wavenet, Griffin-Lim, or a single-layer recurrent neural network model like WaveRNN, etc., to achieve better sound quality, close to that of real human speech.

[0110] The above-mentioned technical solution of the embodiment of the present disclosure separates the fundamental frequency to be synthesized from the spectral envelope, thereby allowing the predicted Mel spectrum to be adjusted by precise control of the fundamental frequency to be synthesized when predicting the Mel spectrum, thereby ultimately achieving the purpose of precise control of the synthesized audio, which is conducive to ensuring the pitch accuracy of the synthesized audio and greatly improving the flexibility of audio synthesis.

[0111] As a possible implementation, the fundamental frequency to be synthesized can be derived from the musical score data. In order to improve the fit between the pitch of the target synthesized audio and the pitch of the musical score and reduce the "out-of-tune" situation of the target synthesized audio, for example Figure 3 As shown, the above step S210 may include the following steps when obtaining the base frequency to be synthesized:

[0112] In step 310 , music score data is acquired.

[0113] Musical notation is a method of recording music using symbols, namely musical notes. The shape of the notes in the musical notation reflects the relative duration, i.e. the length of the note, and their position in the musical notation indicates the specific pitch.

[0114] In step 320, the original fundamental frequency corresponding to the music score data is determined.

[0115] Specifically, the pitch corresponding to each note in the music score data can be analyzed to obtain the original fundamental frequency of the music score data. The original fundamental frequency is actually a fundamental frequency sequence that corresponds one-to-one to the notes in the music score data.

[0116] For example, the parsing of the musical score data may be performed according to the following correspondence between fundamental frequency and pitch: Among them, N represents the pitch of the note and P represents the fundamental frequency.

[0117] In step 330, the original fundamental frequency is adjusted to obtain an adjusted fundamental frequency corresponding to the music score data.

[0118] In the disclosed embodiment, the spectrum envelope is separated from the fundamental frequency, which allows the fundamental frequency of the predicted Mel spectrum to be adjusted, thereby enabling precise control of the predicted Mel spectrum.

[0119] As a possible implementation method, in order to achieve accurate control of the predicted Mel spectrum, we can use Figure 4 The method shown adjusts the original base frequency, such as Figure 4 As shown, the method may include:

[0120] In step S410, the music score data is input into a pre-trained fundamental frequency residual prediction model to obtain the fundamental frequency residual corresponding to the music score data.

[0121] Among them, the fundamental frequency residual prediction model can be a long short-term memory network (Long Short-Term Memory, LSTM), which can predict the fundamental frequency residual corresponding to the music score data based on the input music score data. It can be understood that the fundamental frequency residual is actually a fundamental frequency residual sequence that corresponds one-to-one to the notes in the music score data.

[0122] The training of the fundamental frequency residual prediction model will be described in the subsequent parts of the embodiments of this disclosure.

[0123] In step S420, the fundamental frequency residual is adjusted to obtain an adjusted fundamental frequency residual.

[0124] In the embodiment of the present disclosure, in order to make the pitch of the target synthesized audio more consistent with the pitch of the musical score, in one possible implementation, the predicted fundamental frequency residual can be restricted and adjusted, such as Figure 5 As shown, when adjusting the baseband residual to obtain the adjusted baseband residual, the following steps may be included:

[0125] In step S510 , the semitone fundamental frequency of the music score data is determined according to the original fundamental frequency of the music score data.

[0126] It should be noted that the semitone fundamental frequency of the music score data in the embodiment of the present disclosure is actually a semitone fundamental frequency sequence corresponding one-to-one to the notes in the music score data, and each semitone fundamental frequency in the sequence is a range limited by the upper semitone fundamental frequency and the lower semitone fundamental frequency, wherein the upper semitone fundamental frequency and the lower semitone fundamental frequency are the difference between the original fundamental frequency of the current note and the original fundamental frequencies of the two adjacent notes on the left and right.

[0127] For example, assuming there are three adjacent notes in the music score data, and the pitches corresponding to these three adjacent notes are 64, 65, and 66 respectively, then according to the above-mentioned pitch and fundamental frequency conversion formula, the three original fundamental frequencies corresponding to these three notes can be obtained, such as 340hz, 360hz, and 400hz. Then the semitone distance between 65 and 66 is 40hz (400hz-360hz), and the semitone distance between 65 and 64 is 20hz (360hz-340hz). Therefore, for the note pitch 65, its semitone fundamental frequency is -20hz~40hz.

[0128] In step S520 , it is determined whether the fundamental frequency residual exceeds the semitone fundamental frequency.

[0129] In step S530 , if the fundamental frequency residual exceeds the semitone fundamental frequency, the fundamental frequency residual is adjusted to obtain an adjusted fundamental frequency residual.

[0130] The fundamental frequency residual is adjusted not to exceed the semitone fundamental frequency of the music score data.

[0131] Specifically, the fundamental frequency residual corresponding to each note in the musical score data may be compared with the semitone fundamental frequency. If the fundamental frequency residual of a note exceeds its corresponding semitone fundamental frequency, the fundamental frequency residual of the note may be adjusted so that the adjusted fundamental frequency residual does not exceed the semitone fundamental frequency corresponding to the note. If the fundamental frequency residual of a note does not exceed its corresponding semitone fundamental frequency, the fundamental frequency residual of the note may not be adjusted.

[0132] Continuing with the example of a semitone fundamental frequency, let's assume that for a note pitch of 65, the corresponding fundamental frequency residual is 60 Hz. This fundamental frequency residual exceeds the corresponding semitone fundamental frequency by -20 Hz to 40 Hz. In this case, the fundamental frequency residual needs to be adjusted so that the adjusted fundamental frequency residual is within the range of -20 Hz to 40 Hz. The specific adjustment range of the fundamental frequency residual can be random, as long as the adjusted fundamental frequency residual does not exceed the corresponding semitone fundamental frequency.

[0133] The embodiments of the present disclosure implement limited adjustment of the fundamental frequency residual through the above method, which is conducive to precise control of the synthesized fundamental frequency and ensures the accuracy of the pitch in the target synthesized audio.

[0134] In step S430, a sum of the original fundamental frequency and the residual of the adjusted fundamental frequency is calculated, and the sum is used as the adjusted fundamental frequency corresponding to the music score data.

[0135] In step 340, the adjusted fundamental frequency is used as the fundamental frequency to be synthesized.

[0136] The embodiments of the present disclosure achieve precise control of the fundamental frequency to be synthesized by limiting and adjusting the fundamental frequency residual, thereby enabling precise control of the predicted Mel spectrum obtained based on the fundamental frequency to be synthesized, improving the fit between the target synthesized audio pitch and the musical score pitch, and avoiding the problem of "out of tune" in singing synthesis.

[0137] As a possible implementation method, in order to further improve the flexibility of audio synthesis, realize the synthesis of audio of any tone or style and improve the naturalness of the target synthesized audio, the fundamental frequency to be synthesized can be derived from real person voice, such as real person singing, etc. Based on this, Figure 6 As shown, the above step S210 may include the following steps when obtaining the base frequency to be synthesized:

[0138] In step S610, user voice audio is obtained.

[0139] In step S620, the fundamental frequency of the user's voice audio is determined.

[0140] Specifically, a speech fundamental frequency recognition algorithm may be used to extract the fundamental frequency of the user's speech audio signal. The algorithm may include an autocorrelation algorithm, a parallel processing method, a cepstrum method, a simplified inverse filtering method, and the like.

[0141] In step S630, the above-mentioned speech fundamental frequency is used as the fundamental frequency to be synthesized.

[0142] By using the fundamental frequency of existing real-person speech audio, directly feeding it into the Mel-spectrogram prediction model and combining it with the input spectral envelope information to predict the Mel-spectrogram, the naturalness of the target synthesized audio can be effectively improved. At the same time, it can also synthesize synthesized audio of any different tones or styles.

[0143] As a possible implementation, the audio synthesis method of the embodiment of the present disclosure further includes training an audio synthesis model, which includes a spectrum prediction model, a mel spectrum prediction model and a fundamental frequency residual prediction model. That is, in the embodiment of the present disclosure, the spectrum prediction model, the mel spectrum prediction model and the fundamental frequency residual prediction model are trained as a whole. Figure 7The training process of the audio synthesis model according to the embodiment of the present disclosure is described in detail. The training process may include the following steps:

[0144] (1) Construct Figure 7 The audio synthesis model to be trained is shown, and the audio synthesis model to be trained includes a spectrum prediction model to be trained, a fundamental frequency residual prediction model to be trained, and a Mel spectrum prediction model to be trained.

[0145] The spectrum prediction model to be trained can be a CBHG model, which consists of a 1-D convolutional filter, a high-speed network, and a bidirectional recurrent neural network. The fundamental frequency residual prediction model to be trained can be a long short-term memory (LSTM) model. The mel spectrum prediction model to be trained can be a recurrent neural network (RNN) model.

[0146] (2) Acquire training data, where the training data includes sample data and reference data. The sample data includes sample text and sample music score data. The reference data includes reference Mel spectrum and reference fundamental frequency residual.

[0147] The reference fundamental frequency residual can be determined based on the original fundamental frequency of the sample music score data and the sample speech fundamental frequency of the sample user speech audio. Specifically, the difference between the original fundamental frequency and the sample speech fundamental frequency can be used as the reference fundamental frequency residual. The reference Mel-spectrogram can be extracted from the sample synthesized audio, which can be selected and specifically limited as needed.

[0148] (3) The sample text and the sample music score data are respectively input into the spectrum prediction model to be trained and the fundamental frequency residual prediction model to be trained to obtain the training spectrum envelope information and the training fundamental frequency residual.

[0149] Specifically, during the training process, the sample text is used as the input of the spectrum prediction model to be trained, and the output of the spectrum prediction model to be trained is the training spectrum envelope information; the sample music score data is used as the input of the fundamental frequency residual prediction model to be trained, and the output of the fundamental frequency residual prediction model to be trained is the training fundamental frequency residual.

[0150] (4) Input the training spectrum envelope information and the training fundamental frequency residual into the Mel spectrum prediction model to be trained to obtain the training Mel spectrum.

[0151] (5) The audio synthesis model to be trained is trained according to the training fundamental frequency residual, the training Mel spectrum, the reference fundamental frequency residual and the reference Mel spectrum to obtain the audio synthesis model at the end of the training. The audio synthesis model at the end of the training includes the aforementioned spectrum prediction model, fundamental frequency residual prediction model and Mel spectrum prediction model of the embodiment of the present disclosure.

[0152] In a specific implementation, this step may include:

[0153] A first loss value is determined according to the training fundamental frequency residual and the reference fundamental frequency residual.

[0154] A second loss value is determined based on the training mel-spectrogram and the reference mel-spectrogram.

[0155] A target loss value is determined according to the first loss value and the second loss value.

[0156] According to the above target loss value, the model parameters of the spectrum prediction model to be trained, the fundamental frequency residual prediction model to be trained, and the Mel spectrum prediction model to be trained are adjusted respectively until the preset training end conditions are met to obtain the audio synthesis model.

[0157] In practical applications, the first loss value Loss1 and the second loss value Loss2 can both be mean square error (MSE) losses. The calculation formula of the mean square error MSE loss is as follows:

[0158]

[0159] Among them, yi is the training value, is the reference value, and n is the sample size.

[0160] The target loss value Loss can be the sum of the first loss value Loss1 and the second loss value Loss2, that is:

[0161] Loss = Loss1 + Loss2

[0162] After the target loss value Loss is calculated using the above formula, the model parameters of the spectrum prediction model to be trained, the fundamental frequency residual prediction model to be trained, and the mel spectrum prediction model to be trained can be adjusted in the reverse direction in the direction of minimizing the target loss value Loss until a preset training end condition is met. Exemplarily, the preset training end condition can be reaching a preset number of iterations or reaching a preset minimum loss value.

[0163] The above-mentioned method of the embodiment of the present disclosure can be used to train an audio synthesis model. The audio synthesis model includes the spectrum prediction model, Mel spectrum prediction model and fundamental frequency residual prediction model mentioned in the embodiment of the present disclosure. Therefore, the audio synthesis method of the embodiment of the present disclosure can be implemented based on the audio synthesis model. While improving the flexibility of audio synthesis, it can also achieve fine control of pitch to solve the problem of "out-of-tune" synthesized audio, so that the synthesized audio has a higher degree of fit with the pitch of the musical score. It is very suitable for scenarios such as singing synthesis where pitch accuracy is limited.

[0164] Figure 8 FIG. 1 is a block diagram of an audio synthesis device according to an exemplary embodiment. Figure 8 The device includes an acquisition unit 810, a spectrum envelope prediction unit 820, a Mel spectrum prediction unit 830 and an audio synthesis unit 840.

[0165] The acquisition unit 810 is configured to acquire the fundamental frequency to be synthesized and the text to be synthesized;

[0166] The spectrum envelope prediction unit 820 is configured to input the to-be-synthesized text into a pre-trained spectrum prediction model to obtain spectrum envelope information;

[0167] The mel spectrum prediction unit 830 is configured to input the spectrum envelope information and the base frequency to be synthesized into a pre-trained mel spectrum prediction model to obtain a predicted mel spectrum;

[0168] The audio synthesis unit 840 is configured to obtain target synthesized audio according to the predicted Mel spectrum.

[0169] In an exemplary embodiment, the acquisition unit 810 may include:

[0170] a music score acquiring unit, configured to acquire music score data;

[0171] A first determining unit is configured to determine an original fundamental frequency corresponding to the music score data;

[0172] The first adjustment unit is configured to adjust the original fundamental frequency to obtain an adjusted fundamental frequency corresponding to the music score data; and use the adjusted fundamental frequency as the fundamental frequency to be synthesized.

[0173] In an exemplary embodiment, the first adjustment unit may include:

[0174] a fundamental frequency residual prediction unit configured to input the music score data into a pre-trained fundamental frequency residual prediction model to obtain the fundamental frequency residual corresponding to the music score data;

[0175] A second adjustment unit is configured to adjust the fundamental frequency residual to obtain an adjusted fundamental frequency residual;

[0176] The first calculation unit is configured to calculate a sum of the original fundamental frequency and the residual of the adjusted fundamental frequency, and use the sum as the adjusted fundamental frequency corresponding to the music score data.

[0177] In an exemplary embodiment, the second adjusting unit may include:

[0178] a second determining unit configured to determine a semitone fundamental frequency of the music score data based on the original fundamental frequency of the music score data;

[0179] A judging unit is configured to judge whether the fundamental frequency residual exceeds the semitone fundamental frequency;

[0180] The adjustment subunit is configured to adjust the fundamental frequency residual to obtain an adjusted fundamental frequency residual when the fundamental frequency residual exceeds the semitone fundamental frequency; the adjusted fundamental frequency residual does not exceed the semitone fundamental frequency.

[0181] In an exemplary embodiment, the acquiring unit 810 may include:

[0182] A user voice audio acquisition unit is configured to acquire user voice audio;

[0183] The third determining unit is configured to determine the voice fundamental frequency of the user voice audio; and use the voice fundamental frequency as the fundamental frequency to be synthesized.

[0184] In an exemplary embodiment, the apparatus may further include a training unit for training the audio synthesis model, the training unit including:

[0185] A model construction unit is configured to execute construction of an audio synthesis model to be trained, where the audio synthesis model to be trained includes a spectrum prediction model to be trained, a fundamental frequency residual prediction model to be trained, and a mel spectrum prediction model to be trained;

[0186] A training data acquisition unit is configured to acquire training data, wherein the training data includes sample data and reference data, wherein the sample data includes sample text and sample music score data, and the reference data includes a reference Mel spectrum and a reference fundamental frequency residual;

[0187] The first input unit is configured to input the sample text and the sample music score data into the spectrum prediction model to be trained and the fundamental frequency residual prediction model to be trained, respectively, to obtain training spectrum envelope information and training fundamental frequency residual;

[0188] The second input unit is configured to input the training spectrum envelope information and the training fundamental frequency residual into the Mel spectrum prediction model to be trained to obtain a training Mel spectrum;

[0189] The training subunit is configured to train the audio synthesis model to be trained based on the training fundamental frequency residual, training Mel spectrum, reference fundamental frequency residual and reference Mel spectrum to obtain the audio synthesis model at the end of training; the audio synthesis model at the end of training includes a spectrum prediction model, a fundamental frequency residual prediction model and a Mel spectrum prediction model.

[0190] In an exemplary embodiment, the training subunit may include:

[0191] A first loss determining unit is configured to determine a first loss value according to the training fundamental frequency residual and the reference fundamental frequency residual;

[0192] A second loss determining unit is configured to determine a second loss value according to the training Mel spectrum and the reference Mel spectrum;

[0193] a target loss determining unit, configured to determine a target loss value based on the first loss value and the second loss value;

[0194] The parameter adjustment unit is configured to adjust the model parameters of the spectral prediction model to be trained, the fundamental frequency residual prediction model to be trained and the Mel spectrum prediction model to be trained according to the above-mentioned target loss value until the preset training end conditions are met to obtain the above-mentioned audio synthesis model.

[0195] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0196] In an exemplary embodiment, an electronic device is also provided, including a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the steps of any audio synthesis method in the above embodiments when executing the instructions stored in the memory.

[0197] The electronic device may be a terminal, a server or a similar computing device. Taking the electronic device as an example, Figure 9 1 is a block diagram of a terminal for executing an audio synthesis method according to an exemplary embodiment. Specifically:

[0198] The terminal may include components such as an RF (Radio Frequency) circuit 910, a memory 920 including one or more computer-readable storage media, an input unit 930, a display unit 940, a sensor 950, an audio circuit 960, a WiFi (wireless fidelity) module 970, a processor 980 including one or more processing cores, and a power supply 990. It will be understood by those skilled in the art that Figure 9 The terminal structure shown in the figure does not constitute a limitation on the terminal, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0199] The RF circuit 910 can be used to receive and send signals during information transmission or calls. Specifically, after receiving downlink information from the base station, it is handed over to one or more processors 980 for processing. In addition, uplink data is sent to the base station. Generally, the RF circuit 910 includes but is not limited to an antenna, at least one amplifier, a tuner, one or more oscillators, a subscriber identity module (SIM) card, a transceiver, a coupler, an LNA (low noise amplifier), a duplexer, etc. In addition, the RF circuit 910 can also communicate with the network and other terminals via wireless communication. Such wireless communication can use any communication standard or protocol, including but not limited to GSM (Global System of Mobile Communication), GPRS (General Packet Radio Service), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), LTE (Long Term Evolution), email, SMS (Short Messaging Service), etc.

[0200] The memory 920 can be used to store software programs and modules. The processor 980 executes various functional applications and data processing by running the software programs and modules stored in the memory 920. The memory 920 may mainly include a program storage area and a data storage area. The program storage area may store an operating system, application programs required for functions, etc.; the data storage area may store data created based on the use of the terminal, etc. In addition, the memory 920 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 920 may also include a memory controller to provide access to the memory 920 by the processor 980 and the input unit 930.

[0201] The input unit 930 can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical, or trackball signal input related to user settings and function control. Specifically, the input unit 930 may include a touch-sensitive surface 931 and other input devices 932. The touch-sensitive surface 931, also known as a touch display or touchpad, can detect user touch operations on or near it (for example, operations performed by a user using a finger, stylus, or any other suitable object or accessory on or near the touch-sensitive surface 931) and drive corresponding connected devices according to a pre-set program. Optionally, the touch-sensitive surface 931 may include a touch detection device and a touch controller. The touch detection device detects the user's touch position and detects signals generated by the touch operation, transmitting the signals to the touch controller. The touch controller receives the touch information from the touch detection device, converts it into touch point coordinates, and then sends it to the processor 980. It can also receive and execute commands from the processor 980. In addition, the touch-sensitive surface 931 can be implemented using various types, including resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface 931, the input unit 930 may further include other input devices 932. Specifically, the other input devices 932 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, power keys, etc.), a trackball, a mouse, and a joystick.

[0202] The display unit 940 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the terminal. These graphical user interfaces can be composed of graphics, text, icons, videos, and any combination thereof. The display unit 940 may include a display panel 941. Optionally, the display panel 941 can be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), or the like. Furthermore, the touch-sensitive surface 931 can cover the display panel 941. When the touch-sensitive surface 931 detects a touch operation on or near it, it transmits the information to the processor 980 to determine the type of touch event. The processor 980 then provides a corresponding visual output on the display panel 941 based on the type of touch event. Among them, the touch-sensitive surface 931 and the display panel 941 can be two independent components to implement input and output functions. However, in some embodiments, the touch-sensitive surface 931 and the display panel 941 can also be integrated to implement input and output functions.

[0203] The terminal may also include at least one sensor 950, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 941 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 941 and / or the backlight when the terminal is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the terminal posture (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that the terminal can also be configured with, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be described in detail here.

[0204] Audio circuit 960, speaker 961, and microphone 962 provide an audio interface between the user and the terminal. Audio circuit 960 can convert received audio data into electrical signals and transmit them to speaker 961, which then converts them into sound signals for output. Microphone 962, on the other hand, converts collected sound signals into electrical signals, which are then received by audio circuit 960 and converted into audio data. The audio data is then processed by output processor 980 and transmitted via RF circuit 910 to, for example, another terminal. Alternatively, the audio data can be output to memory 920 for further processing. Audio circuit 960 may also include an earphone jack to allow communication between an external headset and the terminal.

[0205] WiFi is a short-range wireless transmission technology. The terminal can help users send and receive emails, browse web pages and access streaming media through the WiFi module 970, which provides users with wireless broadband Internet access. Figure 9 A WiFi module 970 is shown, but it is understandable that it is not an essential component of the terminal and can be omitted as needed without changing the essence of the invention.

[0206] The processor 980 is the control center of the terminal. It connects all components of the terminal using various interfaces and circuits. By running or executing software programs and / or modules stored in memory 920 and accessing data stored in memory 920, it performs various terminal functions and processes data, thereby providing overall terminal monitoring. Optionally, the processor 980 may include one or more processing cores. Preferably, the processor 980 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into the processor 980.

[0207] The terminal also includes a power supply 990 (e.g., a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 980 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 990 can also include any of one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other components.

[0208] Although not shown, the terminal may also include a camera, a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the terminal also includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors. The one or more programs include instructions for executing the audio synthesis provided in the above method embodiment.

[0209] It can be understood by those skilled in the art that Figure 9 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 9 More or fewer components than shown, or with Figure 9 Different configurations shown.

[0210] In an exemplary embodiment, a storage medium is further provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the steps of any audio synthesis method in the above embodiments.

[0211] In an exemplary embodiment, a computer program product is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the audio synthesis method provided in any of the above-described embodiments.

[0212] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, which can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0213] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0214] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An audio synthesis method, characterized in that: include: Acquiring music score data, and determining an original fundamental frequency corresponding to the music score data; Inputting the music score data into a pre-trained fundamental frequency residual prediction model to obtain the fundamental frequency residual corresponding to the music score data; adjusting the fundamental frequency residual to obtain an adjusted fundamental frequency residual; calculating the sum of the original fundamental frequency and the adjusted fundamental frequency residual to obtain the fundamental frequency to be synthesized; Get the text to be synthesized; Inputting the text to be synthesized into a pre-trained spectrum prediction model to obtain spectrum envelope information; Inputting the spectrum envelope information and the fundamental frequency to be synthesized into a pre-trained Mel spectrum prediction model to obtain a predicted Mel spectrum; A target synthesized audio is obtained according to the predicted Mel spectrum.

2. The audio synthesis method according to claim 1, wherein: The adjusting the fundamental frequency residual to obtain the adjusted fundamental frequency residual includes: determining a semitone fundamental frequency of the music score data according to an original fundamental frequency of the music score data; Determining whether the fundamental frequency residual exceeds the semitone fundamental frequency; If the fundamental frequency residual exceeds the semitone fundamental frequency, the fundamental frequency residual is adjusted to obtain an adjusted fundamental frequency residual; the adjusted fundamental frequency residual does not exceed the semitone fundamental frequency.

3. The audio synthesis method according to claim 1, wherein: The method further comprises the step of training an audio synthesis model; the training of the audio synthesis model comprises: Constructing an audio synthesis model to be trained, wherein the audio synthesis model to be trained includes a spectrum prediction model to be trained, a fundamental frequency residual prediction model to be trained, and a mel spectrum prediction model to be trained; Acquire training data, where the training data includes sample data and reference data, where the sample data includes sample text and sample music score data, and the reference data includes a reference Mel spectrum and a reference fundamental frequency residual; Inputting the sample text and the sample music score data into the spectrum prediction model to be trained and the fundamental frequency residual prediction model to be trained, respectively, to obtain training spectrum envelope information and training fundamental frequency residual; Inputting the training spectrum envelope information and the training fundamental frequency residual into the Mel spectrum prediction model to be trained to obtain a training Mel spectrum; The audio synthesis model to be trained is trained according to the training fundamental frequency residual, the training Mel spectrum, the reference fundamental frequency residual and the reference Mel spectrum to obtain the audio synthesis model at the end of training; the audio synthesis model at the end of training includes the spectrum prediction model, the fundamental frequency residual prediction model and the Mel spectrum prediction model.

4. The audio synthesis method according to claim 3, characterized in that The step of training the audio synthesis model to be trained according to the training fundamental frequency residual, the training Mel spectrum, the reference fundamental frequency residual, and the reference Mel spectrum to obtain the audio synthesis model at the end of the training comprises: Determining a first loss value according to the training fundamental frequency residual and the reference fundamental frequency residual; Determining a second loss value according to the training Mel spectrum and the reference Mel spectrum; determining a target loss value according to the first loss value and the second loss value; According to the target loss value, the model parameters of the spectrum prediction model to be trained, the fundamental frequency residual prediction model to be trained, and the Mel spectrum prediction model to be trained are adjusted respectively until the preset training end conditions are met to obtain the audio synthesis model.

5. An audio synthesis device, characterized in that: include: an acquiring unit configured to acquire music score data and determine an original fundamental frequency corresponding to the music score data; Inputting the music score data into a pre-trained fundamental frequency residual prediction model to obtain the fundamental frequency residual corresponding to the music score data; adjusting the fundamental frequency residual to obtain an adjusted fundamental frequency residual; calculating the sum of the original fundamental frequency and the adjusted fundamental frequency residual to obtain the fundamental frequency to be synthesized; and obtaining the text to be synthesized; a spectrum envelope prediction unit configured to input the to-be-synthesized text into a pre-trained spectrum prediction model to obtain spectrum envelope information; a mel spectrum prediction unit configured to input the spectrum envelope information and the base frequency to be synthesized into a pre-trained mel spectrum prediction model to obtain a predicted mel spectrum; The audio synthesis unit is configured to obtain target synthesized audio according to the predicted Mel spectrum.

6. The audio synthesis device according to claim 5, characterized in that The acquisition unit includes: a second determining unit configured to determine a semitone fundamental frequency of the music score data according to an original fundamental frequency of the music score data; A judging unit is configured to judge whether the fundamental frequency residual exceeds the semitone fundamental frequency; The adjustment subunit is configured to adjust the fundamental frequency residual to obtain an adjusted fundamental frequency residual when the fundamental frequency residual exceeds the semitone fundamental frequency; the adjusted fundamental frequency residual does not exceed the semitone fundamental frequency.

7. The audio synthesis device according to claim 5, characterized in that The apparatus further comprises a training unit for training an audio synthesis model, wherein the training unit comprises: A model construction unit is configured to execute construction of an audio synthesis model to be trained, wherein the audio synthesis model to be trained includes a spectrum prediction model to be trained, a fundamental frequency residual prediction model to be trained, and a mel spectrum prediction model to be trained; A training data acquisition unit is configured to acquire training data, wherein the training data includes sample data and reference data, wherein the sample data includes sample text and sample music score data, and the reference data includes a reference Mel spectrum and a reference fundamental frequency residual; A first input unit is configured to input the sample text and the sample music score data into the spectrum prediction model to be trained and the fundamental frequency residual prediction model to be trained, respectively, to obtain training spectrum envelope information and training fundamental frequency residual; A second input unit is configured to input the training spectrum envelope information and the training fundamental frequency residual into the Mel spectrum prediction model to be trained to obtain a training Mel spectrum; The training subunit is configured to train the audio synthesis model to be trained based on the training fundamental frequency residual, the training Mel spectrum, the reference fundamental frequency residual and the reference Mel spectrum to obtain the audio synthesis model at the end of the training; the audio synthesis model at the end of the training includes the spectrum prediction model, the fundamental frequency residual prediction model and the Mel spectrum prediction model.

8. The audio synthesis device according to claim 7, characterized in that The training subunit includes: A first loss determining unit is configured to determine a first loss value according to the training fundamental frequency residual and the reference fundamental frequency residual; A second loss determining unit is configured to determine a second loss value according to the training Mel spectrum and the reference Mel spectrum; a target loss determining unit, configured to determine a target loss value according to the first loss value and the second loss value; The parameter adjustment unit is configured to adjust the model parameters of the spectrum prediction model to be trained, the fundamental frequency residual prediction model to be trained and the Mel spectrum prediction model to be trained according to the target loss value until the preset training end condition is met to obtain the audio synthesis model.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the audio synthesis method according to any one of claims 1 to 4. 10 . A computer storage medium, when instructions in the computer storage medium are executed by a processor of an electronic device, enables the electronic device to execute the audio synthesis method according to any one of claims 1 to 4.

Images