Fundamental frequency generation method, computer device and computer readable storage medium

Abstract

The application relates to a base frequency generation method, computer equipment and a storage medium, which can improve the accuracy of base frequency extraction results. The method comprises the following steps: obtaining phonemes corresponding to lyrics of a song and sequence information between the phonemes; generating encoding information of each phoneme according to a pitch, a pitch duration and a connected tone mode of each phoneme in a score of the song; inputting the encoding information of the phonemes into a trained generation model according to the sequence information, determining predicted base frequency features of multiple frames based on the encoding information of each phoneme and the encoding information of context phonemes of each phoneme by the generation model; the context phonemes of each phoneme are determined based on the sequence information between the phonemes; obtaining a pitch of each frame in the multiple frames, determining a base frequency of each frame according to the pitch of each frame and the predicted base frequency features, and obtaining the base frequency of the song according to the base frequencies of the multiple frames.

Description

Technical Field

[0001] This application relates to the field of audio technology, and in particular to a fundamental frequency generation method, a computer device, and a computer-readable storage medium. Background Technology

[0002] With the development of computer technology, vocal synthesis technology has been widely used. In order to generate realistic song audio, the fundamental frequency can be used as an acoustic feature input in the vocal synthesis process.

[0003] In related technologies, when synthesizing the audio of a target song, audio sung by other users can be obtained as reference audio, and the fundamental frequency can be extracted from the reference audio using a fundamental frequency extraction tool; alternatively, the sheet music of the target song can be obtained, and the fundamental frequency can be predicted by the pitch corresponding to each word in the sheet music.

[0004] However, the fundamental frequencies extracted by the above methods generally lack detailed information, such as the jitter or intonation unique to the real fundamental frequency, resulting in poor accuracy of the obtained fundamental frequencies. Summary of the Invention

[0005] Therefore, it is necessary to provide a fundamental frequency generation method, computer device, and computer-readable storage medium that can improve the accuracy of fundamental frequency extraction, in order to address the above-mentioned technical problems.

[0006] Firstly, this application provides a fundamental frequency generation method. The method includes:

[0007] Obtain the phonemes corresponding to each lyric in the song, as well as the order information between multiple phonemes;

[0008] Based on the pitch, pitch duration, and legato information of each phoneme in the score of the song, the encoding information of each phoneme is generated;

[0009] The encoded information of the multiple phonemes is input into the trained generative model according to the order information. The generative model determines the predicted fundamental frequency features of multiple frames based on the encoded information of each phoneme and the encoded information of the context phonemes of each phoneme. The context phonemes of each phoneme are determined based on the order information between the multiple phonemes. The total duration of the multiple frames corresponds to the duration of the song.

[0010] The pitch of each frame in the plurality of frames is obtained, and the fundamental frequency of each frame is determined based on the pitch of each frame and the predicted fundamental frequency feature. The fundamental frequency of the song is obtained based on the fundamental frequencies of the plurality of frames.

[0011] In one embodiment, the step of determining the predicted fundamental frequency features of multiple frames by the generation model based on the encoding information of each phoneme and the encoding information of the context phonemes of each phoneme includes:

[0012] The generation model determines the phoneme-level fundamental frequency feature of each phoneme based on the encoding information of each phoneme and the encoding information of the context phonemes of each phoneme;

[0013] The timbre characteristics of the target user are obtained, and the phoneme-level fundamental frequency characteristics of each phoneme are fused with the timbre characteristics of the target user to obtain the fused phoneme-level fundamental frequency characteristics of each phoneme.

[0014] Based on the phoneme duration of each phoneme, the fused phoneme-level fundamental frequency features of each phoneme are mapped to multiple frame-level fundamental frequency features.

[0015] Based on the frame-level fundamental frequency features corresponding to multiple phonemes, the predicted fundamental frequency features of multiple frames are obtained.

[0016] In one embodiment, the generative model includes multiple self-attention modules, which are used to extract features of different dimensions from the encoded information of the multiple phonemes.

[0017] The step of determining the phoneme-level fundamental frequency feature of each phoneme by the generation model based on the encoding information of each phoneme and the encoding information of the context phonemes of each phoneme includes:

[0018] The generative model performs self-attention calculation on the encoded information of each phoneme input to each self-attention module, as well as the encoded information of the context phonemes of each phoneme, to obtain the feature extraction result of each self-attention module in the corresponding dimension.

[0019] Based on the feature extraction results of each self-attention module, the phoneme-level fundamental frequency feature corresponding to each phoneme is determined.

[0020] In one embodiment, mapping the fused phoneme-level fundamental frequency features of each phoneme to multiple frame-level fundamental frequency features based on the phoneme duration of each phoneme includes:

[0021] The number of frames for each phoneme is determined based on the preset frame duration and the phoneme duration of each phoneme.

[0022] Based on the number of frames for each phoneme, the fused phoneme-level fundamental frequency features of each phoneme are copied to obtain multiple frame-level fundamental frequency features corresponding to each phoneme.

[0023] In one embodiment, the generative model further includes a duration warping network and a residual network connected after the duration warping network, wherein the fused phoneme-level fundamental frequency features of each phoneme are mapped to multiple frame-level fundamental frequency features through the duration warping network.

[0024] The step of obtaining predicted fundamental frequency features for multiple frames based on frame-level fundamental frequency features corresponding to multiple phonemes includes:

[0025] The frame-level fundamental frequency features corresponding to the multiple phonemes output by the duration warping network are input into the residual network. The residual network then sequentially performs feature transfer from the first residual block to the last residual block on the frame-level fundamental frequency features corresponding to each phoneme, thereby obtaining the predicted fundamental frequency features of multiple frames.

[0026] In one embodiment, obtaining the pitch of each of the plurality of frames includes:

[0027] The pitch and duration of each note in the song are obtained from the musical score.

[0028] Based on the pitch duration of each note, the pitch of each note is mapped to the pitch of multiple frames.

[0029] In one embodiment, determining the fundamental frequency of each frame based on the pitch of each frame and the predicted fundamental frequency feature includes:

[0030] The initial fundamental frequency of each frame is determined based on the pitch of each frame;

[0031] The initial fundamental frequency of each frame is adjusted based on the predicted fundamental frequency characteristics of each frame to obtain the fundamental frequency of each frame.

[0032] In one embodiment, the generative model is trained through the following steps:

[0033] Obtain the sample phonemes corresponding to each lyric of the sample song;

[0034] Based on the pitch, pitch duration, and legato information of each sample phoneme in the score of the sample song, the encoding information of each sample phoneme is generated;

[0035] A generator and discriminator to be trained are obtained, and the encoding information of the multiple sample phonemes is input into the generator according to the order information between the multiple sample phonemes. The generator determines the sample prediction fundamental frequency features of multiple frames based on the encoding information of each sample phoneme and the encoding information of the context sample phonemes of each sample phoneme. The context sample phonemes of each sample phoneme are determined based on the order information between the multiple sample phonemes, and the total duration of the multiple frames corresponds to the duration of the song sample.

[0036] The pitch of each frame in the plurality of frames corresponding to the sample song is obtained. Based on the pitch of each frame corresponding to the sample song and the sample predicted fundamental frequency feature, the fundamental frequency of each frame is determined. Based on the fundamental frequencies of the plurality of frames, the predicted fundamental frequency of the sample song is obtained.

[0037] Based on the predicted fundamental frequency and the fundamental frequency label of the sample song, the generator and discriminator are subjected to adversarial training, and when the training termination condition is met, the generator is determined as the generative model.

[0038] In one embodiment, the step of performing adversarial training on the generator and discriminator based on the predicted fundamental frequency and the fundamental frequency label of the sample song includes:

[0039] The predicted fundamental frequency of the sample song is input into the current discriminator, and the current discriminator obtains the predicted probability that the predicted fundamental frequency is the true fundamental frequency;

[0040] The current discriminator is adjusted based on the predicted probability; and

[0041] Based on the predicted probability, the predicted fundamental frequency, and the fundamental frequency label, the loss value of the current generator is determined, and the model parameters of the current generator are adjusted according to the loss value.

[0042] In one embodiment, the step of inputting the predicted fundamental frequency of the sample song into the current discriminator, and having the current discriminator obtain the predicted probability that the predicted fundamental frequency is the true fundamental frequency, includes:

[0043] The predicted fundamental frequency of the sample song is input into the current discriminator, which randomly extracts a target fundamental frequency of a preset length from the predicted fundamental frequency and transforms the target fundamental frequency into a two-dimensional matrix with various different structures.

[0044] Each two-dimensional matrix is ​​input into its corresponding sub-discriminator, which then obtains the sub-probability that the predicted fundamental frequency is the true fundamental frequency.

[0045] Based on the sub-probabilities output by each sub-discriminator, the predicted probability that the predicted fundamental frequency of the sample song is the true fundamental frequency is obtained.

[0046] Secondly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0047] Obtain the phonemes corresponding to each lyric in the song, as well as the order information between multiple phonemes;

[0048] Based on the pitch, pitch duration, and legato information of each phoneme in the score of the song, the encoding information of each phoneme is generated;

[0049] The encoded information of the multiple phonemes is input into the trained generative model according to the order information. The generative model determines the predicted fundamental frequency features of multiple frames based on the encoded information of each phoneme and the encoded information of the context phonemes of each phoneme. The context phonemes of each phoneme are determined based on the order information between the multiple phonemes. The total duration of the multiple frames corresponds to the duration of the song.

[0050] The pitch of each frame in the plurality of frames is obtained, and the fundamental frequency of each frame is determined based on the pitch of each frame and the predicted fundamental frequency feature. The fundamental frequency of the song is obtained based on the fundamental frequencies of the plurality of frames.

[0051] Thirdly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0052] Obtain the phonemes corresponding to each lyric in the song, as well as the order information between multiple phonemes;

[0053] Based on the pitch, pitch duration, and legato information of each phoneme in the score of the song, the encoding information of each phoneme is generated;

[0054] The encoded information of the multiple phonemes is input into the trained generative model according to the order information. The generative model determines the predicted fundamental frequency features of multiple frames based on the encoded information of each phoneme and the encoded information of the context phonemes of each phoneme. The context phonemes of each phoneme are determined based on the order information between the multiple phonemes. The total duration of the multiple frames corresponds to the duration of the song.

[0055] The pitch of each frame in the plurality of frames is obtained, and the fundamental frequency of each frame is determined based on the pitch of each frame and the predicted fundamental frequency feature. The fundamental frequency of the song is obtained based on the fundamental frequencies of the plurality of frames.

[0056] Fourthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0057] Obtain the phonemes corresponding to each lyric in the song, as well as the order information between multiple phonemes;

[0058] Based on the pitch, pitch duration, and legato information of each phoneme in the score of the song, the encoding information of each phoneme is generated;

[0059] The encoded information of the multiple phonemes is input into the trained generative model according to the order information. The generative model determines the predicted fundamental frequency features of multiple frames based on the encoded information of each phoneme and the encoded information of the context phonemes of each phoneme. The context phonemes of each phoneme are determined based on the order information between the multiple phonemes. The total duration of the multiple frames corresponds to the duration of the song.

[0060] The pitch of each frame in the plurality of frames is obtained, and the fundamental frequency of each frame is determined based on the pitch of each frame and the predicted fundamental frequency feature. The fundamental frequency of the song is obtained based on the fundamental frequencies of the plurality of frames.

[0061] The aforementioned fundamental frequency generation method, computer device, and computer-readable storage medium can acquire the phonemes corresponding to each lyric of a song, as well as the order information between multiple phonemes. Based on the pitch, pitch duration, and legato information of each phoneme in the song's score, the method generates encoding information for each phoneme. Furthermore, the encoding information of multiple phonemes can be input into a trained generative model according to the order information. The generative model then determines the predicted fundamental frequency features of multiple frames based on the encoding information of each phoneme and the encoding information of its context phonemes. The context phonemes of each phoneme are determined based on the order information between multiple phonemes, and the total duration of the multiple frames corresponds to the duration of the song. The method also acquires the pitch of each frame within the multiple frames, determines the fundamental frequency of each frame based on its pitch and the predicted fundamental frequency features, and obtains the fundamental frequency of the song based on the fundamental frequencies of the multiple frames. In this application, on the one hand, the predicted fundamental frequency features are generated based on the encoding information of phonemes and phoneme context, so that the predicted fundamental frequency features can be affected by the context phonemes, which increases the correlation between the predicted fundamental frequency features and the context phonemes, avoids analyzing the fundamental frequency of each phoneme in isolation, and effectively obtains accurate detailed information of the fundamental frequency. On the other hand, by obtaining the predicted fundamental frequency features of multiple frames and combining the pitch of each frame in multiple frames to determine the fundamental frequency of each frame, the granularity of the fundamental frequency analysis is refined, which can supplement more detailed information of the fundamental frequency of the song as a whole, thereby improving the accuracy of the fundamental frequency extraction results and making the final predicted fundamental frequency closer to the real fundamental frequency curve when the song is sung. Attached Figure Description

[0062] Figure 1 This is a flowchart illustrating a fundamental frequency generation method in one embodiment;

[0063] Figure 2a This is a schematic diagram of a predicted fundamental frequency in one embodiment;

[0064] Figure 2b This is a schematic diagram of another method for predicting the fundamental frequency in one embodiment;

[0065] Figure 3 This is a flowchart illustrating one step in obtaining predicted fundamental frequency features in one embodiment;

[0066] Figure 4 This is a schematic diagram of the structure of a residual network in one embodiment;

[0067] Figure 5 This is a schematic diagram of a training and generation model in one embodiment;

[0068] Figure 6 This is a schematic diagram of the structure of a generator in one embodiment;

[0069] Figure 7 This is a schematic diagram of the structure of a discriminator in one embodiment;

[0070] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0072] To enable those skilled in the art to better understand this application, the following describes the methods for obtaining the fundamental frequency in related technologies. In related technologies, when synthesizing the audio of a target song, the audio of the target song sung by other users can be obtained as reference audio, and the fundamental frequency can be extracted from the reference audio using a fundamental frequency extraction tool; alternatively, the sheet music of the target song can be obtained, and the fundamental frequency can be predicted by using the pitch corresponding to each word in the sheet music.

[0073] However, in the first method, if the reference audio contains accompaniment, the fundamental frequency is difficult to extract accurately. Even if the reference audio does not contain accompaniment, the extracted fundamental frequency heavily depends on the singing level of the user performing the reference audio, and the final extracted fundamental frequency is often too flat. The fundamental frequency obtained by the second method generally lacks the details such as jitter and intonation unique to the true fundamental frequency, especially in the case of long notes (such as a single phoneme lasting more than 2 seconds), where the predicted fundamental frequency will be too flat. Based on this, this application provides a fundamental frequency generation method that can obtain a fundamental frequency with rich detailed information, effectively improving the accuracy of fundamental frequency extraction.

[0074] In one embodiment, such as Figure 1 As shown, a baseband generation method is provided. This embodiment uses the application of this method to a server as an example for illustration. It can be understood that this method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and can be implemented through the interaction between the terminal and the server.

[0075] The server can be implemented using a standalone server or a server cluster consisting of multiple servers; the server can have a data storage system, which can be used to store the data that the server needs to process, such as songs whose base frequencies are to be generated. The data storage system can be integrated on the server or placed in the cloud or other network servers.

[0076] Among them, the terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc.; portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc.

[0077] In this embodiment, the method includes the following steps:

[0078] S101, obtain the phonemes corresponding to each lyric of the song, as well as the order information between multiple phonemes.

[0079] Among them, a phoneme is the smallest unit of speech that is divided according to the natural attributes of speech. It is based on the articulation action in a syllable, and one articulation action can correspond to one phoneme.

[0080] A song can be a complete song or a segment of a complete song, such as one or more lines of lyrics.

[0081] Lyrics may include characters written in actual text, or characters marked by preset symbols, such as characters expressed by ellipses or symbols indicating "repetition".

[0082] The order information between multiple phonemes can be determined based on the singing order of multiple phonemes.

[0083] In practical applications, responding to a baseband extraction request allows us to determine the song for which the request is directed. Specifically, for example, a terminal or other server can request the baseband of a specified song and send a baseband extraction request to the server. The terminal or server can then perform subsequent processing based on the baseband returned by the server, such as song synthesis, spectrum prediction, or model training. Alternatively, a terminal or other server can send a song synthesis request to the server. Upon receiving this request, the server can consider it as having received a baseband extraction request for the specified song.

[0084] After determining the song for which the fundamental frequency extraction request is being made, the lyrics of the song can be obtained. These lyrics can include Chinese lyrics or foreign language lyrics, such as English, Japanese, French, etc. Then, for the obtained lyrics, the phonemes of each character in the lyrics and the order information between multiple phonemes can be determined.

[0085] S102, Generate the encoding information of each phoneme based on the pitch, pitch duration and legato information of each phoneme in the song's score.

[0086] The legato information can indicate the application of legato techniques when singing corresponding lyrics. For example, it can include at least one of the following information: whether legato is used, the start position of the legato, the end position of the legato, and the duration of the legato.

[0087] Encoded information can be the encoded result obtained after encoding processing. In one example, the encoded information can be a multi-dimensional vector generated through an embedding layer.

[0088] In practice, the sheet music of a song can be obtained. The sheet music can record various types of information, such as notes, tuplets, accidentals, fermas, repeat signs, etc. Based on the sheet music, the pitch, duration, and legato information of each phoneme in the score can be determined.

[0089] Specifically, for example, the pitch at the character level can be obtained from the notes in the musical score. For each character, the pitch of that character can be used as the pitch of the phoneme that makes up that character. Or, after determining the pitch of the phoneme, the duration of the pitch can be determined based on the number of beats of the notes in the musical score. Regarding the legato information, the legato method when singing can be determined based on the legato notes in the musical score. For example, the legato information can indicate whether the corresponding phoneme is performed by legato singing. In addition, the transition information can be obtained from the musical score of the song, which can indicate whether the corresponding phoneme is performed by transition singing.

[0090] Then, each phoneme and its pitch, pitch duration, and liaison information can be encoded to obtain the pitch encoding information, pitch duration encoding information, and liaison encoding information of each phoneme. Furthermore, the encoding information of each phoneme can be generated based on the pitch encoding information, pitch duration encoding information, and liaison encoding information of each phoneme.

[0091] S103, the encoded information of multiple phonemes is input into the trained generative model according to the sequential information. The generative model determines the predicted fundamental frequency features of multiple frames based on the encoded information of each phoneme and the encoded information of the context phonemes of each phoneme.

[0092] In this context, the context phonemes of each phoneme are determined based on the order information between multiple phonemes, and the total duration of multiple frames corresponds to the duration of the song.

[0093] In one example, the predicted fundamental frequency features can characterize the variation or jitter features of the predicted fundamental frequency, such as features that characterize details such as fundamental frequency jitter or intonation.

[0094] In practice, the way a user's voice changes varies when singing songs with different content. For example, lyrics with a gentle pitch change (such as "The evening breeze gently caresses Penghu Bay" in "Penghu Bay") differ from lyrics with a significant pitch change (such as "A great river with wide waves" in "My Motherland"). Furthermore, whether legato, trills, or falsetto are used when singing the same lyrics will also affect the final performance and the fundamental frequency change of the song being sung. It's understandable that in reality, the fundamental frequency of a user singing a song is not flat but changes accordingly based on the content being sung.

[0095] In this embodiment, after obtaining the encoded information that can represent the pitch, pitch duration, and liaison information of a phoneme, the encoded information of multiple phonemes can be sequentially input into the trained generative model according to the order information between multiple phonemes.

[0096] In an optional embodiment, the phoneme, pitch, pitch duration, and liaison information can be converted into a multidimensional vector with length M through an embedding layer. Then, the encoding information of T phonemes can be sequentially formed into a vector sequence of length T. When inputting the encoding information into the generative model, this vector sequence can be used as the input content into the generative model.

[0097] After obtaining the encoded information of multiple phonemes input in sequence, the generation model can determine the context phoneme of each phoneme based on the input order of the encoded information, and obtain the encoded information of each context phoneme. Then, it can combine the encoded information of each phoneme and the encoded information of the context phonemes to determine the predicted fundamental frequency features of multiple frames. It can be understood that in this step, by generating predicted fundamental frequency features based on the encoded information of phonemes and their context, the predicted fundamental frequency features of each frame are not only determined based on the encoded information of the corresponding phoneme (i.e., the phoneme to which each frame belongs), but also influenced by the context phonemes before and after that phoneme, making the predicted fundamental frequency features closer to the fundamental frequency features of the real fundamental frequency (the fundamental frequency of a real human voice singing a song).

[0098] S104, obtain the pitch of each frame in multiple frames, determine the fundamental frequency of each frame based on the pitch of each frame and the predicted fundamental frequency characteristics, and obtain the fundamental frequency of the song based on the fundamental frequencies of multiple frames.

[0099] In practical applications, vowels and voiced consonants in speech have corresponding fundamental frequencies, reflecting the frequency of vocal cord vibration, generally ranging from 100 Hz to 400 Hz. The sheet music for each song reflects the pitch of each word when sung, and the unit of pitch is also Hertz (Hz). There is a correspondence between pitch and fundamental frequency; pitch can be determined by the fundamental frequency of the sound. However, the fundamental frequency calculated directly from pitch lacks variation and differs from the fluctuating or melodic fundamental frequency produced when a user sings. It suffers from being too rigid and flat, lacking the detailed information possessed by a true fundamental frequency.

[0100] In this step, in addition to obtaining the predicted fundamental frequency features of multiple frames, the pitch of multiple frames can also be obtained. In one embodiment, the frame-level pitch can be obtained from the song's score, thereby obtaining the pitch of each frame in multiple frames. Since the predicted fundamental frequency features can characterize the variation characteristics of the fundamental frequency of the corresponding frame, and the pitch of the corresponding frame can determine the fundamental frequency corresponding to that pitch, the fundamental frequency can be predicted based on the pitch of each frame, combined with the predicted fundamental frequency features of each frame, to determine the fundamental frequency of each frame. Based on the fundamental frequencies of multiple frames, the fundamental frequency of the song can be obtained. For example, the fundamental frequencies of multiple frames can be concatenated to obtain the fundamental frequency of the song. Figure 2a and Figure 2b Each of the predicted fundamental frequencies is shown in part, from Figure 2a and Figure 2b It can be seen that the predicted fundamental frequency contains more jitter and is closer to the true fundamental frequency curve.

[0101] In the aforementioned fundamental frequency generation method, the phonemes corresponding to each lyric of the song, as well as the order information between multiple phonemes, can be obtained. Based on the pitch, pitch duration, and legato information of each phoneme in the song's score, encoding information for each phoneme is generated. Then, the encoding information of multiple phonemes can be input into a trained generation model according to the order information. The generation model determines the predicted fundamental frequency features of multiple frames based on the encoding information of each phoneme and the encoding information of its context phonemes. The context phonemes of each phoneme are determined based on the order information between multiple phonemes, and the total duration of the multiple frames corresponds to the duration of the song. Furthermore, the pitch of each frame in the multiple frames can be obtained, and then the fundamental frequency of each frame is determined based on the pitch and predicted fundamental frequency features of each frame. Finally, the fundamental frequency of the song is obtained based on the fundamental frequencies of the multiple frames. In this application, on the one hand, the predicted fundamental frequency features are generated based on the encoding information of phonemes and phoneme context, so that the predicted fundamental frequency features can be affected by the context phonemes, which increases the correlation between the predicted fundamental frequency features and the context phonemes, avoids analyzing the fundamental frequency of each phoneme in isolation, and effectively obtains accurate detailed information of the fundamental frequency. On the other hand, by obtaining the predicted fundamental frequency features of multiple frames and combining the pitch of each frame in multiple frames to determine the fundamental frequency of each frame, the granularity of the fundamental frequency analysis is refined, which can supplement more detailed information of the fundamental frequency of the song as a whole, thereby improving the accuracy of the fundamental frequency extraction results and making the final predicted fundamental frequency closer to the real fundamental frequency curve when the song is sung.

[0102] In one embodiment, such as Figure 3 As shown, in S103, the generative model determines the predicted fundamental frequency features of multiple frames based on the encoding information of each phoneme and the encoding information of the context phonemes of each phoneme. This can include the following steps:

[0103] S301, the generative model determines the phoneme-level fundamental frequency feature of each phoneme based on the encoding information of each phoneme and the encoding information of the context phonemes of each phoneme.

[0104] After inputting the encoded information of multiple phonemes into the generative model, the model can perform feature extraction by combining the encoded information of each phoneme with the encoded information of its context phonemes. Based on the feature extraction results, the phoneme-level fundamental frequency feature is obtained for each phoneme. The phoneme-level fundamental frequency feature can characterize the variation features of the fundamental frequency corresponding to a phoneme, and one phoneme-level fundamental frequency feature can correspond to one phoneme.

[0105] S302, acquire the timbre characteristics of the target user, and fuse the phoneme-level fundamental frequency characteristics of each phoneme with the timbre characteristics of the target user to obtain the fused phoneme-level fundamental frequency characteristics of each phoneme.

[0106] As an example, the target user can be the user whose voice is to be synthesized. The target user can request the synthesis of a song with the characteristics of the target user's voice, so that the singing voice of the target user can be simulated even if the target user does not sing.

[0107] In practical applications, the timbre characteristics of the target user can be obtained, and then the phoneme-level fundamental frequency characteristics of each phoneme can be fused with the timbre characteristics of the target user to obtain the fused phoneme-level fundamental frequency characteristics of each phoneme. For example, the phoneme-level fundamental frequency characteristics of each phoneme can be spliced ​​with the timbre characteristics of the target user, and the spliced ​​features can be used as the fused phoneme-level fundamental frequency characteristics.

[0108] S303 maps the fused phoneme-level fundamental frequency features of each phoneme to multiple frame-level fundamental frequency features based on the phoneme duration of each phoneme.

[0109] In practical applications, the duration of a phoneme is often much longer than the duration of a frame. If fundamental frequency prediction is performed directly based on phoneme-level fundamental frequency features, the temporal granularity will be large, resulting in a coarse predicted fundamental frequency. In this step, after obtaining the fused phoneme-level fundamental frequency features, the temporal granularity associated with the fundamental frequency features can be adjusted. Based on the phoneme duration of each phoneme, the phoneme-level fundamental frequency features are mapped to multiple frame-level fundamental frequency features.

[0110] S304 obtains the predicted fundamental frequency features of multiple frames based on the frame-level fundamental frequency features corresponding to multiple phonemes.

[0111] After obtaining the fundamental frequency features at multiple frame levels corresponding to each phoneme, the predicted fundamental frequency features of multiple frames can be obtained based on the fundamental frequency features at multiple frame levels corresponding to each phoneme.

[0112] In this embodiment, on the one hand, the timbre features of the target user can be fused with the phoneme-level fundamental frequency features to improve the matching between the final predicted fundamental frequency and the target user's voice. On the other hand, by mapping the phoneme-level fundamental frequency features to multiple frame-level fundamental frequency features, the granularity of the fundamental frequency features can be refined, providing a basis for determining the fundamental frequency of each frame in the future.

[0113] In one embodiment, the generative model includes multiple self-attention modules, which are used to extract features of different dimensions from the encoded information of multiple phonemes. In one example, the self-attention module can be a Feed-Forward Transformer block (FFT module), which is a feedforward network based on a self-attention mechanism. It is understood that a network based on a self-attention mechanism for feature extraction can also be used, depending on the specific circumstances. Step S301, in which the generative model determines the phoneme-level fundamental frequency feature of each phoneme based on the encoded information of each phoneme and the encoded information of the context phonemes of each phoneme, may include the following steps:

[0114] The generative model performs self-attention calculations on the encoded information of each phoneme input to each self-attention module, as well as the encoded information of the context phonemes of each phoneme, to obtain the feature extraction results of each self-attention module in the corresponding dimension; based on the feature extraction results of each self-attention module, the phoneme-level fundamental frequency features corresponding to each phoneme are determined.

[0115] In practice, after inputting the encoded information of multiple phonemes into the generation model in sequence, the encoded information of the phonemes can be input into the self-attention module. For each of the multiple self-attention modules, the self-attention module can perform self-attention calculation on the encoded information of each phoneme input into the module and the encoded information of the context phonemes of each phoneme, to obtain the feature extraction result of the self-attention module.

[0116] In an optional embodiment, the multiple self-attention modules can be cascaded, that is, multiple self-attention modules can be connected in series. The encoding information of multiple phonemes input to the first self-attention module can be the encoding information generated based on pitch, pitch duration and legato information. The encoding information of multiple phonemes input to other self-attention modules can be the feature extraction results obtained after feature extraction by the previous self-attention module. The feature extraction results of each self-attention module can be sequentially transferred from the first self-attention module to the last self-attention module, and the phoneme-level fundamental frequency feature corresponding to each phoneme is obtained based on the feature extraction result output by the last self-attention module.

[0117] In another optional embodiment, the multiple self-attention modules can also be multiple self-attention modules connected in parallel. For example, multiple self-attention modules can be connected in parallel based on a multi-head self-attention mechanism, so that the feature extraction results output by each self-attention module can be obtained.

[0118] In this embodiment, by using multiple self-attention modules to perform self-attention calculations on the encoding information of each phoneme and the encoding information of the context phonemes, it is possible to extract features from the encoding information of the phonemes and the encoding information of the context phonemes in different dimensions, thereby enhancing the expressive power of the generation model and the final extracted phoneme-level fundamental frequency features.

[0119] In one embodiment, S303 maps the fused phoneme-level fundamental frequency features of each phoneme to multiple frame-level fundamental frequency features based on the phoneme duration of each phoneme, which may include the following steps:

[0120] Based on the preset frame duration and the phoneme duration of each phoneme, the number of frames for each phoneme is determined; according to the number of frames for each phoneme, the fused phoneme-level fundamental frequency features of each phoneme are copied to obtain multiple frame-level fundamental frequency features corresponding to each phoneme.

[0121] In practical implementation, the preset frame duration can be determined based on the duration of a single frame. Since the duration of a phoneme is often longer than the duration of a frame, after determining the preset frame duration, the preset frame duration covered by each phoneme's duration can be determined. For example, the ratio of the phoneme's duration to the preset frame duration can be determined to obtain the number of frames for each phoneme. Then, based on the number of frames for each phoneme, the fused phoneme-level fundamental frequency features of each phoneme can be copied. For example, if the duration of a phoneme is 5 frames, and the corresponding phoneme-level fundamental frequency feature is vector F, this vector can be copied 5 times, thus obtaining multiple frame-level fundamental frequency features for each phoneme. Correspondingly, the length of the vector sequence output by the generation model will change from the original number of phonemes T to the number of frames P corresponding to the song, with the total duration covered by the number of frames P corresponding to the song's duration.

[0122] In this embodiment, by copying the phoneme-level fundamental frequency features after the fusion of each phoneme, multiple frame-level fundamental frequency features corresponding to each phoneme can be obtained quickly.

[0123] In one embodiment, the generative model further includes a duration warping network and a residual network connected after the duration warping network. The duration warping network can be used to establish mapping relationships between features at different temporal granularities. To facilitate differentiation from other duration warping networks, the duration warping network in this embodiment can also be referred to as the first duration warping network. In practical applications, after obtaining the phoneme-level fundamental frequency features, the fused phoneme-level fundamental frequency features of each phoneme can be input into the first duration warping network. The first duration warping network then maps the fused phoneme-level fundamental frequency features of each phoneme into multiple frame-level fundamental frequency features based on the phoneme duration of each phoneme.

[0124] Residual networks, also known as Residual Neural Networks (ResNet), can be understood as convolutional neural networks with added residual blocks. In generative models, the residual network can include multiple residual blocks. Specifically, each residual block consists of a direct mapping part and a residual part. The residual part can contain at least one convolutional layer. For the input x to the residual block... l The input x can be used l The inputs are processed separately into the direct mapping part and the residual part. The residual part can modify the input x. l Performing a convolution operation yields F(x), while the direct mapping part can convert x... l The output can be directly obtained, and thus can be based on the convolution result F(x) of the residual part and the output x of the direct mapping part. l We obtain F(x) + x l The structure of residual blocks allows the input of previous layers to be introduced into the current layer. By cascading multiple residual blocks in the generative model, the network depth can be increased, improving the performance of the generative network in learning fundamental frequency jitter features and outputting accurate results. At the same time, the problem of gradient vanishing or gradient exploding can be avoided during the process of deepening the network.

[0125] The step of obtaining the predicted fundamental frequency features of multiple frames based on the frame-level fundamental frequency features corresponding to multiple phonemes may include the following steps:

[0126] The frame-level fundamental frequency features corresponding to multiple phonemes output by the duration warping network are input into the residual network. The residual network then sequentially passes the frame-level fundamental frequency features corresponding to each phoneme from the first residual block to the last residual block, thereby obtaining the predicted fundamental frequency features of multiple frames.

[0127] After obtaining the frame-level fundamental frequency features corresponding to each phoneme, the multi-frame-level fundamental frequency features corresponding to each phoneme can be input into the residual network. This allows the frame-level fundamental frequency features corresponding to each phoneme to be transferred from the first residual block to the last residual block in multiple cascaded residual blocks of the residual network. Based on the output of the residual network, the fundamental frequency jitter features of multiple frames can be obtained.

[0128] Compared to traditional methods that output the speaker's timbre features at the very beginning of the model (e.g., inputting and immediately processing the timbre features at the outset), this embodiment fuses the timbre features with phoneme-level fundamental frequency features and maps them to frame-level fundamental frequency features. Then, the frame-level fundamental frequency features are input into the residual network. In other words, the timbre features are added to the generative model after obtaining the phoneme-level fundamental frequency features but before outputting the final predicted fundamental frequency features. This processing stage is closer to the stage where the generative model outputs its results; in other words, it is closer to the final output of the generative model (i.e., the predicted fundamental frequency features). By introducing timbre features at this stage before inputting them into the residual network, the speaker's timbre features can more effectively influence the final predicted fundamental frequency features, making the predicted fundamental frequency more consistent with the speaker's vocal characteristics.

[0129] In one embodiment, when the residual blocks in the residual network perform convolution operations on the input, they can be implemented by dilated convolution, which can also be called dilated convolution or dilated convolution. It can be understood as the process of adding some spaces (zeros) between the elements of the convolution kernel to expand the original convolution kernel.

[0130] In some alternative embodiments, the residual network can be structured as follows: Figure 4 As shown, the residual network can include six cascaded residual blocks. The multi-frame spliced ​​features corresponding to multiple phonemes are input into the residual network, then passed through fully connected layers and activation functions before being input into the residual blocks (only one residual block is shown in the figure; the other cascaded residual blocks are not shown). After the multi-frame spliced ​​features have passed through multiple residual blocks for feature propagation, the features output from the last residual block can be convolved, and the resulting features are used as the fundamental frequency jitter features for multiple frames.

[0131] In residual networks, one-dimensional convolution can be performed using the convolution function Conv1d(channel, kernel size, dilation), where channel is the number of channels, kernel size is the kernel size, and dilation is the dilation rate of the dilated convolution. The dilation rate is positively correlated with the kernel size. For example, the dilation rate in different residual blocks of a residual network can be different, and different dilation rates can also be used in the same residual block for different dilated convolutions. For example, the dilation rate used in each residual block of a residual network can be selected from the following values: 1, 3, 5, 7, 9, 11. In some examples, when the kernel size of Conv1d is 13 and dilated convolution is combined, each vector output by the residual network (i.e., the spliced ​​features of each frame) is affected by 516 vectors before and after it. Each vector corresponds to one frame. In practice, a typical frame length is 16ms. Therefore, the frame can be affected by about 8.3s of data before and after it (16ms*516≈8.3s). Since the long notes in a song are generally 2 to 6s, the receptive field of the residual network is sufficient to meet this duration requirement.

[0132] In this embodiment, by using dilated convolution to process the input features in the residual network, the receptive field of the generative model for the features can be effectively expanded, and the generative model's ability to model the contextual information of each phoneme can be improved. This allows the fundamental frequency jitter feature of each frame to be affected by the contextual information of the frames before and after it, effectively obtaining the detailed information of the fundamental frequency of each frame.

[0133] In one embodiment, obtaining the pitch of each frame in a plurality of frames may include the following steps:

[0134] The pitch and duration of each note in the song are obtained from the song's score; based on the duration of each note, the pitch of each note is mapped to the pitch of multiple frames.

[0135] As an example, pitch duration can be the duration of a pitch.

[0136] In practical applications, the sheet music of a song can be obtained, and based on the melody recorded in the sheet music, the pitch sequence of the song can be obtained. This pitch sequence can include multiple pitches arranged sequentially in the sheet music. Furthermore, the duration of each pitch in the pitch sequence can be determined based on the sheet music; for example, the duration of a pitch can be determined based on the number of beats corresponding to each pitch in the sheet music.

[0137] After obtaining the pitch duration of each pitch, the pitch of each note can be mapped to the pitch of multiple frames based on the preset frame duration and the pitch duration of each note. In one embodiment, the pitch mapping between notes and frames can be achieved through a duration warping network. For ease of distinction, the duration warping network in this embodiment can be referred to as the second duration warping network. Specifically, each pitch and its pitch duration can be input into the second duration warping network, which then maps the pitch of each note to the pitch of multiple frames based on the pitch duration of each note.

[0138] In this embodiment, by mapping the pitch of each note to the pitch of multiple frames based on the pitch duration of each note, the frame-level pitch can be predicted in detail, providing a basis for the subsequent generation of fundamental frequencies with more detailed information.

[0139] In one embodiment, determining the fundamental frequency of each frame based on the pitch and predicted fundamental frequency characteristics of each frame in step S104 may include the following steps:

[0140] The initial fundamental frequency of each frame is determined based on the pitch of each frame; the initial fundamental frequency of each frame is adjusted based on the predicted fundamental frequency characteristics of each frame to obtain the fundamental frequency of each frame.

[0141] In practice, there is a correspondence between the pitch of a sound and its fundamental frequency. After obtaining the pitches of multiple frames, the fundamental frequency corresponding to the pitch of each frame can be used as the initial fundamental frequency of that frame. Then, for the initial fundamental frequency of each frame, the predicted fundamental frequency features output by the generative model for that frame can be obtained, and the initial fundamental frequency of the frame can be adjusted according to the predicted fundamental frequency features. For example, the amplitude, frequency, or number of jitters in the initial fundamental frequency can be adjusted according to the predicted fundamental frequency features; and the adjustment result of each frame can then be used as the fundamental frequency of each frame.

[0142] In this embodiment, by adjusting the initial fundamental frequency of each frame based on the predicted fundamental frequency features of each frame, the detailed information of the predicted fundamental frequency can be significantly enriched, the similarity between the predicted fundamental frequency and the true fundamental frequency can be improved, and the accurate fundamental frequency can be obtained.

[0143] In one embodiment, the generative model is trained through the following steps:

[0144] S401, Obtain the sample phonemes corresponding to each lyric of the sample song.

[0145] The sample songs can be songs used for model training. For example, each or more lines of lyrics in a sample song can be used as a training data point.

[0146] Specifically, the lyrics of a sample song can be obtained, and the phonemes of each character in the lyrics can be determined as multiple sample phonemes.

[0147] S402, Generate the encoding information of each sample phoneme based on the pitch, pitch duration and legato information of each sample phoneme in the score of the sample song.

[0148] After determining the sample phonemes of each character in the sample song, the score of the sample song can be further obtained. Specifically, the score of the sample song can record various types of information, such as notes, tuplets, accidentals, fermas, and repeat signs. Based on the score, the pitch, duration, and legato information of each sample phoneme can be determined. Then, the pitch, duration, and legato information of each sample phoneme can be encoded to obtain pitch encoding information, pitch duration encoding information, and legato encoding information for each sample phoneme. Furthermore, based on the pitch encoding information, duration encoding information, and legato encoding information of each sample phoneme, the encoding information for each sample phoneme can be generated.

[0149] S403, obtain the generator and discriminator to be trained, and input the encoding information of the multiple sample phonemes into the generator according to the order information between the multiple sample phonemes. The generator determines the sample prediction fundamental frequency features of multiple frames based on the encoding information of each sample phoneme and the encoding information of the context sample phonemes of each sample phoneme.

[0150] The context sample phoneme for each sample phoneme is determined based on the order information between multiple sample phonemes, and the total duration of multiple frames corresponds to the duration of the song sample.

[0151] In practical applications, models can be trained in various ways. Generative Adversarial Networks (GANs) consist of at least two parts: a generator (Generative Model) and a discriminator (Discriminative Model). As a deep learning model, it can achieve excellent training results through mutual game learning.

[0152] In this step, a generator and a discriminator to be trained can be obtained. The generator can be used to predict fundamental frequency features, and the discriminator can be used to determine the probability that the input fundamental frequency is the true fundamental frequency.

[0153] After obtaining the encoding information of each sample phoneme in multiple sample phonemes, the order information between multiple sample phonemes can be obtained, and the encoding information of multiple sample phonemes is input into the generator to be trained according to the order information. The generator determines the sample prediction fundamental frequency features of multiple frames based on the encoding information of each sample phoneme and the encoding information of the context sample phonemes of each sample phoneme.

[0154] S404: Obtain the pitch of each frame corresponding to the sample song in multiple frames; determine the fundamental frequency of each frame based on the pitch of each frame corresponding to the sample song and the sample predicted fundamental frequency feature; and obtain the predicted fundamental frequency of the sample song based on the fundamental frequencies of multiple frames.

[0155] Furthermore, after acquiring multiple frames, the pitch of each frame in the sample song can be determined based on the score of the sample song. Then, the fundamental frequency of each frame can be determined based on the pitch of each frame corresponding to the sample song and the sample predicted fundamental frequency features of each frame. Finally, the predicted fundamental frequency of the sample song can be obtained by combining the fundamental frequencies of multiple frames.

[0156] The processes of steps S403 and S404 are the same as those of the generative model in obtaining the predicted fundamental frequency features and generating the fundamental frequency of the song based on the predicted fundamental frequency features. For specific processes, please refer to S103, S104 and the related embodiments mentioned above. This embodiment will not elaborate on these processes.

[0157] S405: Based on the predicted fundamental frequency and the fundamental frequency label of the sample song, the generator and discriminator are trained adversarially, and the generator is determined as the generative model when the training termination condition is met.

[0158] In practical applications, after obtaining sample songs, multiple frame-level fundamental frequencies can be extracted from the audio of the sample songs, and fundamental frequency labels for the sample songs can be generated based on the extracted frame-level fundamental frequencies. After obtaining the predicted fundamental frequencies, the generator and discriminator can be adversarially trained based on the predicted fundamental frequencies and the fundamental frequency labels of the sample songs. When the training termination condition is met, the trained generator and discriminator are obtained. Then, the trained generator can be determined as the generative model used subsequently to obtain the predicted fundamental frequency features.

[0159] In this embodiment, by introducing a generative adversarial network, the discriminator can guide the learning of the generative model, making the generated fundamental frequency based on the predicted fundamental frequency features closer to the real fundamental frequency, which helps to restore the detailed information of the real fundamental frequency.

[0160] In one embodiment, S405 performs adversarial training on the generator and discriminator based on the predicted fundamental frequency and the fundamental frequency label of the sample song, which may include the following steps:

[0161] The predicted fundamental frequency of the sample song is input into the current discriminator, which obtains the prediction probability that the predicted fundamental frequency is the true fundamental frequency. The current discriminator is adjusted based on the prediction probability. Based on the prediction probability, the predicted fundamental frequency, and the fundamental frequency label, the loss value of the current generator is determined, and the model parameters of the current generator are adjusted according to the loss value.

[0162] After obtaining the predicted fundamental frequency of the sample song, the predicted fundamental frequency of the sample song can be input into the current discriminator. The current discriminator obtains the prediction probability that the predicted fundamental frequency is the true fundamental frequency. Then, the generator and discriminator can be adjusted according to the prediction probability output by the discriminator.

[0163] Specifically, when adjusting the generator's model parameters, the current discriminator's model parameters can be fixed first, and the generator's loss value can be determined based on the discriminator's output prediction probability, prediction fundamental frequency, and fundamental frequency label. The generator's loss value can include two parts, which can be referred to as the first loss value and the second loss value for ease of distinction: the first loss value can be determined based on the difference between the predicted fundamental frequency and the fundamental frequency label, and the second loss value can be determined based on the difference between the prediction probability and a first preset probability, where the first preset probability indicates that the fundamental frequency is the true fundamental frequency; in one example, the first preset probability is set to 1. The generator's loss function L1 can then be as follows:

[0164] L1 = MSE(predicted fundamental frequency, fundamental frequency label) + MSE(D(predicted fundamental frequency), 1) * w

[0165] Where MSE is the minimum mean squared error loss, D (prediction fundamental frequency) is the prediction probability output by the discriminator, and w is the weighting coefficient.

[0166] After fixing the model parameters of the current discriminator, the above steps can be repeated to adjust the model parameters of the generator multiple times until the switching conditions are met.

[0167] Then, the model parameters of the current generator can be fixed, and the adjustment can be switched to the model parameters of the current discriminator. For the discriminator, the loss value of the discriminator can be determined based on the predicted probability output by the discriminator and the fundamental frequency label. The loss value of the discriminator can include two parts, which can be referred to as the third loss value and the fourth loss value for easy distinction.

[0168] The third loss value can characterize the difference between the discriminator's judgment result and the actual situation when it determines whether the true baseband is real. Specifically, the baseband label can be input into the current discriminator to obtain the discriminator's predicted probability that the baseband label is the true baseband. Then, based on the difference between the predicted probability and the first preset probability, the third loss value is determined.

[0169] The fourth loss value characterizes the difference between the discriminator's judgment result and the actual situation when determining whether the predicted fundamental frequency is false. To determine the fourth loss value, the predicted fundamental frequency can be input into the current discriminator to obtain the prediction probability that the discriminator determines the predicted fundamental frequency to be the true fundamental frequency. Based on the difference between this prediction probability and a second preset probability, the fourth loss value is obtained. The second preset probability indicates that the fundamental frequency is a fake fundamental frequency; in one example, the second preset probability is 0. The discriminator's loss function L2 can then be as follows:

[0170] L1 = MSE(D(baseband tag), 1) * w 1 + MSE(D(predicted baseband), 0) * w2

[0171] Where w1 and w2 are weighting coefficients.

[0172] To enable those skilled in the art to better understand the above steps, the following example illustrates the embodiments of this application, but it should be understood that the embodiments of this application are not limited thereto.

[0173] like Figure 5 As shown, after obtaining the lyrics of the sample song, for each sample phoneme, the encoding information of the sample phoneme can be generated based on the sample phoneme, the pitch of the sample phoneme, the pitch duration, and whether the sample phoneme is connected. This information is then input into the generator to be trained. The generator can determine the fundamental frequency feature of the sample prediction based on the input encoding information.

[0174] In one embodiment, the internal structure of the generator can be as follows: Figure 6 As shown, the sample phoneme, the pitch of the sample phoneme, the pitch duration, and the liaison attribute indicating whether the sample phoneme is liaison can be input into the generator to obtain their respective feature vectors (embedding). The feature vectors of each sample phoneme in each dimension (including sample phoneme, pitch, pitch duration, and liaison attribute) are concatenated to obtain the encoding information of each sample phoneme. The encoding information of multiple sample phonemes can form a vector sequence.

[0175] This vector sequence can then be input into multiple cascaded self-attention modules (e.g., six) to obtain phoneme-level fundamental frequency features. These phoneme-level fundamental frequency features can be fused with the encoded information corresponding to the speaker ID by addition, resulting in fused phoneme-level fundamental frequency features. The encoded information corresponding to the speaker ID can be used to distinguish different speakers. By adding the encoded information corresponding to the speaker ID, the final predicted fundamental frequency can be made more consistent with the voice characteristics of the corresponding speaker. After obtaining the fused phoneme-level fundamental frequency features, a temporal warping network can be used to map the phoneme-level fundamental frequency features into multiple frame-level fundamental frequency features. These features are then input into the residual network inside the generator for feature transfer. The sample predicted fundamental frequency features are obtained based on the output of the residual network.

[0176] like Figure 5 As shown, the pitches of multiple notes in a sample song can also be obtained and input into a duration warping network. This network maps these pitches to multiple frame-level pitches. Then, the frame-level sample predicted fundamental frequency features and the frame-level pitches can be combined to obtain the predicted fundamental frequency of the sample song. In one example, the sample predicted fundamental frequency features can be residuals, representing the amplitude of the fundamental frequency fluctuation. Therefore, the frame-level predicted fundamental frequency can be obtained by adding the frame-level sample predicted fundamental frequency features to the frame-level pitches.

[0177] After obtaining the predicted fundamental frequency of the sample song, the predicted fundamental frequency can be input into the discriminator to obtain the discrimination result. Combined with the fundamental frequency extracted from the training corpus (i.e., the fundamental frequency label mentioned in the above embodiment), the generator is subjected to supervised training. After iterating the generator multiple times, the training switches to the discriminator. Similarly, when training the discriminator, supervised training can also be performed using the fundamental frequency extracted from the training corpus and the predicted fundamental frequency.

[0178] In this embodiment, by alternating the training of the generator and the discriminator, and by determining the loss value of the current generator based on the predicted probability, the predicted fundamental frequency, and the fundamental frequency label, the model parameters of the current generator are adjusted according to the loss value, so that the predicted fundamental frequency features obtained by the generator can gradually approach the fundamental frequency features of the real fundamental frequency, thereby improving the accuracy of fundamental frequency feature extraction.

[0179] In one embodiment, the current discriminator includes multiple sub-discriminators; inputting the predicted fundamental frequency of the sample song into the current discriminator, and having the current discriminator obtain the predicted probability that the predicted fundamental frequency is the true fundamental frequency, may include the following steps:

[0180] The predicted fundamental frequency of the sample song is input into the current discriminator, which randomly extracts a target fundamental frequency of a preset length from the predicted fundamental frequency and transforms the target fundamental frequency into a two-dimensional matrix with various different structures. Each two-dimensional matrix is ​​then input into its corresponding sub-discriminator, which obtains the sub-probability that the predicted fundamental frequency is the true fundamental frequency. Based on the sub-probabilities output by each sub-discriminator, the predicted probability that the predicted fundamental frequency of the sample song is the true fundamental frequency is obtained.

[0181] In practical applications, the predicted fundamental frequency of a sample song can be input into the current discriminator, which then randomly extracts a fundamental frequency segment of a preset length from the predicted fundamental frequency as the target fundamental frequency. For example, for a predicted fundamental frequency of length 200, the fundamental frequency sequence consists of 200 frequency points. During random extraction, a starting point can be randomly generated, and a fundamental frequency segment of a preset length can be extracted from that starting point. If the randomly generated starting point is the 50th frequency point, and the shortest fundamental frequency segment length is 70, then frequencies from the 50th to the 120th can be extracted and then input into each sub-discriminator. Each sub-discriminator can include multiple alternating convolutional layers and activation layers. In one example, the convolutional layers can be implemented using the `conv2d` function, and the activation layers can be implemented using the Leadky ReLU activation function.

[0182] After obtaining the target fundamental frequency, which is a 1-dimensional tensor, in order to improve the generalization ability of the discriminator, the target fundamental frequency can be transformed into a variety of two-dimensional matrices with different structures. These two-dimensional matrices with different structures can refer to two-dimensional matrices with different lengths or widths.

[0183] In one optional embodiment, a 1D tensor can be converted into a 2D matrix using the `reshape(m,n)` function. Before the conversion, the 1D tensor corresponding to the target fundamental frequency can be padded with zeros according to the structure of the 2D matrix to be output. The parameter `m` or `n` in `reshape(m,n)` can be written as "-1", which can be used to instruct the device to automatically calculate the row or column values ​​based on the total number of elements in the original array. For example, if the length of the randomly selected target fundamental frequency sequence is 70, to perform a `reshape(-1,11)` conversion operation, the target fundamental frequency can first be padded with 7 zeros to make its length 77, and then the conversion can be performed to obtain a 2D matrix with a structure of (7,11).

[0184] The discriminator may include multiple sub-discriminators. After obtaining multiple two-dimensional matrices, each sub-discriminator can input each two-dimensional matrix into the sub-discriminator corresponding to the structure of the two-dimensional matrix. The sub-discriminators can obtain the sub-probability that the predicted fundamental frequency is the true fundamental frequency. Based on the sub-probabilities output by each sub-discriminator, the predicted probability that the predicted fundamental frequency of the sample song is the true fundamental frequency can be obtained.

[0185] For example, the structure of the discriminator can be as follows: Figure 7As shown, after obtaining the predicted fundamental frequency, the discriminator can randomly truncate the target fundamental frequency. After adding the corresponding number of zeros according to the reshape(m,n) function used subsequently, it can be converted into a two-dimensional matrix and input into the corresponding sub-discriminator to obtain the sub-probability output by each sub-discriminator. Then, the average of multiple sub-probabilities can be used as the output result of the discriminator to obtain the predicted probability that the predicted fundamental frequency of the sample song is the true fundamental frequency.

[0186] In this embodiment, by randomly sampling the target fundamental frequency and converting it into a two-dimensional matrix with different structures, and having multiple sub-discriminators make judgments respectively, the data augmentation effect can be achieved, improving the generalization ability of the discriminator.

[0187] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0188] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores song data and baseband data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a baseband generation method.

[0189] Those skilled in the art will understand that Figure 8The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0190] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0191] Obtain the phonemes corresponding to each lyric in the song, as well as the order information between multiple phonemes;

[0192] Based on the pitch, pitch duration, and legato information of each phoneme in the score of the song, the encoding information of each phoneme is generated;

[0193] The encoded information of the multiple phonemes is input into the trained generative model according to the order information. The generative model determines the predicted fundamental frequency features of multiple frames based on the encoded information of each phoneme and the encoded information of the context phonemes of each phoneme. The context phonemes of each phoneme are determined based on the order information between the multiple phonemes. The total duration of the multiple frames corresponds to the duration of the song.

[0194] The pitch of each frame in the plurality of frames is obtained, and the fundamental frequency of each frame is determined based on the pitch of each frame and the predicted fundamental frequency feature. The fundamental frequency of the song is obtained based on the fundamental frequencies of the plurality of frames.

[0195] In one embodiment, the processor also performs the steps described in the other embodiments when executing the computer program.

[0196] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0197] Obtain the phonemes corresponding to each lyric in the song, as well as the order information between multiple phonemes;

[0198] Based on the pitch, pitch duration, and legato information of each phoneme in the score of the song, the encoding information of each phoneme is generated;

[0199] The encoded information of the multiple phonemes is input into the trained generative model according to the order information. The generative model determines the predicted fundamental frequency features of multiple frames based on the encoded information of each phoneme and the encoded information of the context phonemes of each phoneme. The context phonemes of each phoneme are determined based on the order information between the multiple phonemes. The total duration of the multiple frames corresponds to the duration of the song.

[0200] The pitch of each frame in the plurality of frames is obtained, and the fundamental frequency of each frame is determined based on the pitch of each frame and the predicted fundamental frequency feature. The fundamental frequency of the song is obtained based on the fundamental frequencies of the plurality of frames.

[0201] In one embodiment, the computer program, when executed by a processor, also implements the steps described in the other embodiments above.

[0202] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0203] Obtain the phonemes corresponding to each lyric in the song, as well as the order information between multiple phonemes;

[0204] Based on the pitch, pitch duration, and legato information of each phoneme in the score of the song, the encoding information of each phoneme is generated;

[0205] The encoded information of the multiple phonemes is input into the trained generative model according to the order information. The generative model determines the predicted fundamental frequency features of multiple frames based on the encoded information of each phoneme and the encoded information of the context phonemes of each phoneme. The context phonemes of each phoneme are determined based on the order information between the multiple phonemes. The total duration of the multiple frames corresponds to the duration of the song.

[0206] The pitch of each frame in the plurality of frames is obtained, and the fundamental frequency of each frame is determined based on the pitch of each frame and the predicted fundamental frequency feature. The fundamental frequency of the song is obtained based on the fundamental frequencies of the plurality of frames.

[0207] In one embodiment, the computer program, when executed by a processor, also implements the steps described in the other embodiments above.

[0208] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0209] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0210] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0211] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A fundamental frequency generation method, characterized in that, The method includes: Obtain the phonemes corresponding to each lyric in the song, as well as the order information between multiple phonemes; Based on the pitch, pitch duration, and legato information of each phoneme in the score of the song, the encoding information of each phoneme is generated; The encoded information of the multiple phonemes is input into the trained generative model according to the order information. The generative model determines the predicted fundamental frequency features of multiple frames based on the encoded information of each phoneme and the encoded information of the context phonemes of each phoneme. The context phonemes of each phoneme are determined based on the order information between the multiple phonemes. The total duration of the multiple frames corresponds to the duration of the song. The pitch of each frame in the plurality of frames is obtained, and the fundamental frequency of each frame is determined based on the pitch of each frame and the predicted fundamental frequency feature. The fundamental frequency of the song is obtained based on the fundamental frequencies of the plurality of frames.

2. The method according to claim 1, characterized in that, The generation model determines the predicted fundamental frequency features of multiple frames based on the encoding information of each phoneme and the encoding information of the context phonemes of each phoneme, including: The generation model determines the phoneme-level fundamental frequency feature of each phoneme based on the encoding information of each phoneme and the encoding information of the context phonemes of each phoneme; The timbre characteristics of the target user are obtained, and the phoneme-level fundamental frequency characteristics of each phoneme are fused with the timbre characteristics of the target user to obtain the fused phoneme-level fundamental frequency characteristics of each phoneme. Based on the phoneme duration of each phoneme, the fused phoneme-level fundamental frequency features of each phoneme are mapped to multiple frame-level fundamental frequency features. Based on the frame-level fundamental frequency features corresponding to multiple phonemes, the predicted fundamental frequency features of multiple frames are obtained.

3. The method according to claim 2, characterized in that, The generative model includes multiple self-attention modules, which are used to extract features of different dimensions from the encoded information of the multiple phonemes. The step of determining the phoneme-level fundamental frequency feature of each phoneme by the generation model based on the encoding information of each phoneme and the encoding information of the context phonemes of each phoneme includes: The generative model performs self-attention calculation on the encoded information of each phoneme input to each self-attention module, as well as the encoded information of the context phonemes of each phoneme, to obtain the feature extraction result of each self-attention module in the corresponding dimension. Based on the feature extraction results of each self-attention module, the phoneme-level fundamental frequency feature corresponding to each phoneme is determined.

4. The method according to claim 2, characterized in that, The step of mapping the fused phoneme-level fundamental frequency features of each phoneme to multiple frame-level fundamental frequency features based on the phoneme duration of each phoneme includes: The number of frames for each phoneme is determined based on the preset frame duration and the phoneme duration of each phoneme. Based on the number of frames for each phoneme, the fused phoneme-level fundamental frequency features of each phoneme are copied to obtain multiple frame-level fundamental frequency features corresponding to each phoneme.

5. The method according to claim 2, characterized in that, The generative model also includes a duration warping network and a residual network connected after the duration warping network. The fused phoneme-level fundamental frequency features of each phoneme are mapped to multiple frame-level fundamental frequency features through the duration warping network. The step of obtaining predicted fundamental frequency features for multiple frames based on frame-level fundamental frequency features corresponding to multiple phonemes includes: The frame-level fundamental frequency features corresponding to the multiple phonemes output by the duration warping network are input into the residual network. The residual network then sequentially performs feature transfer from the first residual block to the last residual block on the frame-level fundamental frequency features corresponding to each phoneme, thereby obtaining the predicted fundamental frequency features of multiple frames.

6. The method according to claim 1, characterized in that, The step of obtaining the pitch of each frame in the plurality of frames includes: The pitch and duration of each note in the song are obtained from the musical score. Based on the pitch duration of each note, the pitch of each note is mapped to the pitch of multiple frames.

7. The method according to claim 1, characterized in that, Determining the fundamental frequency of each frame based on the pitch and the predicted fundamental frequency feature of each frame includes: The initial fundamental frequency of each frame is determined based on the pitch of each frame; The initial fundamental frequency of each frame is adjusted based on the predicted fundamental frequency characteristics of each frame to obtain the fundamental frequency of each frame.

8. The method according to any one of claims 1 to 7, characterized in that, The generative model is trained through the following steps: Obtain the sample phonemes corresponding to each lyric of the sample song; Based on the pitch, pitch duration, and legato information of each sample phoneme in the score of the sample song, the encoding information of each sample phoneme is generated; The generator and discriminator to be trained are obtained, and the encoding information of the multiple sample phonemes is input into the generator according to the order information between the multiple sample phonemes. The generator determines the sample prediction fundamental frequency features of multiple frames based on the encoding information of each sample phoneme and the encoding information of the context sample phonemes of each sample phoneme. The context sample phoneme of each sample phoneme is determined based on the order information between the multiple sample phonemes, and the total duration of the multiple frames corresponds to the duration of the sample song; The pitch of each frame in the plurality of frames corresponding to the sample song is obtained. Based on the pitch of each frame corresponding to the sample song and the sample predicted fundamental frequency feature, the fundamental frequency of each frame is determined. Based on the fundamental frequencies of the plurality of frames, the predicted fundamental frequency of the sample song is obtained. Based on the predicted fundamental frequency and the fundamental frequency label of the sample song, the generator and discriminator are subjected to adversarial training, and when the training termination condition is met, the generator is determined as the generative model.

9. The method according to claim 8, characterized in that, The step of performing adversarial training on the generator and discriminator based on the predicted fundamental frequency and the fundamental frequency tags of the sample songs includes: The predicted fundamental frequency of the sample song is input into the current discriminator, and the current discriminator obtains the predicted probability that the predicted fundamental frequency is the true fundamental frequency; The current discriminator is adjusted based on the predicted probability; and Based on the predicted probability, the predicted fundamental frequency, and the fundamental frequency label, the loss value of the current generator is determined, and the model parameters of the current generator are adjusted according to the loss value.

10. The method according to claim 9, characterized in that, The step of inputting the predicted fundamental frequency of the sample song into the current discriminator, and having the current discriminator obtain the predicted probability that the predicted fundamental frequency is the true fundamental frequency, includes: The predicted fundamental frequency of the sample song is input into the current discriminator, which randomly extracts a target fundamental frequency of a preset length from the predicted fundamental frequency and transforms the target fundamental frequency into a two-dimensional matrix with various different structures. Each two-dimensional matrix is ​​input into its corresponding sub-discriminator, which then obtains the sub-probability that the predicted fundamental frequency is the true fundamental frequency. Based on the sub-probabilities output by each sub-discriminator, the predicted probability that the predicted fundamental frequency of the sample song is the true fundamental frequency is obtained.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 10.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

Images