A method, apparatus, electronic device and storage medium for song synthesis

Abstract

This application provides a song synthesis method, apparatus, electronic device, and storage medium, relating to the field of song synthesis. The method includes: acquiring the dry sound Mel spectrum corresponding to the dry sound to be synthesized, and the accompaniment Mel spectrum corresponding to the accompaniment to be synthesized; extracting features from the dry sound Mel spectrum and the accompaniment Mel spectrum using a neural network model, and aligning the obtained dry sound features and accompaniment features; both the dry sound features and the accompaniment features contain feature information corresponding to the Mel spectrum at each time step; aligning the dry sound Mel spectrum and the accompaniment Mel spectrum based on the time correspondence between the aligned dry sound features and the accompaniment features, and generating a synthesized song using the aligned dry sound Mel spectrum and accompaniment Mel spectrum; the neural network model can be used to automatically align the Mel spectra corresponding to the dry sound to be synthesized and the accompaniment to be synthesized, thereby improving the efficiency and quality of song synthesis.

Description

Technical Field

[0001] This application relates to the field of song synthesis, and in particular to a song synthesis method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Song synthesis often involves multiple manual operations, especially when mixing the dry vocals and the accompaniment. This often requires meticulous manual editing, which not only reduces the efficiency of song synthesis but also makes it difficult to avoid alignment errors caused by manual editing, thus easily affecting the quality of the song. Summary of the Invention

[0003] The purpose of this application is to provide a song synthesis method, apparatus, electronic device, and computer-readable storage medium that can utilize a neural network model to automatically align the dry vocals to be synthesized with the accompaniment to be synthesized, thereby improving the efficiency of song synthesis and the quality of synthesized songs.

[0004] To solve the above-mentioned technical problems, this application provides a song synthesis method, including:

[0005] Obtain the dry sound Mel spectrum corresponding to the dry sound to be synthesized, and the accompaniment Mel spectrum corresponding to the accompaniment to be synthesized;

[0006] The dry sound Mel spectrum and the accompaniment Mel spectrum are used to extract features using a neural network model, and the obtained dry sound features and accompaniment features are aligned; both the dry sound features and the accompaniment features contain the feature information of the corresponding Mel spectrum at each time point;

[0007] Based on the time correspondence between the aligned dry vocal features and the accompaniment features, the dry vocal Mel spectrum and the accompaniment Mel spectrum are aligned, and a synthesized song is generated using the aligned dry vocal Mel spectrum and the accompaniment Mel spectrum.

[0008] Preferably, obtaining the dry sound Mel spectrum corresponding to the dry sound to be synthesized, and the accompaniment Mel spectrum corresponding to the accompaniment to be synthesized, includes:

[0009] Obtain the musical score corresponding to the dry audio to be synthesized and the accompaniment to be synthesized;

[0010] The accompaniment to be synthesized is converted into a spectrum, and the spectrum is then converted into the Mel spectrum of the accompaniment.

[0011] Extract the lyrics, pitch, and duration of each note from the musical score, and integrate the lyrics, pitch, and duration of each note into a triplet.

[0012] According to the order of the notes in the musical score, the triplets of each note are written into a triplet sequence in sequence, and the triplet sequence is converted into the dry sound Mel spectrum using a singing synthesis model.

[0013] Preferably, after converting the triplet sequence into the dry sound Mel spectrum using a singing synthesis model, the method further includes:

[0014] The dry acoustic Mel spectrum is enhanced using an enhancement model, and features are extracted from the enhanced dry acoustic Mel spectrum using the neural network model.

[0015] Preferably, the enhancement processing of the dry acoustic Mel spectrum using the enhancement model includes:

[0016] The dry acoustic Mel spectrum is enhanced using a probability diffusion model.

[0017] Preferably, the step of converting the triplet sequence into the dry vocal Mel spectrum using a vocal synthesis model includes:

[0018] Encode each triplet in the triplet sequence into a corresponding vector to obtain a vector sequence;

[0019] The vector sequence is converted into the dry sound Mel spectrum using the singing synthesis model.

[0020] Preferably, the step of converting the vector sequence into the dry sound Mel spectrum using the singing synthesis model includes:

[0021] The vector sequence is converted into the dry acoustic Mel spectrum using the Fastspeech model or the Fastspeech2 model.

[0022] Preferably, the step of generating a synthesized song using aligned dry Mel spectra and accompaniment Mel spectra includes:

[0023] The aligned dry Mel spectrum and the accompaniment Mel spectrum are mixed to obtain a mixed Mel spectrum;

[0024] The mixed Mel spectrum is converted into the synthesized song using a vocoder.

[0025] Preferably, the step of using a neural network model to extract features from the dry vocal Mel spectrum and the accompaniment Mel spectrum, and then aligning the obtained dry vocal features and accompaniment features, includes:

[0026] The dry sound Mel spectrum and the accompaniment Mel spectrum are used to extract features from the dry sound features and the accompaniment features by using a convolutional recurrent neural network model.

[0027] The attention mechanism model is used to align the dry vocal features and the accompaniment features.

[0028] This application also provides a song synthesis device, including:

[0029] The acquisition module is used to acquire the dry sound Mel spectrum corresponding to the dry sound to be synthesized, and the accompaniment Mel spectrum corresponding to the accompaniment to be synthesized.

[0030] The feature extraction and alignment module is used to extract features from the dry vocal Mel spectrum and the accompaniment Mel spectrum using a neural network model, and to align the obtained dry vocal features and accompaniment features; both the dry vocal features and the accompaniment features contain feature information corresponding to the Mel spectrum at each time point;

[0031] The synthesized song generation module is used to align the dry vocal Mel spectrum and the accompaniment Mel spectrum based on the time correspondence between the aligned dry vocal features and the accompaniment features, and to generate a synthesized song using the aligned dry vocal Mel spectrum and the accompaniment Mel spectrum.

[0032] This application also provides an electronic device, including:

[0033] Memory, used to store computer programs;

[0034] A processor is used to implement the song synthesis method described above when executing the computer program.

[0035] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when loaded and executed by a processor, implement the song synthesis method described above.

[0036] This application provides a song synthesis method, comprising: obtaining the dry vocal Mel spectrum corresponding to the dry vocal to be synthesized, and the accompaniment Mel spectrum corresponding to the accompaniment to be synthesized; extracting features from the dry vocal Mel spectrum and the accompaniment Mel spectrum using a neural network model, and aligning the obtained dry vocal features and accompaniment features; wherein the dry vocal features and the accompaniment features each contain feature information corresponding to the Mel spectrum at each time step; aligning the dry vocal Mel spectrum and the accompaniment Mel spectrum based on the time correspondence between the aligned dry vocal features and the accompaniment features, and generating a synthesized song using the aligned dry vocal Mel spectrum and accompaniment Mel spectrum.

[0037] As can be seen, this application first obtains the Mel spectrum of the dry vocal to be synthesized and the Mel spectrum of the accompaniment to be synthesized. That is, this application can use the Mel spectra of both the dry vocal and the accompaniment to synthesize for song synthesis. Subsequently, this application uses a neural network model to extract features from the two Mel spectra, obtaining dry vocal features and accompaniment features. Since both features contain the corresponding Mel spectra at various times, the neural network model can further automatically align the dry vocal features and accompaniment features based on the similarity of these feature information. Furthermore, the dry vocal and accompaniment Mel spectra can be aligned based on the temporal correspondence between the aligned dry vocal and accompaniment features, and finally, a synthesized song is generated using the aligned dry vocal and accompaniment Mel spectra. In other words, this application can automatically align the dry vocal and accompaniment to be synthesized using the dry vocal Mel spectrum, the accompaniment Mel spectrum, and a neural network model. This not only improves the efficiency of song synthesis but also effectively avoids alignment errors caused by manual editing, thereby improving the quality of the synthesized song. This application also provides a song synthesis device, an electronic device, and a computer-readable storage medium, which have the aforementioned beneficial effects. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0039] Figure 1 A flowchart of a song synthesis method provided in an embodiment of this application;

[0040] Figure 2 A schematic diagram of a neural network accompaniment adding module provided in an embodiment of this application;

[0041] Figure 3 A schematic diagram of a song synthesis model provided in an embodiment of this application;

[0042] Figure 4 This is a structural block diagram of a song synthesis device provided in an embodiment of this application;

[0043] Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] Existing song synthesis processes often involve multiple manual operations. For example, mixing the vocals and accompaniment often requires meticulous manual editing, which significantly reduces the efficiency of song synthesis and makes it difficult to avoid alignment errors, thus easily affecting the quality of the final product. In view of this, this application provides a song synthesis method that uses a neural network model to automatically align the vocals and accompaniment. This not only improves the efficiency of song synthesis but also effectively reduces alignment errors between the vocals and accompaniment, thereby significantly improving the quality of the final product. Please refer to [reference needed]. Figure 1 , Figure 1 A flowchart of a song synthesis method provided in this application embodiment, the method may include:

[0046] S101. Obtain the dry sound Mel spectrum corresponding to the dry sound to be synthesized, and the accompaniment Mel spectrum corresponding to the accompaniment to be synthesized.

[0047] It should be noted that dry voice refers to human voice without accompaniment. In the embodiments of this application, in order to achieve automatic alignment between dry voice and accompaniment, the corresponding Mel spectra of the two are first obtained, and then the Mel spectra are input into the neural network model for automatic alignment. The Mel spectra is an intermediate data in the speech synthesis process and is a representation of audio.

[0048] It should be noted that the embodiments of this application do not limit the methods for obtaining the dry voice Mel spectrum and the accompaniment Mel spectrum. For example, considering that the Mel spectrum can be converted from a general spectrum, the audio files corresponding to the dry voice and the accompaniment can be recorded, and the audio files can be converted to obtain the corresponding dry voice spectrum and the accompaniment spectrum. Then, these two spectra can be converted into the corresponding dry voice Mel spectrum and the accompaniment Mel spectrum. As another example, the singing content usually has a corresponding score, that is, the dry voice to be synthesized has a corresponding score. The score usually contains the lyrics text corresponding to each note, pitch, and pitch duration (indicating the duration that the current pitch should be sustained when singing). These three elements can be used to synthesize speech, and the Mel spectrum can also be used to represent synthesized speech. Therefore, a preset singing synthesis model can be used to synthesize the score of the dry voice to be synthesized to obtain the dry voice Mel spectrum corresponding to the dry voice to be synthesized. It is worth noting that since accompaniment can also be recorded by machine synthesis, as long as the corresponding project file of the accompaniment is obtained (including the structure and composition of the song, the instruments of the accompaniment, the musical score, the lyrics and music score, and the vocal mixing, etc.), the corresponding audio file of the accompaniment can be directly synthesized by machine synthesis. Moreover, the alignment operation and song synthesis operation mentioned in the subsequent embodiments of this application can be completed automatically by the machine without human supervision and operation. Therefore, this application only needs to encapsulate the various models used into a complete song synthesis model to realize end-to-end synthesis of songs. That is, only the corresponding scores of the accompaniment and the dry vocals need to be input into this song synthesis model to obtain the corresponding synthesized song without human operation, which can further improve the efficiency and quality of song synthesis.

[0049] S102. Use a neural network model to extract features from the dry sound Mel spectrum and the accompaniment Mel spectrum, and align the obtained dry sound features and accompaniment features; both the dry sound features and the accompaniment features contain the feature information of the corresponding Mel spectrum at each time step.

[0050] To achieve automatic alignment of the dry vocal and accompaniment features, this embodiment of the application, after obtaining the dry vocal Mel spectrum and the accompaniment Mel spectrum, uses a neural network model to extract features from these two spectra. Then, the neural network model is used to align the obtained dry vocal and accompaniment features, thereby aligning the dry vocal and accompaniment Mel spectra based on the alignment result. It should be noted that both the dry vocal and accompaniment features contain two dimensions: a time dimension and a feature dimension. The dry vocal feature has a size of T1×D1, indicating that the time dimension and feature dimension of the dry vocal feature are T1 and D1 respectively, and its feature dimension contains the feature information corresponding to the dry vocal Mel spectrum at each time step. The accompaniment feature has a size of T2×D2, indicating that the time dimension and feature dimension of the accompaniment feature are T2 and D2 respectively, and its feature dimension contains the feature information corresponding to the accompaniment Mel spectrum at each time step. Because of the similarity between the dry vocals and the accompaniment—that is, the similarity between their features—feature alignment can be performed. This involves sliding these two types of features along the time dimension to align them. Furthermore, it can be understood that the Mel spectrum of the dry vocals and the Mel spectrum of the accompaniment can be aligned based on the temporal correspondence between the aligned features.

[0051] It should be noted that the embodiments of this application do not limit the specific neural network models corresponding to feature extraction and feature alignment. Preferably, feature extraction can adopt a convolutional recurrent neural network model (CRNN), while feature alignment can use an attention mechanism model. This is because the features extracted by the convolutional recurrent neural network are of high quality, and the attention mechanism model can make the computer pay more attention to the similarity between features, thereby improving the feature alignment accuracy between the dry vocal features and the accompaniment features.

[0052] In one possible scenario, a neural network model is used to extract features from the dry vocal Mel spectrum and the accompaniment Mel spectrum, and feature alignment is performed on the obtained dry vocal features and accompaniment features, including:

[0053] Step 11: Use a convolutional recurrent neural network model to extract features from the dry vocal Mel spectrum and the accompaniment Mel spectrum to obtain dry vocal features and accompaniment features;

[0054] Step 12: Use an attention mechanism model to align the dry vocal features and the accompaniment features.

[0055] S103. Based on the time correspondence between the aligned dry vocal features and the accompaniment features, the dry vocal Mel spectrum and the accompaniment Mel spectrum are aligned, and the aligned dry vocal Mel spectrum and the accompaniment Mel spectrum are used to generate a synthesized song.

[0056] As described above, after feature alignment is completed, the dry vocal Mel spectrum and the accompaniment Mel spectrum can be aligned based on the temporal correspondence between the aligned dry vocal features and the accompaniment features, so as to generate a synthesized song using the aligned dry vocal Mel spectrum and the accompaniment Mel spectrum. Specifically, the aligned dry vocal Mel spectrum and the accompaniment Mel spectrum can be mixed to obtain a mixed Mel spectrum, and then a vocoder can be used to convert the mixed Mel spectrum into a synthesized song, where the vocoder is a model that can generate a spectrum based on the Mel spectrum.

[0057] In one possible scenario, generating a synthesized song using aligned dry Mel spectra and accompaniment Mel spectra could include:

[0058] Step 21: Mix the aligned dry Mel spectrum and the accompaniment Mel spectrum to obtain the mixed Mel spectrum;

[0059] Step 22: Use a vocoder to convert the mixed Mel spectrum into a synthesized song.

[0060] The following section will describe the complete implementation process of steps S102 and S103 in conjunction with a specific structural diagram. Please refer to the diagram. Figure 2 , Figure 2 This is a schematic diagram of a neural network accompaniment addition module provided in an embodiment of this application. The accompaniment Mel spectrum and the dry vocal Mel spectrum are first input to the accompaniment encoder and the dry vocal encoder, respectively, for feature extraction. Both the accompaniment encoder and the dry vocal encoder are convolutional recurrent neural network structures. Subsequently, the dry vocal features and accompaniment features output from the two encoders are input to an attention mechanism module for alignment, thereby aligning the dry vocal Mel spectrum and the accompaniment Mel spectrum according to the alignment result. Finally, the aligned dry vocal Mel spectrum and the accompaniment Mel spectrum are mixed to obtain a mixed Mel spectrum, which is then converted into a synthesized song using a vocoder.

[0061] Based on the above embodiments, this application first obtains the Mel spectrum of the dry vocal to be synthesized and the Mel spectrum of the accompaniment to be synthesized. That is, this application can use the Mel spectra of both the dry vocal and the accompaniment to be synthesized for song synthesis. Subsequently, this application can use a neural network model to extract features from the two Mel spectra to obtain dry vocal features and accompaniment features. Since both features contain feature information corresponding to the Mel spectra at various times, the neural network model can be used to automatically align the dry vocal features and accompaniment features based on the similarity of these feature information. Furthermore, the dry vocal Mel spectrum and the accompaniment Mel spectrum can be aligned based on the temporal correspondence of the aligned dry vocal features and accompaniment features, and finally, the synthesized song can be generated using the aligned dry vocal Mel spectrum and the accompaniment Mel spectrum. In other words, this application can use the dry vocal Mel spectrum, the accompaniment Mel spectrum, and the neural network model to achieve automatic alignment of the dry vocal and the accompaniment to be synthesized, which not only improves the efficiency of song synthesis but also effectively avoids alignment errors caused by manual editing, thereby improving the quality of the synthesized song.

[0062] Based on the above embodiments, the process of obtaining the dry sound Mel spectrum and the accompaniment Mel spectrum is described in detail below. In one possible scenario, obtaining the dry sound Mel spectrum corresponding to the dry sound to be synthesized, and the accompaniment Mel spectrum corresponding to the accompaniment to be synthesized, may include:

[0063] S201. Obtain the sheet music and accompaniment corresponding to the dry audio to be synthesized.

[0064] S202. Convert the accompaniment to be synthesized into a spectrum, and then convert the spectrum into the accompaniment Mel spectrum.

[0065] It should be noted that the embodiments of this application do not limit the specific process of spectrum conversion and Mel spectrum conversion, and relevant technologies of spectrum and Mel spectrum can be referred to.

[0066] S203. Extract the lyrics, pitch, and duration of each note from the musical score, and integrate the lyrics, pitch, and duration of each note into a triplet.

[0067] The musical score information can be extracted using a toolkit to obtain the lyrics text, pitch, and pitch duration corresponding to each note, and can be integrated into a (text, pitch, pitch duration) triple, for example (you, 83, 0.5). It should be noted that the embodiments in this application do not limit the specific method of extracting musical score information. Since musical score information is usually saved in musicxml format, relevant technologies for parsing musicxml files can be referred to.

[0068] S204. Write the triplets of each note into a triplet sequence according to the order of the notes in the score, and use the singing synthesis model to convert the triplet sequence into a dry sound Mel spectrum.

[0069] It is understandable that the triplet sequence is a form of musical score representation, obtained by integrating the triplets of each note according to their order in the score; while the vocal synthesis model is a neural network model used to convert the triplet sequence into a dry vocal metronome spectrum. Of course, to facilitate the conversion of the triplet sequence by the vocal synthesis model, each triplet in the sequence can also be encoded into a vector form that the neural network model can process, and then the resulting vector sequence can be input into the vocal synthesis model for conversion. It should be noted that the embodiments of this application do not limit the specific encoding method and can be set according to actual application needs.

[0070] In one possible scenario, converting a triplet sequence into a dry Mel spectrum using a singing voice synthesis model could include:

[0071] Step 31: Encode each triplet in the triplet sequence into a corresponding vector to obtain a vector sequence;

[0072] Step 32: Use the singing synthesis model to convert the vector sequence into a dry sound Mel spectrum.

[0073] Furthermore, this application does not limit the specific type of singing voice synthesis model. It allows selection from relevant speech synthesis models based on actual application needs, or even the design of a custom singing voice synthesis model. In this application, considering the good quality of the dry audio synthesized by the Fastspeech model and Fastspeech2 model, the Fastspeech model or Fastspeech2 model can be used to convert the vector sequence into a dry audio Mel spectrum. The Fastspeech model is an acoustic model, while the Fastspeech2 model is an upgraded version of the Fastspeech model. This application also does not limit the training process of the singing voice synthesis model. It is typically trained using a training set consisting of real human singing audio and the corresponding Mel spectrum. For details, please refer to the relevant techniques of the Fastspeech model and Fastspeech2 model.

[0074] Furthermore, considering that the synthesized dry Mel spectra may still have issues such as unnaturalness and insufficient fullness, the dry Mel spectra output by the singing synthesis model can be enhanced to obtain higher-quality dry Mel spectra. Specifically, an enhancement model can be used to enhance the dry Mel spectra, where the model can predict the corresponding high-quality Mel spectra based on the input Mel spectra. It is understood that the training set of this model consists of low-quality Mel spectra and corresponding high-quality Mel spectra. The low-quality Mel spectra suffers from defects such as unnaturalness, excessive noise, and insufficient fullness, while the high-quality Mel spectra does not have these defects. Using this training set to train the enhancement model allows it to predict the corresponding high-quality Mel spectra based on the input low-quality Mel spectra.

[0075] In one possible scenario, after converting the triplet sequence into a dry vocal Mel spectrum using a singing synthesis model, the process may further include:

[0076] Step 41: Enhance the dry acoustic Mel spectrum using an enhancement model, and then use a neural network model to extract features from the enhanced dry acoustic Mel spectrum.

[0077] Preferably, the enhancement model can be a probability diffusion model, which treats the output of the vocal synthesis module as noise and models it from a probabilistic perspective, thereby obtaining a higher quality Mel spectrum through a noise reduction process. In other words, the probability diffusion model can be used to enhance the dry vocal Mel spectrum.

[0078] Based on the above embodiments, the embodiments of this application can use a neural network model to automatically synthesize the score corresponding to the dry sound to be synthesized and the accompaniment to be synthesized, and obtain the corresponding dry sound Mel spectrum and accompaniment Mel spectrum, which can further improve the efficiency and quality of song synthesis.

[0079] The following section will describe the complete implementation process of the song synthesis method described above, based on a complete structural diagram. Please refer to the provided text. Figure 3 , Figure 3 This is a schematic diagram of a song synthesis model provided in an embodiment of this application. The various modules shown in the diagram will be described below.

[0080] 1. Extraction Module: This module extracts the lyrics, pitch, and corresponding pitch duration from the musical score. The score is usually saved in musicxml format. A toolkit can be used to extract the (text, pitch, pitch duration) corresponding to each note, for example (you, 83, 0.5). Similar triplets form the basic input for the vocal synthesis module.

[0081] 2. Vocal Synthesis Module: This module converts a given sequence of (text, pitch, pitch duration) into a corresponding Mel spectrum. This process typically involves encoding each item in the triplet into a vector, concatenating these vectors, and then feeding them into the vocal synthesis module. During training, the module needs to be provided with authentic Mel spectra to guide model learning. These authentic Mel spectra are usually obtained by directly converting the spectrum of a human singing a song into a Mel spectrum.

[0082] 3. Enhancement Module: Typical vocal synthesis modules are represented by Fastspeech or Fastspeech2 models. These models have relatively limited audio synthesis quality. In order to make the results closer to the real spectrum or closer to human hearing, an enhancement model can be used, such as the probability diffusion model. This model treats the output of the vocal synthesis module as noise and models it from a probabilistic perspective. Through the noise reduction process, a better Mel spectrum can be obtained.

[0083] 4. Neural Network Accompaniment Addition Module: Please refer to the following for the structure of this module. Figure 2 Unlike traditional synthesis methods that linearly superimpose the accompaniment and synthesized dry vocals, this application uses neural networks for higher-quality synthesis. In this module, the accompaniment is first preprocessed and converted into a Mel spectrum M. acc Then, it is processed by an accompaniment encoder to obtain a series of transformed accompaniment features F. acc The accompaniment encoder here is a CRNN neural network structure, and the output of all time steps constitutes F. acc Similarly, the dry acoustic Mel spectrum M ugc It will also pass through a dry acoustic encoder to extract its features F ugc Next, an attention mechanism module is used to align the features of the dry vocals of the accompaniment, and the alignment result is then converted into the sound of the artwork by a vocoder.

[0084] The following describes the song synthesis apparatus, electronic device, and computer-readable storage medium provided in the embodiments of this application. The song synthesis apparatus, electronic device, and computer-readable storage medium described below can be referred to in correspondence with the song synthesis method described above.

[0085] Please refer to Figure 4 , Figure 4 This is a structural block diagram of a song synthesis device provided in an embodiment of this application. The device may include:

[0086] The acquisition module 401 is used to acquire the dry sound Mel spectrum corresponding to the dry sound to be synthesized, and the accompaniment Mel spectrum corresponding to the accompaniment to be synthesized.

[0087] The feature extraction and alignment module 402 is used to extract features from the dry vocal Mel spectrum and the accompaniment Mel spectrum using a neural network model, and to align the obtained dry vocal features and accompaniment features; both the dry vocal features and the accompaniment features contain the feature information corresponding to the Mel spectrum at each time step.

[0088] The synthesized song generation module 403 is used to align the dry vocal Mel spectrum and the accompaniment Mel spectrum based on the time correspondence between the aligned dry vocal features and the accompaniment features, and to generate a synthesized song using the aligned dry vocal Mel spectrum and the accompaniment Mel spectrum.

[0089] Preferably, the acquisition module 401 may include:

[0090] The acquisition submodule is used to acquire the sheet music and accompaniment to be synthesized for the dry audio to be synthesized.

[0091] The first conversion submodule is used to convert the accompaniment to be synthesized into a spectrum, and to convert the spectrum into the accompaniment Mel spectrum;

[0092] The extraction submodule is used to extract the lyrics text, pitch and pitch duration corresponding to each note from the musical score, and integrate the lyrics text, pitch and pitch duration of the notes into a triplet;

[0093] The second conversion submodule is used to write the triplets of each note into a triplet sequence according to the order of each note in the score, and to convert the triplet sequence into a dry sound Mel spectrum using a vocal synthesis model.

[0094] Preferably, the acquisition module 401 may further include:

[0095] The enhancement submodule is used to enhance the dry acoustic Mel spectrum using an enhancement model, and then use a neural network model to extract features from the enhanced dry acoustic Mel spectrum.

[0096] Preferably, the enhancement submodule is specifically used for:

[0097] The dry acoustic Mel spectrum is enhanced using a probability diffusion model.

[0098] Preferably, the second conversion submodule may include:

[0099] The vector transformation unit is used to encode each triplet in the triplet sequence into a corresponding vector, thereby obtaining a vector sequence.

[0100] The Mel spectrum conversion unit is used to convert a vector sequence into a dry Mel spectrum using a vocal synthesis model.

[0101] Preferably, the Mel spectrum conversion unit is specifically used for:

[0102] The vector sequence is converted into a dry acoustic Mel spectrum using the Fastspeech model or the Fastspeech2 model.

[0103] Preferably, the song synthesis generation module 403 includes:

[0104] The mixing submodule is used to mix the aligned dry Mel spectrum and the accompaniment Mel spectrum to obtain the mixed Mel spectrum;

[0105] The song generation submodule is used to convert mixed Mel spectra into synthesized songs using a vocoder.

[0106] Preferably, the feature extraction and alignment module 402 may include:

[0107] The feature extraction submodule is used to extract features from the dry vocal Mel spectrum and the accompaniment Mel spectrum using a convolutional recurrent neural network model to obtain dry vocal features and accompaniment features.

[0108] The feature alignment submodule is used to perform feature alignment between the dry vocal features and the accompaniment features using an attention mechanism model.

[0109] This application also provides an electronic device that may include a processor and a memory. The relationship between the processor and the memory in this electronic device can be referenced. Figure 5 .

[0110] The processor 11 of the electronic device 10 is used to execute the program stored in the memory 12;

[0111] The memory 12 of the electronic device 10 is used to store a program, which is used for at least:

[0112] Obtain the dry sound Mel spectrum corresponding to the dry sound to be synthesized, and the accompaniment Mel spectrum corresponding to the accompaniment to be synthesized;

[0113] A neural network model was used to extract features from the dry vocal Mel spectrum and the accompaniment Mel spectrum, and the obtained dry vocal features and accompaniment features were aligned. Both the dry vocal features and the accompaniment features contain the feature information of the corresponding Mel spectrum at each time step.

[0114] Based on the time correspondence between aligned dry vocal features and accompaniment features, the dry vocal Mel spectrum and the accompaniment Mel spectrum are aligned, and a synthesized song is generated using the aligned dry vocal Mel spectrum and accompaniment Mel spectrum.

[0115] Of course, the electronic device may also include a communication bus 16, a communication interface 13, a display 15, and an input unit 14, etc., without limitation.

[0116] On the other hand, this application also provides a storage medium storing a computer program, which, when loaded and executed by a processor, is used to implement the song synthesis method described in any of the above embodiments.

[0117] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0118] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0119] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0120] The foregoing has provided a detailed description of a song synthesis method, apparatus, electronic device, and computer-readable storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for synthesizing songs, characterized in that, include: Obtain the dry sound Mel spectrum corresponding to the dry sound to be synthesized, and the accompaniment Mel spectrum corresponding to the accompaniment to be synthesized; The dry sound Mel spectrum and the accompaniment Mel spectrum are used to extract features using a neural network model, and the obtained dry sound features and accompaniment features are aligned. Both the dry sound feature and the accompaniment feature contain the feature information of the corresponding Mel spectrum at each time point; Based on the time correspondence between the aligned dry vocal features and the accompaniment features, the dry vocal Mel spectrum and the accompaniment Mel spectrum are aligned, and a synthesized song is generated using the aligned dry vocal Mel spectrum and the accompaniment Mel spectrum.

2. The song synthesis method according to claim 1, characterized in that, The step of obtaining the dry sound Mel spectrum corresponding to the dry sound to be synthesized, and the accompaniment Mel spectrum corresponding to the accompaniment to be synthesized, includes: Obtain the musical score corresponding to the dry audio to be synthesized and the accompaniment to be synthesized; The accompaniment to be synthesized is converted into a spectrum, and the spectrum is then converted into the Mel spectrum of the accompaniment. Extract the lyrics, pitch, and duration of each note from the musical score, and integrate the lyrics, pitch, and duration of each note into a triplet. According to the order of the notes in the musical score, the triplets of each note are written into a triplet sequence in sequence, and the triplet sequence is converted into the dry sound Mel spectrum using a singing synthesis model.

3. The song synthesis method according to claim 2, characterized in that, After converting the triplet sequence into the dry vocal Mel spectrum using a singing synthesis model, the method further includes: The dry acoustic Mel spectrum is enhanced using an enhancement model, and features are extracted from the enhanced dry acoustic Mel spectrum using the neural network model.

4. The song synthesis method according to claim 3, characterized in that, The enhancement process of the dry acoustic Mel spectrum using the enhancement model includes: The dry acoustic Mel spectrum is enhanced using a probability diffusion model.

5. The song synthesis method according to claim 2, characterized in that, The process of converting the triplet sequence into the dry vocal Mel spectrum using a singing synthesis model includes: Encode each triplet in the triplet sequence into a corresponding vector to obtain a vector sequence; The vector sequence is converted into the dry sound Mel spectrum using the singing synthesis model.

6. The song synthesis method according to claim 5, characterized in that, The process of converting the vector sequence into the dry Mel spectrum using the singing synthesis model includes: The vector sequence is converted into the dry acoustic Mel spectrum using the Fastspeech model or the Fastspeech2 model.

7. The song synthesis method according to claim 1, characterized in that, The method of generating a synthesized song using aligned dry Mel spectra and accompaniment Mel spectra includes: The aligned dry Mel spectrum and the accompaniment Mel spectrum are mixed to obtain a mixed Mel spectrum; The mixed Mel spectrum is converted into the synthesized song using a vocoder.

8. The song synthesis method according to any one of claims 1 to 7, characterized in that, The process of extracting features from the dry vocal Mel spectrum and the accompaniment Mel spectrum using a neural network model, and then aligning the obtained dry vocal features and accompaniment features, includes: The dry sound Mel spectrum and the accompaniment Mel spectrum are used to extract features from the dry sound features and the accompaniment features by using a convolutional recurrent neural network model. The attention mechanism model is used to align the dry vocal features and the accompaniment features.

9. A song synthesis device, characterized in that, include: The acquisition module is used to acquire the dry sound Mel spectrum corresponding to the dry sound to be synthesized, and the accompaniment Mel spectrum corresponding to the accompaniment to be synthesized. The feature extraction and alignment module is used to extract features from the dry sound Mel spectrum and the accompaniment Mel spectrum using a neural network model, and to align the obtained dry sound features and accompaniment features. Both the dry sound feature and the accompaniment feature contain the feature information of the corresponding Mel spectrum at each time point; The synthesized song generation module is used to align the dry vocal Mel spectrum and the accompaniment Mel spectrum based on the time correspondence between the aligned dry vocal features and the accompaniment features, and to generate a synthesized song using the aligned dry vocal Mel spectrum and the accompaniment Mel spectrum.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the song synthesis method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when loaded and executed by a processor, implement the song synthesis method as described in any one of claims 1 to 8.

Images