Song synthesis method and component, song synthesis model training method and component

Abstract

The application discloses a song synthesis method and component in the field of computer technology, a song synthesis model training method and component. Based on the application, a user can train a song synthesis model of the user, so that the model training is more flexible; the user uses the song synthesis model to change the tone color in any song to the tone color of the user, while retaining the singing skills such as warble, slide and breath in the original song, so that a synthesized song with the tone color of the user is obtained, and the operability and synthesis quality of the song synthesis are improved. Accordingly, the song synthesis component, the song synthesis model training method and component provided by the application also have the above technical effects.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a singing synthesis method and component, a singing synthesis model training method and component. BACKGROUND

[0002] The existing singing synthesis model takes complex information such as phonemes and musical scores as synthesis elements, and the pitch information comes from the musical score, so the synthesis effect is generally mechanical. At the same time, it is difficult to construct singing skills such as trill, slide and breath in the synthesized song in the existing singing synthesis model. When training, the existing singing synthesis model also needs to use songs annotated with phonemes, phoneme duration and musical scores as training sets, and the annotation of the training set also consumes a lot of cost. Moreover, the training and application of the model are not controlled by the user, but by the model provider to train the same model for multiple users. That is, the user cannot customize the model. SUMMARY

[0003] Therefore, the purpose of the present application is to provide a singing synthesis method and component, a singing synthesis model training method and component, so that the user can customize the model, while ensuring the naturalness and authenticity of the synthesized song and improving the synthesis effect. The specific scheme is as follows:

[0004] To achieve the above purpose, on the one hand, the present application provides a singing synthesis method, comprising:

[0005] providing a singing synthesis interface; the singing synthesis interface is used to receive song synthesis instructions of a target song by a user;

[0006] in response to the song synthesis instructions, providing a model training interface; the model training interface is provided with a plurality of user audios for the user to select;

[0007] based on the user-selected song with user timbre, a singing synthesis model is trained;

[0008] calling the singing synthesis model to change the timbre in the target song to the user timbre to obtain a synthesized song;

[0009] displaying the synthesized song in the singing synthesis interface.

[0010] Optionally, it further comprises:

[0011] In the process of calling the singing synthesis model, the singing synthesis progress is displayed in the singing synthesis interface in real time.

[0012] Optionally, before providing the model training interface, it further comprises:

[0013] determining whether the singing synthesis model exists;

[0014] If the song synthesis model exists, the calling of the song synthesis model is performed to change the tone in the target song to the user tone to obtain a synthesized song; and the synthesized song is displayed on the song synthesis interface; otherwise, a model training interface is provided.

[0015] Optionally, the method further comprises:

[0016] After the song synthesis model training is completed, the user tone is extracted and stored by using the song synthesis model, so that the user tone is queried when the song synthesis model is called.

[0017] In another aspect, the application also provides a song synthesis method, comprising:

[0018] In response to a model training instruction, a model training interface is provided; and the model training interface provides a plurality of user audio for user selection;

[0019] Based on the song with the user tone selected by the user, a song synthesis model is trained;

[0020] A song synthesis interface is provided so that the user inputs a song synthesis instruction for a target song;

[0021] In response to the song synthesis instruction, the song synthesis model is called to change the tone in the target song to the user tone to obtain a synthesized song;

[0022] The synthesized song is displayed on the song synthesis interface.

[0023] Optionally, the plurality of user audio provided by the model training interface is: a song recorded by the user in real time, a song uploaded by the user in real time, and / or a song recorded by the user in the past.

[0024] In another aspect, the application also provides a song synthesis method, comprising:

[0025] A song synthesis request sent by a client is received;

[0026] A target audio of a song audio and a target tone specified by the song synthesis request is determined;

[0027] A preset song synthesis model is called to construct a synthesis spectrum including a tone feature of the target tone, a song content feature of the song audio, and a song fundamental frequency feature based on a target spectrum of the target audio, a song spectrum of the song audio, and a song fundamental frequency;

[0028] The synthesis spectrum is input into a preset vocoder to make the vocoder output a synthesized song including the target tone, song content, and song melody of the song audio.

[0029] Optionally, the calling the preset song synthesis model is configured to construct a synthesis spectrum comprising a timbre feature of the target timbre, a song content feature of the song audio, and a song fundamental frequency feature based on a target spectrum of the target audio, a song spectrum of the song audio, and a song fundamental frequency.

[0030] extracting the song spectrum and the song fundamental frequency from the song audio;

[0031] extracting the target spectrum from the target audio;

[0032] inputting the song spectrum into a content encoder in the song synthesis model to obtain the song content feature;

[0033] inputting the target spectrum into a timbre encoder in the song synthesis model to obtain the timbre feature;

[0034] inputting the song fundamental frequency into a word embedding module in the song synthesis model to obtain the song fundamental frequency feature;

[0035] superimposing the song content feature, the timbre feature, and the song fundamental frequency feature, and inputting a superimposition result into a decoder in the song synthesis model to obtain the synthesis spectrum.

[0036] Optionally, before the inputting the song fundamental frequency into the word embedding module in the song synthesis model to obtain the song fundamental frequency feature, if a pitch raising or pitch lowering request sent by the client is received, the method further comprises:

[0037] raising or lowering the song fundamental frequency.

[0038] Optionally, the inputting the song spectrum into the content encoder in the song synthesis model to obtain the song content feature comprises:

[0039] inputting the song spectrum into the content encoder to enable the content encoder to extract continuous content features in the song spectrum and to perform vector quantization on the continuous content features to obtain the song content feature.

[0040] Optionally, the inputting the target spectrum into the timbre encoder in the song synthesis model to obtain the timbre feature comprises:

[0041] inputting the target spectrum into the timbre encoder to enable the timbre encoder to extract continuous timbre features in the target spectrum and to perform mean value calculation on the continuous timbre features to obtain the timbre feature.

[0042] Optionally, the inputting the song fundamental frequency into the word embedding module in the song synthesis model to obtain the song fundamental frequency feature comprises:

[0043] inputting the song fundamental frequency into the word embedding module, so that the word embedding module maps the song fundamental frequency into the song fundamental frequency feature in vector representation.

[0044] Optionally, the method further comprises:

[0045] returning the synthesized song to the client and displaying and playing the synthesized song.

[0046] In another aspect, the application also provides a song synthesis model training method, comprising:

[0047] receiving a model training request sent by a client;

[0048] determining a training set specified by the model training request; the training set is a song set selected by a user and carrying a user voice;

[0049] training a model based on the training set to obtain the song synthesis model.

[0050] Optionally, the training a model based on the training set to obtain the song synthesis model comprises:

[0051] obtaining a training song from the training set;

[0052] extracting a trainable frequency spectrum and a training fundamental frequency from the training song;

[0053] inputting the trainable frequency spectrum into a content encoder in an initial model to obtain a training content feature of the training song;

[0054] inputting the trainable frequency spectrum into a timbre encoder in the initial model to obtain a training timbre feature of the training song;

[0055] inputting the training fundamental frequency into a word embedding module in the initial model to obtain a training fundamental frequency feature of the training song;

[0056] superimposing the training content feature, the training timbre feature, and the training fundamental frequency feature, and inputting a superimposed result into a decoder in the initial model to obtain a training synthesized frequency spectrum;

[0057] calculating a loss between the training synthesized frequency spectrum and the trainable frequency spectrum;

[0058] if the loss meets a preset convergence condition, outputting a current model as the song synthesis model; otherwise, updating parameters of the content encoder, the timbre encoder, the decoder, and the word embedding module in the initial model based on the loss, and iteratively training the model with another training song from the training set.

[0059] In yet another aspect, the present application also provides an electronic device, comprising a processor and a memory; wherein the memory is configured to store a computer program, the computer program is loaded and executed by the processor to implement any of the foregoing methods.

[0060] In yet another aspect, the present application also provides a storage medium, wherein the storage medium stores computer executable instructions, the computer executable instructions are loaded and executed by a processor to implement any of the foregoing methods.

[0061] The present application can provide a song synthesis interface; the song synthesis interface is configured to receive a song synthesis instruction of a target song from a user; in response to the song synthesis instruction, a model training interface is provided; the model training interface provides a plurality of user audios for the user to select; based on the user-selected song with the user's voice, a song synthesis model is trained; then the song synthesis model is called to change the voice in the target song to the user's voice to obtain a synthesized song; finally, the synthesized song is displayed on the song synthesis interface. It can be seen that the present application allows the user to independently select the training data (i.e., the user-selected song with the user's voice) for training the song synthesis model, and then trains the song synthesis model based on the training data. The song synthesis model can change the voice in the target song to the user's voice while retaining the original song content and melody in the song, so the vibrato, glissando and breathy singing skills in the song are also retained, making the synthesized song more real and natural. Based on the present application, the user can train his own song synthesis model, making the model training more flexible; the user uses the song synthesis model to change the voice in any song to his own voice while retaining the melody and other contents in the song, thus obtaining a synthesized song with the user's voice, thereby improving the operability and synthesis quality of song synthesis.

[0062] The song synthesis model provided in the application can convert the tone in a song to another tone while retaining the original song content and melody in the song, so that the vibrato, slide and breath in the song are also retained, making the synthesized song more real and natural. In specific implementation, a user only needs to specify a song and audio with a target tone based on a client, and then sends a song synthesis request accordingly, so that the background can determine the target audio of the song audio (i.e., the original song) and the target tone (i.e., the target tone) specified in the song synthesis request; call the preset song synthesis model to construct a synthesis spectrum including the tone characteristics of the target tone, the song content characteristics of the song audio and the song fundamental frequency based on the target spectrum of the target audio, the song spectrum of the song audio and the song fundamental frequency; input the synthesis spectrum into the preset vocoder to make the vocoder output a synthesized song including the target tone, the song content and the song melody of the song audio. It can be seen that the scheme can retain the original song content and melody in the song audio, and add the target tone to obtain the synthesized song. The song synthesis model in the scheme takes the audio spectrum as the input data, and does not involve complex phonemes and music scores, so that the model training process can avoid excessive music score labeling and phoneme labeling work, and has high training effect.

[0063] Correspondingly, the song synthesis component, the song synthesis model training method and the component provided in the application also have the above technical effects. The component includes a device and a storage medium. BRIEF DESCRIPTION OF DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0065] Figure 1 A physical architecture schematic diagram applicable to the application is provided for the application.

[0066] Figure 2 A first song synthesis method flowchart is provided for the application.

[0067] Figure 3 A song synthesis interface schematic diagram is provided for the application.

[0068] Figure 4 A model training interface schematic diagram is provided for the application.

[0069] Figure 5 A prompt interface schematic diagram after model training is completed is provided for the application.

[0070] Figure 6 A song synthesis progress diagram is provided for the present application;

[0071] Figure 7 A second song synthesis method flowchart is provided for the present application;

[0072] Figure 8 A third song synthesis method flowchart is provided for the present application;

[0073] Figure 9 A song synthesis model training method flowchart is provided for the present application;

[0074] Figure 10 A song synthesis model training step diagram is provided for the present application;

[0075] Figure 11 A song synthesis model application flowchart is provided for the present application;

[0076] Figure 12 A song synthesis model structure diagram is provided for the present application;

[0077] Figure 13 A server structure diagram is provided for the present application;

[0078] Figure 14 A terminal structure diagram is provided for the present application. DETAILED DESCRIPTION

[0079] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application. In addition, in the embodiments of the present application, "first", "second", etc. are used to distinguish similar objects, not necessarily to describe a specific order or sequence.

[0080] The existing song synthesis model takes complex information such as phonemes and music scores as synthesis elements, and the pitch information comes from the music score, so the synthesis effect is generally mechanical. At the same time, the singing skills such as trill, slide and breath in the song are difficult to construct in the synthesized song. The existing song synthesis model also needs to use songs with annotated phonemes, phoneme duration and music score as training set when training, and the annotation of the training set also consumes a lot of cost. Moreover, the training and application of the model are not controlled by the user, but by the model provider to train the same model for multiple users. That is, the user cannot customize the model.

[0081] In view of the above-mentioned problems, this application proposes a singing synthesis solution that allows users to independently train the model. The trained model can convert the timbre of a song into another timbre while retaining the original song content and melody. Therefore, singing techniques such as vibrato, glissando, and breath in the song are also retained, making the synthesized song more realistic and natural. The singing synthesis model in the solution does not involve complex elements such as phonemes and music scores. Therefore, the model training process can avoid excessive music score annotation and phoneme annotation work, and the training effect is relatively high.

[0082] To facilitate understanding, the physical framework to which this application applies is first introduced.

[0083] It should be understood that the singing voice synthesis method provided in this application can be applied to a system or program having a singing voice synthesis function. Specifically, the system or program having a singing voice synthesis function can be run on a device such as a server or a personal computer. Correspondingly, the singing voice synthesis model training method can be applied to a system or program having a singing voice synthesis model training function. Specifically, the system or program having a singing voice synthesis model training function can be run on a device such as a server or a personal computer.

[0084] like Figure 1 As shown, Figure 1 This is a diagram of the physical architecture applicable to this application. Figure 1 In the present invention, a system or program with a singing voice synthesis function can be run on a server, which receives a song synthesis request sent by a client from other terminal devices through a network; determines the song audio and target audio of the target timbre specified in the song synthesis request; calls a preset singing voice synthesis model to construct a synthetic spectrum including the timbre characteristics of the target timbre, the song content characteristics of the song audio and the song fundamental frequency characteristics based on the target spectrum of the target audio, the song spectrum of the song audio and the song fundamental frequency; inputs the synthetic spectrum into a preset vocoder so that the vocoder outputs a synthetic song including the target timbre, the song content of the song audio and the song melody. Among them, the singing voice synthesis solution provided in this application does not change the arrangement structure of the original song, but only changes the timbre. Of course, the singing voice synthesis is carried out after the consent of the song owner.

[0085] Accordingly, a system or program with a singing synthesis model training function can run on a server, which receives a model training request sent by a client from other terminal devices through the network; determines a training set specified by the model training request; the training set is: a song set selected by the user with the user's timbre; and performs model training based on the training set to obtain a singing synthesis model.

[0086] When a system or program with a singing synthesis function runs on a client, the client provides a singing synthesis interface; the singing synthesis interface is used to receive a user's song synthesis instruction for a target song; in response to the song synthesis instruction, a model training interface is provided; the model training interface provides multiple user audios for the user to select; based on the song with the user's timbre selected by the user, a singing synthesis model is trained; the singing synthesis model is called to change the timbre in the target song to the user's timbre to obtain a synthesized song; and the synthesized song is displayed on the singing synthesis interface.

[0087] Depend on Figure 1 It can be seen that the server can establish communication connections with multiple devices, and the server obtains song audio and target audio of target timbre from these devices, and synthesizes the song based on the obtained audio.

[0088] Figure 1 A variety of terminal devices are shown in FIG. In actual scenarios, more or fewer types of terminal devices may participate in the process of singing synthesis or model training. The specific number and type depend on the actual scenario and are not limited here. In addition, Figure 1 One server is shown in the figure, but in actual scenarios, multiple servers may be involved, and the specific number of servers depends on the actual scenario.

[0089] It should be noted that the singing voice synthesis method and model training method provided in this embodiment can be performed online or offline.

[0090] It is understood that the above-mentioned systems and programs with singing synthesis functions can be used as a type of cloud service program. The specific operating mode depends on the actual scenario and is not limited here. Naturally, the above-mentioned systems and programs with model training functions can also be used as a type of cloud service program. The specific operating mode depends on the actual scenario and is not limited here.

[0091] Specifically, after the song synthesis is completed, users can share and play the synthesized song on the client, and can also compare the synthesized song with the original song. The synthesized song can participate in activities such as scoring and liking on the client under the user's operation.

[0092] In combination with the above commonalities, please refer to Figure 2 , Figure 2 The first method flow chart of singing synthesis provided in the embodiment of the present application is as follows. Figure 2 As shown, the singing voice synthesis method is applied to the client and may include the following steps:

[0093] S201. Provide a singing synthesis interface; the singing synthesis interface is used to receive a song synthesis instruction from a user for a target song.

[0094] In this embodiment, the singing synthesis interface may include a song list, and each song in the song list corresponds to a button for initiating a song synthesis instruction. When a user clicks the button corresponding to any song in the song list, the user initiates a song synthesis instruction for the song. Figure 3 .like Figure 3 As shown, one song corresponds to one "synthesis button".

[0095] Among them, the user can enter the singing synthesis interface by clicking the entrance of the singing synthesis function provided by the client.

[0096] S202. In response to the song synthesis instruction, a model training interface is provided; the model training interface provides multiple user audios for the user to select.

[0097] S203: Based on the song with the user's timbre selected by the user, a singing synthesis model is trained.

[0098] S204: Call the singing synthesis model to change the timbre of the target song to the user's timbre to obtain a synthesized song.

[0099] S205: Display the synthesized song on the singing synthesis interface.

[0100] If the user clicks Figure 3 The client then provides a model training interface based on the song synthesis instruction of song 1. The model training interface provides multiple user audios for the user to choose from. Figure 4 As shown, users can select songs in this interface. It can be seen that the user audio provided by the model training interface is each song. These songs can be: songs recorded by the user in real time, songs uploaded by the user in real time, and / or songs recorded by the user in the past. These songs have one thing in common, that is, they all include the user's own voice, so these songs are actually songs sung by the user himself. Figure 4 After selecting multiple songs on the interface shown, you can click the "Start Model Training" button at the bottom of the interface to initiate model training. The trained singing synthesis model is bound to the user account.

[0101] The client performs model training based on the model training instructions input by the user and the songs selected by the user. After the training is completed, the user can be notified of the training completion message and prompted to start singing synthesis. Figure 5 As shown, after the training is completed, it can be displayed on the client Figure 5 If the user clicks Figure 5 Click the "Yes" button to jump directly to Figure 3 The singing synthesis interface shown in the figure allows users to select songs to be synthesized; if the user clicksFigure 5 If the "No" button is clicked, the client jumps to the home page.

[0102] In one embodiment, during the process of calling the song synthesis model, the song synthesis progress is displayed in real time in the song synthesis interface. For example, if the user clicks the "Song 1" corresponding synthesis button, the song synthesis progress bar is displayed at the position of "Song 1", and the user is prompted that the song synthesis is in progress. See FIG. 6 for details. Figure 3 Figure 3 The "Song 1" corresponding synthesis button is modified to a "Share" button. After the song synthesis, the user can click the "Share" button to share the synthesized song. The subsequent sharing process can refer to the existing related technology, and the present embodiment will not be described in detail. Figure 6 Figure 3 It should be noted that if the user has his own song synthesis model before selecting a song in the song synthesis interface, after the user clicks the "Song 1" button, the client determines whether the current user has a model. If it is determined that the current user has a model, the client does not provide a model training interface, but calls the song synthesis model to change the timbre in the target song to the user's timbre to obtain a synthesized song, and displays the synthesized song in the song synthesis interface. Therefore, in one embodiment, before providing the model training interface, it further includes: determining whether a song synthesis model exists; if the song synthesis model exists, calling the song synthesis model to change the timbre in the target song to the user's timbre to obtain a synthesized song, and displaying the synthesized song in the song synthesis interface; otherwise, providing the model training interface.

[0103] It should be noted that if the user has his own song synthesis model before selecting a song in the song synthesis interface, after the user clicks the "Song 1" button, the client determines whether the current user has a model. If it is determined that the current user has a model, the client does not provide a model training interface, but calls the song synthesis model to change the timbre in the target song to the user's timbre to obtain a synthesized song, and displays the synthesized song in the song synthesis interface. Therefore, in one embodiment, before providing the model training interface, it further includes: determining whether a song synthesis model exists; if the song synthesis model exists, calling the song synthesis model to change the timbre in the target song to the user's timbre to obtain a synthesized song, and displaying the synthesized song in the song synthesis interface; otherwise, providing the model training interface. Figure 3 In one embodiment, when the client calls the song synthesis model to change the timbre in the target song to the user's timbre, the user's timbre needs to be known. For this purpose, the song synthesis model can be provided with audio with the user's timbre for timbre extraction. In order to avoid the song synthesis model extracting the user's timbre every time the song synthesis model is called, the user's timbre can be stored in advance and bound to the user's account. For example, after the song synthesis model is trained, the user's timbre is extracted and stored by using the song synthesis model, so that the user's timbre can be queried when the song synthesis model is called. The user's timbre is stored in the form of timbre features.

[0104]

[0105] ​​​It can be seen that the embodiment allows the user to independently select the training data (i.e., the song with the user's voice selected by the user) for training the song synthesis model, and then trains the song synthesis model based on the training data, so that the song synthesis model can change the timbre in the target song to the user's voice while retaining the original song content and melody in the song, so that the vibrato, slide and breath in the song are also retained, making the synthesized song more real and natural. Based on the embodiment, the user can independently train the song synthesis model of the user, making the model training more flexible; the user uses the song synthesis model to change the timbre in any song to the user's voice while retaining the melody and other content in the song, so as to obtain a synthesized song with the user's voice, thereby improving the operability and synthesis quality of the song synthesis.

[0106] Please refer to Figure 7 , Figure 7 The second song synthesis method flowchart provided by the embodiment of the present application. As shown in Figure 7 , the song synthesis method is applied to a client and can include the following steps:

[0107] S701, in response to a model training instruction, a model training interface is provided; the model training interface provides a plurality of user audio for the user to select.

[0108] Among them, the user audio provided by the model training interface is each song sung by the user himself, which can be: a song sung by the user in real time, a song uploaded by the user in real time and / or a song sung by the user in the past.

[0109] S702, based on the song with the user's voice selected by the user, a song synthesis model is trained.

[0110] S703, a song synthesis interface is provided so that the user inputs a song synthesis instruction for a target song.

[0111] S704, in response to the song synthesis instruction, the song synthesis model is called to change the timbre in the target song to the user's voice to obtain a synthesized song.

[0112] S705, the synthesized song is displayed on the song synthesis interface.

[0113] In the embodiment, the user first performs model training on the client, and after the song synthesis model is trained, the user jumps to the song synthesis interface for song synthesis. That is, the user clicks the entry of the model training function provided by the client to enter the model training interface shown in Figure 4 , and then the user can check the song on the interface and click the "initiate model training" button below the interface to independently initiate model training.

[0114] After the model training is completed, the client displaysFigure 5 As shown, the user can click Figure 5 the "Yes" button as shown, and directly jump to Figure 3 the song voice synthesis interface shown, so that the user can select a song to be subjected to song voice synthesis. If the user clicks the synthesis button corresponding to Figure 3 "Song 1", a synthesis progress bar is displayed at the position of Figure 3 "Song 1", and the user is prompted that the song synthesis is being performed, as shown in Figure 6 At this time Figure 3 the song 1 is subjected to song synthesis, and the "synthesis button" corresponding to the song 1 is modified to a "share" button. After the song synthesis, the user can click the "share" button to share the synthesized song. The subsequent sharing process can refer to the related art, and will not be described herein.

[0115] Based on the embodiment, the user can train the song voice synthesis model by himself, so that the model training is more flexible. The user uses the song voice synthesis model to change the tone in any song to his own tone, while retaining the melody and other contents in the song, so as to obtain a synthesized song with the user's tone, thereby improving the operability and synthesis quality of the song synthesis.

[0116] Please refer to Figure 8 , Figure 8 a third song voice synthesis method flowchart provided by the embodiment of the present application. As shown in Figure 8 , the song voice synthesis method can include the following steps:

[0117] S801, receiving a song synthesis request sent by a client.

[0118] S802, determining a target audio of a song audio and a target tone specified by the song synthesis request.

[0119] In the embodiment, the client can be an application installed on a terminal, and the user can send a song synthesis request to the server through the client. Of course, when sending the song synthesis request, the user needs to specify a certain song and another audio with a target tone on the interface of the client. The other audio with the target tone can be a song with the target tone or a segment of human voice. Therefore, the server can determine the song audio and the target tone specified by the user after receiving the song synthesis request. It can be seen that the method provided by the embodiment can be applied to the server. Of course, if the client has sufficient computer resources locally, the terminal installing the client can also execute the method provided by the embodiment.

[0120] S803, calling a preset song synthesis model to construct a synthesis spectrum comprising timbre features of the target timbre, song content features of the song audio, and song fundamental frequency features based on the target spectrum of the target audio, the song spectrum of the song audio, and the song fundamental frequency.

[0121] In the embodiment, the input of the song synthesis model is the target spectrum of the target audio, the song spectrum of the song audio, and the song fundamental frequency, and the output is the synthesis spectrum comprising the timbre features of the target timbre, the song content features of the song audio, and the song fundamental frequency features. Therefore, in one specific implementation, calling the preset song synthesis model to construct the synthesis spectrum comprising the timbre features of the target timbre, the song content features of the song audio, and the song fundamental frequency features based on the target spectrum of the target audio, the song spectrum of the song audio, and the song fundamental frequency comprises: extracting the song spectrum and the song fundamental frequency from the song audio; extracting the target spectrum from the target audio; inputting the song spectrum into a content encoder in the song synthesis model to obtain the song content features; inputting the target spectrum into a timbre encoder in the song synthesis model to obtain the timbre features; inputting the song fundamental frequency into a word embedding module in the song synthesis model to obtain the song fundamental frequency features; superimposing the song content features, the timbre features, and the song fundamental frequency features, and inputting the superimposition result into a decoder in the song synthesis model to obtain the synthesis spectrum.

[0122] In one specific implementation, inputting the song spectrum into the content encoder in the song synthesis model to obtain the song content features comprises: inputting the song spectrum into the content encoder to enable the content encoder to extract continuous content features in the song spectrum and perform vector quantization on the continuous content features to obtain the song content features. In one specific implementation, inputting the target spectrum into the timbre encoder in the song synthesis model to obtain the timbre features comprises: inputting the target spectrum into the timbre encoder to enable the timbre encoder to extract continuous timbre features in the target spectrum and perform mean value calculation on the continuous timbre features to obtain the timbre features. In one specific implementation, inputting the song fundamental frequency into the word embedding module in the song synthesis model to obtain the song fundamental frequency features comprises: inputting the song fundamental frequency into the word embedding module to enable the word embedding module to map the song fundamental frequency into the song fundamental frequency features represented by a vector.

[0123] It can be seen that the song synthesis model comprises a content encoder, a timbre encoder, a word embedding module, and a decoder. The content encoder is configured to obtain song content features from a song spectrum. The timbre encoder is configured to obtain timbre features of a target audio from a target spectrum of the target audio. The word embedding module is configured to obtain song base frequency features from a song base frequency. The decoder is configured to decode and synthesize a superimposed result of the song content features, the timbre features, and the song base frequency features to obtain a synthesis spectrum comprising the timbre features of the target timbre, the song content features of the song audio, and the song base frequency features of the song melody. Accordingly, the synthesis spectrum can obtain a new song that only changes the timbre, but retains the song content features and the song base frequency.

[0124] In S804, the synthesis spectrum is input into a preset vocoder, so that the vocoder outputs a synthesized song comprising the target timbre, the song content of the song audio, and the song melody.

[0125] The vocoder in the embodiment can convert the synthesis spectrum into a synthesized song audible to human ears.

[0126] Of course, if the user wants to change the pitch of the original song when synthesizing the song, the user can issue a pitch raising or lowering request in the relevant interface of the client. In this way, the server can adjust the song base frequency of the original song according to the pitch raising or lowering request, and then extract the base frequency features. Therefore, in a specific implementation, if a pitch raising or lowering request sent by the client is received, before the song base frequency is input into the word embedding module in the song synthesis model to obtain the song base frequency features, the song base frequency is also subjected to pitch raising or lowering. Specifically, the following situations can change the pitch of the original song: for example, the original song is a male lead singer, and the target timbre is a female timbre. In this case, it is appropriate to raise the pitch of the original song a little. Conversely, the original song is a female lead singer, and the target timbre is a male timbre. In this case, it is appropriate to lower the pitch of the original song a little.

[0127] After obtaining the synthesized song, the server can return the synthesized song to the client, so that the synthesized song can be displayed and played on the client under the operation of the user. If necessary, the user can also play the original song to compare the differences before and after synthesis. The user can also share and play the synthesized song based on the client. The synthesized song can also participate in activities such as scoring and liking on the client under the operation of the user.

[0128] It can be seen that the song synthesis model provided in the embodiment can convert the tone in the song to another tone while retaining the original song content and melody in the song, so the vibrato, slide and breath singing skills in the song are also retained, making the synthesized song more real and natural. In specific implementation, a user only needs to specify a song and an audio with a target tone based on a client, and then sends a song synthesis request accordingly, so that the background can determine the target audio of the song audio (i.e., the original song) and the target tone (i.e., the target tone) specified in the song synthesis request; call the preset song synthesis model to construct a synthesis spectrum including the tone features of the target tone, the song content features of the song audio and the song fundamental frequency features based on the target spectrum of the target audio, the song spectrum of the song audio and the song fundamental frequency; input the synthesis spectrum into the preset vocoder to make the vocoder output a synthesized song including the target tone, the song content and the song melody of the song audio. It can be seen that the scheme can retain the original song content and melody in the song audio while adding the target tone to obtain the synthesized song. In addition, the song synthesis model in the scheme takes the audio spectrum as the input data and does not involve complex elements such as phonemes and scores, so the model training process can avoid excessive score labeling and phoneme labeling work, and the training effect is higher.

[0129] See Figure 9 , Figure 9 A song synthesis model training method flowchart is provided in the embodiment of the present application. As shown in Figure 9 , a song synthesis model training method can include the following steps:

[0130] S901, receiving a model training request sent by a client.

[0131] S902, determining a training set specified in the model training request.

[0132] The training set is: a set of songs selected by a user carrying a user tone. It can be seen that the training set includes a plurality of training songs of the same tone for training the song synthesis model.

[0133] The song synthesis model used in the embodiment is the song synthesis model described in the above embodiment.

[0134] In the embodiment, the client can also be an application installed on a terminal, and the user can send a model training request to the server through the client. Of course, when sending the model training request, the user needs to specify a plurality of songs of the same tone on the interface of the client to form a training set. After receiving the model training request, the server can determine the training set for training the model, and then train the model accordingly. It can be seen that the method provided in the embodiment can be applied to the server. Of course, if the client has sufficient computer resources locally, the terminal installed with the client can also execute the method provided in the embodiment.

[0135] Please refer to Figure 4 , the user can select any song to sing on the interface of the client to form a training set, and then initiate model training with one key. Figure 4 The songs with check marks are the songs selected into the training set, and the songs without check marks are the songs not selected into the training set.

[0136] S903, model training based on the training set to obtain a singing synthesis model.

[0137] As can be seen from the foregoing embodiments, the singing synthesis model includes a content encoder, a timbre encoder, a word embedding module, and a decoder. Therefore, when training the singing synthesis model, the structure of each component part needs to be determined first. Specifically, the structure of the content encoder and the timbre encoder can be completely the same, and when training, the parameters of the content encoder are adapted for song content extraction, and the parameters of the timbre encoder are adapted for timbre extraction. The word embedding module and the decoder can use existing trained models, or use initialized states, that is, the content encoder, the timbre encoder, the word embedding module, and the decoder in the singing synthesis model can all be trained from the initial state.

[0138] Please refer to Figure 10 , the specific model training steps include:

[0139] S1001, obtaining a training song from the training set;

[0140] S1002, extracting a trainable spectrum and a training fundamental frequency from the training song;

[0141] S1003, inputting the trainable spectrum into the content encoder in the initial model to obtain a training content feature of the training song;

[0142] S1004, inputting the trainable spectrum into the timbre encoder in the initial model to obtain a training timbre feature of the training song;

[0143] S1005, inputting the training fundamental frequency into the word embedding module in the initial model to obtain a training fundamental frequency feature of the training song;

[0144] S1006, superimposing the training content feature, the training timbre feature, and the training fundamental frequency feature, and inputting the superimposed result into the decoder in the initial model to obtain a training synthesis spectrum;

[0145] S1007, calculating the loss between the training synthesis spectrum and the trainable spectrum;

[0146] S1008, determining whether the loss meets a preset convergence condition; if yes, performing S1009; otherwise, performing S510;

[0147] S1009, output the current model as a song synthesis model;

[0148] S1010, update the parameters of the content encoder, timbre encoder, decoder and word embedding module in the initial model based on the loss, and perform S1001 to iteratively train the model from another song in the training set until the loss between the training synthesized spectrum and the trainable spectrum meets the preset convergence condition.

[0149] As shown in Figure 10 S1003, S1004 and S1005 can be executed in parallel, and of course can also be executed in no particular order.

[0150] In this embodiment, the content encoder can extract continuous content features in the trainable spectrum and vector quantize the continuous content features to obtain training content features. The timbre encoder can extract continuous timbre features in the trainable spectrum and perform mean value calculation on the continuous timbre features to obtain training timbre features. The word embedding module can map the training fundamental frequency to training fundamental frequency features represented by vectors. It can be seen that the functions of the components of the model in the training process are exactly the same as those in the model use process, and they can be referred to each other when necessary.

[0151] It can be seen that the song synthesis model provided in this embodiment can convert the timbre in a song to another timbre, while also retaining the original song content and melody in the song, so the singing skills such as trill, slide and breath in the song are also retained, making the synthesized song more real and natural. The training scheme can retain the original song content and melody in the song audio, and then add the target timbre to obtain the synthesized song. The song synthesis model in this scheme takes audio spectrum as input data and does not involve complex phonemes, scores and other elements, so the model training process can avoid excessive score labeling and phoneme labeling work, and the training effect is higher.

[0152] The song synthesis scheme provided in this application is described below through a specific application scenario example. The scheme can only change the timbre in a song, but also retains the original song content and melody in the song.

[0153] Please refer to Figure 11 , the specific process of the song synthesis scheme includes:

[0154] S1101, the user specifies a song and an audio with a target timbre on the terminal, and the terminal generates a song synthesis request based on the user's specification;

[0155] S1102, the terminal sends the song synthesis request to the server;

[0156] S1103. The server determines the song specified by the user and an audio segment with a target timbre based on the song synthesis request, and calls a preset singing synthesis model to change the spectrum of the song specified by the user into a synthesized spectrum including the target timbre, the original song content and the fundamental frequency, and inputs the synthesized spectrum into a preset vocoder so that the vocoder outputs a synthesized song with a different timbre from the song specified by the user but with exactly the same melody and content.

[0157] S1104, the server sends the synthesized song to the terminal;

[0158] S1105: After the terminal receives the synthesized song, the user can play the synthesized song.

[0159] The terminal may be a smart phone, TV, tablet computer, laptop computer, desktop computer or other device.

[0160] In actual application, the present application can be set in a music playing tool, which is installed on a user terminal.

[0161] See Figure 12 The singing synthesis model in this embodiment includes a content encoder, a timbre encoder, a word embedding module (for fundamental frequency extraction, i.e. Figure 12 PitchExtract) and decoder (decoder).

[0162] It should be noted that the spectrum features of the original song need to be extracted, such as Figure 12 Mel-spectral features (Log-Mel Spec) are extracted from the audio signal to be fed into the content encoder and timbre encoder. Specifically, the audio signal is framed and windowed, then the linear spectrum is calculated using a Fourier transform. Finally, a Mel-scale filter bank is used to transform the linear spectrum into a Mel-scale spectrum, effectively converting the linear scale of the spectrum into a Mel-scale spectrum.

[0163] At the same time, the fundamental frequency of the original song can be extracted using a commonly used fundamental frequency extraction tool (such as pYIN). Then, through the word embedding module, each fundamental frequency value is mapped to a vector to obtain the fundamental frequency feature P. Fundamental frequency information is important information about the original song.

[0164] In one example, the content encoder and the timbre encoder have the same structure. Figure 12As shown, the two modules are composed of multiple residual connections. A Log-Mel Spec feature (denoted as X) is processed through a convolution (conv), batch normalization (batchnorm) and a rectified linear unit (ReLU) module to obtain Y, and then X and Y are added to avoid gradient disappearance of the model. X+Y is taken as the input of the next residual connection structure, and so on. The residual connection structure set here is 5, the convolution kernel size is 3, and the channel number is 128. After the encoding is completed, a linear layer (Linear) calculation is performed to reduce the dimension of the feature.

[0165] It should be noted that although the content encoder and the timbre encoder use the same basic structure as described above, the content encoder will output Z after vector quantization of the continuous features output by the linear layer, which converts the continuous latent space into a discrete latent space, allowing the codebook information to be shared between different timbres. The timbre encoder will perform a mean calculation on the linear layer output and output S as the target person's timbre representation, which avoids interference from different pronunciation content at different positions in the audio to the representation of the timbre.

[0166] After superimposing the fundamental frequency feature P, the Z output by the content encoder, and the S output by the timbre encoder, the superimposed result is input into the decoder to obtain the newly synthesized spectral feature (Predict Log-Mel Spec) in the audio signal. Figure 12 The decoder uses the decoder of the FASTSPEECH model, which is composed of 6 feed-forward networks (FFN). FFN can realize non-autoregressive feature synthesis and has good performance. In addition, FFN also has better performance in non-autoregressive synthesis tasks. The output of the decoder is also a mel spectrum feature.

[0167] Based on the above model structure, the model training process includes: for a piece of audio signal, first extract the fundamental frequency information and the mel spectrum feature. The fundamental frequency information will first pass through the word embedding module to obtain a P vector. The mel spectrum feature will pass through the content encoder and the timbre encoder respectively to obtain Z vector and S vector.

[0168] In the content encoder, the mel spectrum feature passes through 5 residual connection structures and is dimensionally reduced, and then a latent variable CEnc(x) is obtained through nearest neighbor search, CEnc(x) = e k , where k = argmin j || CEnc(x) - e j || 2, that is, the final output Z after vector quantization. Where e kFor the kth vector in the codebook with dimension [N, D], e j For the kth vector in the codebook with dimension [N, D], e

[0169] In the timbre encoder, the mel-spectrogram feature will also pass through 5 linear residual connection structures to obtain SEnc(x), and then pass through mean processing to obtain the timbre representation S: S = mean(CEnc(x)). Wherein, mean represents mean processing in the time dimension of a sequence. For example: a [T, D] sequence, after mean processing, a [1, D] vector is obtained.

[0170] Then the content vector Z, the timbre vector S and the fundamental frequency vector P are added to obtain Z+S+P, which is input to the decoder to obtain the predicted mel-spectrogram feature.

[0171] The loss function used by the model is the minimum mean square error (MES Loss). The mean square error is calculated between the predicted mel-spectrogram obtained by the model and the mel-spectrogram calculated from the audio signal, and then the parameters of each component in the model are updated based on the loss value through gradient back propagation. The learning rate is 0.001, and the optimizer is the Adam optimizer. When the loss value of the model converges basically, the training is completed, and thus the song synthesis model is obtained.

[0172] In the model inference stage, the input audio of the content encoder and the timbre encoder is audio A and audio B respectively, and the input of the fundamental frequency encoding module is audio A. Therefore, the song synthesis model can change the timbre of audio A to audio B, thereby obtaining a synthesized audio with the timbre of audio B, the content and melody of audio A. It should be noted that the mel-spectrogram feature output by the song synthesis model can be input into a vocoder model to obtain the corresponding audio signal. Common vocoder models include HifiGan and MelGan models.

[0173] In addition, in order to solve the problem of different vocal ranges of men and women, the fundamental frequency information can be edited to realize the modification of the pitch of the song. For example: after adjusting or lowering the fundamental frequency information, the fundamental frequency encoding is performed.

[0174] As can be seen, the scheme provided by the embodiment can train a model for changing the timbre of audio while preserving the content and melody of the audio. The model can convert songs with other timbres to a target timbre, and the model does not need to perform phoneme labeling on song data during training. Through the timbre encoder, any timbre can be converted, and through the decoder, the synthesized song can not only ensure that there is no pitch deviation, but also support the modification of the pitch of the song. In specific applications, the user can select a work he has sung as the training set to train a song synthesis model with his own timbre as the target timbre, such as Figure 4After the model training is completed, you can select the song you want to synthesize in the singing synthesis interface to synthesize a song with your own voice.

[0175] An electronic device provided in an embodiment of the present application is introduced below. The electronic device described below and the related implementation steps of the above embodiment can be referenced to each other.

[0176] Furthermore, the embodiment of the present application also provides an electronic device. The electronic device can be Figure 13 The server 50 shown may also be Figure 14 Terminal 60 is shown. Figure 13 and Figure 14 Each of the diagrams is a structural diagram of an electronic device according to an exemplary embodiment, and the contents in the diagrams cannot be considered as any limitation on the scope of use of the present application.

[0177] Figure 13 This is a schematic diagram of the structure of a server provided in an embodiment of the present application. The server 50 may include: at least one processor 51, at least one memory 52, a power supply 53, a communication interface 54, an input / output interface 55, and a communication bus 56. The memory 52 is used to store a computer program, which is loaded and executed by the processor 51 to implement the relevant steps of the singing synthesis disclosed in any of the aforementioned embodiments.

[0178] In this embodiment, the power supply 53 is used to provide operating voltage for each hardware device on the server 50; the communication interface 54 can create a data transmission channel between the server 50 and external devices. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 55 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0179] In addition, the memory 52, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon include an operating system 521, a computer program 522 and data 523, etc. The storage method can be temporary storage or permanent storage.

[0180] The operating system 521 is used to manage and control the various hardware devices on the server 50 and the computer program 522, and can be Windows Server, Netware, Unix, Linux, etc. The computer program 522 can further include computer programs for completing other specific work in addition to the computer programs for completing the song synthesis method disclosed in any of the embodiments. The data 523 can include the developer information of the application program and other data in addition to the update information of the application program.

[0181] Figure 14 A structure schematic diagram of a terminal provided in the embodiments of the present application is shown in FIG. 6. The terminal 60 can specifically include, but is not limited to, a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.

[0182] Generally, the terminal 60 in the embodiments includes a processor 61 and a memory 62.

[0183] The processor 61 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 61 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). The processor 61 can also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 61 can be integrated with a GPU (Graphics Processing Unit) that is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 61 can further include an AI (Artificial Intelligence) processor for processing machine learning-related computing operations.

[0184] The memory 62 can include one or more computer-readable storage media. The memory 62 can also include high-speed random access memory and can include non-volatile, computer-readable storage media such as one or more magnetic disk storage devices, optical storage devices, flash memory devices, solid-state memory devices, or the like. In this embodiment, the memory 62 stores the following computer programs 621, which are loaded and executed by the processor 61 to implement the steps of the song synthesis method performed by the terminal side disclosed in any of the preceding embodiments. In addition, the memory 62 can also store an operating system 622, data 623, and the like, and the storage mode can be temporary storage or permanent storage. The operating system 622 can include Windows, Unix, Linux, and the like. The data 623 can include, but is not limited to, application update information.

[0185] In some embodiments, the terminal 60 can further include a display screen 63, an input / output interface 64, a communication interface 65, a sensor 66, a power supply 67, and a communication bus 68.

[0186] Those skilled in the art can understand that the structure shown in the above embodiments does not constitute a limitation on the terminal 60, and can include more or fewer components than those shown in the figure. Figure 14

[0187] The following describes a storage medium provided by the embodiments of the present application. The storage medium described below can be referred to the related implementation steps of the above embodiments.

[0188] Further, the embodiments of the present application also disclose a storage medium, which stores computer executable instructions. When the computer executable instructions are loaded and executed by a processor, the song synthesis method disclosed in any of the preceding embodiments is implemented. For the specific steps of the method, refer to the corresponding content disclosed in the preceding embodiments, which will not be repeated here.

[0189] It should be noted that the above are only the preferred embodiments of the present application, and do not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0190] In the specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other.

[0191] ​The principles and implementation manners of the present application are described by using specific examples in the present application, and the above examples are only used to help understand the method and its core idea of the present application; meanwhile, for the general technical personnel in the art, the specific implementation manners and application ranges will be changed according to the idea of the present application, and the above description should not be understood as the limitation of the present application.

Claims

1. A singing voice synthesis method, characterized in that: include: Receive song synthesis request sent by the client; Determine the target audio of the song audio and the target timbre specified in the song synthesis request; Calling a preset singing synthesis model to construct a synthetic spectrum including the timbre characteristics of the target timbre, the song content characteristics of the song audio, and the song fundamental frequency characteristics based on the target spectrum of the target audio, the song spectrum of the song audio, and the song fundamental frequency; Inputting the synthesized spectrum into a preset vocoder so that the vocoder outputs a synthesized song including the target timbre, the song content of the song audio, and the song melody; The calling of the preset singing synthesis model constructs a synthetic spectrum including the timbre characteristics of the target timbre, the song content characteristics of the song audio, and the song fundamental frequency characteristics based on the target spectrum of the target audio, the song spectrum of the song audio, and the song fundamental frequency, including: Extracting the song spectrum and the song fundamental frequency from the song audio; extracting the target spectrum from the target audio; Inputting the song spectrum into the content encoder in the singing synthesis model to obtain the song content features; Inputting the target spectrum into a timbre encoder in the singing voice synthesis model to obtain the timbre feature; Inputting the song fundamental frequency into the word embedding module in the singing synthesis model to obtain the song fundamental frequency feature; The song content feature, the timbre feature and the song fundamental frequency feature are superimposed, and the superposition result is input into the decoder in the singing synthesis model to obtain the synthesized spectrum.

2. The method according to claim 1, characterized in that If a pitch-up or pitch-down request is received from the client, the method further includes: inputting the song fundamental frequency into the word embedding module in the singing synthesis model to obtain the song fundamental frequency feature. The fundamental frequency of the song is pitched up or down.

3. The method according to claim 1, characterized in that The step of inputting the song spectrum into the content encoder of the singing synthesis model to obtain the song content features includes: The song spectrum is input into the content encoder so that the content encoder extracts continuous content features from the song spectrum and performs vector quantization on the continuous content features to obtain the song content features.

4. The method according to claim 1, wherein The step of inputting the target spectrum into a timbre encoder in the singing synthesis model to obtain the timbre feature comprises: The target spectrum is input into the timbre encoder so that the timbre encoder extracts continuous timbre features from the target spectrum and performs mean calculation on the continuous timbre features to obtain the timbre features.

5. The method according to claim 1, wherein The step of inputting the song fundamental frequency into the word embedding module in the singing synthesis model to obtain the song fundamental frequency feature includes: The song fundamental frequency is input into the word embedding module so that the word embedding module maps the song fundamental frequency into the song fundamental frequency feature represented by a vector.

6. The method according to any one of claims 1 to 4, characterized in that Also includes: The synthesized song is returned to the client for display and playback.

7. A singing synthesis model training method, characterized in that: include: Receive model training requests sent by the client; Determine a training set specified by the model training request; the training set is: a song set with the user's timbre selected by the user; Model training is performed based on the training set to obtain the singing synthesis model; the singing synthesis model is the singing synthesis model used in the singing synthesis method according to any one of claims 1 to 6.

8. The method according to claim 7, characterized in that The model training is performed based on the training set to obtain the singing synthesis model, including: Obtaining training songs from the training set; Extracting a trainable spectrum and a training fundamental frequency from the training song; Inputting the trainable spectrum into a content encoder in an initial model to obtain training content features of the training song; Inputting the trainable spectrum into the timbre encoder in the initial model to obtain the training timbre features of the training song; Inputting the training fundamental frequency into the word embedding module in the initial model to obtain the training fundamental frequency features of the training song; Superimposing the training content feature, the training timbre feature, and the training fundamental frequency feature, and inputting the superimposed result into a decoder in the initial model to obtain a training synthetic spectrum; Calculating a loss between the training synthetic spectrum and the trainable spectrum; If the loss meets the preset convergence conditions, the current model is output as the singing synthesis model; otherwise, the parameters of the content encoder, the timbre encoder, the decoder and the word embedding module in the initial model are updated based on the loss, and another training song is selected from the training set to iteratively train the model.

9. A singing voice synthesis method, characterized in that: include: Providing a singing synthesis interface; the singing synthesis interface is used to receive a song synthesis instruction from a user for a target song; In response to the song synthesis instruction, a model training interface is provided; the model training interface provides multiple user audios for user selection; Based on the user-selected songs with the user's timbre, a singing synthesis model is trained; Calling the singing synthesis model to change the timbre of the target song to the user timbre to obtain a synthesized song; the singing synthesis model is the singing synthesis model used in the singing synthesis method according to any one of claims 1 to 6; The synthesized song is displayed on the singing synthesis interface.

10. The method according to claim 9, characterized in that Also includes: During the process of calling the singing synthesis model, the singing synthesis progress is displayed in real time on the singing synthesis interface.

11. The method according to claim 9, characterized in that Before providing the model training interface, it also includes: Determining whether the singing voice synthesis model already exists; If the singing synthesis model already exists, the steps of calling the singing synthesis model to change the timbre of the target song to the user timbre to obtain a synthesized song are executed; and the steps of displaying the synthesized song on the singing synthesis interface are displayed; otherwise, a model training interface is provided.

12. The method according to any one of claims 9 to 11, characterized in that Also includes: After the singing voice synthesis model is trained, the user timbre is extracted and stored using the singing voice synthesis model so that the user timbre can be queried when the singing voice synthesis model is called.

13. A singing voice synthesis method, characterized in that: include: In response to the model training instruction, providing a model training interface; The model training interface provides multiple user audios for the user to select; Based on the user-selected songs with the user's timbre, a singing synthesis model is trained; Provide a singing synthesis interface so that users can input singing synthesis instructions for target songs; In response to the song synthesis instruction, calling the singing synthesis model to change the timbre of the target song to the user timbre to obtain a synthesized song; the singing synthesis model is the singing synthesis model used in the singing synthesis method according to any one of claims 1 to 6; The synthesized song is displayed on the singing synthesis interface.

14. The method according to claim 13, characterized in that The multiple user audios provided by the model training interface are: songs recorded by the user in real time, songs uploaded by the user in real time and / or songs recorded by the user in the past.

15. An electronic device, characterized in that: The electronic device comprises a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the method according to any one of claims 1 to 14.

16. A storage medium, characterized in that The storage medium stores computer-executable instructions, which, when loaded and executed by a processor, implement the method according to any one of claims 1 to 14.

Images