Reversed singing song generation method, computer equipment, storage medium and program product

Abstract

The invention relates to a turning song generation method, computer equipment, a storage medium and a program product. The method comprises the following steps: acquiring lyric information of a song to be turned and sung and pitch information of the song to be turned and sung; aligning the lyric information and the pitch information through a pre-trained lyric pitch alignment model to obtain music score information corresponding to the to-be-turned song; and according to the music score information corresponding to the to-be-turned song, generating a dry singing sound with the target tone, and according to the dry singing sound and the song accompaniment corresponding to the to-be-turned song, generating a turned song corresponding to the to-be-turned song. By adopting the method, the incidence relation and / or the corresponding relation between the lyric information and the pitch information can be determined by utilizing the lyric pitch alignment model, so that the singing sound is generated instead of purely depending on the separated dry sound, the singing effect of the singing sound is prevented from being influenced by the quality of the separated dry sound, and the singing efficiency is improved. And the turning and singing effect of the turning and singing song is optimized.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus, computer device, computer-readable storage medium and computer program product for generating a cover song. Background Art

[0002] Singing synthesis can generate songs by analyzing and extracting the user's timbre characteristics, combining the song's music score to simulate and synthesize the user's voice. Singing synthesis can generate songs sung with a timbre different from the original singer's timbre, thereby realizing song cover.

[0003] Traditionally, song covers are mainly achieved by separating the original singer's dry voice and converting the singing voice timbre. However, the cover effect of traditional technology is limited by the quality of the separated original singer's dry voice. The poor quality of the original singer's dry voice will affect the singing effect of the final cover song, which is not conducive to optimizing the cover effect of the cover song. Summary of the invention

[0004] Based on this, it is necessary to provide a cover song generation method, device, computer equipment, computer-readable storage medium and computer program product that can optimize the cover effect of cover songs in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for generating a cover song, comprising:

[0006] Obtaining lyrics information of a song to be covered and pitch information of the song to be covered; the timestamp information carried by the lyrics information of the song to be covered represents the start time and end time of each text in the song to be covered; the timestamp information carried by the pitch information of the song to be covered represents the start time and end time of each pitch in the song to be covered;

[0007] The lyrics information of the song to be covered and the pitch information of the song to be covered are aligned by a pre-trained lyrics pitch alignment model to obtain music score information corresponding to the song to be covered; the music score information represents the correlation between the lyrics information of the song to be covered and the pitch information of the song to be covered;

[0008] A dry singing voice with a target timbre is generated according to the music score information corresponding to the song to be covered, and a cover song corresponding to the song to be covered is generated according to the dry singing voice and the song accompaniment corresponding to the song to be covered.

[0009] In one embodiment, aligning the lyrics information of the song to be covered and the pitch information of the song to be covered to obtain the music score information corresponding to the song to be covered includes:

[0010] Generate a lyrics time sequence according to the lyrics information of the song to be covered and the timestamp information carried by the lyrics information of the song to be covered; generate a pitch time sequence according to the pitch information of the song to be covered and the timestamp information carried by the pitch information of the song to be covered;

[0011] The music score information corresponding to the song to be covered is determined according to the similarity between the lyrics time series and the pitch time series.

[0012] In one embodiment, determining the music score information corresponding to the song to be covered according to the similarity between the lyrics time series and the pitch time series includes:

[0013] Generate a target matrix according to the lyrics time sequence and the pitch time sequence, wherein the elements of the target matrix are used to represent the distance between the lyrics information in the lyrics time sequence and the pitch information in the pitch time sequence;

[0014] The shortest path between the lyrics time series and the pitch time series is determined according to the target matrix, and the shortest path is used as the music score information corresponding to the song to be covered.

[0015] In one embodiment, generating a dry singing voice with a target timbre according to the music score information corresponding to the song to be covered includes:

[0016] Generate a dry voice with a preset timbre according to the music score information corresponding to the song to be covered, and obtain timbre characteristic information of a target timbre, where the target timbre is the timbre selected by the user;

[0017] The timbre feature information of the dry singing voice with the preset timbre and the target timbre is input into a pre-trained timbre conversion model to generate the dry singing voice with the target timbre.

[0018] In one embodiment, before the step of aligning the lyrics information of the song to be covered and the pitch information of the song to be covered by using a pre-trained lyrics pitch alignment model, the method further includes:

[0019] Modifying the lyrics information of the song to be covered to obtain modified lyrics information;

[0020] The modified lyrics information and the pitch information of the song to be covered are aligned by the pre-trained lyrics pitch alignment model to obtain the music score information corresponding to the song to be covered.

[0021] In one embodiment, before the step of aligning the lyrics information of the song to be covered and the pitch information of the song to be covered by using a pre-trained lyrics pitch alignment model, the method further includes:

[0022] Modifying the pitch information of the song to be covered to obtain modified pitch information;

[0023] The lyrics information and the pitch information of the song to be covered are aligned through the pre-trained lyrics pitch alignment model to obtain the music score information corresponding to the song to be covered.

[0024] In a second aspect, the present application also provides a method for training a lyrics pitch alignment model, comprising:

[0025] Acquire lyrics information of a sample song and pitch information of the sample song; the timestamp information carried by the lyrics information of the sample song represents the start time and end time of each text in the sample song; the timestamp information carried by the pitch information of the sample song represents the start time and end time of each pitch in the sample song;

[0026] Aligning the lyrics information of the sample song and the pitch information of the sample song through a lyrics pitch alignment model to obtain music score information corresponding to the sample song;

[0027] According to the music score information corresponding to the sample song and the actual music score information of the sample song, the lyrics pitch alignment model is trained to obtain a pre-trained lyrics pitch alignment model, wherein the pre-trained lyrics pitch alignment model is used in the cover song generation method as described.

[0028] In a third aspect, the present application also provides a cover song generation device, comprising:

[0029] An acquisition module is used to acquire lyrics information of a song to be covered and pitch information of the song to be covered; the timestamp information carried by the lyrics information of the song to be covered represents the start time and end time of each text in the song to be covered; the timestamp information carried by the pitch information of the song to be covered represents the start time and end time of each pitch in the song to be covered;

[0030] An alignment module, used for aligning the lyrics information of the song to be covered and the pitch information of the song to be covered by using a pre-trained lyrics pitch alignment model to obtain music score information corresponding to the song to be covered; the music score information represents the correlation between the lyrics information of the song to be covered and the pitch information of the song to be covered;

[0031] A generation module is used to generate a dry singing voice with a target timbre according to the music score information corresponding to the song to be covered, and to generate a cover song corresponding to the song to be covered according to the dry singing voice and the song accompaniment corresponding to the song to be covered.

[0032] In a fourth aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the above method are implemented.

[0033] In a fifth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps of the above method when executed by a processor.

[0034] In a sixth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0035] The above-mentioned cover song generation method, device, computer equipment, computer-readable storage medium and computer program product respectively obtain the lyrics information and pitch information of the song to be covered, and use the pre-trained lyrics pitch alignment model to align the lyrics information and the pitch information based on the timestamp information carried by the lyrics information and the timestamp information carried by the pitch information, and obtain the music score information for generating the cover song, so as to use the music score information to generate a dry singing voice with a preset timbre, and generate a cover song according to the dry singing voice and the song accompaniment, and can use the lyrics pitch alignment model to determine the correlation and / or correspondence between the lyrics information and the pitch information, and then generate the cover singing voice, instead of simply relying on the separated dry voice to generate the cover singing voice, so as to avoid the quality of the separated dry voice affecting the singing effect of the cover singing voice, and optimize the cover effect of the cover song. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 A diagram of an application environment of a method for generating a cover song in one embodiment;

[0038] Figure 2 A schematic diagram of a process for generating a cover song in one embodiment;

[0039] Figure 3 A schematic diagram of a process of singing a song cover in one embodiment;

[0040] Figure 4 A schematic diagram of another process of singing a song cover in one embodiment;

[0041] Figure 5 A schematic diagram of aligning lyrics and pitch in one embodiment;

[0042] Figure 6 A schematic diagram of a process of singing synthesis in one embodiment;

[0043] Figure 7 It is a structural block diagram of a cover song generating device in one embodiment;

[0044] Figure 8 The figure is a diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0046] The cover song generation method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 obtains the lyrics information of the song to be covered, and the pitch information of the song to be covered; the timestamp information carried by the lyrics information of the song to be covered represents the start time and end time of each text in the song to be covered; the timestamp information carried by the pitch information of the song to be covered represents the start time and end time of each pitch in the song to be covered; the server 104 aligns the lyrics information of the song to be covered and the pitch information of the song to be covered through the pre-trained lyrics pitch alignment model, and obtains the music score information corresponding to the song to be covered; the music score information represents the association between the lyrics information of the song to be covered and the pitch information of the song to be covered; the server 104 generates a dry voice with a target timbre according to the music score information corresponding to the song to be covered, and generates a cover song corresponding to the song to be covered according to the dry voice and the song accompaniment corresponding to the song to be covered. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0047] In an exemplary embodiment, Figure 2 As shown, a method for generating a cover song is provided, and the method is applied to a server as an example for explanation, including the following steps S202 to S206. Among them:

[0048] Step S202, obtaining lyrics information and pitch information of the song to be covered; the timestamp information carried by the lyrics information of the song to be covered represents the start time and end time of each text in the song to be covered; the timestamp information carried by the pitch information of the song to be covered represents the start time and end time of each pitch in the song to be covered.

[0049] The songs to be covered may refer to songs that need to be covered. In practical applications, the songs to be covered may include any songs in a preset song database.

[0050] In actual applications, the lyrics information and pitch information of the song to be covered can be pre-extracted and stored in a preset storage location, the lyrics information of the song to be covered can also be extracted by a lyrics information extraction model, and the pitch information of the song to be covered can also be extracted by a pitch information extraction model.

[0051] The lyrics information extraction model may refer to a model used to determine the lyrics in a song. In practical applications, the lyrics information extraction model may include but is not limited to an automatic speech recognition model (Automatic Speech Recognition, ASR).

[0052] Among them, the lyric information may refer to the text of the lyrics in the song. In actual applications, the lyrics may have corresponding syllables, and the syllables may include several phonemes. In a specific implementation, the lyric information of the song to be covered carries timestamp information, and the timestamp information carried by the lyric information of the song to be covered can represent the start time and end time of each text in the song to be covered.

[0053] Among them, the pitch information extraction model may refer to a model used to determine the pitch in a song. In practical applications, the pitch information extraction model may include but is not limited to a pitch transcription model. The pitch information of the song to be covered may also carry timestamp information. The timestamp information carried by the pitch information of the song to be covered may represent the start time and end time of each pitch in the song to be covered.

[0054] Among them, pitch information can refer to information that characterizes the frequency of sound. In practical applications, pitch can include fundamental frequency. When a sound-emitting body makes sound due to vibration, the sound can generally be decomposed into many simple sine waves. Natural sounds are basically composed of many sine waves with different frequencies. Among them, the sine wave with the lowest frequency is the fundamental frequency, and other sine waves with higher frequencies are overtones.

[0055] As an example, when performing a song cover, the traditional technology usually separates the original dry voice of the song to obtain the original dry voice file, and then inputs the original dry voice file into the singing voice conversion model to generate the cover singing voice. Although the traditional technology has a short link and high production efficiency, the cover singing effect of the generated cover singing dry voice depends entirely on the quality of the separated dry voice. If the quality of the separated dry voice is not good, the singing effect of the cover singing voice is poor. In order to generate a song cover with a good cover effect, the server can search for the lyrics information of the song to be covered, the timestamp information corresponding to the lyrics information of the song to be covered, and the pitch information of the song to be covered (and the timestamp information corresponding to the pitch information of the song to be covered) from a preset storage location. The server can also input the song to be covered into the lyrics information extraction model, which can analyze the song to be covered, determine the lyrics information of the song to be covered and the timestamp information corresponding to the lyrics information of the song to be covered. The server can also input the song to be covered into the pitch information extraction model, which can analyze the song to be covered, determine the pitch information of the song to be covered (and the timestamp information corresponding to the pitch information of the song to be covered).

[0056] Step S204, aligning the lyrics information of the song to be covered and the pitch information of the song to be covered by a pre-trained lyrics pitch alignment model to obtain the music score information corresponding to the song to be covered; the music score information represents the correlation between the lyrics information of the song to be covered and the pitch information of the song to be covered.

[0057] Among them, the lyrics pitch alignment model can refer to a model used to determine the association / correspondence between lyrics and pitch. In practical applications, the lyrics pitch alignment model can be used to analyze the similarity between lyrics and pitch, so as to determine the association / correspondence between lyrics and pitch.

[0058] The score information may refer to information representing the correlation between the lyrics information of the song to be covered and the pitch information of the song to be covered. In practical applications, the score information output by the lyrics pitch alignment model may be used as information similar to the score.

[0059] As an example, the server can input the lyrics information of the song to be covered and the pitch information of the song to be covered into a pre-trained lyrics pitch alignment model. The lyrics pitch alignment model can integrate the lyrics information of the song to be covered and the pitch information of the song to be covered according to the time dimension, analyze the correlation / correspondence between the lyrics information and the pitch information sentence by sentence, and determine the music score information corresponding to the song to be covered.

[0060] Step S206, generating a dry vocal with a target timbre according to the music score information corresponding to the song to be covered, and generating a cover song corresponding to the song to be covered according to the dry vocal and the song accompaniment corresponding to the song to be covered.

[0061] The target timbre may refer to the timbre that is ultimately required to be presented when a cover song is sung. In practical applications, the target timbre may be the timbre selected by the user.

[0062] Among them, the dry voice of the singing voice with the target timbre may refer to the dry voice of the song to be covered with the target timbre.

[0063] As an example, the server can input the score information corresponding to the song to be covered into the pre-trained singing voice generation model, and the singing voice generation model can directly generate the singing voice with the target timbre in combination with the target timbre selected by the user and the score information corresponding to the song to be covered, and mix the singing voice with the target timbre and the song accompaniment corresponding to the song to be covered to generate the cover song corresponding to the song to be covered. In practical applications, the server can also combine the score information corresponding to the song to be covered with the preset timbre in the preset timbre database to generate the singing voice with the preset timbre singing the song to be covered, and then the server can mix the singing voice with the preset timbre singing the song to be covered and the song accompaniment corresponding to the song to be covered to generate the cover song corresponding to the song to be covered. In specific implementation, the server can also adjust the timbre of the singing voice with the preset timbre singing the song to be covered to the target timbre selected by the user, generate the singing voice with the target timbre singing the song to be covered, and generate the cover song with the target timbre singing the song to be covered in combination with the song accompaniment corresponding to the song to be covered.

[0064] In the above-mentioned cover song generation method, the lyrics information and pitch information of the song to be covered are respectively obtained, and the pre-trained lyrics pitch alignment model is used to align the lyrics information and the pitch information based on the timestamp information carried by the lyrics information and the timestamp information carried by the pitch information, so as to obtain the music score information used to generate the cover song, thereby using the music score information to generate a dry singing voice with a target timbre, and generating a cover song based on the dry singing voice and the song accompaniment. The lyrics pitch alignment model can be used to determine the correlation and / or correspondence between the lyrics information and the pitch information, and then generate the cover singing voice. It does not simply rely on the separated dry voice to generate the cover singing voice, so as to avoid the quality of the separated dry voice affecting the singing effect of the cover singing voice, and optimize the cover effect of the cover song.

[0065] In an exemplary embodiment, the lyrics information of the song to be covered and the pitch information of the song to be covered are aligned to obtain the score information corresponding to the song to be covered, including: generating a lyrics time series according to the lyrics information of the song to be covered and the timestamp information carried by the lyrics information of the song to be covered; generating a pitch time series according to the pitch information of the song to be covered and the timestamp information carried by the pitch information of the song to be covered; and determining the score information corresponding to the song to be covered according to the similarity between the lyrics time series and the pitch time series.

[0066] The lyrics time sequence may refer to information obtained by arranging the lyrics information of the cover songs in chronological order. In practical applications, the lyrics time sequence may be regarded as a character sequence.

[0067] The pitch time series may refer to information obtained by arranging the pitch information of the cover songs in chronological order.

[0068] As an example, the server can arrange the lyrics information of the song to be covered according to the timestamp information carried in the lyrics information of the song to be covered, and the arrangement result can be used as the lyrics time series. The server can also arrange the pitch information of the song to be covered according to the timestamp information carried in the pitch information of the song to be covered, and the arrangement result can be used as the pitch time series. The server can then analyze the similarity between the lyrics time series and the pitch time series (such as the best matching path between the time series), determine the association / correspondence between the lyrics information of the song to be covered and the pitch information of the song to be covered, and then determine the music score information corresponding to the song to be covered.

[0069] In this embodiment, a lyrics time series and a pitch time series are generated based on the lyrics information and the pitch information respectively, and the score information is determined by analyzing the similarity between the lyrics time series and the pitch time series. The similarity between the time series can be combined to accurately analyze the correlation / correspondence between the lyrics information and the pitch information, so as to obtain accurate score information, improve the accuracy of the score information, and provide a data basis for the subsequent generation of cover songs in combination with the score information, thereby optimizing the singing effect of the cover songs.

[0070] In some embodiments, based on the degree of similarity between the lyrics time sequence and the pitch time sequence, the music score information corresponding to the song to be covered is determined, including: generating a target matrix based on the lyrics time sequence and the pitch time sequence; determining the shortest path between the lyrics time sequence and the pitch time sequence based on the target matrix, and using the shortest path as the music score information corresponding to the song to be covered.

[0071] Among them, the target matrix can refer to a distance matrix. In practical applications, the elements in the target matrix are determined by the lyrics time series and the pitch time series. The elements of the target matrix can be used to represent the distance between the lyrics information in the lyrics time series and the pitch information in the pitch time series.

[0072] Among them, the shortest path can be regarded as a best matching path.

[0073] As an example, the server may use a dynamic time warping (DTW) algorithm to determine the shortest path between the lyrics time series and the pitch time series. Specifically, the server may use the lyrics time series Q = [q1, q2, ..., q n ] and the length n of the pitch time series C=[c1,c2,…,c m ] length m, construct a target matrix D of size n*m, and the element d in the target matrix D i,j It can be expressed as d i,j =dist(q i , c j ), dist() can be a distance calculation function, and the distance calculated by the distance calculation function can include but is not limited to the Euclidean distance. Then the server can use dynamic programming to search for the target matrix D from d 1,1 to d n,m The shortest path is taken as the shortest path between the lyrics time series and the pitch time series. In practical applications, i,j In terms of position, the path search direction is monotonous and cannot be turned back. The server can use the shortest path between the lyrics time series and the pitch time series as the score information corresponding to the song to be covered.

[0074] In this embodiment, the shortest path between the lyrics time series and the pitch time series is analyzed to determine the music score information corresponding to the song to be covered, which can accurately analyze the similarities and differences between the time series, improve the accuracy of the shortest path, and thus improve the accuracy of the music score information.

[0075] In some embodiments, generating a dry singing voice with a target timbre according to the score information corresponding to the song to be covered includes: generating a dry singing voice with a preset timbre according to the score information corresponding to the song to be covered, and obtaining the timbre feature information of the target timbre; the target timbre is the timbre selected by the user; inputting the dry singing voice with the preset timbre and the timbre feature information of the target timbre into a pre-trained timbre conversion model to obtain a converted dry singing voice (i.e., a dry singing voice with the target timbre); the converted dry singing voice is a singing voice of the song to be covered with the target timbre. Then, the converted dry singing voice and the song accompaniment corresponding to the song to be covered can be mixed to obtain a cover song corresponding to the song to be covered.

[0076] The preset timbre may refer to a preset timbre. In practical applications, the preset timbre may be any timbre in a preset timbre database.

[0077] The timbre characteristic information may refer to information characterizing the timbre characteristics of the target timbre. In practical applications, the timbre characteristics may include but are not limited to fundamental frequency, harmonics, overtones, etc. The target timbre is the timbre selected by the user through the timbre selection interface.

[0078] Among them, the timbre conversion model may refer to a model used to change or adjust the timbre of a singing voice.

[0079] Among them, the converted dry voice can refer to the singing voice of the song to be covered with the target timbre.

[0080] As an example, the server obtains the music score information corresponding to the song to be covered, and can combine the preset timbre and the music score information corresponding to the song to be covered to generate a dry voice of the singing voice with the preset timbre, and then obtain the timbre feature information of the target timbre selected by the user. The server can then input the singing dry voice and the timbre feature information into a pre-trained timbre conversion model. The timbre conversion model can adjust the timbre of the dry voice of the singing voice that sings the song to be covered with the preset timbre to the target timbre to obtain the converted dry voice. The server can then mix the converted dry voice with the song accompaniment corresponding to the song to be covered to obtain a cover song corresponding to the song to be covered.

[0081] In this embodiment, the timbre of a singing voice with a preset timbre is adjusted through a timbre conversion model, and mixed with the song accompaniment to obtain a cover song. The timbre of the cover singing voice can be flexibly adjusted to optimize the singing effect of the cover song.

[0082] In some embodiments, before the step of aligning the lyrics information of the song to be covered and the pitch information of the song to be covered through a pre-trained lyrics pitch alignment model, the above method also includes: modifying the lyrics information of the song to be covered to obtain modified lyrics information; aligning the modified lyrics information and the pitch information of the song to be covered through a pre-trained lyrics pitch alignment model to obtain the music score information corresponding to the song to be covered.

[0083] The modified lyrics information may refer to the lyrics obtained after modifying the lyrics information of the cover song.

[0084] As an example, since the traditional technology separates the original dry voice of the song and then inputs the original dry voice file into the singing voice conversion model to generate the cover singing voice, it is impossible to edit the song. In order to realize song editing during the song cover process, the server can, after obtaining the lyrics information of the song to be covered, respond to the user's lyrics modification operation, modify the lyrics information of the song to be covered, and align the modified lyrics information with the pitch information of the song to be covered through a pre-trained lyrics pitch alignment model, determine the association / correspondence between the modified lyrics information and the pitch information of the song to be covered, and obtain the music score information corresponding to the song to be covered. The cover song generated based on the music score information at this time can be used as the cover song after song editing.

[0085] In this embodiment, by modifying the lyrics information and combining the lyrics modification results and the pitch information to determine the music score information, song editing can be achieved during the song cover process, personalized song cover can be achieved, and the poor quality of the original singer's dry voice can be avoided to affect the singing effect of the cover song, thereby optimizing the singing effect of the cover song.

[0086] In some embodiments, before the step of aligning the lyrics information and the pitch information of the song to be covered by a pre-trained lyrics pitch alignment model, the above method also includes: modifying the pitch information of the song to be covered to obtain modified pitch information; aligning the lyrics information and the pitch information of the song to be covered by a pre-trained lyrics pitch alignment model to obtain the music score information corresponding to the song to be covered.

[0087] The modified pitch information may refer to the pitch obtained after modifying the pitch information of the cover song.

[0088] As an example, in order to implement song editing during the song cover process, the server can also modify the pitch of the song. Specifically, after obtaining the pitch information of the song to be covered, the server can modify the pitch information of the song to be covered in response to the user's pitch modification operation, and align the modified pitch information with the lyrics information of the song to be covered through the pre-trained lyrics pitch alignment model, determine the association / correspondence between the modified pitch information and the lyrics information of the song to be covered, and obtain the music score information corresponding to the song to be covered. The cover song generated based on the music score information at this time can be used as the cover song after song editing. It is understandable that the server can modify only the lyrics information or the pitch information, and the server can also modify the lyrics and pitch at the same time. For example: after obtaining the lyrics information and pitch information of the song to be covered, the server can modify the lyrics information and pitch information of the song to be covered in response to the user's lyrics modification operation and pitch modification operation, and align the modified pitch information and the modified lyrics information through a pre-trained lyrics pitch alignment model, determine the association / correspondence between the modified pitch information and the modified lyrics information, and obtain the music score information corresponding to the song to be covered. The cover song generated based on the music score information at this time can be used as the cover song after song editing.

[0089] In this embodiment, by modifying the pitch information and determining the music score information in combination with the pitch modification result and the lyrics information, song editing can be implemented during the song cover process, personalized song cover can be achieved, and the poor quality of the original singer's dry voice can be avoided from affecting the singing effect of the cover song, thereby optimizing the singing effect of the cover song.

[0090] In an exemplary embodiment, a method for training a lyrics pitch alignment model is provided, and the training method for the lyrics pitch alignment model includes: obtaining lyrics information and pitch information of a sample song; the timestamp information carried by the lyrics information of the sample song represents the start time and end time of each text in the sample song; the timestamp information carried by the pitch information of the sample song represents the start time and end time of each pitch in the sample song; aligning the lyrics information and the pitch information of the sample song through the lyrics pitch alignment model to obtain the score information corresponding to the sample song; training the lyrics pitch alignment model according to the score information corresponding to the sample song and the actual score information of the sample song to obtain a pre-trained lyrics pitch alignment model.

[0091] The sample song may refer to any song in a preset song database. In practical applications, the actual score information of the sample song may be known information.

[0092] Among them, the music score information corresponding to the sample song may refer to the information output by the lyrics pitch alignment model, which characterizes the similarity between the lyrics information of the sample song and the pitch information of the sample song.

[0093] The actual score information of the sample song may refer to the real score information of the sample song.

[0094] As an example, the server can obtain at least one sample song from a preset sample song database. For each sample song, the server can obtain the lyrics information of the sample song and the pitch information of the sample song respectively. Then, the server can input the lyrics information of the sample song and the pitch information of the sample song into the lyrics pitch alignment model to be trained. The lyrics pitch alignment model can analyze the lyrics information of the sample song and the pitch information of the sample song, so that the alignment model aligns the lyrics information of the sample song and the pitch information of the sample song. The lyrics pitch alignment model can output the score information corresponding to the sample song. The server can obtain the real score information of the sample song as the actual score information. Then, the server can train the lyrics pitch alignment model according to the score information corresponding to the sample song and the actual score information corresponding to the sample song. Specifically, the server can determine the model loss function value corresponding to the lyrics pitch alignment model according to the score information corresponding to the sample song and the actual score information corresponding to the sample song, and modify the model parameters of the lyrics pitch alignment model according to the model loss function value until the model loss function value meets the preset requirements, and obtain the trained lyrics pitch alignment model.

[0095] In this embodiment, the lyrics information, pitch information and actual music score information of the sample song are used to train the lyrics pitch alignment model, which can enable the lyrics pitch alignment model to fully learn the association / correspondence between the lyrics and the pitch of the song, thereby improving the accuracy of the music score information output by the lyrics pitch alignment model. It can also provide a data basis for the subsequent generation of cover songs in combination with the music score information, thereby optimizing the singing effect of the cover songs.

[0096] In some embodiments, Figure 3 As shown, a flowchart of a song cover is provided. When a song needs to be covered and the real score information of the song cannot be obtained, the server can extract text information (such as lyrics information) and pitch information from the original song file of the song to be covered through a speech recognition module (such as an ASR model) and a pitch transcription model. Both the text information and the pitch information are timestamped, so the text (such as lyrics) and the pitch can be matched to form a score file (such as score information). The singing voice synthesis model reads the model input from the score file and can output an audio dry voice with a fixed timbre in the library. Then the singing voice conversion model (such as a timbre conversion model) can process the audio dry voice and output an audio dry voice with a specified timbre (such as a target timbre). After that, it can be mixed with the song accompaniment of the song to be covered to complete the finished cover and obtain the cover song of the song to be covered.

[0097] In practical applications, such as Figure 4 As shown, a process diagram of song cover is provided, the server can separate the vocal accompaniment of the original singer's work (such as the original singer's audio) of the song to be covered, and separate the vocal part and the accompaniment part of the song to be covered, and then the server can extract the lyrics and pitch of the vocal part of the cover song, and align the lyrics pitch of the extracted lyrics information and pitch information to obtain the score information, and then the server can perform singing synthesis based on the score information to obtain the dry vocal of the song to be covered with a preset timbre / target timbre, and the server can mix the dry vocal with the song accompaniment of the song to be covered to obtain a cover work (such as a cover song).

[0098] As a known example, the server can first obtain key information of the original song to be covered (such as lyrics information and pitch information). Specifically, the server can recognize the lyrics of the cover song through a speech recognition tool or a speech recognition model. The speech recognition tool or the speech recognition model can return ms-level time information (timestamp information) while outputting the lyrics information. For example, the lyrics information returned by the speech recognition tool or the speech recognition model for the lyrics "yes" and the timestamp information corresponding to the lyrics information can be expressed as "text: yes, start time: 0.5, end time: 1.2, confidence: 0.51", wherein the speech recognition tool or the speech recognition model has a high recognition accuracy and can recognize whispers and sensevoice in speech. The server can perform pitch recognition on the cover song through a pitch extraction tool or a pitch transcription model. The pitch extraction tool or the pitch transcription model can extract the main melody of the human voice from the original song to be covered as the pitch information. For example, the pitch information output by the pitch extraction tool or the pitch transcription model can be expressed as "start time 20444, end time 21555, pitch value 60".

[0099] like Figure 5 As shown in the figure, a schematic diagram of aligning lyrics and pitch is provided. After the server obtains the lyrics information and pitch information, it needs to integrate the lyrics information and pitch information together according to the time dimension, so that each word in the lyrics has corresponding pitch information. Specifically, the server can use the lyrics pitch alignment model, take sentences as units, and use the dynamic time warping (DTW) algorithm to align the pitch information and lyrics information. The server can adjust the DTW parameters according to the actual alignment situation to obtain an accurate alignment effect. In practical applications, the server can generate a lyrics time series Q=[q1,q2,…,q n ], the length of the lyrics time series is n, and the server can generate a pitch time series C=[c1,c2,…,c m ], the length of the pitch time series is m, the similarity between the lyrics time series Q and the pitch time series C can be used as music score information, and the process of calculating the similarity between the lyrics time series Q and the pitch time series C can include: constructing a matrix D of size n*m, the matrix element d i,j =dist(q i , c j ), dist() can be a distance calculation function, usually using Euclidean distance, to search for the distance from d in the matrix D. 1,1 to d n,m The shortest path is usually calculated using dynamic programming. i,jIn terms of position, the path search direction is monotonous and cannot be turned back; the search in matrix D starts from d 1,1 to d n,m The shortest path is used as the alignment relationship (such as similarity) between the Q and C sequences. The calculation expression of the shortest path can be expressed as D(i,j)=dist(i,j)+min[D(i-1,j),D(i,j-1),D(i-1,j-1)]. After DTW alignment, the shortest path (such as similarity) between the lyrics time series Q and the pitch time series C can be used as a file similar to a music score (that is, the music score information output by the lyrics pitch alignment model). The music score information output by the lyrics pitch alignment model can be used for singing synthesis.

[0100] like Figure 6 As shown, a flow chart of singing synthesis is provided, the server can input the score information output by the lyrics pitch alignment model into the acoustic model (such as fastpitch, tacotron, etc.) and the vocoder (such as hifigan, bigvgan, etc.), the acoustic model can analyze the lyrics information and pitch information in the score information output by the lyrics pitch alignment model, generate audio features (such as mel spectrum features), and the vocoder can convert the audio features into the dry voice of the singing voice that sings the song to be covered with the preset timbre in the preset timbre database. Furthermore, the server can also adjust the timbre of the dry voice of the singing voice to the target timbre specified by the user (such as the user's own timbre, etc.) based on the singing voice conversion model (such as the timbre conversion model), and then the server can mix the dry voice of the singing voice with the target timbre / preset timbre and the song accompaniment of the song to be covered to obtain the cover song. It can be understood that in the process of generating a cover song, the server can modify the lyrics and / or pitch of the cover song in response to the user's lyrics modification operation and / or pitch modification operation. The server can determine the music score information based on the lyrics modification results and / or pitch modification results, and then generate a cover song based on the music score information.

[0101] In this embodiment, by automatically extracting the lyrics and pitch of the song, aligning the lyrics and pitch to determine the score information, and then generating a cover song based on the score information, it is possible to efficiently generate a cover song of the song. At the same time, in the process of generating the cover song, it also supports modifications to the lyrics, pitch, and timbre, which facilitates secondary creation and flexible adjustment of the song. Moreover, since the dry voice in the cover song is synthesized from the singing voice, the sound quality problem introduced by separation is avoided, thereby optimizing the singing effect of the cover song.

[0102] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0103] Based on the same inventive concept, the embodiment of the present application also provides a cover song generation device for implementing the above-mentioned cover song generation method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more cover song generation device embodiments provided below can refer to the above-mentioned limitations on the cover song generation method, and will not be repeated here.

[0104] In an exemplary embodiment, Figure 7 As shown, a cover song generation device is provided, including: an acquisition module 702, an alignment module 704 and a generation module 706, wherein:

[0105] The acquisition module 702 is used to obtain the lyrics information of the song to be covered and the pitch information of the song to be covered; the timestamp information carried by the lyrics information of the song to be covered represents the start time and end time of each text in the song to be covered; the timestamp information carried by the pitch information of the song to be covered represents the start time and end time of each pitch in the song to be covered.

[0106] The alignment module 704 is used to align the lyrics information of the song to be covered and the pitch information of the song to be covered through a pre-trained lyrics pitch alignment model to obtain the music score information corresponding to the song to be covered; the music score information represents the correlation between the lyrics information of the song to be covered and the pitch information of the song to be covered.

[0107] The generation module 706 is used to generate a dry singing voice with a target timbre according to the music score information corresponding to the song to be covered, and to generate a cover song corresponding to the song to be covered according to the dry singing voice and the song accompaniment corresponding to the song to be covered.

[0108] In one of the exemplary embodiments, the alignment module 704 is further specifically used to generate a lyrics time series based on the lyrics information of the song to be covered and the timestamp information carried by the lyrics information of the song to be covered; generate a pitch time series based on the pitch information of the song to be covered and the timestamp information carried by the pitch information of the song to be covered; and determine the music score information corresponding to the song to be covered based on the degree of similarity between the lyrics time series and the pitch time series.

[0109] In one of the exemplary embodiments, the alignment module 704 is further specifically used to generate a target matrix based on the lyrics time series and the pitch time series, and the elements of the target matrix are used to represent the distance between the lyrics information in the lyrics time series and the pitch information in the pitch time series; determine the shortest path between the lyrics time series and the pitch time series based on the target matrix, and use the shortest path as the music score information corresponding to the song to be covered.

[0110] In one of the exemplary embodiments, the above-mentioned generation module 706 is specifically used to generate a dry singing voice with a preset timbre according to the music score information corresponding to the song to be covered, and obtain the timbre characteristic information of the target timbre, wherein the target timbre is the timbre selected by the user; the dry singing voice with the preset timbre and the timbre characteristic information of the target timbre are input into a pre-trained timbre conversion model to generate a dry singing voice with the target timbre.

[0111] In one of the exemplary embodiments, the above-mentioned device also includes a modification module, which is specifically used to modify the lyrics information of the song to be covered to obtain the modified lyrics information; align the modified lyrics information with the pitch information of the song to be covered through the pre-trained lyrics pitch alignment model to obtain the music score information corresponding to the song to be covered.

[0112] In one of the exemplary embodiments, the modification module is specifically used to modify the pitch information of the song to be covered to obtain the modified pitch information; the lyrics information and the pitch information of the song to be covered are aligned through the pre-trained lyrics pitch alignment model to obtain the music score information corresponding to the song to be covered.

[0113] Each module in the above-mentioned cover song generation device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0114] In an exemplary embodiment, a training device for a lyrics pitch alignment model is provided, comprising: a sample acquisition module, an information alignment module and a training module, wherein:

[0115] The sample acquisition module is used to obtain the lyrics information of the sample song and the pitch information of the sample song; the timestamp information carried by the lyrics information of the sample song represents the start time and end time of each text in the sample song; the timestamp information carried by the pitch information of the sample song represents the start time and end time of each pitch in the sample song.

[0116] The information alignment module is used to align the lyrics information of the sample song and the pitch information of the sample song through a lyrics pitch alignment model to obtain the music score information corresponding to the sample song.

[0117] The training module is used to train the lyrics pitch alignment model to obtain a pre-trained lyrics pitch alignment model based on the music score information corresponding to the sample song and the actual music score information of the sample song.

[0118] Each module in the training device of the lyrics pitch alignment model can be implemented in whole or in part by software, hardware and a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0119] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as lyrics information, pitch information and music score information. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for generating a cover song is implemented.

[0120] Those skilled in the art will understand that Figure 8The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0121] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0122] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0123] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0124] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0125] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0126] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0127] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for generating a cover song, characterized in that: The method comprises: Obtaining lyrics information of a song to be covered and pitch information of the song to be covered; the timestamp information carried by the lyrics information of the song to be covered represents the start time and end time of each text in the song to be covered; the timestamp information carried by the pitch information of the song to be covered represents the start time and end time of each pitch in the song to be covered; The lyrics information of the song to be covered and the pitch information of the song to be covered are aligned by a pre-trained lyrics pitch alignment model to obtain music score information corresponding to the song to be covered; the music score information represents the correlation between the lyrics information of the song to be covered and the pitch information of the song to be covered; A dry singing voice with a target timbre is generated according to the music score information corresponding to the song to be covered, and a cover song corresponding to the song to be covered is generated according to the dry singing voice and the song accompaniment corresponding to the song to be covered.

2. The method according to claim 1, characterized in that The aligning the lyrics information of the song to be covered and the pitch information of the song to be covered to obtain the music score information corresponding to the song to be covered includes: Generate a lyrics time sequence according to the lyrics information of the song to be covered and the timestamp information carried by the lyrics information of the song to be covered; generate a pitch time sequence according to the pitch information of the song to be covered and the timestamp information carried by the pitch information of the song to be covered; The music score information corresponding to the song to be covered is determined according to the similarity between the lyrics time series and the pitch time series.

3. The method according to claim 2, characterized in that The step of determining the music score information corresponding to the song to be covered according to the similarity between the lyrics time series and the pitch time series comprises: Generate a target matrix according to the lyrics time sequence and the pitch time sequence, wherein the elements of the target matrix are used to represent the distance between the lyrics information in the lyrics time sequence and the pitch information in the pitch time sequence; The shortest path between the lyrics time series and the pitch time series is determined according to the target matrix, and the shortest path is used as the music score information corresponding to the song to be covered.

4. The method according to claim 1, characterized in that The step of generating a dry singing voice with a target timbre according to the music score information corresponding to the song to be covered includes: Generate a dry voice with a preset timbre according to the music score information corresponding to the song to be covered, and obtain timbre characteristic information of a target timbre, where the target timbre is the timbre selected by the user; The timbre feature information of the dry singing voice with the preset timbre and the target timbre is input into a pre-trained timbre conversion model to generate the dry singing voice with the target timbre.

5. The method according to claim 1, characterized in that Before the step of aligning the lyrics information of the song to be covered and the pitch information of the song to be covered by the pre-trained lyrics pitch alignment model, the method further includes: Modifying the lyrics information of the song to be covered to obtain modified lyrics information; The modified lyrics information and the pitch information of the song to be covered are aligned by the pre-trained lyrics pitch alignment model to obtain the music score information corresponding to the song to be covered.

6. The method according to claim 1, characterized in that Before the step of aligning the lyrics information of the song to be covered and the pitch information of the song to be covered by the pre-trained lyrics pitch alignment model, the method further includes: Modifying the pitch information of the song to be covered to obtain modified pitch information; The lyrics information and the pitch information of the song to be covered are aligned through the pre-trained lyrics pitch alignment model to obtain the music score information corresponding to the song to be covered.

7. A method for training a lyrics pitch alignment model, characterized in that: The method comprises: Acquire lyrics information of a sample song and pitch information of the sample song; the timestamp information carried by the lyrics information of the sample song represents the start time and end time of each text in the sample song; the timestamp information carried by the pitch information of the sample song represents the start time and end time of each pitch in the sample song; Aligning the lyrics information of the sample song and the pitch information of the sample song through a lyrics pitch alignment model to obtain music score information corresponding to the sample song; According to the score information corresponding to the sample song and the actual score information of the sample song, the lyrics pitch alignment model is trained to obtain a pre-trained lyrics pitch alignment model, wherein the pre-trained lyrics pitch alignment model is used in the cover song generation method as described in any one of claims 1 to 6.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images