Reversed singing song generation method, computer equipment, storage medium and program product

Abstract

The invention relates to a turning song generation method, computer equipment, a storage medium and a program product. The method comprises the steps of identifying a to-be-restored human voice fragment in original human voice of a target song for the target song selected to be sung by a user; obtaining a target human voice fragment matched with the to-be-repaired human voice fragment; the target human voice segment is a singing human voice segment without reverberation and harmony, and the target human voice segment and the human voice segment to be repaired correspond to the same singing content in the target song; replacing a to-be-repaired human voice segment in the original human voice by using the target human voice segment to obtain a repaired original human voice, and extracting a singing feature of the repaired original human voice to obtain a singing guide feature; extracting timbre features in sound materials provided by the user, and generating a turning song of the target song according to the timbre features and the singing guide features; the flipped song is a song generated by singing the target song by simulating the timbre of the user. By adopting the method, the quality of the generated flipped song can be improved.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to a method for generating a cover song, a computer device, a storage medium and a program product. Background Art

[0002] With the rapid development of artificial intelligence technology, the field of music production has also ushered in new changes, especially in the field of song generation and cover. In the past, cover songs needed to rely on the singer's singing level and complex audio processing technology, which took a lot of time and effort. In traditional technology, artificial intelligence technology can be used to process the audio of the original song to generate cover songs that simulate other singers. However, the generated cover songs may have problems such as mute, strange sound or unbalanced volume, resulting in low quality of the generated cover songs. Summary of the invention

[0003] Based on this, it is necessary to provide a cover song generation method, computer device, storage medium and program product that can improve the quality of generated cover songs in response to the above technical problems.

[0004] In a first aspect, the present application provides a method for generating a cover song, comprising:

[0005] For a target song selected by a user for cover, a vocal segment to be repaired in the original vocal of the target song is identified; the vocal segment to be repaired is a vocal segment with reverberation and / or harmony;

[0006] Obtaining a target vocal segment that matches the vocal segment to be repaired; the target vocal segment is a singing vocal segment without reverberation and harmony, and the target vocal segment and the vocal segment to be repaired correspond to the same singing content in the target song;

[0007] Using the target vocal segment to replace the vocal segment to be repaired in the original vocal, to obtain the repaired original vocal, and extracting the singing feature of the repaired original vocal to obtain the lead vocal feature;

[0008] The timbre features in the sound material provided by the user are extracted, and a cover song of the target song is generated according to the timbre features and the lead singing features; the cover song is a song generated by simulating the timbre of the user singing the target song.

[0009] In one embodiment, the step of identifying the vocal segment to be repaired in the original vocal of the target song includes:

[0010] Detecting interference segments in the original vocals of the target song; the interference segments are vocal segments with reverberation and / or harmony;

[0011] For any of the interference segments, if the interference segment partially falls into the non-singing time period of the original singer's vocals, or if the interference segment completely falls into the singing time period of the original singer's vocals and the endpoint timestamp of the interference segment is not aligned with the endpoint timestamp of the pronunciation unit, adjust the two endpoint timestamps of the interference segment to obtain a corrected interference segment; the two endpoint timestamps of the corrected interference segment are both aligned with the endpoint timestamp of a pronunciation unit and the corrected interference segment does not include the non-singing time period;

[0012] According to the corrected interference segment, the vocal segment to be repaired in the original vocal of the target song is obtained.

[0013] In one embodiment, adjusting the timestamps of two endpoints of the interference segment to obtain a corrected interference segment includes:

[0014] If the starting timestamp of the interference segment falls between the two endpoint timestamps of any pronunciation unit in the original singer's voice, the starting timestamp of the interference segment is corrected to the starting timestamp of any pronunciation unit to obtain the corrected interference segment;

[0015] If the starting timestamp of the interference segment falls within the non-singing time period, the starting timestamp of the interference segment is corrected to the starting timestamp of the next pronunciation unit adjacent to the non-singing time period to obtain the corrected interference segment;

[0016] If the end timestamp of the interference segment falls between the two endpoint timestamps of any pronunciation unit, the end timestamp of the interference segment is corrected to the end timestamp of any pronunciation unit to obtain the corrected interference segment;

[0017] If the end timestamp of the interference segment falls within the non-singing time period, the end timestamp of the interference segment is corrected to the end timestamp of the previous pronunciation unit adjacent to the non-singing time period to obtain the corrected interference segment.

[0018] In one embodiment, obtaining a target vocal segment matching the vocal segment to be repaired includes:

[0019] If there is a pure segment in the original vocals that matches the vocal segment to be repaired, the pure segment is determined as a target vocal segment that matches the vocal segment to be repaired; the pure segment is a vocal segment without reverberation and harmony;

[0020] If there is no pure segment in the original vocals that matches the vocal segment to be repaired, a target vocal segment that matches the vocal segment to be repaired is determined from the candidate vocal set.

[0021] In one of the embodiments, determining a target vocal segment matching the vocal segment to be repaired from a candidate vocal set includes:

[0022] Acquire a candidate voice set; the candidate voice set includes a plurality of candidate voices;

[0023] A target vocal segment matching the vocal segment to be repaired is determined from the multiple candidate vocals according to at least one of the singer's gender, pitch, and loudness of the candidate vocals.

[0024] In one embodiment, obtaining a candidate voice set includes:

[0025] Acquire multiple dry voices corresponding to the target song; the dry voices are non-original vocals without reverberation and harmony and with an audio length consistent with the audio length of the original vocals;

[0026] According to the sound quality index information of the dry sound, the multiple candidate voices are determined from the multiple dry sounds to obtain the candidate voice set; the sound quality index information includes at least one of loudness, clarity, and pitch.

[0027] In one embodiment, before identifying the vocal segment to be repaired in the original vocal of the target song, the method further includes:

[0028] Separating the accompaniment and vocals of the target song to obtain the separated original vocals;

[0029] According to the position information of each pronunciation unit in the lyrics of the target song, the non-singing time period between each line in the lyrics is identified; the position information includes the endpoint timestamp of the pronunciation unit and the line identification information of the pronunciation unit in the lyrics;

[0030] The audio corresponding to the non-singing time period in the separated original singer's vocals is replaced with blank audio to obtain the original singer's vocals of the target song.

[0031] In one embodiment, replacing the audio corresponding to the non-singing time period in the separated original vocals with blank audio to obtain the original vocals of the target song includes:

[0032] Determine a beginning reserved time period and an ending reserved time period in a non-singing time period between any two rows in each row;

[0033] The other time periods in the non-singing time period between any two rows except the beginning reserved time period and the ending reserved time period are used as the cutting time period;

[0034] The audio corresponding to the cropped time period in the separated original singer's vocals is replaced with blank audio to obtain the original singer's vocals of the target song.

[0035] In a second aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0036] In a third aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0037] In a fourth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0038] The above-mentioned cover song generation method, computer device, storage medium and program product, for a target song selected by a user for cover, identifies a singing vocal segment with reverberation and harmony from the original singer's vocal in the target song as a vocal segment to be repaired; determines a target vocal segment that matches the vocal segment to be repaired, the target vocal segment being a singing vocal segment with the same singing content as the target song corresponding to the vocal segment to be repaired and without reverberation and / or harmony; replaces the vocal segment to be repaired in the original singer's vocal with the target vocal segment to obtain a repaired original singer's vocal, and extracts singing skills features from the repaired original singer's vocal to obtain a lead singing feature; and extracts singing skills features from the repaired original singer's vocal according to the sound provided by the user. The timbre features and lead-in features extracted from the material are used to generate a cover song of the target song. The cover song is a song generated by simulating the user's timbre to sing the target song. The vocal segment to be repaired with reverberation and harmony in the original singer's vocals is identified and repaired, and the vocal segment to be repaired is replaced with a high-quality target vocal segment that matches the vocal segment to be repaired. The defective parts of the original singer's vocals with reverberation and harmony are optimized, thereby achieving refined optimization processing of the original singer's vocals and retaining the singing characteristics of the original singer's vocals. Furthermore, by extracting the timbre features of the user's voice, a cover song with user-personalized timbre and high quality is generated, thereby improving the quality of the generated cover song. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0040] FIG1 is an application environment diagram of a cover song generation method in one embodiment;

[0041] FIG2 is a schematic flow chart of a method for generating a cover song in one embodiment;

[0042] FIG3 is a schematic diagram of a user interface for generating a cover song in one embodiment;

[0043] FIG4 is a schematic diagram of another user interface for generating a cover song in another embodiment;

[0044] FIG5 is a schematic diagram of a process of cutting out the original vocals in one embodiment;

[0045] FIG6 is a schematic diagram of correcting an interference segment in one embodiment;

[0046] FIG7 is a flowchart of determining an original vocal and determining a vocal segment to be restored in the original vocal in one embodiment;

[0047] FIG8 is a flowchart of generating a candidate voice set in one embodiment;

[0048] FIG9 is a flowchart of generating a lead-in feature in one embodiment;

[0049] FIG10 is a schematic flow chart of another method for generating a cover song in one embodiment;

[0050] FIG11 is a structural block diagram of a cover song generating device according to an embodiment;

[0051] FIG. 12 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] The cover song generation method provided in the embodiment of the present application can be applied in the application environment shown in FIG1 . The terminal 102 communicates with the server 104 via a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 identifies the vocal segment to be repaired in the original vocals of the target song selected by the user for cover; the vocal segment to be repaired is a singing vocal segment with reverberation and / or harmony; the server 104 obtains a target vocal segment that matches the vocal segment to be repaired; the target vocal segment is a singing vocal segment without reverberation and harmony, and the target vocal segment and the vocal segment to be repaired correspond to the same singing content in the target song; the server 104 replaces the vocal segment to be repaired in the original vocals with the target vocal segment to obtain the repaired original vocals, and extracts the singing characteristics of the repaired original vocals to obtain the lead singing characteristics; the server 104 extracts the timbre characteristics in the sound material provided by the user, and the terminal 102 or the server 104 generates a cover song of the target song based on the timbre characteristics and the lead singing characteristics; the cover song is a song generated by simulating the timbre of the user singing the target song. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0054] In an exemplary embodiment, as shown in FIG. 2 , a method for generating a cover song is provided, and the method is described by taking the application of the method to the server 104 in FIG. 1 as an example, including:

[0055] Step S202, for the target song selected by the user for cover, identifying the vocal segment to be repaired in the original singer's vocal of the target song.

[0056] The vocal segment to be restored is a singing vocal segment with reverberation and / or harmony.

[0057] The target song may refer to the original song selected by the user for cover, including complete accompaniment and vocals.

[0058] The original vocal may refer to the voice of the singer in the original recording of the target song.

[0059] The vocal segment to be repaired can be the part of the original vocal that is disturbed by harmony and reverberation. Reverberation refers to the sound reflection and delay caused by the recording environment or other technical reasons in the original vocal, which may affect the clarity of the vocal; harmony refers to other parts or tones accompanying the original vocal, which are usually added to increase the richness of the music, but may affect the clarity of the vocal. The presence of harmony and reverberation will make it impossible to generate the lead vocal feature with high-quality singing characteristics in the subsequent process, so it is necessary to reduce the reverberation and harmony in the original vocal as much as possible.

[0060] Optionally, the server can remove reverberation and harmony from the original vocals by inverse filtering, spectral subtraction, recursive neural networks, etc. However, after removing the reverberation and harmony by these methods, reverberation and harmony will still remain in the original vocals, and they cannot be completely removed. Therefore, a binary classification detection model can be used to detect the original vocals that still have reverberation and harmony after the reverberation and harmony are removed once, as the vocal segment to be repaired. Therefore, the vocal segment to be repaired in the embodiment of the present application may refer to a vocal segment that still has reverberation and harmony after the reverberation and harmony are removed once from the original vocals. Among them, the binary classification detection model can be an artificial intelligence model trained by using audio segments with reverberation and harmony as positive samples and audio segments without reverberation and harmony as negative samples.

[0061] To facilitate understanding by those skilled in the art, FIG3 exemplarily provides a user interface diagram for generating a cover song. In a specific implementation, a user can open a music application of a terminal, which includes a function of making cover songs; the music application of the terminal can display a music library interface 300, which may include multiple songs 302 for the user to audition, and the music library interface includes song making prompt information 304, which can be used to instruct the user to make a cover song, and the user can select a target song that he wants to cover from multiple songs 302, and the terminal can enter a song making interface 306 in response to the selection operation of the target song, and optionally, the selection operation of the target song may include but is not limited to triggering operations such as clicking and long pressing the control of the target song; the song making interface 306 includes a cover song selection control 308 and a cover song generation entry 310, and the cover song selection control 308 can be used to select and generate high-quality cover songs in the embodiment of the present application, and the cover song generation entry 310 can be used to trigger the production of cover songs, and high-quality cover songs are produced by the cover song generation method of the embodiment of the present application, thereby meeting the user's demand for professional singing and high-quality cover songs.

[0062] In order to facilitate the understanding of those skilled in the art, FIG. 4 exemplarily provides another user interface schematic diagram for generating cover songs. In a specific implementation, the music application of the terminal can display a video recommendation interface 400. When the user browses the video displayed on the video recommendation interface 400, the user can produce a corresponding high-quality cover song for any song in the favorite video through the cover song generation method of the embodiment of the present application. The video recommendation interface 400 may include a target song confirmation entry 402. The user can perform a user trigger operation on the target song confirmation entry 402 corresponding to the song in the video by taking the song in the video he likes as the target song. The target song confirmation entry 402 is used to enter the cover song confirmation interface 404; the cover song confirmation interface 404 includes a cover song confirmation entry 406, and the cover song confirmation entry 406 is used to trigger the generation of the cover song. The terminal can respond to the user trigger operation on the cover song confirmation entry 406, and produce a high-quality cover song through the cover song generation method of the embodiment of the present application. Therefore, for any song that the user likes, whether it is in the music library or in the video, the user can use the method of the embodiment of the present application to produce a cover version with his own timbre, thereby meeting the user's personalized cover song production needs.

[0063] Through the above solution, users can be provided with a higher quality and more personalized cover song production experience.

[0064] Step S204: obtaining a target vocal segment that matches the vocal segment to be repaired.

[0065] The target vocal segment is a vocal segment without reverberation and harmony, and the target vocal segment and the vocal segment to be repaired correspond to the same singing content in the target song. That is to say, the target vocal segment can be a segment without reverberation and harmony in the original vocals, or a segment without reverberation and harmony in the non-original vocals. The non-original vocals can refer to the sound of a non-original singer singing the target song. The same singing content can include the same lyrics, the same rhythm, the same pitch, etc.

[0066] Optionally, the target vocal segment and the vocal segment to be repaired correspond to the same singing content in the target song, which may mean that the target vocal segment and the vocal segment to be repaired are in the same singing time period in the target song, or, it may mean that the target vocal segment and the vocal segment to be repaired are in different singing time periods in the target song but correspond to the same singing content. For example, a song may contain two chorus parts with the same singing content, and the target vocal segment of one chorus part can be used to replace the vocal segment to be repaired in the other chorus part, and the replaced part corresponds to the same singing content.

[0067] In a specific implementation, when the server identifies the vocal segment to be repaired in the original vocals of the target song, it can determine the singing time period corresponding to the vocal segment to be repaired in the target song, and determine the singing content of the target song in the singing time period; then obtain the original vocals corresponding to the target song, and obtain the non-singing vocals corresponding to the target song, wherein the non-singing vocals can be sung by a non-original singer with excellent singing skills, and as similar as possible to the original singer's voice, with good sound quality and less noise; identify the target vocal segment with the same singing content corresponding to the vocal segment to be repaired in the target song from these original vocals and non-singing vocals. If there are multiple segments matching the vocal segment to be repaired among these original vocals and non-singing vocals, the best quality, least noise and other most satisfactory segments can be selected from them as the target vocal segment; or, the target vocal segment can be obtained from the original vocals first, and if there is no satisfactory segment among the original vocals, the target vocal segment can be obtained from the non-singing vocals.

[0068] Step S206, using the target vocal segment to replace the vocal segment to be repaired in the original singer's vocals to obtain the repaired original singer's vocals, and extracting the singing characteristics of the repaired original singer's vocals to obtain the lead singing characteristics.

[0069] The lead-in singing feature carries the singing skill features of the original singer's voice, and the lead-in singing feature retains the singing skill features of the original singer's voice without the interference of reverberation and harmony. Among them, the singing skill features may include, but are not limited to, features used to reflect the original singer's singing skills such as intonation, pitch, breath, melody, rhythm, singing skills, and emotional expression. Optionally, the singing skill features may include phonetic posteriorgrams (PPG) and fundamental frequency (F0). PPG can reflect the time distribution and phoneme category information of phonemes in the human voice, and can be used to reflect the singing skills such as pronunciation clarity, rhythm, and melody; the fundamental frequency is the basic vibration frequency of the human voice, which can be used to reflect the singing skills such as pitch, intonation, range, and vibrato.

[0070] In the specific implementation, there may be multiple vocal segments to be repaired in the original vocals. For example, the first vocal segment to be repaired in the original vocals corresponds to the singing time period from 1 minute 30 seconds to 1 minute 45 seconds in the target song; the second vocal segment to be repaired in the original vocals corresponds to the singing time period from 2 minutes 40 seconds to 2 minutes 50 seconds in the target song. Therefore, the corresponding target vocal segment can be selected for each vocal segment to be repaired, so that each vocal segment to be repaired in the original vocals is replaced with the corresponding target vocal segment to obtain the repaired original vocals, and the lead singing feature is determined based on the singing performance features such as the phoneme posterior probability graph PPG or fundamental frequency F0 extracted from the repaired original vocals.

[0071] By finely optimizing the original singer's vocals, the existing cover song generation methods can solve the problems of mute, strange sound, and volume imbalance in the cover songs generated. In addition, users can choose any target song they want to cover, which can meet the user's production demands in terms of song personalization and song quality.

[0072] Step S208, extracting the timbre features from the sound material provided by the user, and generating a cover song of the target song based on the timbre features and the lead singing features.

[0073] Among them, the cover song is a song generated by simulating the user's timbre to sing the target song.

[0074] Optionally, the sound material provided by the user may be an audio clip of the user singing any song.

[0075] Among them, timbre characteristics can refer to the unique quality or characteristics of a sound, which can be the main factor that distinguishes one sound from other sounds and can be used to identify the characteristics of different human voices.

[0076] In the specific implementation, the terminal can collect the user's sound material or obtain the sound material uploaded by the user, and store the user's sound material to the Clustered File System (CFS) shared network disk, so that the user's sound material can be shared among multiple servers, which can reduce the complexity of development; the server detects whether the effective human voice duration in the user's sound material reaches the duration threshold (such as 20 seconds or 30 seconds) through dry voice voice activity detection (Voice Activity Detection, VAD), and the sound material whose effective human voice duration does not reach the duration threshold can be discarded; the sound material of the user whose effective human voice duration meets the duration threshold is subjected to noise reduction processing to minimize the interference of noise caused by the environment, equipment, abnormal operations during recording, etc. on subsequent calculations; the timbre extraction model obtained through training extracts timbre features from the sound material provided by the user. Optionally, the timbre extraction model may include but is not limited to deep learning models such as convolutional neural networks, variational autoencoders, recurrent neural networks, and models based on self-attention mechanisms. Taking the convolutional neural network model as an example, by applying a convolution operation to the spectrogram of the audio signal (such as a Mel-spectrogram), local patterns and features related to timbre, such as frequency distribution and harmonic structure, can be effectively extracted.

[0077] After extracting the timbre features from the sound material provided by the user, the terminal or server can generate the cover vocals of the target song based on the timbre features and the lead-singing features, and then professionally mix the cover vocals and the accompaniment of the target song to generate a high-quality cover song. Optionally, the terminal can input the timbre features and the lead-singing features into an artificial intelligence model such as a singing voice conversion model, and the artificial intelligence model can generate the cover vocals of the target song based on the timbre features and the lead-singing features, and the cover vocals simulate the user's timbre while retaining the singing skills features in the lead-singing features; after obtaining the cover vocals, the cover vocals can be professionally mixed with the accompaniment of the target song to generate a high-quality cover song that simulates the user's timbre.

[0078] In the above-mentioned cover song generation method, for the target song selected by the user for cover, a singing vocal segment with reverberation and harmony is identified from the original vocals in the target song as the vocal segment to be repaired; a target vocal segment matching the vocal segment to be repaired is determined, and the target vocal segment is a singing vocal segment with the same singing content as the target song corresponding to the vocal segment to be repaired and without reverberation and / or harmony; the vocal segment to be repaired in the original vocal is replaced with the target vocal segment to obtain the repaired original vocal, and the singing skill features are extracted from the repaired original vocal to obtain the lead singing features; based on the features extracted from the sound material provided by the user The timbre characteristics and lead-in singing characteristics are used to generate a cover song of the target song. The cover song is a song generated by simulating the timbre of the user singing the target song. The vocal segment to be repaired with reverberation and harmony in the original singer's vocals is identified and repaired, and the vocal segment to be repaired is replaced with a high-quality target vocal segment that matches the vocal segment to be repaired. The defective part of the original singer's vocal with reverberation and harmony is optimized, thereby achieving refined optimization processing of the original singer's vocals and retaining the singing characteristics of the original singer's vocals. Furthermore, by extracting the timbre characteristics of the user's voice, a cover song with the user's personalized timbre and high quality is generated, thereby improving the quality of the generated cover song.

[0079] In an exemplary embodiment, before identifying the vocal segment to be repaired in the original vocals of the target song, it also includes: separating the accompaniment and vocals of the target song to obtain the separated original vocals; identifying the non-singing time period between each line in the lyrics according to the position information of each pronunciation unit in the lyrics of the target song; the position information includes the endpoint timestamp of the pronunciation unit and the line identification information of the pronunciation unit in the lyrics; replacing the audio corresponding to the non-singing time period in the separated original vocals with blank audio to obtain the original vocals of the target song.

[0080] In a specific implementation, the server can perform spectrum analysis on the target song and separate the accompaniment and vocals by using the different features of phonemes and accompaniment on the spectrum graph, or separate the accompaniment and vocals through a deep learning model to obtain the separated original vocals. The server can parse the lyrics in the separated original vocals to obtain the position information of each pronunciation unit in the lyrics of the target song. Specifically, the timestamps of the lyrics can be parsed word by word to obtain the parsing results, which include the position information of each pronunciation unit in the lyrics, wherein the position information includes the endpoint timestamp of the pronunciation unit and the row identification information of the pronunciation unit in the lyrics. The endpoint timestamp includes the start timestamp corresponding to the pronunciation start point of the pronunciation unit in the target song and the end timestamp corresponding to the pronunciation end point of the pronunciation unit in the target song.

[0081] In the specific implementation, the parsing result can be expressed as the following structure :

[0082] ;

[0083] Among them, start is the starting timestamp of the pronunciation unit, end is the ending timestamp of the pronunciation unit, word is the content of the pronunciation unit, and line is the line mark of the pronunciation unit in the lyrics. It is the position information of a pronunciation unit.

[0084] Optionally, if the target song is a Chinese song, the pronunciation unit may be a character; if the target song is an English song, the pronunciation unit may be a word.

[0085] Since the lyrics of the target song can be divided into multiple paragraphs of lyrics in lines, each line of lyrics may include multiple pronunciation units or a sentence composed of multiple pronunciation units. Therefore, the line identification information of the pronunciation unit in the lyrics is used to indicate which line the pronunciation unit is in the lyrics.

[0086] In the specific implementation, the terminal identifies the non-singing time period between each line in the lyrics based on the position information of each pronunciation unit. The non-singing time period between each line can be the time period between the end timestamp of the last pronunciation unit in the nth line of the lyrics and the start timestamp of the first pronunciation unit in the n+1th line.

[0087] The server replaces the audio corresponding to the non-singing time period of the separated original singer's vocals with blank audio. The server may first crop the audio corresponding to the non-singing time period of the separated original singer's vocals, and then use the blank audio to fill the cropped part of the separated original singer's vocals to ensure that the audio length of the original singer's vocals after cropping is consistent with that of the separated original singer's vocals before cropping. This ensures that there is no interference from reverberation and harmony in the non-singing time period, thereby reducing the reverberation and harmony part in the original singer's vocals as much as possible in the early stage, which helps to improve the quality of the subsequently generated cover songs.

[0088] Further, in an exemplary embodiment, replacing the audio corresponding to the non-singing time period in the separated original vocals with blank audio to obtain the original vocals of the target song may include: determining the beginning reserved time period and the ending reserved time period in the non-singing time period between any two rows in each row; taking the other time periods except the beginning reserved time period and the ending reserved time period in the non-singing time period between any two rows as the trimming time period; replacing the audio corresponding to the trimming time period in the separated original vocals with blank audio to obtain the original vocals in the target song.

[0089] In order to facilitate the understanding of those skilled in the art, FIG5 exemplifies a flow chart of the original vocal cutting process. Optionally, the beginning reserved time period and the ending reserved time period can be 100 milliseconds or 200 milliseconds. By retaining the beginning reserved time period and the ending reserved time period in the non-singing time period, it is ensured that these reserved time periods will not be cut, thereby avoiding some tail sounds and prolonged sounds from being cut off, and ensuring that useful vocals are avoided from being cut off when cutting the non-singing time period, thereby improving the integrity and quality of the original vocals, and helping to improve the quality of the subsequently generated cover songs.

[0090] Similarly, after cropping the audio corresponding to the cropped time period in the separated original singer's vocals, blank audio can be used to fill the cropped part of the separated original singer's vocals to ensure that the audio length of the original singer's vocals after cropping is consistent with that of the separated original singer's vocals before cropping. This can ensure that there is no interference from reverberation and harmony during the non-singing time period, and also avoid cutting off useful tail sounds, prolonged sounds and other vocals, which helps to improve the quality of subsequently generated cover songs.

[0091] In an exemplary embodiment, identifying a vocal segment to be repaired in the original vocals of a target song includes: detecting an interference segment in the original vocals of the target song; the interference segment is a singing vocal segment with reverberation and / or harmony; for any interference segment, if the interference segment partially falls into the non-singing time period of the original vocals, or if the interference segment completely falls into the singing time period of the original vocals and the endpoint timestamp of the interference segment is not aligned with the endpoint timestamp of the pronunciation unit, adjusting the two endpoint timestamps of the interference segment to obtain a corrected interference segment; the two endpoint timestamps of the corrected interference segment are both aligned with the endpoint timestamp of a pronunciation unit and the corrected interference segment does not include the non-singing time period; according to the corrected interference segment, obtaining the vocal segment to be repaired in the original vocals of the target song.

[0092] In the specific implementation, the server can remove reverberation and harmony from the original vocals through inverse filtering, spectral subtraction, recursive neural network and other methods. However, after removing reverberation and harmony through these methods, reverberation and harmony will still remain in the original vocals and cannot be completely removed. Therefore, a binary classification detection model can be used to detect the interference fragments of reverberation and harmony that remain after the reverberation and harmony are removed from the original vocals. The multiple interference fragments detected can be used with the following structure express:

[0093] ;

[0094] Among them, begin is the starting timestamp of the interference segment, finish is the ending timestamp of the interference segment, and each group Corresponds to an interference fragment.

[0095] In order to facilitate the understanding of those skilled in the art, FIG6 exemplifies a schematic diagram of correcting an interference segment. Among them, "Listen to the Sound of the Sea Crying" is the singing time period of the original singer's voice, each word is regarded as a pronunciation unit, interference segment A partially falls into the non-singing time period of the original singer's voice, interference segment B does not fall into the non-singing time period of the original singer's voice but is not aligned with the endpoint timestamp of the pronunciation unit, and interference segment C partially falls into the non-singing time period of the original singer's voice. Since the two endpoint timestamps of these interference segments are not aligned with the endpoint timestamps of any pronunciation unit, this may cause the replaced segment to be repaired to be unable to smoothly and naturally connect with the unreplaced segment in the original singer's voice when the segment to be repaired is replaced later. Therefore, it is necessary to correct the interference segment to obtain a corrected interference segment, and use the corrected interference segment as the vocal segment to be repaired.

[0096] The two endpoint timestamps of the interference segment include a start timestamp corresponding to the start point of the interference segment in the target song and an end timestamp corresponding to the end point of the interference segment in the target song.

[0097] In a specific implementation, for any interference segment, if the interference segment completely falls into the non-singing time period of the original vocal, the interference segment can be discarded. Specifically, in the above exemplary embodiment, the non-singing time period of the original vocal will be cropped and replaced with blank audio. Therefore, if the interference segment completely falls into the non-singing time period of the original vocal, it does not need to be used as a vocal segment to be repaired.

[0098] In one of the embodiments, adjusting the two endpoint timestamps of the interference segment to obtain a corrected interference segment may include: if the starting timestamp of the interference segment falls between the two endpoint timestamps of any pronunciation unit in the original singer's voice, then correcting the starting timestamp of the interference segment to the starting timestamp of any pronunciation unit to obtain a corrected interference segment; if the starting timestamp of the interference segment falls into the non-singing time period, then correcting the starting timestamp of the interference segment to the starting timestamp of the next pronunciation unit adjacent to the non-singing time period to obtain a corrected interference segment; if the ending timestamp of the interference segment falls between the two endpoint timestamps of any pronunciation unit, then correcting the ending timestamp of the interference segment to the ending timestamp of any pronunciation unit to obtain a corrected interference segment; if the ending timestamp of the interference segment falls into the non-singing time period, then correcting the ending timestamp of the interference segment to the ending timestamp of the previous pronunciation unit adjacent to the non-singing time period to obtain a corrected interference segment.

[0099] In a specific implementation, there may be multiple vocal segments to be repaired in the original vocals. Multiple vocal segments to be repaired can generate a list of vocal segments to be repaired. The list of vocal segments to be repaired can be represented as the following structure: :

[0100] ;

[0101] Among them, initiate is the starting timestamp of the vocal segment to be repaired, and terminate is the ending timestamp of the vocal segment to be repaired. Corresponding to a vocal segment to be repaired, the vocal segment list to be repaired can be used to efficiently determine the singing time period corresponding to the vocal segment to be repaired in the original singer's voice.

[0102] The above scheme can ensure that when the vocal segment to be repaired is replaced later, the replaced vocal segment to be repaired can be smoothly and naturally connected with other unreplaced singing vocal segments, thereby improving the fluency and quality of the generated lead singing features, thereby improving the quality of the subsequently generated cover songs.

[0103] In order to facilitate the understanding of those skilled in the art, FIG. 7 exemplarily provides a flow chart for determining the original vocals and determining the vocal segments to be repaired in the original vocals. It can be seen that when the server determines the target song that the user wants to cover in the music library, it can query the pre-made lead-in feature corresponding to the target song from the database. The process of pre-making the lead-in feature by the server can be to separate the accompaniment and vocals of any target song in the music library, remove the reverberation and harmony, and detect the reverberation and harmony that remain after removing the reverberation and harmony, and obtain the reverberation and harmony detection result, that is, the above-mentioned interference segment; parse the word-by-word timestamp of the original vocal lyrics, thereby cutting out the non-singing time period in the original vocals and replacing the non-singing time period with blank audio to obtain the original vocals; at the same time, correct the reverberation and harmony detection result, obtain a list of vocal segments to be repaired, and replace the vocal segments to be repaired in the list of vocal segments with the target vocal segments, obtain the repaired original vocals, and determine the lead-in feature according to the singing feature extracted from the repaired original vocals.

[0104] Further optionally, based on the corrected interference fragments, the vocal fragments to be repaired in the original vocals of the target song are obtained, which may include: if any corrected interference fragment is a number of pronunciation units in the target line of lyrics in the target song, then the fragments corresponding to the target line of lyrics are all used as the vocal fragments to be repaired in the original vocals of the target song, and the target line can be the line where any corrected interference fragment is located in the lyrics. In other words, in order to ensure natural connection of breath, volume, etc., if the interference fragment is a few words in a line, the fragments corresponding to the entire line of lyrics can be used as the vocal fragments to be repaired, and then a few more words can be replaced or the entire line of lyrics where these words are located can be replaced, so that smooth and natural high-quality lead-in features can be generated, thereby generating high-quality cover songs.

[0105] In another exemplary embodiment, a target vocal segment that matches the vocal segment to be repaired is obtained, including: if there is a pure segment that matches the vocal segment to be repaired in the original vocal, the pure segment is determined as the target vocal segment that matches the vocal segment to be repaired; the pure segment is a singing vocal segment without reverberation and harmony; if there is no pure segment that matches the vocal segment to be repaired in the original vocal, then the target vocal segment that matches the vocal segment to be repaired is determined from a set of candidate vocals.

[0106] A pure segment is a vocal segment without reverberation and harmony in the original vocals. A target song may have repeated lyrics. For example, the lyrics of the first chorus and the second chorus in the target song are repeated. Assuming that the first chorus does not have reverberation and harmony, the first chorus is a pure segment; while the second chorus has reverberation and harmony, the second chorus is a vocal segment to be repaired; since the first chorus and the second chorus correspond to the same singing content in the target song, the first chorus can be used as a target vocal segment that matches the second chorus. In other words, the first chorus (pure segment) can be used to replace the second chorus (target segment). Since the second chorus has been replaced with the first chorus without reverberation and harmony, there is no need to search for a target vocal segment that matches the second chorus from the candidate vocal set. The original vocal segment can be used to generate the lead-in feature to the greatest extent, improving the fluency and consistency of the subsequent generation of the lead-in feature, thereby improving the quality of the cover song.

[0107] If there is no pure segment in the original vocals that matches the vocal segment to be repaired, a target vocal segment that matches the vocal segment to be repaired is determined from the candidate vocal set. The candidate vocal set may include the vocals of non-original singers who have excellent singing skills and similar voices to the original singers of the target song, and these singers have good vocal quality and less noise.

[0108] In an exemplary embodiment, determining a target vocal segment that matches a vocal segment to be repaired from a candidate vocal set includes: obtaining a candidate vocal set; the candidate vocal set includes multiple candidate vocals; and determining a target vocal segment that matches the vocal segment to be repaired from multiple candidate vocals based on at least one of information including the singer's gender, pitch, and loudness of the candidate vocals.

[0109] Further, in an exemplary embodiment, obtaining a set of candidate vocals includes: obtaining multiple dry voices corresponding to the target song; the dry voice is a non-original vocal without reverberation and harmony and the audio length is consistent with the audio length of the original vocal; according to the sound quality index information of the dry voice, determining multiple candidate vocals from the multiple dry voices to obtain a set of candidate vocals; the sound quality index information includes at least one of loudness, clarity, and pitch.

[0110] Among them, the dry voice obtained is a high-quality singing vocal clip obtained by a non-original singer singing the target song completely, so that the high-quality singing vocal clip of the non-original singer can replace the vocal clip to be repaired in the original singer's vocal.

[0111] In a specific implementation, multiple candidate vocals are determined from multiple dry voices based on the sound quality index information of the dry voice. For any dry voice, the loudness of the lyrics is detected, and the volume level of each paragraph or single word of the lyrics is evaluated to ensure that the loudness of the lyrics remains consistent during singing, so as to avoid affecting the overall audio quality due to unbalanced volume. For example, the short-time energy (STE) of each word in the lyrics of the dry voice can be calculated to measure the energy of each word of the dry voice in each time window to reflect the loudness of each word in the dry voice; thus, the volume of the dry voice can be determined to be balanced based on the average loudness of each word in a paragraph of the lyrics. If the volume of several voices is balanced, they can be used as candidate vocals.

[0112] Alternatively, based on the sound quality index information of the dry voice, multiple candidate voices are determined from multiple dry voices. The clarity of the lyrics of any dry voice may be detected. The speech in the dry voice may be converted into text through an automatic speech recognition (ASR) model, and the converted text is compared with the actual lyrics to determine whether the lyrics are sung correctly. Furthermore, the converted text may be refined into text at the phoneme level to detect the clarity of the phoneme pronunciation, so as to detect whether easily confused phoneme pairs "s" and "z", or "p" and "b" are sung clearly.

[0113] Alternatively, based on the sound quality index information of the dry voice, multiple candidate vocals are determined from multiple dry voices. Pitch detection can be performed on any dry voice to generate a pitch score. The pitch detection can be performed by comparing the pitch of the lyrics in the dry voice with the standard pitch of the lyrics of the target song to obtain the pitch score. The pitch scores of a single sentence of lyrics can be filtered, and the dry voice segments with pitch scores higher than the pitch threshold are retained as candidate vocals; the average pitch score of a section of lyrics can also be filtered, and the entire dry voice with an average pitch score lower than the pitch threshold is discarded, and the dry voice with an average pitch score higher than the pitch threshold is used as a candidate vocal.

[0114] Alternatively, based on the sound quality index information of the dry voice, multiple candidate voices are determined from multiple dry voices. For any dry voice, a machine learning model is used to detect multiple dimensions of the dry voice, such as breath and pitch, to obtain a pleasantness score, and the dry voice with a pleasantness score higher than the pleasantness threshold is used as a candidate voice.

[0115] The terminal selects a target vocal segment that matches the vocal segment to be repaired from multiple candidate voices based on at least one of the gender of the singer, the pitch, and the loudness of the candidate voices. In a specific implementation, the terminal may select candidate voices that are in the same singing time period as the vocal segment to be repaired from multiple candidate voices, and then prioritize the candidate voices with the same gender as the singer of the vocal segment to be repaired from the candidate voices in the same singing time period, and prioritize the candidate voices with the same pitch (key) as the vocal segment to be repaired. Optionally, candidate voices with different keys may be adjusted to the key of the vocal segment to be repaired, and prioritize the candidate voices with not too low loudness, such as candidate voices with loudness higher than a loudness threshold, and determine the target vocal segment that matches the vocal segment to be repaired from these prioritized candidate voices.

[0116] To facilitate understanding by those skilled in the art, FIG8 exemplifies a flowchart for generating a set of candidate vocals, in which the collected dry voice encoding is uploaded, and the audio length of the dry voice is verified to be consistent with the original vocals, and then loudness detection, clarity detection, filtering of pitch scores and pleasantness scores are performed to obtain a set of candidate vocals.

[0117] To facilitate understanding by those skilled in the art, FIG9 exemplarily provides a flowchart for generating a guide vocal feature, traversing each vocal segment to be repaired in the list of vocal segments to be repaired, and first detecting whether there is a reusable segment in the original vocal, that is, whether there is a pure segment in the original vocal with the same singing content as the target song corresponding to the vocal segment to be repaired; if so, the reusable segment in the original vocal replaces the current vocal segment to be repaired, and the next vocal segment to be repaired is replaced; if not, a set of candidate vocals is obtained, and the candidate vocals are sorted according to the singer's gender, pitch, and loudness, and the best candidate vocals (for example, the above-mentioned singer's gender, pitch, and loudness conditions are all met) are selected as the target vocal segment, and the target vocal segment replaces the vocal segment to be repaired, and then the next vocal segment to be repaired is replaced, and so on.

[0118] Through the above scheme, candidate vocals can be determined from multiple dry voices according to the sound quality index information of the dry voice, and a high-quality candidate vocal set can be generated. Then, suitable target vocal segments can be screened out from the high-quality candidate vocal set to repair the vocal segments to be repaired, thereby achieving refined optimization processing of the original vocals. Therefore, high-quality lead vocal features can be generated based on the original vocals with repaired reverberation and harmony, and then, based on the high-quality lead vocal features and the timbre features in the user's voice material, a high-quality cover song with user-personalized timbre can be generated.

[0119] FIG. 10 is another method for generating a cover song according to an exemplary embodiment. As shown in FIG. 10 , the method is applied to the server 104 in FIG. 1 as an example for explanation, and includes:

[0120] Step S1002, for the target song selected by the user for cover, detecting interference segments in the original singer's vocals in the target song.

[0121] Interference clips are sung vocal clips with reverb and / or harmony present.

[0122] Step S1004, for any interference segment, if the interference segment partially falls into the non-singing time period of the original singer's vocals, or if the interference segment completely falls into the singing time period of the original singer's vocals and the endpoint timestamp of the interference segment is not aligned with the endpoint timestamp of the pronunciation unit, adjust the two endpoint timestamps of the interference segment to obtain a corrected interference segment.

[0123] Both endpoint timestamps of the corrected interference segment are aligned with the endpoint timestamp of a pronunciation unit and the corrected interference segment does not include a non-singing time period.

[0124] In one of the embodiments, the two endpoint timestamps of the interference segment are adjusted to obtain a corrected interference segment, including: if the start timestamp of the interference segment falls between the two endpoint timestamps of any pronunciation unit in the original singer's voice, the start timestamp of the interference segment is corrected to the start timestamp of any pronunciation unit to obtain a corrected interference segment; if the start timestamp of the interference segment falls into a non-singing time period, the start timestamp of the interference segment is corrected to the start timestamp of the next pronunciation unit adjacent to the non-singing time period to obtain a corrected interference segment; if the end timestamp of the interference segment falls between the two endpoint timestamps of any pronunciation unit, the end timestamp of the interference segment is corrected to the end timestamp of any pronunciation unit to obtain a corrected interference segment; if the end timestamp of the interference segment falls into a non-singing time period, the end timestamp of the interference segment is corrected to the end timestamp of the previous pronunciation unit adjacent to the non-singing time period to obtain a corrected interference segment.

[0125] Step S1006, obtaining the vocal segment to be repaired in the original vocal of the target song according to the corrected interference segment.

[0126] The vocal segment to be restored is a singing vocal segment with reverberation and / or harmony.

[0127] In one of the embodiments, before step S902, it also includes: separating the accompaniment and vocals of the target song to obtain the separated original vocals; identifying the non-singing time period between each line in the lyrics according to the position information of each pronunciation unit in the lyrics of the target song; the position information includes the endpoint timestamp of the pronunciation unit and the line identification information of the pronunciation unit in the lyrics; replacing the audio corresponding to the non-singing time period in the separated original vocals with blank audio to obtain the original vocals of the target song.

[0128] Furthermore, in one of the embodiments, the audio corresponding to the non-singing time period in the separated original vocals is replaced with blank audio to obtain the original vocals of the target song, including: determining the beginning reserved time period and the ending reserved time period in the non-singing time period between any two rows in each row; taking the other time periods except the beginning reserved time period and the ending reserved time period in the non-singing time period between any two rows as the trimming time period; replacing the audio corresponding to the trimming time period in the separated original vocals with blank audio to obtain the original vocals of the target song.

[0129] Step S1008, determining whether there is a pure segment in the original vocals that matches the vocal segment to be repaired, if so, executing step S910; if not, executing step S912.

[0130] Among them, the pure clip is a singing vocal clip without reverberation and harmony.

[0131] Step S1010 , determining the clean segment as a target vocal segment that matches the vocal segment to be repaired, and executing step S1016 .

[0132] Step S1012: Obtain a set of candidate voices.

[0133] The candidate voice set includes multiple candidate voices.

[0134] As an exemplary embodiment, obtaining a set of candidate vocals includes: obtaining multiple dry voices corresponding to a target song; a dry voice is a non-original vocal without reverberation and harmony and whose audio length is consistent with the audio length of the original vocal; based on sound quality index information of the dry voice, determining multiple candidate vocals from the multiple dry voices to obtain a set of candidate vocals; the sound quality index information includes at least one of loudness, clarity, and pitch.

[0135] Step S1014, determining a target vocal segment that matches the vocal segment to be repaired from multiple candidate vocals based on at least one of the singer's gender, pitch, and loudness of the candidate vocals, and executing step S1016.

[0136] Step S1016, using the target vocal segment to replace the vocal segment to be repaired in the original singer's vocals, to obtain the repaired original singer's vocals, and extracting the singing characteristics of the repaired original singer's vocals to obtain the lead singing characteristics.

[0137] Step S1018, extracting the timbre features from the sound material provided by the user, and generating a cover song of the target song based on the timbre features and the lead singing features.

[0138] A cover song is a song generated by simulating the user's timbre to sing the target song.

[0139] It should be noted that the specific limitations of the above steps can be found in the specific limitations of a cover song generation method mentioned above, which will not be repeated here.

[0140] Exemplarily, the method for generating cover songs can be divided into two processes, the first process is to generate lead-in features, and the second process is to generate cover songs. The first process: the original vocals of the target song can be obtained by vocal separation technology, and then the de-harmony algorithm and de-reverberation algorithm are used to remove and separate the harmony and reverberation of the original vocals, so as to improve the quality of generating lead-in features using the original vocals; then, based on the multi-dimensional singing evaluation score and audio detection technology, high-quality singing vocal segments of non-original singers are screened in the candidate vocal set; next, a binary classification model is used to detect the time regions corresponding to the severely residual harmony and reverberation in the original vocals, and the time regions corresponding to these severely residual harmony and reverberation are replaced with high-quality singing vocal segments of non-original singers, and then the phoneme posterior probability map PPG and fundamental frequency F0 are extracted from them to generate lead-in features for the subsequent production of cover songs. The second process: First, collect the user's voice materials and perform feature extraction and training to obtain a timbre extraction model, and use the timbre extraction model to extract the user's timbre features, and then generate a cover vocal based on the timbre features and lead singing features; then professionally mix the cover vocal and the accompaniment of the target song to generate a professional and high-quality cover song.

[0141] In summary, since the generation of cover songs must be based on the lead-in feature, the existing cover song generation method has a relatively limited range of songs that users can choose from. The cover song generation method of the present application is based on multi-dimensional singing evaluation scores, audio detection technology for screening vocal segments, original vocal extraction technology based on vocal separation, audio noise reduction algorithm, harmony reverberation detection algorithm, and sound mixing technology based on cover vocals and accompaniment, which can allow users to provide personalized songs they want to cover, and generate personalized high-quality cover songs that users want. Therefore, the cover song generation method of the present application only requires users to input their own voice materials and provide songs they want to cover, and can produce high-quality cover songs that use user timbre and perform according to the original song professional singing method, and are processed by exquisite mixing. It can solve the problems of damaged audio points and volume imbalance in the existing cover song generation method, and can detect and repair damaged audio points (such as vocal segments with reverberation and harmony), and cooperate with professional mixing processing, which can significantly improve the singing and listening quality of cover songs, and meet the user's more personalized cover song production demands.

[0142] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0143] Based on the same inventive concept, the embodiment of the present application also provides a cover song generation device for implementing the above-mentioned cover song generation method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more cover song generation device embodiments provided below can refer to the above-mentioned limitations on the cover song generation method, and will not be repeated here.

[0144] In an exemplary embodiment, as shown in FIG11 , a cover song generation device is provided, comprising: an identification module 1110, an acquisition module 1120, a replacement module 1130 and a generation model 1140, wherein:

[0145] The identification module 1110 is used to identify the vocal segment to be repaired in the original vocal of the target song selected by the user for cover; the vocal segment to be repaired is a singing vocal segment with reverberation and / or harmony.

[0146] The acquisition module 1120 is used to obtain a target vocal segment that matches the vocal segment to be repaired; the target vocal segment is a singing vocal segment without reverberation and harmony, and the target vocal segment and the vocal segment to be repaired correspond to the same singing content in the target song.

[0147] The replacement module 1130 is used to replace the to-be-repaired vocal segment in the original singer's vocal with the target vocal segment to obtain the repaired original singer's vocal, and extract the singing skill features of the repaired original singer's vocal to obtain the lead vocal feature.

[0148] The generation module 1140 is used to extract the timbre features in the sound material provided by the user, and generate a cover song of the target song based on the timbre features and the lead singing features; the cover song is a song generated by simulating the timbre of the user singing the target song.

[0149] In an exemplary embodiment, the identification module 1110 is used to detect interference segments in the original vocals of the target song; the interference segments are singing vocal segments with reverberation and / or harmony; for any of the interference segments, if the interference segment partially falls into the non-singing time period of the original vocals, or if the interference segment completely falls into the singing time period of the original vocals and the endpoint timestamp of the interference segment is not aligned with the endpoint timestamp of the pronunciation unit, adjust the two endpoint timestamps of the interference segment to obtain a corrected interference segment; the two endpoint timestamps of the corrected interference segment are both aligned with the endpoint timestamp of a pronunciation unit and the corrected interference segment does not include the non-singing time period; based on the corrected interference segment, the vocal segment to be repaired in the original vocals of the target song is obtained.

[0150] In an exemplary embodiment, the identification module 1110 is used to correct the start timestamp of the interference segment to the start timestamp of any pronunciation unit in the original singer's vocals if the start timestamp of the interference segment falls between the two endpoint timestamps of any pronunciation unit in the original singer's vocals, so as to obtain the corrected interference segment; if the start timestamp of the interference segment falls into the non-singing time period, the start timestamp of the interference segment is corrected to the start timestamp of the next pronunciation unit adjacent to the non-singing time period, so as to obtain the corrected interference segment; if the end timestamp of the interference segment falls between the two endpoint timestamps of any pronunciation unit, the end timestamp of the interference segment is corrected to the end timestamp of any pronunciation unit, so as to obtain the corrected interference segment; if the end timestamp of the interference segment falls into the non-singing time period, the end timestamp of the interference segment is corrected to the end timestamp of the previous pronunciation unit adjacent to the non-singing time period, so as to obtain the corrected interference segment.

[0151] In an exemplary embodiment, the acquisition module 1120 is used to determine the pure segment as the target vocal segment that matches the vocal segment to be repaired if there is a pure segment in the original vocal that matches the vocal segment to be repaired; the pure segment is a singing vocal segment without reverberation and harmony; if there is no pure segment in the original vocal that matches the vocal segment to be repaired, determine the target vocal segment that matches the vocal segment to be repaired from the candidate vocal set.

[0152] In an exemplary embodiment, the acquisition module 1120 is used to obtain a candidate vocal set; the candidate vocal set includes multiple candidate vocals; based on at least one of the singer's gender, pitch, and loudness of the candidate vocals, a target vocal segment that matches the vocal segment to be repaired is determined from the multiple candidate vocals.

[0153] In an exemplary embodiment, the acquisition module 1120 is used to acquire multiple dry voices corresponding to the target song; the dry voice is a non-original vocal without reverberation and harmony and the audio length is consistent with the audio length of the original vocal; according to the sound quality index information of the dry voice, the multiple candidate vocals are determined from the multiple dry voices to obtain the candidate vocal set; the sound quality index information includes at least one of loudness, clarity, and pitch.

[0154] In an exemplary embodiment, the device also includes a separation module, which is used to separate the accompaniment and vocals of the target song to obtain the separated original vocals; identify the non-singing time period between each line in the lyrics according to the position information of each pronunciation unit in the lyrics of the target song; the position information includes the endpoint timestamp of the pronunciation unit and the line identification information of the pronunciation unit in the lyrics; replace the audio corresponding to the non-singing time period in the separated original vocals with blank audio to obtain the original vocals of the target song.

[0155] In an exemplary embodiment, the separation module is used to determine the beginning reserved time period and the ending reserved time period in the non-singing time period between any two rows in the rows; use the other time periods in the non-singing time period between any two rows except the beginning reserved time period and the ending reserved time period as the trimming time period; replace the audio corresponding to the trimming time period in the separated original singing vocals with blank audio to obtain the original singing vocals of the target song.

[0156] Each module in the above-mentioned cover song generation device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0157] In an exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG12. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a method for generating a cover song is implemented. The display unit of the computer device is used to form a visually visible picture, which may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0158] Those skilled in the art will understand that the structure shown in FIG. 12 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0159] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above-mentioned method for generating a cover song. The steps of the method for generating a cover song here can be the steps of the method for generating a cover song in each of the above-mentioned embodiments.

[0160] In one embodiment, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the processor executes the steps of the above-mentioned method for generating a cover song. The steps of the method for generating a cover song here can be the steps of the method for generating a cover song in each of the above-mentioned embodiments.

[0161] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, causes the processor to perform the steps of the above-mentioned method for generating a cover song. The steps of the method for generating a cover song here may be the steps of the method for generating a cover song in each of the above-mentioned embodiments.

[0162] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0163] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0164] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0165] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for generating a cover song, characterized in that: The method comprises: For a target song selected by a user for cover, a vocal segment to be repaired in the original vocal of the target song is identified; the vocal segment to be repaired is a vocal segment with reverberation and / or harmony; Obtaining a target vocal segment that matches the vocal segment to be repaired; the target vocal segment is a singing vocal segment without reverberation and harmony, and the target vocal segment and the vocal segment to be repaired correspond to the same singing content in the target song; Using the target vocal segment to replace the vocal segment to be repaired in the original vocal to obtain the repaired original vocal, and extracting the singing feature of the repaired original vocal to obtain the lead vocal feature; The timbre features in the sound material provided by the user are extracted, and a cover song of the target song is generated according to the timbre features and the lead singing features; the cover song is a song generated by simulating the timbre of the user singing the target song.

2. The method according to claim 1, characterized in that: The step of identifying the vocal segment to be repaired in the original vocal of the target song comprises: Detecting interference segments in the original vocals of the target song; the interference segments are vocal segments with reverberation and / or harmony; For any of the interference segments, if the interference segment partially falls into the non-singing time period of the original singer, or if the interference segment completely falls into the singing time period of the original singer and the endpoint timestamp of the interference segment is not aligned with the endpoint timestamp of the pronunciation unit, adjust the two endpoint timestamps of the interference segment to obtain a corrected interference segment; the endpoint timestamp of the corrected interference segment is aligned with the endpoint timestamp of the pronunciation unit and the corrected interference segment does not include the non-singing time period; According to the corrected interference segment, the vocal segment to be repaired in the original vocal of the target song is obtained.

3. The method according to claim 2, characterized in that The adjusting the timestamps of two endpoints of the interference segment includes: If the starting timestamp of the interference segment falls between the two endpoint timestamps of any pronunciation unit in the original singer's voice, the starting timestamp of the interference segment is corrected to the starting timestamp of any pronunciation unit; If the starting timestamp of the interference segment falls within the non-singing time period, the starting timestamp of the interference segment is corrected to the starting timestamp of the next pronunciation unit adjacent to the non-singing time period; If the end timestamp of the interference segment falls between the two endpoint timestamps of any pronunciation unit, the end timestamp of the interference segment is corrected to the end timestamp of any pronunciation unit; If the end timestamp of the interference segment falls within the non-singing time period, the end timestamp of the interference segment is corrected to the end timestamp of the previous pronunciation unit adjacent to the non-singing time period.

4. The method according to claim 1, characterized in that: The step of obtaining a target vocal segment that matches the vocal segment to be repaired includes: If there is a pure segment in the original vocals that matches the vocal segment to be repaired, the pure segment is determined as a target vocal segment that matches the vocal segment to be repaired; the pure segment is a vocal segment without reverberation and harmony; If there is no pure segment in the original vocals that matches the vocal segment to be repaired, a target vocal segment that matches the vocal segment to be repaired is determined from the candidate vocal set.

5. The method according to claim 4, characterized in that The step of determining a target vocal segment matching the vocal segment to be repaired from a candidate vocal set includes: Acquire a candidate voice set; the candidate voice set includes a plurality of candidate voices; A target vocal segment matching the vocal segment to be repaired is determined from the multiple candidate vocals according to at least one of the singer's gender, pitch, and loudness of the candidate vocals.

6. The method according to claim 5, characterized in that The step of obtaining a candidate voice set includes: Acquire multiple dry voices corresponding to the target song; the dry voices are non-original vocals without reverberation and harmony and with an audio length consistent with the audio length of the original vocals; According to the sound quality index information of the dry sound, the multiple candidate voices are determined from the multiple dry sounds to obtain the candidate voice set; the sound quality index information includes at least one of loudness, clarity, and pitch.

7. The method according to claim 1, characterized in that Before identifying the vocal segment to be repaired in the original vocal of the target song, the method further includes: Separating the accompaniment and vocals of the target song to obtain the separated original vocals; According to the position information of each pronunciation unit in the lyrics of the target song, the non-singing time period between each line in the lyrics is identified; the position information includes the endpoint timestamp of the pronunciation unit and the line identification information of the pronunciation unit in the lyrics; The audio corresponding to the non-singing time period in the separated original singer's vocals is replaced with blank audio to obtain the original singer's vocals of the target song.

8. The method according to claim 7, characterized in that The step of replacing the audio corresponding to the non-singing time period in the separated original vocals with blank audio to obtain the original vocals of the target song includes: Determine a beginning reserved time period and an ending reserved time period in a non-singing time period between any two rows in each row; The other time periods in the non-singing time period between any two rows except the beginning reserved time period and the ending reserved time period are used as the cutting time period; The audio corresponding to the cropped time period in the separated original singer's vocals is replaced with blank audio to obtain the original singer's vocals of the target song.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images