A method, device, electronic device and storage medium for aligning melody and lyrics

Abstract

The embodiments of the present application provide a melody and lyrics alignment method, device, electronic device and storage medium, the melody and lyrics alignment method comprising: inputting song audio into a melody extraction multi-task model to obtain a main melody pitch sequence, the probability of each pitch in the main melody pitch sequence being a human voice, and the start time of each word in the lyrics, the melody extraction multi-task model is obtained by training each sample song audio in the sample song audio set according to a preset multi-task neural network model; obtaining a human voice main melody pitch sequence according to the main melody pitch sequence and the probability that each pitch in the main melody pitch sequence is a human voice; determining a target start time for each word in the lyrics according to the start time of each pitch in the human voice main melody pitch sequence, the start time of each word in the lyrics and a reference lyrics file; aligning the melody and lyrics according to the start time of each pitch in the human voice main melody pitch sequence and the target start time of each word in the lyrics to obtain an alignment result.

Description

Technical Field

[0001] The present application relates to the field of network technology, and in particular to a method, device, electronic device and storage medium for aligning melody and lyrics. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the present application. No description herein is admitted to be prior art by virtue of its inclusion in this section.

[0003] The main melody of a song audio refers to an ordered sequence of pitches in the song audio, which consists of pitches and the duration of each pitch. The main melody in a song audio includes the instrumental melody (i.e. the prelude, interlude and coda) and the vocal melody (i.e. the vocals and accompaniment of the vocal part).

[0004] The melody and lyrics alignment is to match the time information of each word in the lyrics with the pitch information to obtain the start time, end time and corresponding pitch of each word in the lyrics. In the related art, the melody and lyrics alignment method is: using the melody extraction model to separate the human voice and melody in the song audio, extract the main melody, and achieve the time alignment of the main melody and the lyrics text. However, the melody extraction model cannot accurately determine whether the extracted main melody is the instrumental main melody or the vocal main melody. Therefore, the accuracy of aligning the extracted main melody with the lyrics is low. Summary of the invention

[0005] In response to the problems in the background technology, the embodiments of the present application provide a method, device, electronic device and storage medium for aligning melody and lyrics.

[0006] On the one hand, an embodiment of the present application provides a method for aligning melody and lyrics, including:

[0007] Inputting the song audio into a melody extraction multi-task model to obtain a main melody pitch sequence, a probability that each pitch in the main melody pitch sequence is a human voice, and a start time of each word in the lyrics, wherein the melody extraction multi-task model is obtained by training the main melody pitch of each sample song audio in the sample song audio set, the main melody pitch of the human voice, and the start time of the word in the sample lyrics according to a preset multi-task neural network model;

[0008] Obtaining a main melody pitch sequence of human voice according to the main melody pitch sequence and the probability that each pitch in the main melody pitch sequence is human voice;

[0009] Determine the target start time of each word in the lyrics according to the start time of each pitch in the main melody pitch sequence of the human voice, the start time of each word in the lyrics and a reference lyrics file;

[0010] The melody and lyrics are aligned according to the start time of each pitch in the pitch sequence of the main melody of the human voice and the target start time of each word in the lyrics to obtain an alignment result.

[0011] In a possible implementation, the song audio is input into the melody extraction multi-task model to obtain the main melody pitch sequence, the probability of each pitch in the main melody pitch sequence being a human voice, and the start time of each word in the lyrics, specifically including:

[0012] Input the song audio into the melody extraction multi-task model to obtain the probability of each pitch at each set time point, the probability that the pitch at each set time point is a human voice, and the probability that the pitch at each set time point is the start time of a word in the lyrics;

[0013] Obtaining a main melody pitch sequence according to the pitch probabilities at the set time points;

[0014] Determining the start time of each word in the lyrics according to the probability that each set time point is the start time of the word in the lyrics; and

[0015] The method further comprises: obtaining a main melody pitch sequence of human voice according to the main melody pitch sequence and the probability that each pitch in the main melody pitch sequence is human voice;

[0016] A main melody pitch sequence of human voice is obtained according to the main melody pitch sequence and the probability that the pitch at each set time point is human voice.

[0017] In a possible implementation, obtaining the main melody pitch sequence according to the pitch probabilities at the set time points specifically includes:

[0018] For each currently set time point, respectively calculate the interval between each pitch and the target pitch at the last set time point;

[0019] Determine the weight corresponding to each calculated interval according to the preset correspondence between the interval and the weight;

[0020] Multiplying each pitch probability at the current set time point by the weight corresponding to each corresponding interval to obtain each candidate pitch probability at the current set time point;

[0021] Determine the pitch corresponding to the maximum candidate pitch probability as the target pitch at the current set time point;

[0022] The target pitch at each set time point is determined as the corresponding main melody pitch to obtain the main melody pitch sequence.

[0023] In a possible implementation, obtaining a main melody pitch sequence of human voice according to the main melody pitch sequence and the probability that the pitch at each set time point is human voice specifically includes:

[0024] Determine the pitch of the main melody at a set time point at which the probability of being a human voice is greater than a first preset threshold as the pitch of the main melody of the human voice;

[0025] The vocal melody pitch sequence is obtained according to the vocal main melody pitch.

[0026] In a possible implementation manner, determining the start time of each word in the lyrics according to the probability that each set time point is the start time of the word in the lyrics specifically includes:

[0027] Obtaining a corresponding probability curve according to the probability that each set time point is the start time of a word in the lyrics;

[0028] The set time points corresponding to the peak values ​​in the probability curve are determined as the start times of the words in the lyrics.

[0029] In a possible implementation, determining the target start time of each word in the lyrics according to the start time of each pitch in the main melody pitch sequence of the human voice, the start time of each word in the lyrics and a reference lyrics file specifically includes:

[0030] Matching the start time of each pitch in the main melody pitch sequence of the human voice with the start time of each word in the lyrics to determine a candidate start time for each word in the lyrics;

[0031] The candidate start time of each word in the lyrics is matched with the reference lyrics file to obtain the target start time of each word in the lyrics.

[0032] In a possible implementation, matching the start time of each pitch in the main melody pitch sequence of the human voice with the start time of each word in the lyrics to determine the candidate start time of each word in the lyrics specifically includes:

[0033] Matching the start time of each pitch in the pitch sequence of the main melody of the human voice with the start time of each word in the lyrics according to a first preset time error range;

[0034] If the start time of any pitch matches the start time of any word in the lyrics within the first preset time error range, the start time of any pitch is determined to be a pitch switching point and a lyrics switching point, and the start time of the pitch is determined to be a candidate start time of the matched any word;

[0035] If the start time of any pitch does not match the start time of any word in the lyrics within the first preset time error range, the start time of any pitch is determined as the pitch switching point under one word with multiple tones;

[0036] If within the first preset time error range, the start time of any word in the lyrics does not match the start time of any pitch, then the start time of any word is determined as the lyrics switching point under one-tone multiple-words, and the start time of any word is determined as the candidate start time of any word.

[0037] In a possible implementation, matching the candidate start time of each word in the lyrics with the reference lyrics file to obtain the target start time of each word in the lyrics specifically includes:

[0038] Matching the closest candidate start time among the candidate start times of each word in the lyrics according to the start time of each word in the reference lyrics file within a second preset time error range;

[0039] The closest matched candidate start times are respectively determined as the target start time of each word in the lyrics.

[0040] In a possible implementation, the method further includes:

[0041] Identify the time point corresponding to each beat point in the song audio according to a preset beat detection algorithm;

[0042] The start time of each pitch in the pitch sequence of the main melody of the human voice and the target start time of each word in the lyrics are matched to the time point corresponding to each beat point.

[0043] On the one hand, an embodiment of the present application provides a melody and lyrics alignment device, including:

[0044] A melody extraction unit, used to input the song audio into a melody extraction multi-task model, to obtain a main melody pitch sequence, a probability that each pitch in the main melody pitch sequence is a human voice, and a start time of each word in the lyrics, wherein the melody extraction multi-task model is obtained by training the main melody pitch of each sample song audio in the sample song audio set, the main melody pitch of the human voice, and the start time of the word in the sample lyrics according to a preset multi-task neural network model;

[0045] An obtaining unit, configured to obtain a main melody pitch sequence of human voice according to the main melody pitch sequence and the probability that each pitch in the main melody pitch sequence is human voice;

[0046] A determination unit, for determining a target start time of each word in the lyrics according to the start time of each pitch in the main melody pitch sequence of the human voice, the start time of each word in the lyrics and a reference lyrics file;

[0047] The processing unit is used to align the melody and lyrics according to the start time of each pitch in the main melody pitch sequence of the human voice and the target start time of each word in the lyrics to obtain an alignment result.

[0048] In a possible implementation manner, the melody extraction unit is specifically used to:

[0049] Input the song audio into the melody extraction multi-task model to obtain the probability of each pitch at each set time point, the probability that the pitch at each set time point is a human voice, and the probability that the pitch at each set time point is the start time of a word in the lyrics;

[0050] Obtaining a main melody pitch sequence according to the pitch probabilities at the set time points;

[0051] Determining the start time of each word in the lyrics according to the probability that each set time point is the start time of the word in the lyrics; and

[0052] The obtaining unit is specifically used to obtain a main melody pitch sequence of human voice according to the main melody pitch sequence and the probability that the pitch at each set time point is human voice.

[0053] In a possible implementation manner, the melody extraction unit is specifically used to:

[0054] For each currently set time point, respectively calculate the interval between each pitch and the target pitch at the last set time point;

[0055] Determine the weight corresponding to each calculated interval according to the preset correspondence between the interval and the weight;

[0056] Multiplying each pitch probability at the current set time point by the weight corresponding to each corresponding interval to obtain each candidate pitch probability at the current set time point;

[0057] Determine the pitch corresponding to the maximum candidate pitch probability as the target pitch at the current set time point;

[0058] The target pitch at each set time point is determined as the corresponding main melody pitch to obtain the main melody pitch sequence.

[0059] In a possible implementation manner, the obtaining unit is specifically configured to:

[0060] Determine the pitch of the main melody at a set time point at which the probability of being a human voice is greater than a first preset threshold as the pitch of the main melody of the human voice;

[0061] The vocal melody pitch sequence is obtained according to the vocal main melody pitch.

[0062] In a possible implementation manner, the melody extraction unit is specifically used to:

[0063] Obtaining a corresponding probability curve according to the probability that each set time point is the start time of a word in the lyrics;

[0064] The set time points corresponding to the peak values ​​in the probability curve are determined as the start times of the words in the lyrics.

[0065] In a possible implementation manner, the determining unit is specifically configured to:

[0066] Matching the start time of each pitch in the main melody pitch sequence of the human voice with the start time of each word in the lyrics to determine a candidate start time for each word in the lyrics;

[0067] The candidate start time of each word in the lyrics is matched with the reference lyrics file to obtain the target start time of each word in the lyrics.

[0068] In a possible implementation manner, the determining unit is specifically configured to:

[0069] Matching the start time of each pitch in the pitch sequence of the main melody of the human voice with the start time of each word in the lyrics according to a first preset time error range;

[0070] If the start time of any pitch matches the start time of any word in the lyrics within the first preset time error range, the start time of any pitch is determined to be a pitch switching point and a lyrics switching point, and the start time of the pitch is determined to be a candidate start time of the matched any word;

[0071] If the start time of any pitch does not match the start time of any word in the lyrics within the first preset time error range, the start time of any pitch is determined as the pitch switching point under one word with multiple tones;

[0072] If within the first preset time error range, the start time of any word in the lyrics does not match the start time of any pitch, then the start time of any word is determined as the lyrics switching point under one-tone multiple-words, and the start time of any word is determined as the candidate start time of any word.

[0073] In a possible implementation manner, the determining unit is specifically configured to:

[0074] Matching the closest candidate start time among the candidate start times of each word in the lyrics according to the start time of each word in the reference lyrics file within a second preset time error range;

[0075] The closest matched candidate start times are respectively determined as the target start time of each word in the lyrics.

[0076] In a possible implementation manner, the device further includes:

[0077] An identification unit, used to identify a time point corresponding to each beat point in the song audio according to a preset beat detection algorithm;

[0078] The beat quantization unit is used to match the start time of each pitch in the pitch sequence of the main melody of the human voice and the target start time of each word in the lyrics to the time point corresponding to each beat point.

[0079] On the one hand, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the melody and lyrics alignment method described in the present application when executing the computer program.

[0080] On the one hand, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the steps in the melody and lyrics alignment method described in the present application are implemented.

[0081] The melody and lyrics alignment method, device, electronic device and storage medium provided in the embodiments of the present application input the song audio into the melody extraction multi-task model to obtain the main melody pitch sequence, the probability that each pitch in the main melody pitch sequence is a human voice, and the start time of each word in the lyrics. The melody extraction multi-task model is obtained by training the main melody pitch of each sample song audio in the sample song audio set, the main melody pitch of the human voice and the start time of the word in the sample lyrics according to a preset multi-task neural network model. The human voice main melody pitch sequence is obtained according to the main melody pitch sequence and the probability that each pitch in the main melody pitch sequence is a human voice. The target start time of each word in the lyrics is determined according to the start time of each pitch in the human voice main melody pitch sequence, the start time of each word in the lyrics and a reference lyrics file. The melody and lyrics are aligned according to the target start time of each word in the lyrics to obtain an alignment result. In the embodiment of the present application, the main melody pitch, the probability of each pitch in the main melody being a human voice, and the start time of each word in the lyrics can be more accurately identified through the melody extraction multi-task model. Therefore, the main melody pitch of the human voice can be more accurately determined according to the main melody pitch and the probability of each pitch in the main melody being a human voice. The start time of each word in the lyrics is corrected according to the identified start time of each pitch of the main melody of the human voice, the start time of each word in the lyrics, and the reference lyrics file to obtain a more accurate target start time for each word in the lyrics. The melody and lyrics are then aligned according to the start time of each pitch of the main melody of the human voice and the target start time of each word in the lyrics, thereby achieving direct alignment of the main melody pitch of the human voice with the lyrics, and the alignment effect is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 A schematic diagram of an application scenario of the melody and lyrics alignment method provided in an embodiment of the present application;

[0083] Figure 2 A flowchart of a method for aligning melody and lyrics provided in an embodiment of the present application;

[0084] Figure 3 A structural diagram of a melody extraction multi-task model provided in an embodiment of the present application;

[0085] Figure 4 A flowchart of obtaining a main melody pitch sequence according to the probability of each pitch at each set time point provided in an embodiment of the present application;

[0086] Figure 5 A flowchart for determining the start time of each word in the lyrics according to the probability that each set time point is the start time of the word in the lyrics provided in an embodiment of the present application;

[0087] Figure 6A flowchart for determining the target start time of each word in lyrics provided in an embodiment of the present application;

[0088] Figure 7 A flowchart of performing beat quantization on the song audio after the melody and lyrics are aligned provided in an embodiment of the present application;

[0089] Figure 8 A schematic diagram of the structure of a melody and lyrics alignment device provided in an embodiment of the present application;

[0090] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0091] The principles and spirit of the present application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present application, and are not intended to limit the scope of the present application in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0092] Those skilled in the art know that the embodiments of the present application can be implemented as a system, device, equipment, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0093] It should be understood herein that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction rather than having any limiting meaning.

[0094] The following is a detailed explanation of the principles and spirit of the present application with reference to several representative embodiments of the present application. For ease of understanding, the technical terms involved in the embodiments of the present application are explained as follows:

[0095] 1. Main melody: refers to an ordered sequence of pitches in a song audio, consisting of pitches and the duration of each pitch. The main melody in a song audio includes the instrumental melody (i.e. the prelude, interlude and coda) and the vocal melody (i.e. the vocals and accompaniment of the vocal part).

[0096] 2. Onset of pitch: the starting time of each note in the main melody.

[0097] 3. Lyrics file (YRC): A format of lyrics data that records the start time and duration of each word in the lyrics, accurate to milliseconds.

[0098] 4. Reference Lyrics File (YRC): The artificial lyrics file published on the Internet is called the reference lyrics file. The timestamp of the reference lyrics file can reflect the time information of each word in the lyrics. The reference lyrics file can contain redundant information that cannot be matched to the lyrics and pitch, such as song title, singer, song segment, etc.

[0099] 5. Lyrics start time (onset): the start time of each word in the lyrics file.

[0100] 6. Align the melody with the lyrics: Match the time information and pitch information of each word in the lyrics to obtain the start time, end time and corresponding pitch of each word in the lyrics.

[0101] 7. Beat: An organizational form that indicates fixed unit time value and strength rules in music. For example, most popular songs are in 4 / 4 time, with a measure containing four beats, and each beat is worth a quarter note.

[0102] 8. Beat quantization: The lyrics and melody in the song audio are generally switched at fixed beat points. Beat quantization refers to matching the time information of the lyrics and melody to the beat points of the song. SUMMARY OF THE INVENTION

[0104] The main melody of a song audio refers to an ordered sequence of pitches in the song audio, which consists of pitches and the duration of each pitch. The main melody in a song audio includes the main melody of an instrument and the main melody of a human voice. The melody and lyrics alignment is to match the time information of each word in the lyrics with the pitch information to obtain the start time, end time and corresponding pitch of each word in the lyrics. In the related art, the melody and lyrics are aligned in the following way: the human voice and melody in the song audio are separated by a melody extraction model, the main melody is extracted, and the time alignment of the main melody and the lyrics text is achieved. However, the melody extraction model cannot accurately determine whether the extracted main melody is the main melody of an instrument or the main melody of a human voice. Therefore, the accuracy of aligning the extracted main melody with the lyrics is low.

[0105] Based on this, the embodiments of the present application provide a melody and lyrics alignment method, device, electronic device and storage medium, which inputs the song audio into a melody extraction multi-task model to obtain a main melody pitch sequence, the probability that each pitch in the main melody pitch sequence is a human voice, and the start time of each word in the lyrics. The melody extraction multi-task model is obtained by training the main melody pitch of each sample song audio in the sample song audio set, the main melody pitch of the human voice, and the start time of the word in the sample lyrics according to a preset multi-task neural network model. The human voice main melody pitch sequence is obtained according to the main melody pitch sequence and the probability that each pitch in the main melody pitch sequence is a human voice. The human voice main melody pitch sequence is obtained according to the start time of each pitch in the human voice main melody pitch sequence, the start time of each word in the lyrics and a reference lyrics file. The melody and lyrics are aligned according to the start time and the target start time of each word in the lyrics to obtain an alignment result. In the embodiment of the present application, the main melody pitch, the probability of each pitch in the main melody being a human voice, and the start time of each word in the lyrics can be more accurately identified through the melody extraction multi-task model. Therefore, the main melody pitch of the human voice can be more accurately determined according to the main melody pitch and the probability of each pitch in the main melody being a human voice. The start time of each word in the lyrics is corrected according to the identified start time of each pitch of the main melody of the human voice, the start time of each word in the lyrics and the reference lyrics file to obtain a more accurate target start time for each word in the lyrics. The melody and lyrics are then aligned according to the start time of each pitch of the main melody of the human voice and the target start time of each word in the lyrics, thereby achieving direct alignment of the main melody pitch of the human voice with the lyrics, and the alignment effect is more accurate.

[0106] After introducing the basic principles of the present application, various non-limiting implementation methods of the present application are described in detail below.

[0107] Application Scenario Overview

[0108] refer to Figure 1, which is a schematic diagram of an application scenario of the melody and lyrics alignment method provided in an embodiment of the present application. The application scenario may include a server 101 and a terminal 102, the server 101 and the terminal 102 are connected via a communication network, the server 101 may align the melody and lyrics for the song audio sent by the terminal 102, and return the aligned song audio to the terminal 102. The server 101 inputs the song audio sent by the terminal 102 into the melody extraction multi-task model to obtain the main melody pitch sequence, the probability that each pitch in the main melody pitch sequence is a human voice, and the start time of each word in the lyrics. The melody extraction multi-task model is obtained by training the main melody pitch of each sample song audio in the sample song audio set, the main melody pitch of the human voice, and the start time of the word in the sample lyrics according to a preset multi-task neural network model. The human voice main melody pitch sequence is obtained according to the main melody pitch sequence and the probability that each pitch in the main melody pitch sequence is a human voice. The target start time of each word in the lyrics is determined according to the start time of each pitch in the main melody pitch sequence of the human voice, the start time of each word in the lyrics and the reference lyrics file. The melody and lyrics are aligned according to the start time of each pitch in the main melody pitch sequence of the human voice and the target start time of each word in the lyrics to obtain an alignment result.

[0109] Among them, the communication network can be a local area network, a wide area network, a mobile Internet, etc., the server 101 can include an independent physical server, and can also include a cloud server that provides basic cloud computing services such as cloud servers, cloud databases, and cloud storage. The terminal 102 can be but is not limited to: smart phones, tablet computers, laptops, desktop computers and other devices, and the embodiments of the present application are not limited to this.

[0110] The application scenario of the melody and lyrics alignment method provided in the embodiment of the present application may also only include the server 101, and the server 101 aligns the melody and lyrics for the acquired song audio, which is not limited in the embodiment of the present application.

[0111] Exemplary Methods

[0112] The following describes the melody and lyrics alignment method according to the exemplary embodiment of the present application in combination with the above application scenario. It should be noted that the above application scenario is only shown to facilitate understanding of the spirit and principle of the present application, and the implementation of the present application is not limited in this regard. On the contrary, the implementation of the present application can be applied to any applicable scenario.

[0113] refer to Figure 2 The present application embodiment provides a method for aligning melody and lyrics, which can be applied to Figure 1 The server 101 shown may specifically include the following steps:

[0114] S21. Input the song audio into the melody extraction multi-task model to obtain the main melody pitch sequence, the probability that each pitch in the main melody pitch sequence is a human voice, and the start time of each word in the lyrics.

[0115] In specific implementation, the server inputs the song audio into the melody extraction multi-task model, obtains the probability of each pitch at each set time point, the probability of the pitch being a human voice at each set time point, and the probability of the start time of a word in the lyrics at each set time point, and then obtains the main melody pitch sequence according to the probability of each pitch being the start time of a word in the lyrics at each set time point, and determines the start time of each word in the lyrics according to the probability of the start time of a word in the lyrics at each set time point. Among them, the melody extraction multi-task model is obtained by training the main melody pitch of each sample song audio in the sample song audio set, the main melody pitch of the human voice, and the start time of the word in the sample lyrics according to the preset multi-task neural network model.

[0116] As a possible implementation, the structure diagram of the melody extraction multi-task model in the embodiment of the present application is as follows: Figure 3 As shown, the melody extraction multi-task model may include: an input layer, a shared network layer, a first task network layer, a second task network layer and a third task network layer, and the shared network layer is connected to the first task network layer and the second task network layer respectively.

[0117] As a possible implementation, the first task network layer may include a first fully connected layer and a first classifier, the second task network layer may include a second fully connected layer and a second classifier, the third task network layer may include a third fully connected layer and a third classifier, the input layer is connected to the shared network layer, the shared network layer is respectively connected to the first fully connected layer of the first task network layer, the second fully connected layer of the second task network layer, and the third fully connected layer of the third task network layer, the first fully connected layer is connected to the first classifier, the first classifier is connected to the first output layer, the second fully connected layer is connected to the second classifier, the second classifier is connected to the second output layer, the third fully connected layer is connected to the third classifier, and the third classifier is connected to the third output layer. In this implementation, the shared network layer may be, but is not limited to, a CNN (Convolutional Neural Network) model + LSTM (Long Short Term Memory Network) model, wherein the input layer inputs the CQT spectrum of the audio, the spectrum features are first extracted by the CNN model, and then the temporal features are extracted by the LSTM model. The first fully connected layer, the second fully connected layer and the third fully connected layer can all be fully connected layers with the same initial parameters, and the first classifier, the second classifier and the third classifier can all be Softmax classifiers.

[0118] As a possible implementation, the fully connected layers of the first task, the second task, and the third task can also be shared. In this implementation, the shared network layer may include, but is not limited to, the following structure: CNN model + LSTM model + fully connected layer. The first task network layer may include a first classifier, the second task network layer may include a second classifier, and the third task network layer may include a third classifier. The input layer is connected to the shared network layer, and the shared network layer is respectively connected to the first classifier of the first task network layer, the second classifier of the second task network layer, and the third classifier of the third task network layer. The first classifier is connected to the first output layer, the second classifier is connected to the second output layer, and the third classifier is connected to the third output layer. The first classifier, the second classifier, and the third classifier can all use Softmax classifiers.

[0119] The above-mentioned melody extraction multi-task model is obtained by training based on a preset multi-task neural network model based on a sample song audio set, that is to say: the structure of the trained multi-task deep neural network model is the same as the structure of the melody extraction multi-task model. During the training process, a large number of song audios can be used to form a sample song audio set, and each sample song audio in the sample object text set is used as the input of the multi-task neural network model. According to the set time interval, the first task network, the second task network, and the third task network are respectively used to predict the probability of each pitch at each set time point, the probability of the pitch at each set time point being a human voice, and the probability of the start time of the word in the lyrics at each set time point, wherein the set time interval can be set as needed, and the embodiment of the present application does not limit this. For example, the set time interval can be set to 64ms, and each interval of 64ms is a set time point, and the main melody pitch of the sample song audio and the start time of each pitch are marked, the main melody pitch of the human voice is marked, and the start time of each word in the sample lyrics (i.e., the lyrics of the sample song audio) is marked, and the multi-task neural network model is trained according to the predicted value and the marked value of each task network. During the training process of the multi-task neural network model, the first task, the second task and the third task can respectively but not limited to use cross entropy as the loss function. The loss function of the multi-task neural network model can be the weighted sum of the loss function of the first task, the loss function of the second task and the loss function of the third task. During the training process, the parameters of the multi-task neural network model are continuously adjusted until the model converges to obtain a melody extraction multi-task model.

[0120] The specific formula of the loss function of the multi-task learning deep neural network model is as follows:

[0121] Loss=W1*Loss task1 +W2*Loss task2 +W3*Loss task3

[0122] Among them, Loss represents the loss function of the multi-task neural network model;

[0123] Loss task1 Represents the loss function of the first task, W1 represents Loss task1 The weight of

[0124] Loss task2 Represents the loss function of the second task, W2 represents Loss task2 The weight of

[0125] Loss task3 represents the loss function of the second task, W3 represents Loss task3 The weight of .

[0126] Among them, the values ​​of W1, W2 and W3 can be set according to the importance of the first task, the second task and the third task, W1+W2+W3=1.

[0127] Specifically, the song audio is input into the input layer of the melody extraction multi-task model, and the first output layer outputs the probability of each pitch at each set time point, the second output layer outputs the probability that the pitch at each set time point is a human voice, and the third output layer outputs the probability of the start time of a word in the lyrics at each set time point.

[0128] During the implementation process, Figure 4 The process shown in the figure obtains the main melody pitch sequence according to the pitch probabilities at each set time point, and may include the following steps:

[0129] S31. For each currently set time point, respectively calculate the interval between each pitch and the target pitch at the last set time point.

[0130] In specific implementation, the probability of all pitches is included at each set time point. Initially, the pitch with the highest pitch probability can be determined as the target pitch at the first set time point. Starting from the second set time point, the target pitch at the current set time point is determined based on the pitches at the current set time point and the target pitches at the previous set time point.

[0131] Specifically, for each currently set time point, the intervals between each pitch at the currently set time point and the target pitch at the last set time point are calculated, wherein the interval between two pitches is the distance between the two pitches, that is, the absolute value of the difference between the two pitches.

[0132] S32. Determine the weights corresponding to the calculated intervals according to the preset correspondence between the intervals and the weights.

[0133] In a specific implementation, the weights corresponding to the intervals are preset, and the weights corresponding to the calculated intervals are determined according to the preset corresponding relationship between the intervals and the weights.

[0134] S33, multiplying each pitch probability at the current set time point by the weight corresponding to each corresponding interval, to obtain each candidate pitch probability at the current set time point.

[0135] In specific implementation, each pitch probability at the current set time point is multiplied by the weight corresponding to each corresponding interval to obtain each candidate pitch probability at the current set time point.

[0136] S34: Determine the pitch corresponding to the maximum candidate pitch probability as the target pitch at the current set time point.

[0137] For example, if the probability of pitch A at the current set time point multiplied by the weight a corresponding to the interval |AB| between pitch A and the target pitch B at the previous set time point gives the largest candidate pitch probability, then pitch A is determined as the target pitch at the current set time point.

[0138] S35 , determining the target pitch at each set time point as the corresponding main melody pitch, and obtaining a main melody pitch sequence.

[0139] During specific implementation, the target pitch at each set time point is determined as the main melody pitch corresponding to each set time point, and a main melody pitch sequence is obtained.

[0140] As a possible implementation, when determining the start time of each word in the lyrics according to the probability that each set time point is the start time of the word in the lyrics, the following method can be used: Figure 5 The process shown determines the start time of each word in the lyrics, including the following steps:

[0141] S41. Obtain a corresponding probability curve according to the probability that each set time point is the start time of a word in the lyrics.

[0142] S42, determining the set time points corresponding to the peak values ​​in the probability curve as the start times of the words in the lyrics.

[0143] In this way, each set time point corresponding to the obtained peak probability curve corresponds to the start time of each word in the lyrics.

[0144] S22. Obtain a main melody pitch sequence of human voice according to the main melody pitch sequence and the probability that each pitch in the main melody pitch sequence is human voice.

[0145] In a specific implementation, the pitch sequence of the main melody of human voice is obtained according to the pitch sequence of the main melody and the probability that the pitch at each set time point is human voice.

[0146] Specifically, the main melody pitch at the set time point when the probability of the pitch being a human voice is greater than the first preset threshold is determined as the main melody pitch of the human voice, and a human voice melody pitch sequence is obtained according to the main melody pitch of the human voice, that is, the main melody pitches of the human voice are arranged in chronological order to form the main melody pitch sequence of the human voice. The first preset threshold can be set according to the needs, for example, the first preset threshold can be set to any value between greater than or equal to 0.6 and less than or equal to 1, and the embodiment of the present application does not limit this.

[0147] S23, determining a target start time for each word in the lyrics according to the start time of each pitch in the main melody pitch sequence of the human voice, the start time of each word in the lyrics and a reference lyrics file.

[0148] Considering that the start time of each word in the lyrics predicted by the melody extraction multi-task model can cover all the real lyrics switching points, but may also include misjudgments, we can first establish a lyrics time candidate pool to store the candidate start time of each word in the lyrics, and then accurately match the candidate start time of each word in the lyrics with the reference lyrics file to determine the target start time of each word in the lyrics, thereby improving the accuracy of the start time of each word in the lyrics.

[0149] As a possible implementation method, the following Figure 6 The process shown determines the target start time of each word in the lyrics, including the following steps:

[0150] S51, matching the start time of each pitch in the main melody pitch sequence of the human voice with the start time of each word in the lyrics, and determining a candidate start time for each word in the lyrics.

[0151] During specific implementation, in the pitch sequence of the main melody of the human voice, if the pitch of a set time point is different from the pitch of the previous set time point, then the set time point is the start time of the pitch. If the pitches of at least two adjacent set time points are the same, then the first set time point corresponding to the pitch is determined as the start time of the pitch. The start time of each pitch in the main melody pitch sequence of the human voice is matched with the start time of each word in the lyrics according to the first preset time error range, and the candidate start time of each word in the lyrics is determined according to the matching result. Among them, the first preset time error range can be set by itself, for example, it can be set to 100 milliseconds, and the embodiment of the present application does not limit this. The matching results may include at least the following three situations:

[0152] Case 1: If the start time of any pitch matches the start time of any word in the lyrics within the first preset time error range, the start time of any pitch is determined as the pitch switching point and the lyrics switching point, and the start time of the pitch is determined as the candidate start time of any matched word.

[0153] In specific implementation, if the start time of any current pitch in the main melody pitch sequence of the human voice matches the start time of any word in the lyrics within the first preset time error range (such as within 100 milliseconds), then the current start time of any pitch is a pitch switching point and also a lyrics switching point. The start time of any pitch is determined as the candidate start time of the matched any word and is added to the lyrics time candidate pool for storage.

[0154] Case 2: If the start time of any pitch does not match the start time of any word in the lyrics within the first preset time error range, the start time of any pitch is determined as the pitch switching point under one word with multiple tones.

[0155] In specific implementation, if within the first preset time error range, the start time of any current pitch in the main melody pitch sequence of the human voice does not match the start time of any word in the lyrics, then the current start time of any pitch is the pitch switching point in the case of multiple tones for one word, but the word has not been switched, so it is not a candidate start time for the word in the lyrics, and the start time of any pitch will not be added to the lyrics time candidate pool.

[0156] Case three: if within the first preset time error range, the start time of any word in the lyrics does not match the start time of any pitch, then the start time of any word is determined as the lyrics switching point under one-tone multiple-words, and the start time of any word is determined as the candidate start time of any word.

[0157] In specific implementation, if within the first preset time error range, if the start time of any word in the lyrics does not match the start time of any pitch in the main melody pitch sequence of the human voice, then the start time of the any word is determined as the lyrics switching point under one-tone multiple words, but the pitch is not switched, then the start time of the any word is used as the candidate start time of the any word, and it is added to the lyrics time candidate pool for storage.

[0158] According to the second and third cases, the candidate start time for one tone and multiple words and one word with multiple tones is processed, so that the candidate start time of the words in the lyrics stored in the lyrics time candidate pool is more accurate. The lyrics time candidate pool processed by the above three cases contains all the start times of the words in the lyrics that may be the real lyrics switching points and can correspond to the pitch start time in the main melody pitch sequence of the human voice.

[0159] S52: Match the candidate start time of each word in the lyrics with the reference lyrics file to obtain the target start time of each word in the lyrics.

[0160] In specific implementation, within the second preset time error range, the start time of each word in the reference lyrics file is matched with the closest candidate start time among the candidate start times of each word in the lyrics, and each matched closest candidate start time is respectively determined as the target start time for each word in the lyrics.

[0161] Specifically, the second preset time error range can be set according to the needs, for example, it can be set to within 1s, and the embodiment of the present application does not limit this. Within the second preset time error range, each closest candidate start time that can match the start time of each word in the reference lyrics file is determined as the target start time of each word in the lyrics. This is because when predicting the start time of each word in the lyrics according to the melody extraction multi-task model, in order to ensure that the start time of each word in the real lyrics is included in the prediction result, there will also be misjudged time points such as the start time of the instrument drum beats that are misjudged, and the candidate start time of the words in the lyrics stored in the lyrics candidate pool includes the start time of these redundant instrument drum beats that are misjudged by the melody extraction multi-task model. By matching the start time of each word in the reference lyrics file, the candidate start time of the word closest to the start time of each word in the reference lyrics is retained as the final start time of each word in the lyrics, that is: the target start time of each word in the lyrics, which further improves the accuracy of the predicted start time of each word in the lyrics.

[0162] S24, aligning the melody and the lyrics according to the start time of each pitch in the pitch sequence of the main melody of the human voice and the target start time of each word in the lyrics to obtain an alignment result.

[0163] In specific implementation, based on each set time point, the start time of each pitch in the main melody pitch sequence of the human voice can be matched one by one with the target start time of each word in the lyrics. In this way, if there are multiple words for one note, the current note will be cut, and if there are multiple notes for one word, multiple consecutive notes will correspond to one word.

[0164] As a possible implementation, after the melody and lyrics of the song audio are aligned, the beat quantization of the aligned song audio may be performed.

[0165] When implementing it specifically, you can follow the following steps: Figure 7 The process shown in the figure performs beat quantization on the song audio after the melody and lyrics are aligned, and includes the following steps:

[0166] S61. Identify the time point corresponding to each beat point in the song audio according to a preset beat detection algorithm.

[0167] In specific implementation, the open source Madmom beat detection algorithm can be used, but is not limited to, to identify the time point corresponding to each beat point in the song audio.

[0168] S62, matching the start time of each pitch in the pitch sequence of the main melody of the human voice and the target start time of each word in the lyrics to the time point corresponding to each beat point.

[0169] The melody and lyrics alignment method provided in the embodiment of the present application is that the server inputs the song audio into the melody extraction multi-task model to obtain the main melody pitch sequence, the probability that each pitch in the main melody pitch sequence is a human voice, and the start time of each word in the lyrics. The melody extraction multi-task model is obtained by training the main melody pitch of each sample song audio in the sample song audio set, the main melody pitch of the human voice, and the start time of the word in the sample lyrics according to a preset multi-task neural network model. The human voice main melody pitch sequence is obtained according to the main melody pitch sequence and the probability that each pitch in the main melody pitch sequence is a human voice. The target start time of each word in the lyrics is determined according to the start time of each pitch in the human voice main melody pitch sequence, the start time of each word in the lyrics and the reference lyrics file. The melody and lyrics are aligned according to the target start time of each word to obtain an alignment result. In the embodiment of the present application, the main melody pitch, the probability of each pitch in the main melody being a human voice, and the start time of each word in the lyrics can be more accurately identified through the melody extraction multi-task model. Therefore, the pitch of the main melody of the human voice can be more accurately determined according to the main melody pitch and the probability of each pitch in the main melody being a human voice. The start time of each word in the lyrics is corrected according to the identified start time of each pitch of the main melody of the human voice, the start time of each word in the lyrics and the reference lyrics file to obtain a more accurate target start time for each word in the lyrics. The melody and lyrics are then aligned according to the start time of each pitch of the main melody of the human voice and the target start time of each word in the lyrics, thereby achieving direct alignment of the main melody pitch of the human voice with the lyrics, and the alignment effect is more accurate.

[0170] Exemplary Devices

[0171] After introducing the melody and lyrics alignment method according to the exemplary embodiment of the present application, next, the melody and lyrics alignment device according to the exemplary embodiment of the present application will be introduced.

[0172] like Figure 8 As shown, it is a schematic diagram of the structure of the melody and lyrics alignment device 70 provided in an embodiment of the present application. In one embodiment, the melody and lyrics alignment device 70 includes:

[0173] The melody extraction unit 701 is used to input the song audio into a melody extraction multi-task model to obtain a main melody pitch sequence, a probability that each pitch in the main melody pitch sequence is a human voice, and a start time of each word in the lyrics, wherein the melody extraction multi-task model is obtained by training the main melody pitch of each sample song audio in the sample song audio set, the main melody pitch of the human voice, and the start time of the word in the sample lyrics according to a preset multi-task neural network model;

[0174] An obtaining unit 702 is used to obtain a main melody pitch sequence of human voice according to the main melody pitch sequence and the probability that each pitch in the main melody pitch sequence is human voice;

[0175] A determination unit 703, configured to determine a target start time for each word in the lyrics according to the start time of each pitch in the main melody pitch sequence of the human voice, the start time of each word in the lyrics and a reference lyrics file;

[0176] The processing unit 704 is used to align the melody and lyrics according to the start time of each pitch in the pitch sequence of the main melody of the human voice and the target start time of each word in the lyrics to obtain an alignment result.

[0177] In a possible implementation, the melody extraction unit 701 is specifically configured to:

[0178] Input the song audio into the melody extraction multi-task model to obtain the probability of each pitch at each set time point, the probability that the pitch at each set time point is a human voice, and the probability that the pitch at each set time point is the start time of a word in the lyrics;

[0179] Obtaining a main melody pitch sequence according to the pitch probabilities at the set time points;

[0180] Determining the start time of each word in the lyrics according to the probability that each set time point is the start time of the word in the lyrics; and

[0181] The obtaining unit is specifically used to obtain a main melody pitch sequence of human voice according to the main melody pitch sequence and the probability that the pitch at each set time point is human voice.

[0182] In a possible implementation, the melody extraction unit 701 is specifically configured to:

[0183] For each currently set time point, respectively calculate the interval between each pitch and the target pitch at the last set time point;

[0184] Determine the weight corresponding to each calculated interval according to the preset correspondence between the interval and the weight;

[0185] Multiplying each pitch probability at the current set time point by the weight corresponding to each corresponding interval to obtain each candidate pitch probability at the current set time point;

[0186] Determine the pitch corresponding to the maximum candidate pitch probability as the target pitch at the current set time point;

[0187] The target pitch at each set time point is determined as the corresponding main melody pitch to obtain the main melody pitch sequence.

[0188] In a possible implementation manner, the obtaining unit 702 is specifically configured to:

[0189] Determine the pitch of the main melody at a set time point at which the probability of being a human voice is greater than a first preset threshold as the pitch of the main melody of the human voice;

[0190] The vocal melody pitch sequence is obtained according to the vocal main melody pitch.

[0191] In a possible implementation, the melody extraction unit 701 is specifically configured to:

[0192] Obtaining a corresponding probability curve according to the probability that each set time point is the start time of a word in the lyrics;

[0193] The set time points corresponding to the peak values ​​in the probability curve are determined as the start times of the words in the lyrics.

[0194] In a possible implementation manner, the determining unit 703 is specifically configured to:

[0195] Matching the start time of each pitch in the main melody pitch sequence of the human voice with the start time of each word in the lyrics to determine a candidate start time for each word in the lyrics;

[0196] The candidate start time of each word in the lyrics is matched with the reference lyrics file to obtain the target start time of each word in the lyrics.

[0197] In a possible implementation manner, the determining unit 703 is specifically configured to:

[0198] Matching the start time of each pitch in the pitch sequence of the main melody of the human voice with the start time of each word in the lyrics according to a first preset time error range;

[0199] If the start time of any pitch matches the start time of any word in the lyrics within the first preset time error range, the start time of any pitch is determined to be a pitch switching point and a lyrics switching point, and the start time of the pitch is determined to be a candidate start time of the matched any word;

[0200] If the start time of any pitch does not match the start time of any word in the lyrics within the first preset time error range, the start time of any pitch is determined as the pitch switching point under one word with multiple tones;

[0201] If within the first preset time error range, the start time of any word in the lyrics does not match the start time of any pitch, then the start time of any word is determined as the lyrics switching point under one-tone multiple-words, and the start time of any word is determined as the candidate start time of any word.

[0202] In a possible implementation manner, the determining unit 703 is specifically configured to:

[0203] Matching the closest candidate start time among the candidate start times of each word in the lyrics according to the start time of each word in the reference lyrics file within a second preset time error range;

[0204] The closest matched candidate start times are respectively determined as the target start time of each word in the lyrics.

[0205] In a possible implementation manner, the device further includes:

[0206] An identification unit, used to identify a time point corresponding to each beat point in the song audio according to a preset beat detection algorithm;

[0207] The beat quantization unit is used to match the start time of each pitch in the pitch sequence of the main melody of the human voice and the target start time of each word in the lyrics to the time point corresponding to each beat point.

[0208] Based on the same inventive concept as the above-mentioned melody and lyrics alignment method, the embodiment of the present application also provides an electronic device 80. Fig. 9 As shown, the electronic device 80 may include a processor 801 and a memory 802 .

[0209] The processor 801 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and may implement or execute the melody and lyrics alignment methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the melody and lyrics alignment method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0210] The memory 802 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 802 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0211] It should be noted that although several units or sub-units of the melody and lyrics alignment device are mentioned in the above detailed description, this division is only exemplary and not mandatory. In fact, according to the embodiment of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units for embodiment.

[0212] In addition, although the operations of the melody and lyrics alignment method of the present application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0213] Although the spirit and principle of the present application have been described with reference to several specific embodiments, it should be understood that the present application is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present application is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims.

Claims

1. A method for aligning melody and lyrics, characterized in that: include: Inputting the song audio into a melody extraction multi-task model to obtain a main melody pitch sequence, a probability that each pitch in the main melody pitch sequence is a human voice, and a start time of each word in the lyrics, wherein the melody extraction multi-task model is obtained by training the main melody pitch of each sample song audio in the sample song audio set, the main melody pitch of the human voice, and the start time of the word in the sample lyrics according to a preset multi-task neural network model; Obtaining a main melody pitch sequence of human voice according to the main melody pitch sequence and the probability that each pitch in the main melody pitch sequence is human voice; According to the start time of each pitch in the main melody pitch sequence of the human voice, the start time of each word in the lyrics and the reference lyrics file, the target start time of each word in the lyrics is determined, specifically comprising: matching the start time of each pitch in the main melody pitch sequence of the human voice with the start time of each word in the lyrics to determine the candidate start time of each word in the lyrics; matching the candidate start time of each word in the lyrics with the reference lyrics file to obtain the target start time of each word in the lyrics; Matching the start time of each pitch in the main melody pitch sequence of the human voice with the start time of each word in the lyrics to determine the candidate start time of each word in the lyrics, specifically comprising: matching the start time of each pitch in the main melody pitch sequence of the human voice with the start time of each word in the lyrics according to a first preset time error range; if within the first preset time error range, the start time of any pitch matches the start time of any word in the lyrics, then determining the start time of any pitch as a pitch switching point and a lyrics switching point, and determining the start time of the pitch as the candidate start time of the matched any word; if within the first preset time error range, the start time of any pitch does not match the start time of any word in the lyrics, then determining the start time of any pitch as a pitch switching point under one word with multiple tones; if within the first preset time error range, the start time of any word in the lyrics does not match the start time of any pitch, then determining the start time of any word as a lyrics switching point under one tone with multiple words, and determining the start time of any word as the candidate start time of any word; The melody and lyrics are aligned according to the start time of each pitch in the pitch sequence of the main melody of the human voice and the target start time of each word in the lyrics to obtain an alignment result.

2. The method according to claim 1, characterized in that The song audio is input into the melody extraction multi-task model to obtain the main melody pitch sequence, the probability of each pitch in the main melody pitch sequence being a human voice, and the start time of each word in the lyrics, specifically including: Input the song audio into the melody extraction multi-task model to obtain the probability of each pitch at each set time point, the probability that the pitch at each set time point is a human voice, and the probability that the pitch at each set time point is the start time of a word in the lyrics; Obtaining a main melody pitch sequence according to the pitch probabilities at the set time points; Determining the start time of each word in the lyrics according to the probability that each set time point is the start time of the word in the lyrics; and The method further comprises: obtaining a main melody pitch sequence of human voice according to the main melody pitch sequence and the probability that each pitch in the main melody pitch sequence is human voice; A main melody pitch sequence of human voice is obtained according to the main melody pitch sequence and the probability that the pitch at each set time point is human voice.

3. The method according to claim 2, characterized in that Obtaining the main melody pitch sequence according to the pitch probabilities at the set time points specifically includes: For each currently set time point, respectively calculate the interval between each pitch and the target pitch at the last set time point; Determine the weight corresponding to each calculated interval according to the preset correspondence between the interval and the weight; Multiplying each pitch probability at the current set time point by the weight corresponding to each corresponding interval to obtain each candidate pitch probability at the current set time point; Determine the pitch corresponding to the maximum candidate pitch probability as the target pitch at the current set time point; The target pitch at each set time point is determined as the corresponding main melody pitch to obtain the main melody pitch sequence.

4. The method according to claim 2, characterized in that Obtaining a human voice main melody pitch sequence according to the main melody pitch sequence and the probability that the pitch at each set time point is human voice specifically includes: Determine the pitch of the main melody at a set time point at which the probability of being a human voice is greater than a first preset threshold as the pitch of the main melody of the human voice; The vocal melody pitch sequence is obtained according to the vocal main melody pitch.

5. The method according to claim 2, characterized in that Determining the start time of each word in the lyrics according to the probability that each set time point is the start time of the word in the lyrics specifically includes: Obtaining a corresponding probability curve according to the probability that each set time point is the start time of a word in the lyrics; The set time points corresponding to the peak values ​​in the probability curve are determined as the start times of the words in the lyrics.

6. The method according to claim 1, characterized in that Matching the candidate start time of each word in the lyrics with the reference lyrics file to obtain the target start time of each word in the lyrics, specifically including: Matching the closest candidate start time among the candidate start times of each word in the lyrics according to the start time of each word in the reference lyrics file within a second preset time error range; The closest matched candidate start times are respectively determined as the target start time of each word in the lyrics.

7. The method according to claim 1, characterized in that Also includes: Identify the time point corresponding to each beat point in the song audio according to a preset beat detection algorithm; The start time of each pitch in the pitch sequence of the main melody of the human voice and the target start time of each word in the lyrics are matched to the time point corresponding to each beat point.

8. A melody and lyrics alignment device, characterized in that: include: A melody extraction unit, used to input the song audio into a melody extraction multi-task model, to obtain a main melody pitch sequence, a probability that each pitch in the main melody pitch sequence is a human voice, and a start time of each word in the lyrics, wherein the melody extraction multi-task model is obtained by training the main melody pitch of each sample song audio in the sample song audio set, the main melody pitch of the human voice, and the start time of the word in the sample lyrics according to a preset multi-task neural network model; An obtaining unit, configured to obtain a main melody pitch sequence of human voice according to the main melody pitch sequence and the probability that each pitch in the main melody pitch sequence is human voice; A determination unit, for determining a target start time of each word in the lyrics according to the start time of each pitch in the main melody pitch sequence of the human voice, the start time of each word in the lyrics and a reference lyrics file; The determination unit is specifically used to: match the start time of each pitch in the main melody pitch sequence of the human voice with the start time of each word in the lyrics to determine the candidate start time of each word in the lyrics; match the candidate start time of each word in the lyrics with the reference lyrics file to obtain the target start time of each word in the lyrics; The determination unit is specifically used to: match the start time of each pitch in the pitch sequence of the main melody of the human voice with the start time of each word in the lyrics according to a first preset time error range; if the start time of any pitch matches the start time of any word in the lyrics within the first preset time error range, then determine the start time of any pitch as a pitch switching point and a lyrics switching point, and determine the start time of the pitch as a candidate start time of the matched any word; If within the first preset time error range, the start time of any pitch does not match the start time of any word in the lyrics, the start time of any pitch is determined to be a pitch switching point under one-word multi-tone; if within the first preset time error range, the start time of any word in the lyrics does not match the start time of any pitch, the start time of any word is determined to be a lyrics switching point under one-word multi-tone, and the start time of any word is determined as a candidate start time of any word; The processing unit is used to align the melody and lyrics according to the start time of each pitch in the main melody pitch sequence of the human voice and the target start time of each word in the lyrics to obtain an alignment result.

9. The device according to claim 8, characterized in that The melody extraction unit is specifically used for: Input the song audio into the melody extraction multi-task model to obtain the probability of each pitch at each set time point, the probability that the pitch at each set time point is a human voice, and the probability that the pitch at each set time point is the start time of a word in the lyrics; Obtaining a main melody pitch sequence according to the pitch probabilities at the set time points; Determining the start time of each word in the lyrics according to the probability that each set time point is the start time of the word in the lyrics; and The obtaining unit is specifically used to obtain a main melody pitch sequence of human voice according to the main melody pitch sequence and the probability that the pitch at each set time point is human voice.

10. The device according to claim 9, characterized in that The melody extraction unit is specifically used for: For each currently set time point, respectively calculate the interval between each pitch and the target pitch at the last set time point; Determine the weight corresponding to each calculated interval according to the preset correspondence between the interval and the weight; Multiplying each pitch probability at the current set time point by the weight corresponding to each corresponding interval to obtain each candidate pitch probability at the current set time point; Determine the pitch corresponding to the maximum candidate pitch probability as the target pitch at the current set time point; The target pitch at each set time point is determined as the corresponding main melody pitch to obtain the main melody pitch sequence.

11. The device according to claim 9, characterized in that The obtaining unit is specifically used for: Determine the pitch of the main melody at a set time point at which the probability of being a human voice is greater than a first preset threshold as the pitch of the main melody of the human voice; The vocal melody pitch sequence is obtained according to the vocal main melody pitch.

12. The device according to claim 9, characterized in that The melody extraction unit is specifically used for: Obtaining a corresponding probability curve according to the probability that each set time point is the start time of a word in the lyrics; The set time points corresponding to the peak values ​​in the probability curve are determined as the start times of the words in the lyrics.

13. The device according to claim 8, characterized in that The determining unit is specifically configured to: Matching the closest candidate start time among the candidate start times of each word in the lyrics according to the start time of each word in the reference lyrics file within a second preset time error range; The closest matched candidate start times are respectively determined as the target start time of each word in the lyrics.

14. The device according to claim 8, characterized in that Also includes: An identification unit, used to identify a time point corresponding to each beat point in the song audio according to a preset beat detection algorithm; The beat quantization unit is used to match the start time of each pitch in the pitch sequence of the main melody of the human voice and the target start time of each word in the lyrics to the time point corresponding to each beat point.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the melody and lyrics alignment method according to any one of claims 1 to 7 are implemented.

16. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the steps of the melody and lyrics alignment method described in any one of claims 1 to 7 are implemented.

Images