Method for synchronizing music score and audio and computer-readable storage medium

Abstract

The present invention discloses a method for synchronizing music score with audio and a computer-readable storage medium. The method comprises: extracting a main melody track of the audio to be synchronized; extracting the pitch based on the main melody track to obtain a note sequence and the start and end times of each group of main melody notes in the note sequence, wherein the start and end times of the same group of main melody notes are consistent; comparing the note sequence with the music score corresponding to the audio to be synchronized, determining the correspondence between each group of main melody notes in the note sequence and each group of music score notes in the music score, wherein the same group of music score notes are music score notes that need to be played at the same time; and when playing the audio to be synchronized, determining the music score notes corresponding to the current playback time based on the start and end times and the correspondence between each group of main melody notes, and highlighting the corresponding music score notes or the measure in which they are located in the music score. The present invention can achieve music score and audio synchronization.

Description

Technical Field

[0001] The present invention relates to the field of audio processing technology, and in particular to a method for synchronizing music scores and audio and a computer-readable storage medium. Background Art

[0002] During song playback, the music score is played synchronously with the audio to allow users to intuitively see which note the audio is currently playing. That is, as the audio plays, the score moves in sync with the music score. However, due to various factors, the audio resources that accompany the music score may have different overall offsets (first note start time), local speeds, and segment repetitions. This can cause the music score and audio to be out of sync and out of alignment when played simultaneously. Therefore, synchronization of the music score and audio is necessary.

[0003] In addition, in some practical applications, the music score does not have repeat signs, or the marked repeat signs are incorrect, which results in the inability to synchronize the singing audio and the music score.

[0004] Chinese patent publication CN110570834A discloses a method, apparatus, and device for synchronously displaying music scores during song playback. The method includes parsing a song file to extract lyrics, time stamps corresponding to the lyrics, and the sound frequencies of the music scores corresponding to the lyrics. During song playback, the current time of the song is obtained, the lyrics corresponding to the current time and the music scores associated with the sound frequencies of the music scores corresponding to the lyrics are determined as display content, and the determined lyrics and the music scores associated with the sound frequencies of the music scores corresponding to the lyrics are synchronously displayed.

[0005] In this solution, the song file already contains the lyrics and the time stamp corresponding to the lyrics. Then the time of playing the song is obtained, combined with the time stamp of the lyrics (as the standard time), the lyrics in the song file and the corresponding music score are displayed synchronously.

[0006] In other words, the aforementioned patent does not address how to obtain the timing information of lyrics in song files, but rather utilizes existing lyrics timing information. In the current prior art, the timing information of lyrics in song files is generally generated based on the music score. As the previous analysis shows, when the performance audio is out of sync with the music score, the timing information of the lyrics generated from the music score is incorrect, requiring the lyrics or note timing information to be regenerated based on the performance audio.

[0007] Furthermore, the aforementioned patent primarily involves converting the sound frequencies of a musical score into musical notes and outputting them in the form of musical notes. This means that the musical score in this solution only represents pitch, without repeat marks that affect the order in which measures are played. Therefore, this solution does not address how to synchronize musical score with repeat marks. Furthermore, this solution cannot achieve step-by-step playback of the score. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide a method for synchronizing music score and audio and a computer-readable storage medium, which can realize music score and audio synchronization.

[0009] In order to solve the above technical problems, the present invention adopts a technical solution: a method for synchronizing music score and audio, comprising:

[0010] Acquire the audio to be synchronized, and extract the main melody track of the audio to be synchronized;

[0011] Extracting pitches from the main melody track to obtain a note sequence and the start and end times of each group of main melody notes in the note sequence, wherein the start and end times of the same group of main melody notes are consistent;

[0012] Comparing the note sequence with the music score corresponding to the audio to be synchronized, determining the correspondence between each group of main melody notes in the note sequence and each group of music score notes in the music score, wherein the same group of music score notes are music score notes that need to be played at the same time;

[0013] When playing the audio to be synchronized, the music score notes corresponding to the current playback time are determined based on the start and end times of each group of main melody notes and the corresponding relationship, and the corresponding music score notes or the measure where the corresponding music score notes are located are highlighted in the music score.

[0014] The present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned method when the program is executed by a processor.

[0015] The present invention has the following beneficial effects: by identifying the order in which the notes in the main melody track are played and controlling the order in which the score is played, it is possible to achieve synchronization between the score and the audio, without being affected by repeat marks in the score; by highlighting the score note or measure corresponding to the current playback time, it is possible to achieve accurate tracking of the score at the note level, as well as accurate tracking of the score on a measure-by-measure basis, thus meeting diverse needs. The present invention can achieve synchronization between the score and the audio, allowing users to more accurately check the score when playing audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flowchart of a method for synchronizing music score and audio according to the first embodiment of the present invention;

[0017] Figure 2 This is a flowchart of step S3 of the first embodiment of the present invention. DETAILED DESCRIPTION

[0018] To illustrate the technical content, achieved objectives and effects of the present invention in detail, the following is a detailed description in conjunction with the embodiments and accompanying drawings.

[0019] See also Figure 1 , a method for synchronizing music score and audio, comprising:

[0020] Acquire the audio to be synchronized, and extract the main melody track of the audio to be synchronized;

[0021] Extracting pitches from the main melody track to obtain a note sequence and the start and end times of each group of main melody notes in the note sequence, wherein the start and end times of the same group of main melody notes are consistent;

[0022] Comparing the note sequence with the music score corresponding to the audio to be synchronized, determining the correspondence between each group of main melody notes in the note sequence and each group of music score notes in the music score, wherein the same group of music score notes are music score notes that need to be played at the same time;

[0023] When playing the audio to be synchronized, the music score notes corresponding to the current playback time are determined based on the start and end times of each group of main melody notes and the corresponding relationship, and the corresponding music score notes or the measure where the corresponding music score notes are located are highlighted in the music score.

[0024] From the above description, it can be seen that the beneficial effects of the present invention are: it can achieve score-sound synchronization mainly with audio, without being affected by repeat marks in the score; it can achieve accurate score tracking at the note level, and it can also achieve accurate score tracking measure by measure.

[0025] Furthermore, the pitch is extracted according to the main melody track to obtain a note sequence and the start and end time of each group of main melody notes in the note sequence are specifically:

[0026] Extracting a fundamental frequency sequence from the main melody track;

[0027] Performing pitch fitting on the fundamental frequency sequence to obtain a note sequence, and obtaining the start and end time of each main melody note in the note sequence;

[0028] Divide the main melody notes with the same start and end time into the same group.

[0029] It can be seen from the above description that by classifying main melody notes with consistent start and end times into the same group, the present invention is also applicable to the synchronization of musical instruments and multi-part music scores.

[0030] Furthermore, the note sequence is compared with the music score corresponding to the audio to be synchronized, and the corresponding relationship between each group of main melody notes in the note sequence and each group of music score notes in the music score is determined as follows:

[0031] Obtaining the i-th group of main melody notes from the note sequence as the current main melody notes, where the initial value of i is 1;

[0032] Determine whether the current main melody note is the same as the j-th group of music score notes in the music score corresponding to the audio to be synchronized, where the initial value of j is 1;

[0033] If they are the same, the current main melody note is associated with the jth group of music score notes, and i is increased by 1, j is increased by 1, and the step of obtaining the ith group of main melody notes from the note sequence as the current main melody note is continued;

[0034] If they are different, the best matching note is obtained from the music score through a matching algorithm, and the current main melody note is associated with the best matching note.

[0035] From the above description, it can be seen that since some main melody notes and some music score notes have a certain correspondence in sequence, the correspondence can be quickly determined by synchronously increasing the sequence numbers of the main melody notes and the music score notes and comparing them.

[0036] Furthermore, the method of obtaining the best matching note from the music score through the matching algorithm and associating the current main melody note with the best matching note is specifically as follows:

[0037] Get the current main melody note and the K consecutive main melody notes thereafter to obtain the note string to be matched, where the initial value of K is a preset value;

[0038] searching for a best matching note string in the music score according to the note string to be matched, wherein the best matching note string is the same as the note string to be matched;

[0039] If the best matching note string is not found, then add 1 to i or add K+1 to i, and continue to execute the step of obtaining the i-th group of main melody notes from the note sequence as the current main melody notes;

[0040] If the number of the best matching note strings is 1, then each main melody note in the note string to be matched is associated with each group of music score notes in the best matching note string in order, and K+1 is added to i, and j=A+1 is set, where A is the serial number of the last group of music score notes in the best matching note string in the music score, and the step of obtaining the i-th group of main melody notes from the note sequence as the current main melody notes is continued.

[0041] From the above description, it can be seen that if the current main melody notes are different from the music score notes, it means that the audio is not playing the next measure of the music score, but has jumped to other measures. Therefore, the best match of the note string is found in the music score to obtain the corresponding music score notes from the music score.

[0042] Furthermore, after searching for the best matching note string in the music score according to the note string to be matched, the method further includes:

[0043] If the number of the best matching note strings is greater than 1, determining whether K reaches a preset threshold;

[0044] If yes, then add 1 to i or add K+1 to i, and continue to execute the step of obtaining the i-th group of main melody notes from the note sequence as the current main melody notes;

[0045] If not, then K is incremented by one, and the step of obtaining the current main melody note and the subsequent K groups of main melody notes is continued to obtain the note string to be matched.

[0046] As can be seen from the above description, if multiple best matching note strings are matched, the note strings to be matched are expanded and the search is performed again until a unique best matching note string is matched.

[0047] Furthermore, the main melody track is a vocal track;

[0048] After extracting the pitch according to the main melody track to obtain a note sequence and the start and end time of each group of main melody notes in the note sequence, the method further includes:

[0049] Generate a lyrics sequence based on the main melody track using audio-to-text technology, and obtain the start and end time of each lyric in the lyrics sequence;

[0050] The corresponding relationship between the main melody notes and the lyrics is determined according to the start and end time of each group of main melody notes in the note sequence and the start and end time of each lyric in the lyric sequence.

[0051] Furthermore, after searching for the best matching note string in the music score according to the note string to be matched, the method further includes:

[0052] If the number of the best matching note strings is greater than 1, obtaining the lyrics string to be matched corresponding to the note string to be matched according to the correspondence between the main melody notes and the lyrics;

[0053] According to the music score, respectively obtain the lyrics string corresponding to each best matching note string;

[0054] Calculating the matching degree between the lyrics string to be matched and the lyrics string corresponding to each best matching note string, and obtaining the best matching note string corresponding to the lyrics string with the highest matching degree as the only best matching note string;

[0055] Associate each group of main melody notes in the to-be-matched note string with each music score note in the unique best-matched note string in order, add K+1 to i, let j=B+1, B is the serial number of the last group of music score notes in the unique best-matched note string in the music score, and continue to execute the step of obtaining the i-th main melody note from the note sequence as the current main melody note.

[0056] As can be seen from the above description, when multiple best matching note strings are matched, the matching degree of the lyrics can be further calculated, and the only best matching note string can be determined through note matching and lyrics matching.

[0057] Furthermore, when playing the audio to be synchronized, the music score note corresponding to the current playback time is determined according to the start and end time of each group of main melody notes and the corresponding relationship, and the corresponding music score note or the section where the corresponding music score note is located is highlighted in the music score. Specifically:

[0058] Play the audio to be synchronized, obtain the playback progress in real time, and obtain the current playback time;

[0059] Determining the main melody note corresponding to the current playback time according to the current playback time and the start and end times of each group of main melody notes in the note sequence;

[0060] According to the corresponding relationship, the music score note associated with the main melody note corresponding to the current playback time is obtained to obtain the music score note corresponding to the current playback time;

[0061] Display the music score and highlight the music note or measure corresponding to the current playback time.

[0062] From the above description, it can be seen that by highlighting the music note corresponding to the current playback time or the measure in which it is located, accurate music playing at the note level can be achieved, and accurate music playing measure by measure can also be achieved to meet diverse needs.

[0063] Furthermore, the audio to be synchronized is an audio of a musical instrument performance;

[0064] The method of extracting the main melody track of the audio to be synchronized; extracting the pitch according to the main melody track, and obtaining a note sequence and the start and end time of each group of main melody notes in the note sequence is specifically as follows:

[0065] Extracting the solo track of the instrument performance audio as the main melody track;

[0066] Generate a MIDI file corresponding to the main melody track through music transcription technology;

[0067] According to the MIDI file, a note sequence and the start and end time of each group of main melody notes in the note sequence are obtained.

[0068] From the above description, it can be seen that by using the recorded audio of a master or demonstration mechanical piano as a piano performance demonstration and playing it synchronously, the shortcomings of MIDI demonstration such as poor sound quality and strong mechanical feel can be effectively improved.

[0069] The present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned method when the program is executed by a processor.

[0070] Example 1

[0071] Please refer to Figure 1-2 , the embodiment 1 of the present invention is: a method for synchronizing music score and audio, which can realize music score synchronization during the playing process of music or song, such as Figure 1 As shown, the following steps are included:

[0072] S1: Acquire the audio to be synchronized, and extract the main melody track of the audio to be synchronized; wherein the main melody track refers to the track that is consistent with the notes in the music score.

[0073] In this embodiment, if the audio to be synchronized is a song audio, the vocal track can be separated from the original singing audio by spleeter as the main melody track.

[0074] S2: Extracting pitches according to the main melody track to obtain a note sequence and the start and end times of each main melody note in the note sequence.

[0075] Specifically, a fundamental frequency sequence is extracted from the main melody track, and then the fundamental frequency sequence is fitted with similar pitches to obtain a note sequence and the start and end time of each main melody note; then the main melody notes with the same start and end time in the note sequence are divided into a group. At this time, the note sequence includes multiple groups of main melody notes, each group of main melody notes contains at least one main melody note, and the start and end times of the main melody notes in the same group are the same.

[0076] For audio containing only a single part, each group of main melody notes contains only one main melody note; while for some instrument audio or audio containing multiple parts, a group of main melody notes may contain two or even more main melody notes.

[0077] Furthermore, in other optional embodiments, the audio to be synchronized is an instrument performance audio, and the instrument solo track of the instrument performance audio is extracted as the main melody track; then, through music transcription technology, such as CNN (Convolutional Neural Networks) and LSTM (Long Short-Term Memory), a MIDI file corresponding to the main melody track is generated, and the MIDI file is used as a note sequence, and the start and end time of each main melody note is calculated from the MIDI file.

[0078] By using the audio recordings of mechanical pianos played by masters or demonstrations as piano performance demonstrations and playing them synchronously, the shortcomings of MIDI demonstrations such as poor sound quality and a strong mechanical feel can be effectively improved.

[0079] S3: Compare the note sequence with the music score corresponding to the audio to be synchronized, and determine the correspondence between each group of main melody notes in the note sequence and each group of music score notes in the music score.

[0080] A group of musical notes refers to notes that need to be played at the same time. For a single-part score, a group of musical notes contains only one note. However, for scores played on some instruments or with multiple parts, a group of musical notes may contain two or even more notes. For example, piano music typically has a double staff, a treble clef and a bass clef. The treble and bass notes that need to be played at the same time are grouped together.

[0081] Specifically, if Figure 2 As shown, this step includes the following steps:

[0082] S301: Obtain the i-th group of main melody notes from the note sequence as the current main melody notes, where the initial value of i is 1.

[0083] S302: Determine whether the current main melody notes are the same as the j-th group of music notes in the music score corresponding to the audio to be synchronized, where the initial value of j is 1. That is, determine whether the i-th group of main melody notes are the same as the j-th group of music notes. If so, execute step S303; if not, execute step S305.

[0084] S303: Associating the current main melody notes with the j-th group of music score notes, that is, associating the i-th group of main melody notes with the j-th group of music score notes.

[0085] S304: Let i=i+1, j=j+1, and then continue to execute step S301 until the note sequence is traversed.

[0086] S305: Obtain the current main melody note and the K consecutive main melody notes thereafter to obtain a string of notes to be matched, where the initial value of K is a preset value.

[0087] That is, the i-th group of main melody notes to the i+K-th group of main melody notes are obtained, and there are K+1 groups of main melody notes in the note string to be matched. Preferably, the initial value of K is 5.

[0088] S306: searching for a best matching note string in the music score according to the note string to be matched, where the best matching note string is the same as the note string to be matched.

[0089] That is, starting from the first group of music notes in the music score, note string matching is performed to find the music score note string that is consistent with the note string to be matched, which is used as the best matching note string; each group of music score notes in the best matching note string is respectively the same as each group of main melody notes in the note string to be matched.

[0090] S307: Determine whether the best matching note string is matched, if so, execute step S309, if not, execute step S308.

[0091] S308: Set i = i + 1, and then continue with step S301 until the note sequence is traversed. If the best matching note string is not found, it means that the current main melody note may have been calculated incorrectly, resulting in no corresponding music score note being found in the music score. In this case, the current main melody note can be skipped and the note sequence traversal can continue.

[0092] In other optional embodiments, i may be set to i+K+1, and then step S301 is continued until the note sequence is traversed; that is, the current main melody note and the subsequent K consecutive groups of main melody notes are skipped, and the note sequence is continued to be traversed.

[0093] S309: Determine whether the number of the best matching note strings is 1. If so, execute step S310. If not, that is, the number of the best matching note strings is greater than or equal to 2, execute step S312.

[0094] S310: Associating each group of main melody notes in the to-be-matched note string with each group of music score notes in the best matching note string in a one-to-one correspondence order.

[0095] For example, assuming that the best matching note string is the Lth group of music notes to the L+Kth group of music notes in the music score, then the i-th group of main melody notes is associated with the L-th group of music notes, the i+1-th group of main melody notes is associated with the L+1-th group of music notes, the i+2-th group of main melody notes is associated with the L+2-th group of music notes, and so on, until the i+K-th group of main melody notes is associated with the L+K-th group of music notes.

[0096] S311: Let i=i+K+1, j=A+1, where A is the serial number of the last group of music notes in the best matching note string, that is, j=L+K+1, and then continue to execute step S301 until the note sequence is traversed.

[0097] S312: Determine whether K reaches a preset threshold. If so, execute step S308; if not, execute step S313.

[0098] S313: Let K=K+1, and then continue to execute step S305; that is, expand the note string to be matched, and search again until the number of the best matching note string is 1.

[0099] S4: When playing the audio to be synchronized, the music score notes corresponding to the current playing time are determined according to the start and end times of each group of main melody notes and the corresponding relationship, and the corresponding music score notes are highlighted in the music score.

[0100] Specifically, this step includes the following steps:

[0101] S401: Play the audio to be synchronized, and obtain the playback progress in real time to obtain the current playback time; that is, obtain the audio playback progress in real time.

[0102] S402: Determine the main melody note corresponding to the current playing time according to the current playing time and the start and end time of each group of main melody notes in the note sequence.

[0103] That is, if the current playing time is between the start and end time of a group of main melody notes, the group of main melody notes will be used as the main melody notes corresponding to the current playing time, that is, the main melody notes currently being played.

[0104] S403: According to the corresponding relationship, the music score notes associated with the main melody notes corresponding to the current playing time are obtained to obtain the music score notes corresponding to the current playing time.

[0105] That is, according to the correspondence between the main melody note and the music score note, the music score note corresponding to the main melody note currently being played is obtained as the music score note corresponding to the current playing time.

[0106] S404: Display the music score and highlight the music score notes corresponding to the current playing time.

[0107] In this embodiment, only the music score notes corresponding to the current playback time are highlighted. In other optional embodiments, all music score notes in the measure where the music score notes corresponding to the current playback time are located can also be highlighted, that is, the highlighting of a measure will start from the time the first group of music score notes in the measure is played until the last group of music score notes in the measure is played.

[0108] This embodiment identifies the order in which notes in the main melody track are played and controls the order in which the score is highlighted. This can achieve score-sound synchronization based on audio, without being affected by repeat marks in the score, so that users can more accurately compare and check the score when listening to the audio; by highlighting the score note corresponding to the current playback time or the measure in which it is located, accurate score tracking at the note level can be achieved, as can accurate score tracking measure by measure, to meet diverse needs.

[0109] Example 2

[0110] This embodiment is another implementation of step S313 in step S3 in embodiment 1, that is, another implementation when more than two best matching note strings are found.

[0111] In this embodiment, the audio to be synchronized is song audio, and the main melody track is the vocal track.

[0112] After step S2, it further includes: generating a lyrics sequence based on the main melody audio track through audio-to-text technology, and obtaining the start and end time of each lyrics in the lyrics sequence; then determining the correspondence between the main melody notes and the lyrics based on the start and end time of each group of main melody notes in the note sequence and the start and end time of each lyrics in the lyrics sequence.

[0113] That is to say, in addition to extracting the note sequence, you can also use audio-to-text technology, such as TDNN (time-delay neural network), to extract the lyrics sequence, and then mark the correspondence between the notes and lyrics in chronological order.

[0114] Step S313 of this embodiment includes the following steps:

[0115] S3131: According to the correspondence between the main melody notes and the lyrics, obtain the lyrics string to be matched corresponding to the note string to be matched; that is, according to the correspondence between the main melody notes and the lyrics, determine the lyrics corresponding to each group of main melody notes in the note string to be matched, and obtain the lyrics string to be matched.

[0116] S3132: According to the music score, obtain the lyrics string corresponding to each best matching note string.

[0117] Generally, in a music score, there are corresponding lyrics under each note. This step is to obtain the lyrics under each best matching note string in the music score and obtain the corresponding lyrics string.

[0118] S3133: Calculate the matching degree between the lyrics string to be matched and the lyrics string corresponding to each best matching note string respectively, and obtain the best matching note string corresponding to the lyrics string with the highest matching degree as the only best matching note string.

[0119] Specifically, each word in the lyrics string to be matched is extracted in sequence, and then the pronunciation of the word is obtained by searching word by word, or according to the preset association between the word and the pronunciation, to obtain the pronunciation string of the lyrics string to be matched; similarly, the pronunciation string of the lyrics string corresponding to each best matching note string is obtained; then the pronunciation string of the lyrics string to be matched is compared with the pronunciation string of the lyrics string corresponding to each best matching note string to determine the number of pronunciation matches. The higher the number of matches, the higher the degree of match; finally, the best matching note string corresponding to the lyrics string with the largest number of matches is used as the only best matching note string.

[0120] For example, assuming that the pronunciation string of the lyrics string to be matched is "wo shi zhong guo ren", the pronunciation string of the lyrics string corresponding to the best matching note string 1 is "wo shi fu jian ren", the pronunciation string of the lyrics string corresponding to the best matching note string 2 is "wo lai zi di qiu", and the pronunciation string of the lyrics string corresponding to the best matching note string 3 is "wo shi zhong guoren". Since the pronunciation string of the lyrics string corresponding to the best matching note string 3 has more identical pronunciations with the pronunciation string of the lyrics string to be matched, the best matching note string 3 is taken as the only best matching note string.

[0121] S3134: Associating each group of main melody notes in the to-be-matched note string with each group of music score notes in the unique best matching note string in a one-to-one correspondence order. This step may refer to step S310.

[0122] S3135: Let i=i+K+1, j=B+1, B is the serial number of the last group of music notes in the unique best matching note string in the music score, and continue to execute step S301 until the note sequence is traversed.

[0123] This embodiment adds lyrics recognition on the basis of the first embodiment. When multiple best matching note strings are matched, the lyrics matching degree is further calculated without increasing K for expanded search.

[0124] Example 3

[0125] This embodiment is a computer-readable storage medium corresponding to the above embodiment, on which a computer program is stored. When the computer program is executed by the processor, the various processes in the above-mentioned music score and audio synchronization method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0126] In summary, the present invention provides a method and computer-readable storage medium for synchronizing music score with audio. By identifying the note playback order in the main melody track and controlling the music score playback order, this method can achieve music score-audio synchronization primarily based on the audio, unaffected by repeat marks in the music score. By highlighting the music score note or the measure in which it corresponds to the current playback time, accurate music score tracking can be achieved at the note level, as well as on a measure-by-measure basis, meeting diverse needs. The present invention achieves music score-audio synchronization, allowing users to more accurately compare and view music scores while playing audio.

[0127] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's description and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for synchronizing music score and audio, characterized in that: include: Acquire the audio to be synchronized, and extract the main melody track of the audio to be synchronized; Extracting pitches from the main melody track to obtain a note sequence and the start and end times of each group of main melody notes in the note sequence, wherein the start and end times of the same group of main melody notes are consistent; Comparing the note sequence with the music score corresponding to the audio to be synchronized, determining the correspondence between each group of main melody notes in the note sequence and each group of music score notes in the music score, wherein the same group of music score notes are music score notes that need to be played at the same time; When playing the audio to be synchronized, determining the music score note corresponding to the current playback time according to the start and end times of each group of main melody notes and the corresponding relationship, and highlighting the corresponding music score note or the measure where the corresponding music score note is located in the music score; The step of comparing the note sequence with the music score corresponding to the audio to be synchronized and determining the correspondence between each group of main melody notes in the note sequence and each group of music score notes in the music score is as follows: Obtaining the i-th group of main melody notes from the note sequence as the current main melody notes, where the initial value of i is 1; Determine whether the current main melody note is the same as the j-th group of music score notes in the music score corresponding to the audio to be synchronized, where the initial value of j is 1; If they are the same, the current main melody note is associated with the jth group of music score notes, and i is increased by 1, j is increased by 1, and the step of obtaining the ith group of main melody notes from the note sequence as the current main melody note is continued; If they are different, the best matching note is obtained from the music score through a matching algorithm, and the current main melody note is associated with the best matching note; The method of obtaining the best matching note from the music score by using a matching algorithm and associating the current main melody note with the best matching note is as follows: Get the current main melody note and the K consecutive main melody notes thereafter to obtain the note string to be matched, where the initial value of K is a preset value; searching for a best matching note string in the music score according to the note string to be matched, wherein the best matching note string is the same as the note string to be matched; If the best matching note string is not found, then add 1 to i or add K+1 to i, and continue to execute the step of obtaining the i-th group of main melody notes from the note sequence as the current main melody notes; If the number of the best matching note strings is 1, then each main melody note in the note string to be matched is associated with each group of music score notes in the best matching note string in order, and K+1 is added to i, and j=A+1 is set, where A is the serial number of the last group of music score notes in the best matching note string in the music score, and the step of obtaining the i-th group of main melody notes from the note sequence as the current main melody notes is continued.

2. The method for synchronizing music score and audio according to claim 1, wherein: The pitch is extracted according to the main melody track to obtain a note sequence and the start and end time of each group of main melody notes in the note sequence are specifically: Extracting a fundamental frequency sequence from the main melody track; Performing pitch fitting on the fundamental frequency sequence to obtain a note sequence, and obtaining the start and end time of each main melody note in the note sequence; Divide the main melody notes with the same start and end time into the same group.

3. The method for synchronizing music score and audio according to claim 1, wherein: After searching for the best matching note string in the music score according to the note string to be matched, the method further includes: If the number of the best matching note strings is greater than 1, determining whether K reaches a preset threshold; If yes, then add 1 to i or add K+1 to i, and continue to execute the step of obtaining the i-th group of main melody notes from the note sequence as the current main melody notes; If not, then K is incremented by one, and the step of obtaining the current main melody note and the subsequent K groups of main melody notes is continued to obtain the note string to be matched.

4. The method for synchronizing music score and audio according to claim 1, wherein: The main melody track is a vocal track; After extracting the pitch according to the main melody track to obtain a note sequence and the start and end time of each group of main melody notes in the note sequence, the method further includes: Generate a lyrics sequence based on the main melody track using audio-to-text technology, and obtain the start and end time of each lyric in the lyrics sequence; The corresponding relationship between the main melody notes and the lyrics is determined according to the start and end time of each group of main melody notes in the note sequence and the start and end time of each lyric in the lyric sequence.

5. The method for synchronizing music score and audio according to claim 4, characterized in that: After searching for the best matching note string in the music score according to the note string to be matched, the method further includes: If the number of the best matching note strings is greater than 1, obtaining the lyrics string to be matched corresponding to the note string to be matched according to the correspondence between the main melody notes and the lyrics; According to the music score, respectively obtain the lyrics string corresponding to each best matching note string; Calculating the matching degree between the lyrics string to be matched and the lyrics string corresponding to each best matching note string, and obtaining the best matching note string corresponding to the lyrics string with the highest matching degree as the only best matching note string; Associate each group of main melody notes in the note string to be matched with each music score note in the unique best matching note string in sequence, add K+1 to i, let j=B+1, B is the serial number of the last group of music score notes in the unique best matching note string in the music score, and continue to execute the step of obtaining the i-th main melody note from the note sequence as the current main melody note.

6. The method for synchronizing music score and audio according to claim 1, wherein: When playing the audio to be synchronized, the music score note corresponding to the current playback time is determined according to the start and end time of each group of main melody notes and the corresponding relationship, and the corresponding music score note or the section where the corresponding music score note is located is highlighted in the music score. Specifically: Play the audio to be synchronized, obtain the playback progress in real time, and obtain the current playback time; Determining the main melody note corresponding to the current playback time according to the current playback time and the start and end times of each group of main melody notes in the note sequence; According to the corresponding relationship, the music score note associated with the main melody note corresponding to the current playback time is obtained to obtain the music score note corresponding to the current playback time; Display the music score and highlight the music note or measure corresponding to the current playback time.

7. The method for synchronizing music score and audio according to claim 1, characterized in that: The audio to be synchronized is an audio of a musical instrument performance; The method of extracting the main melody track of the audio to be synchronized; extracting the pitch according to the main melody track, and obtaining a note sequence and the start and end time of each group of main melody notes in the note sequence is specifically as follows: Extracting the solo track of the instrument performance audio as the main melody track; Generate a MIDI file corresponding to the main melody track through music transcription technology; According to the MIDI file, a note sequence and the start and end time of each group of main melody notes in the note sequence are obtained.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images