A spectrum synchronization method and a terminal

Abstract

The present invention discloses a method for synchronizing spectrum and a terminal, which determines the first start and end time of each word in the singing lyrics of the audio to be played according to the audio to be played; obtains the sheet music file corresponding to the audio to be played, and determines the second start and end time of each word in the singing lyrics corresponding to the sheet music file according to the sheet music file and the first start and end time; determines the start and end time of each note in the sheet music file according to the second start and end time; aligns each word in the singing lyrics of the audio to be played with each word in the singing lyrics corresponding to the sheet music file, and then determines the start and end time of the corresponding note in the sheet music file based on the start and end time of each word in the aligned singing lyrics, which can realize the alignment of each sound of the audio to be played with each sound of its corresponding sheet music file, improve the synchronization of the sound played during playback with the sound of the sheet music, and is applicable to the playback of various songs.

Description

Technical Field

[0001] The present invention relates to the field of audio processing, and particularly to a method for synchronizing sheet music and audio and a terminal. Background Art

[0002] During the playback of a song, in order to enable users to intuitively see which note the currently played audio in the song has reached, the sheet music is played synchronously with the audio. That is, as the audio is played, the sheet music progresses synchronously. In most cases, the time of each beat in the audio is constant. Therefore, as long as the speed given in the sheet music is the same as the playback speed of the song in the audio, and the sheet music progresses at this given speed, it is possible to maintain the synchronization of the sheet music and the audio when played simultaneously. That is, when the audio currently plays to a certain note, the sheet music also reaches the position of that note in the sheet music.

[0003] However, there are also some special cases, that is, the time of each beat in the audio is not constant, or the time of each beat of some notes in the audio is inconsistent with that of other notes. For example, the speed of a song is one quarter note per beat, and there are 60 beats per minute, that is, the time of one beat is 1 second. However, in some parts or all of the song, the singer sings freely. The time of each beat sung by the singer in these freely sung parts fluctuates between 0.5 seconds and 2 seconds. This results in the time of each beat in some parts or all of the song not being 1 second. At this time, if the sheet music progresses at the speed given in the sheet music file, when played simultaneously with the audio, there will be a problem that the start and end times of some or all of the notes in the sheet music are inconsistent with the start and end times of the corresponding notes in the audio, that is, the sheet music and the audio are not synchronized. That is to say, the note currently played by the audio is not the same as the note currently reached by the sheet music. Summary of the Invention

[0004] The technical problem to be solved by the present invention is: to provide a method for synchronizing sheet music and audio and a terminal, which can improve the synchronization between the note played by the audio during playback and the note reached by the sheet music.

[0005] To solve the above technical problem, a technical solution adopted by the present invention is:

[0006] A method for synchronizing sheet music and audio, comprising the steps of:

[0007] S1. Determine the first start and end times of each word in the singing lyrics of the audio to be played according to the audio to be played;

[0008] S2. Obtain the sheet music file corresponding to the audio to be played, and determine the second start and end times of each word in the singing lyrics corresponding to the sheet music file according to the sheet music file and the first start and end times;

[0009] S3. Determine the start and end times of the corresponding notes in the sheet music file according to the second start and end times.

[0010] To solve the above technical problem, another technical solution adopted by the present invention is:

[0011] A score-audio synchronization terminal includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, each step in the above score-audio synchronization method is implemented.

[0012] The beneficial effects of the present invention are as follows: First, determine the first start and end times of each word in the audio to be played according to the audio to be played. Then, determine the second start and end times of each word in the singing lyrics corresponding to the sheet music file according to the sheet music file corresponding to the audio to be played and the first start and end times. Then, determine the start and end times of the corresponding notes in the sheet music file according to the second start and end times; Align each word in the singing lyrics of the audio to be played with each word in the singing lyrics corresponding to the sheet music file, and then determine the start and end times of the corresponding notes in the sheet music file based on the start and end times of each word in the aligned singing lyrics, which can achieve the alignment of each sound of the audio to be played with each sound of its corresponding sheet music file, improve the synchronization of the sound played during playback with the sound of the sheet music, and is applicable to the playback of various songs, such as variable-speed songs, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a flowchart of the steps of a score-audio synchronization method according to an embodiment of the present invention;

[0014] Figure 2 It is a schematic structural diagram of a score-audio synchronization terminal according to an embodiment of the present invention;

[0015] Figure 3 It is a schematic diagram of the sheet music of the song "Water Village Grandma's Bridge" cited in an embodiment of the present invention;

[0016] Figure 4 For an embodiment of the present invention Figure 3 A corresponding schematic diagram of the waveform and lyrics corresponding to the sheet music shown;

[0017] Figure 5 It is a schematic diagram of the penultimate measure of the sheet music of the song "Falling Leaves Fluttering" cited in an embodiment of the present invention;

[0018] Figure 6 For an embodiment of the present invention Figure 4 A corresponding schematic diagram of the waveform and lyrics corresponding to the sheet music shown;

[0019] Figure 7 For an embodiment of the present invention Figure 4 A corresponding schematic diagram of the waveform and notes corresponding to the sheet music shown;

[0020] Figure 8 For the embodiment of the present invention Figure 7 The schematic diagram after the adjustment of the note time length shown in the figure;

[0021] Figure 9 The schematic diagram of the prelude of the song "The Leaves Are Fluttering" cited in the embodiment of the present invention;

[0022] Figure 10 For the embodiment of the present invention Figure 9 The schematic diagram of the position of the prelude before adjustment shown in the figure;

[0023] Figure 11 For the embodiment of the present invention Figure 9 The schematic diagram of the position of the prelude after adjustment shown in the figure;

[0024] Figure 12 The schematic diagram of a musical score cited in the embodiment of the present invention;

[0025] Figure 13 The schematic diagram of another musical score cited in the embodiment of the present invention;

[0026] Figure 14 The schematic diagram of the musical score including a multi - voice musical score in the embodiment of the present invention;

[0027] Figure 15 The schematic diagram of the musical score including a multi - voice musical score and having a voice with independent lyrics except for the first voice in the embodiment of the present invention. Detailed implementation manners

[0028] To describe in detail the technical content, the achieved purpose and the effects of the present invention, the following is described in conjunction with the embodiments and with reference to the accompanying drawings.

[0029] Please refer to Figure 1 , A spectrum - sound synchronization method, comprising the steps of:

[0030] S1. Determine the first start - end time of each word in the singing lyrics of the audio to be played according to the audio to be played;

[0031] S2. Obtain the musical score file corresponding to the audio to be played, and determine the second start - end time of each word in the singing lyrics corresponding to the musical score file according to the musical score file and the first start - end time;

[0032] S3. Determine the start - end time of the corresponding notes in the musical score file according to the second start - end time.

[0033] As can be seen from the above description, the beneficial effects of the present invention are as follows: First, determine the first start and end times of each word in the audio to be played according to the audio to be played. Then, determine the second start and end times of each word in the singing lyrics corresponding to the musical score file according to the musical score file corresponding to the audio to be played and the first start and end times. Then, determine the start and end times of the corresponding notes in the musical score file according to the second start and end times; Align each word in the singing lyrics of the audio to be played with each word in the singing lyrics corresponding to the musical score file, and then determine the start and end times of the corresponding notes in the musical score file based on the start and end times of each word in the aligned singing lyrics, which can achieve the alignment of each sound of the audio to be played with each sound of its corresponding musical score file, improve the synchronization of the sound played during playback with the sound of the musical score, and is applicable to the playback of various songs, such as variable-speed songs, etc.

[0034] Further, the S1 further includes:

[0035] Determine the third start and end times of each note in the pure note part of the audio to be played according to the audio to be played;

[0036] The S3 further includes:

[0037] Determine the fourth start and end times of each note in the pure note part corresponding to the musical score file according to the musical score file and the third start and end times.

[0038] As can be seen from the above description, since the audio to be played includes both a lyrics part and a pure note part without lyrics but with notes, therefore, for the pure note part, first determine the third start and end times of each note it contains according to the audio to be played, and then determine the fourth start and end times of each note in the pure note part according to the musical score file and the third start and end times, further ensuring the synchronization of the sound played during playback with the sound of the musical score.

[0039] Further, the determining the second start and end times of each word in the singing lyrics corresponding to the musical score file according to the musical score file and the first start and end times includes:

[0040] Expand the singing lyrics corresponding to the musical score file in chronological order according to the preset speed in the musical score file;

[0041] Adjust the singing lyrics corresponding to the musical score file expanded in chronological order according to the first start and end times of each word in the singing lyrics of the audio to be played to obtain the second start and end times of each word in the singing lyrics corresponding to the musical score file.

[0042] As can be seen from the above description, first, the singing lyrics corresponding to the music score file are expanded in chronological order according to the preset speed, and then the expanded singing lyrics are adjusted by the first start and end time, so as to achieve the alignment of the expanded singing lyrics with the first start and end time of each word, ensuring the synchronization of the sound played during playback with the sound reached by the music score.

[0043] Further, before determining the start and end time of each note in the music score file in step S3, it further includes:

[0044] Calibrating the target start and end time according to the audio to be played, where the target start and end time includes at least one of the second start and end time, the fourth start and end time, and the start and end time of the corresponding note determined by the second start and end time.

[0045] As can be seen from the above description, before determining the start and end time of each note in the music score file, calibrating at least one of the second start and end time, the fourth start and end time, and the start and end time of the corresponding note determined by the second start and end time according to the audio to be played further ensures the synchronization of the sound played during playback with the sound reached by the music score. During the calibration process, since the second start and end time affects the start and end time of the corresponding note determined by the second start and end time, therefore, the calibration of the second start and end time can be carried out first, and then the calibration of the start and end time of the corresponding note determined by the second start and end time can be carried out.

[0046] Further, the calibrating the target start and end time according to the audio to be played includes:

[0047] Converting the audio to be played into a waveform in the time domain;

[0048] Determining the start point and end point of each target object of the audio to be played according to the waveform. When the target start and end time is the second start and end time, the target object is the word in the singing lyrics. When the target start and end time is the fourth start and end time, the target object is the note in the pure note part. When the target start and end time is the start and end time of the corresponding note determined by the second start and end time, the target object is the corresponding note determined by the second start and end time;

[0049] Calibrating the target start and end time corresponding to each target object according to the start point and end point of each target object.

[0050] As can be seen from the above description, by converting the audio to be played into a time-domain waveform, determining the start and end points of each note corresponding to each character in the sung lyrics (i.e., the corresponding notes determined by the second start and end times) or each note in the pure note part based on the waveform, and then correcting the second start and end times corresponding to each character or the third start and end times corresponding to each note, and the start and end times of the corresponding notes determined by the second start and end times based on the start and end points of each character or each note, the synchronization between the sound of the audio played during the playback and the sound of the music score is further ensured by means of waveform correction.

[0051] Further, the determining the start and end points of each target object of the audio to be played according to the waveform includes:

[0052] Traverse each time point in the waveform in chronological order. For the traversed time point:

[0053] Judge whether the amplitude of the waveform increases to more than a first preset ratio of the maximum amplitude of the waveform within a first preset time starting from the time point. If so, the time point is the start point of a target object;

[0054] If not, judge whether the amplitude of the waveform decreases to less than a second preset ratio of the maximum amplitude of the waveform within a second preset time starting from the time point. If so, the time point is the end point of a target object;

[0055] Otherwise, continue to traverse the next time point in the waveform until all time points in the waveform are traversed;

[0056] Determine the start and end points of each target object of the audio to be played according to the start and end points of all target objects determined after the traversal is completed.

[0057] As can be seen from the above description, by traversing each time point in the waveform, for the traversed time point, by comparing the magnitude relationship between the amplitude of the waveform within a preset time starting from the time point and the maximum amplitude of the waveform, the start and end points of each target object in the sung lyrics of the audio to be played can be accurately identified.

[0058] Further, the correcting the target start and end times according to the audio to be played includes:

[0059] Play the audio to be played, perform speech recognition or pitch recognition on the audio to be played, determine the start point and end point of each target object in the singing lyrics or the played music of the audio to be played. When the target start and end time is the second start and end time, the target object is a character in the singing lyrics; when the target start and end time is the fourth start and end time, the target object is a note in the pure note part of the played music; when the target start and end time is the start and end time of the corresponding note determined by the second start and end time, the target object is the corresponding note determined by the second start and end time.

[0060] Correct the target start and end time corresponding to each target object according to the start point and end point of each target object.

[0061] As can be seen from the above description, by performing automatic speech recognition or pitch recognition on the audio to be played, determining the start point and end point of each target object in the singing lyrics or the played music, and correcting the target start and end time based on the determined start point and end point of each target object, the correction of the target start and end time is quickly and conveniently achieved in the way of speech recognition or pitch recognition.

[0062] Further, the correction of the target start and end time according to the audio to be played includes:

[0063] Correspondingly display each target object of the audio to be played according to the target start and end time in the playing order. Adjacent target objects are distinguished by different color blocks. The start and end positions of the color blocks on the time axis represent the start and end time corresponding to each target object, and the length of the color blocks on the time axis represents the time length corresponding to each target object. When the target start and end time is the second start and end time, the target object is a character in the singing lyrics; when the target start and end time is the fourth start and end time, the target object is a note in the pure note part; when the target start and end time is the start and end time of the corresponding note determined by the second start and end time, the target object is the corresponding note determined by the second start and end time.

[0064] Play the audio to be played, and synchronously display a progress bar on the color block during the playing process.

[0065] Receive an adjustment instruction sent by the operating object according to the deviation between the played audio and the target object indicated by the progress bar.

[0066] Correct the target start and end time according to the adjustment instruction.

[0067] As described above, the time lengths corresponding to each target object in the played audio are displayed in the form of different color blocks, and a progress bar is synchronously displayed. The operating object can dynamically adjust the color blocks corresponding to each target object in real time based on the audio it hears and the progress bar to correct the target start and end times, which is intuitive and convenient.

[0068] Further, the adjustment instruction includes the color block to be adjusted and the adjustment method;

[0069] The correcting the target start and end times according to the adjustment instruction includes:

[0070] Adjusting the start and end time boundaries of the color block to be adjusted in the time axis direction according to the color block to be adjusted and the adjustment method; or dragging the whole color block to be adjusted along the time axis direction;

[0071] Determining the target start and end times according to the start and end positions of the adjusted color block on the time axis.

[0072] As described above, when adjusting the color block, different adjustment methods can be set for corresponding adjustments, thereby realizing the correction of the target start and end times and achieving the convenience and flexibility of correcting the target start and end times.

[0073] Further, the determining the start and end times of the corresponding notes in the music score file according to the second start and end times includes:

[0074] Determining the start and end times of each note in the music score file according to the correspondence between the lyrics and the notes, the proportional relationship of the note durations, and the second start and end times.

[0075] As described above, when determining the start and end times of the notes, both the correspondence between the lyrics and the notes and the proportional relationship of the note durations are considered, which can ensure the accuracy of the determined start and end times of the notes.

[0076] Further, the determining the start and end times of each note in the music score file according to the correspondence between the lyrics and the notes, the proportional relationship of the note durations, and the second start and end times includes:

[0077] Performing the following steps for each word in the audio to be played:

[0078] Determining the number of notes corresponding to the word according to the correspondence between the lyrics and the notes;

[0079] If the number of notes corresponding to the word is one, the start and end times of the note corresponding to the word are the second start and end times corresponding to the word;

[0080] If the number of musical notes corresponding to the character is more than two, determine the proportional relationship of the note durations between the musical notes corresponding to the character;

[0081] According to the proportional relationship of the note durations and the second start and end time corresponding to the character, determine the start and end time corresponding to each musical note corresponding to the character.

[0082] As can be seen from the above description, by determining the corresponding relationship between each character of the singing lyrics in the audio to be played and the audio and the proportional relationship of the note durations between the musical notes corresponding to each character, the accuracy of the start and end time corresponding to each musical note determined is ensured.

[0083] Further, the S3 further includes:

[0084] Determine whether the score file contains multiple part scores;

[0085] If so, after determining the start and end time corresponding to each musical note of one of the first part scores, determine the start and end time corresponding to each musical note of the remaining part scores according to the corresponding relationship between the remaining part scores and the musical notes of the first part score.

[0086] As can be seen from the above description, for a multi-part score, after determining the start and end time corresponding to each musical note in one of the part scores, the start and end time corresponding to each musical note in the remaining part scores can be directly determined based on the determined part score, which is convenient and fast.

[0087] Further, the determining the start and end time corresponding to each musical note of the remaining part scores according to the corresponding relationship between the remaining part scores and the musical notes of the first part score includes:

[0088] Determine whether the relationship between the musical notes of the remaining part scores and the musical notes of the first part score is one-to-one corresponding and the note durations are the same within the same time period. If so, determine the start and end time corresponding to each musical note of the corresponding remaining part scores according to the start and end time corresponding to each musical note of the first part score within the same time period;

[0089] If not, determine the start and end time corresponding to each musical note of the remaining part scores according to the start and end time corresponding to each musical note of the first part score within the same time period, the note durations of the musical notes of the first part score, and the proportional relationship of the note durations of the musical notes of the remaining part scores.

[0090] As can be seen from the above description, when determining the start and end time corresponding to each musical note of the part score to be determined according to the relationship between the musical notes of different part scores, both the mutual relationship between the musical notes of different part scores and the proportional relationship of the note durations of the musical notes of the part score to be determined are considered, ensuring the accuracy of the start and end time corresponding to each musical note determined.

[0091] Further, determine whether the music score file contains multiple voice music scores, and whether there is a voice with independent lyrics except for the first voice;

[0092] If so, after determining the start and end times of each note of the eligible voice music score, according to the correspondence between each lyric of the eligible voice music score and the notes of the eligible voice music score within the same time period, and the proportional relationship between the start and end times and note durations of each note of the eligible voice music score, determine the start and end times of each lyric of the eligible voice music score.

[0093] As can be seen from the above description, when the music score file contains multiple voice music scores and there is a voice with independent lyrics except for the first voice, after determining the start and end times of each note of the eligible voice music score, the start and end times of each lyric of the eligible voice music score can be determined according to the correspondence between each lyric of the eligible voice music score and the notes of the eligible voice music score, and the proportional relationship between the start and end times and note durations of each note of the eligible voice music score, which can conveniently and quickly determine the start and end times of each lyric of each voice music score.

[0094] Such as Figure 2 shown, a score-sound synchronization terminal includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, each step in the above score-sound synchronization method is implemented.

[0095] Embodiment 1

[0096] Please refer to Figure 1 , a score-sound synchronization method, including the steps:

[0097] S1. Determine the first start and end times of each word in the singing lyrics of the to-be-played audio according to the to-be-played audio;

[0098] Specifically, separate the human voice in the to-be-played audio, and determine the first start and end times of each word in the singing lyrics of the to-be-played audio according to the separated human voice;

[0099] Among them, the D3NET algorithm or the Spleeter algorithm can be used to separate the human voice in the song audio; then, the GOP algorithm is used to determine the first start and end times of each word in the singing lyrics of the to-be-played audio from the separated human voice;

[0100] S2. Obtain the music score file corresponding to the to-be-played audio, and determine the second start and end times of each word in the singing lyrics corresponding to the music score file according to the music score file and the first start and end times;

[0101] Specifically, the singing lyrics corresponding to the music score file are expanded in chronological order according to the preset speed in the music score file;

[0102] Among them, if there are repeat signs in the music score file, the music score is expanded according to the repeat sign information;

[0103] Adjust the singing lyrics corresponding to the music score file expanded in chronological order according to the first start and end time of each word in the singing lyrics of the audio to be played, and obtain the second start and end time of each word in the singing lyrics corresponding to the music score file;

[0104] Among them, the adjustment is specifically: align the expanded lyrics with the first start and end time of each word in chronological order, so as to re-determine the start and end time of the expanded lyrics in the audio;

[0105] S3. Determine the start and end time of the corresponding notes in the music score file according to the second start and end time.

[0106] Embodiment 2

[0107] This embodiment further defines the determination of the start and end time of the notes in the pure note part of the audio to be played. Specifically:

[0108] The S1 further includes:

[0109] Determine the third start and end time of each note in the pure note part of the audio to be played according to the audio to be played;

[0110] Among them, the pure note part of the audio to be played includes the part of the audio to be played without singing lyrics and only playing notes, such as the prelude, the transition between the main song and the chorus, etc.;

[0111] The S3 further includes:

[0112] Determine the fourth start and end time of each note in the pure note part corresponding to the music score file according to the music score file and the third start and end time.

[0113] Embodiment 3

[0114] This embodiment further defines that before determining the start and end time of each note in the music score file in the S3, it further includes:

[0115] Calibrate the target start and end time according to the audio to be played, where the target start and end time includes at least one of the second start and end time, the fourth start and end time, and the start and end time of the corresponding notes determined by the second start and end time;

[0116] That is to say, the second start and end time can be corrected, or the fourth start and end time can be corrected, or the start and end time of the corresponding note determined by the second start and end time can be corrected. It is also possible to correct the second start and end time, the fourth start and end time, and the start and end time of the corresponding note determined by the second start and end time simultaneously;

[0117] In an alternative embodiment, when correcting the second start and end time, S3 includes:

[0118] Correct the second start and end time according to the audio to be played, and determine the start and end time of the corresponding note in the score file based on the corrected second start and end time;

[0119] In an alternative embodiment, when correcting the fourth start and end time, S3 includes:

[0120] Correct the fourth start and end time according to the audio to be played;

[0121] In an alternative embodiment, when correcting the start and end time of the corresponding note determined by the second start and end time, S3 includes:

[0122] Correct the second start and end time according to the audio to be played, and determine the start and end time of the corresponding note in the score file based on the corrected second start and end time; then correct the start and end time of the corresponding note determined by the corrected second start and end time.

[0123] Embodiment III

[0124] This embodiment specifically defines a method for correcting the target start and end time according to the audio to be played:

[0125] Convert the audio to be played into a waveform in the time domain;

[0126] Determine the start point and end point of each target object of the audio to be played according to the waveform. When the target start and end time is the second start and end time, the target object is the word in the sung lyrics. When the target start and end time is the fourth start and end time, the target object is the note in the pure note part; when the target start and end time is the start and end time of the corresponding note determined by the second start and end time, the target object is the corresponding note determined by the second start and end time;

[0127] Correct the target start and end time corresponding to each target object according to the start point and end point of each target object;

[0128] Among them, determining the start point and end point of each target object in the audio to be played according to the waveform includes:

[0129] Traverse each time point in the waveform in chronological order. For the traversed time point:

[0130] Determine whether the amplitude of the waveform increases to more than a first preset ratio of the maximum amplitude of the waveform within a first preset time starting from the time point. If so, the time point is the starting point of a target object;

[0131] If not, determine whether the amplitude of the waveform decreases to less than a second preset ratio of the maximum amplitude of the waveform within a second preset time starting from the time point. If so, the time point is the ending point of a target object;

[0132] Otherwise, continue to traverse the next time point in the waveform until all time points in the waveform are traversed;

[0133] Determine the starting point and ending point of each target object of the audio to be played based on the starting points and ending points of all target objects determined after the traversal is completed;

[0134] Wherein, the first preset time, the first preset ratio, the second preset time, and the second preset ratio can be flexibly and adaptively changed according to different songs;

[0135] Take Figure 3 the song "Water Village Grandma's Bridge" shown as an example. Figure 4 The top black waveform in it is the waveform in the time domain corresponding to the audio to be played. Its center is set to 0, and its amplitude fluctuates up and down with the change of sound. The louder the volume, the greater the amplitude of the up and down fluctuation. When a certain lyric starts to be sung, the fluctuation amplitude will suddenly increase to more than half of the maximum amplitude of the waveform within 0.05 seconds, and the start time of the lyric singing can be determined accordingly. When a certain lyric ends singing, the fluctuation amplitude will decrease to less than 20% of the maximum amplitude of the waveform as the singing volume decreases, and the end time of the lyric singing can be determined accordingly. The differently colored color blocks below the black waveform are the start and end times of each lyric corresponding to when the musical score is unfolded (the left and right edges of each color block represent the start and end times of each lyric, the length of the color block represents the length of a lyric singing, and the words in the color block represent the lyrics). Below the color block is the time scale (the minimum scale unit is 100 milliseconds). There is a vertical line from the black waveform to the time scale, which is the progress bar of the current playback progress. Below the time scale is the playback and playback progress control, that is, after determining the start and end times of each word in the sung lyrics according to the waveform diagram, the start and end positions of the color blocks corresponding to each word on the time axis can be directly adjusted dynamically according to the displayed content;

[0136] Among them, when the audio is played, the playback progress bar will also move to the waveform moment corresponding to the current audio playback moment, so that the relationship between the audio and the waveform can be observed more intuitively, thereby determining the corresponding relationship among the lyrics, the sung words in the audio, and the waveform.

[0137] Embodiment 4

[0138] This embodiment specifically defines another way to correct the target start and end times according to the to-be-played audio:

[0139] Play the to-be-played audio, perform speech recognition or pitch recognition on the to-be-played audio, and determine the start point and end point of each target object in the sung lyrics or played music of the to-be-played audio. When the target start and end time is the second start and end time, the target object is the word in the sung lyrics; when the target start and end time is the fourth start and end time, the target object is the note in the pure note part; when the target start and end time is the start and end time of the corresponding note determined by the second start and end time, the target object is the corresponding note determined by the second start and end time.

[0140] Correct the target start and end time corresponding to each target object according to the start point and end point of each target object.

[0141] This embodiment gives a way to automatically correct the target start and end times, that is, first identify whether the lyrics sung currently are consistent with the lyrics in the corresponding musical score or whether the notes played currently are consistent with the notes in the corresponding musical score through speech recognition or pitch recognition. If not, automatic adjustment is performed.

[0142] In another optional implementation manner, after the automatic adjustment, the waveform correction method of Embodiment 3 can be used for another correction.

[0143] Embodiment 5

[0144] This embodiment specifically defines another way to correct the target start and end times according to the to-be-played audio:

[0145] Correspondingly display each target object of the to-be-played audio in the playback order according to the target start and end times, and distinguish adjacent target objects with different color blocks. The start and end positions of the color blocks on the time axis represent the start and end times corresponding to each target object, and the length of the color blocks on the time axis represents the time length corresponding to each target object. When the target start and end time is the second start and end time, the target object is the word in the sung lyrics; when the target start and end time is the fourth start and end time, the target object is the note in the pure note part; when the target start and end time is the start and end time of the corresponding note determined by the second start and end time, the target object is the corresponding note determined by the second start and end time.

[0146] Play the audio to be played, and synchronously display a progress bar on the color block during the playing process;

[0147] Receive an adjustment instruction sent by an operation object based on the deviation between the played audio and the target object indicated by the progress bar;

[0148] Correct the target start and end times according to the adjustment instruction;

[0149] Wherein, the adjustment instruction includes the color block to be adjusted and the adjustment method;

[0150] The correcting the second start and end times according to the adjustment instruction includes:

[0151] Adjust the start and end time boundaries of the color block to be adjusted in the time axis direction according to the color block to be adjusted and the adjustment method; or drag the whole color block to be adjusted along the time axis direction;

[0152] Determine the target start and end times according to the start and end positions of the adjusted color block on the time axis;

[0153] Specifically, as Figure 4 shown, when the audio is played, the progress bar moves forward. The user can determine whether the current singing time of the lyrics being played is consistent with the lyrics of the corresponding color block in the progress bar by listening to the audio. If not, the start and end of the color block can be adjusted (the left and right boundaries of each color block can extend left and right, the color block can also be dragged left and right as a whole, or all color blocks can be offset as a whole or some of the color blocks can be offset) to ensure that the lyrics at the corresponding moment in the audio are consistent with the start and end times of the lyrics in the color block, and the lyrics played continuously before and after in the audio are consistent with the lyrics that appear continuously before and after in the color block. Through adjustment, ensure that the times of all the lyrics after expansion in the music score are consistent with the times of the corresponding lyrics in the audio, and save;

[0154] In an optional implementation manner, the manual adjustment process of this embodiment can be further executed on the basis of Embodiment 4:

[0155] For example, taking the penultimate measure of the song "The Leaves Are Flutterin'" as shown Figure 5 as an example, the start and end times corresponding to the lyrics "Look" and "Wind" are as Figure 6 shown, and the times of the start and end of each note after the operation of Embodiment 3 are as Figure 7 shown; at this time, if listening to the audio and finding that the time of the note 5 of "Look" is not that much, the left and right boundaries of the color block of note 5 can be dragged to align it with the audio, as Figure 8 shown;

[0156] Among them, when adjusting the start and end times of the notes corresponding to the prelude, the above-mentioned method based on the adjustment of the colored block lyrics can also be used. Since there are no lyrics in the prelude, when performing spectral audio synchronization, one can only listen to the audio, look at the waveform, and adjust the start and end positions of the colored blocks at the same time. For example, Figure 9 As shown in the schematic diagram of the prelude of the song "The Leaves Are Fluttering", Figure 10 is the position of the prelude before adjustment, Figure 11 is the position of the prelude after adjustment.

[0157] Example Six

[0158] This example specifically defines how to determine the start and end times of each note in the sheet music file based on the second start and end times:

[0159] Determine the start and end times of each note in the sheet music file according to the correspondence between the lyrics and the notes, the proportional relationship of the note durations, and the second start and end times of each corrected character;

[0160] Specifically, the following steps are performed for each word in the audio to be played:

[0161] Determine the number of notes corresponding to the word according to the correspondence between the lyrics and the notes;

[0162] If the number of notes corresponding to the word is one, the start and end times of the note corresponding to the word are the second start and end times corresponding to the word;

[0163] If the number of notes corresponding to the word is two or more, determine the proportional relationship of the note durations between the notes corresponding to the word;

[0164] Determine the start and end times corresponding to each note corresponding to the word according to the proportional relationship of the note durations and the second start and end times corresponding to the word;

[0165] For example Figure 3 In the third lyric "Yang", there is only one note 5, so the start and end times of note 5 are the same as the start and end times of the lyric "Yang";

[0166] For another example Figure 12 As shown in, the third lyric "I" corresponds to four notes, namely 5, 1, 3, 5. Since these four notes are all eighth notes, that is, the singing durations are the same, so these four notes divide the time of the lyric "I" equally, and the start and end times of these four notes can be obtained;

[0167] For another example Figure 13, the fourth lyric "song" corresponds to four musical notes, namely the dotted eighth note 5, the sixteenth note 6, the eighth note 7, and the eighth note 6. Since the durations of these four musical notes are different, the time lengths they occupy for this lyric are also different. The first note occupies 37.5% of the duration of this lyric, the second note occupies 12.5% of the duration of this lyric, and the third and fourth notes each occupy 25% of the duration of this lyric. Thus, the start and end times of these four musical notes can be obtained;

[0168] After that, when playing the audio and the musical score, the musical score progresses according to the time of the lyrics and musical notes determined in this embodiment, and synchronization with the audio playback can be achieved (that is, when the audio plays to a certain note and lyric, the musical score also reaches the position of that note and lyric at the same time), that is, the synchronization of the score and the sound.

[0169] The three methods for correcting the start and end times of the target described in Embodiments 3 to 6 above can be combined and applied as needed to further improve the synchronization effect.

[0170] Embodiment 7

[0171] This embodiment describes how to determine the start and end times of each musical note in the application scenario of a multi-voice musical score. Specifically:

[0172] The step S3 further includes:

[0173] Judging whether the musical score file contains multiple voice musical scores;

[0174] If so, after determining the start and end times of each musical note corresponding to one of the voice musical scores, determine the start and end times of each musical note corresponding to the remaining voice musical scores according to the corresponding relationship between the remaining voice musical scores and the musical notes of the one voice musical score;

[0175] Among them, determining the start and end times of each musical note corresponding to the remaining voice musical scores according to the corresponding relationship between the remaining voice musical scores and the musical notes of the first voice musical score includes:

[0176] Judging whether the relationship between the musical notes of the remaining voice musical scores and the musical notes of the first voice musical score is one-to-one and the note durations are the same within the same time period. If so, determine the start and end times of each musical note of the remaining voice musical scores according to the start and end times of each musical note of the first voice musical score within the same time period;

[0177] If not, determine the start and end times of each musical note of the remaining voice musical scores according to the start and end times and note durations of each musical note of the first voice musical score within the same time period and the proportional relationship of the note durations of each musical note of the remaining voice musical scores;

[0178] Such as Figure 14As shown, it is part of the musical score information of the song "Hunter's Chorus". It can be seen from the figure that this line consists of two voices. Assuming that the notes in the previous line have been pitch-synchronized and the notes in the next line have not been pitch-synchronized, when calculating the time of the audio in the next line, synchronize the time of the notes in the previous line to achieve the purpose of synchronously displaying the notes when the two voices are played;

[0179] The specific approach is as follows:

[0180] Extract the duration and start and end times of each note corresponding to the previous line of the corresponding measure in sequence; also analyze the duration of each note in the corresponding measure of the next line in sequence. Determine which notes in the previous line correspond to a certain note in the next line based on the duration (for different voices, the notes in the same time period may be one-to-one, many-to-one, or one-to-many), and assign the time of the corresponding note in the previous line to the corresponding note in the next line;

[0181] For example, in the first measure of the first voice in the figure, there are three notes 4, 3, 3, and the durations of the notes are quarter note, eighth note, and eighth note. In the corresponding measure of the second voice, there are also three notes 6, 1, 1, and the corresponding durations of the notes are quarter note, eighth note, and eighth note. Since the durations of the corresponding notes are the same, the start and end times of these three notes in the second voice are the same as those of the corresponding three notes in the first voice;

[0182] In the fourth measure of the first voice, there are two notes 0, 0, and the durations of the notes are quarter note, quarter note. In the fourth measure of the second voice, there are six notes 5, 5, 5, 5, 5, 5, and the durations of the notes are eighth note, sixteenth note, sixteenth note, eighth note, sixteenth note, sixteenth note. Then, for the first three notes in the fourth measure of the second voice, determine the start and end times of each note according to the duration ratio and based on the start and end time of the first note in the fourth measure of the first voice. The same treatment is done for the last three notes in the fourth measure of the second voice.

[0183] Embodiment 8

[0184] This embodiment describes how to determine the start and end times of the lyrics of the voice part with independent lyrics except for the first voice part in the application scenario of a multi-voice musical score. Specifically:

[0185] Judge whether the musical score file contains multiple voice part musical scores and whether there is a voice part with independent lyrics except for the first voice part;

[0186] If so, after determining the start and end times of each note corresponding to the eligible vocal score, based on the correspondence between each lyric of the eligible vocal score and the notes of the eligible vocal score within the same time period, and the proportional relationship between the start and end times and note durations of each note of the eligible vocal score, determine the start and end times of each lyric of the eligible vocal score. As Figure 14 shown, after obtaining the start and end times of each note of the second voice according to Embodiment 7, the start and end times of several lyrics corresponding to the second voice can be obtained;

[0187] The specific approach is as follows:

[0188] Figure 14 In the second voice, only the third and fourth measures have lyrics, and each lyric corresponds to one note. Then the start and end time of the corresponding note is the start and end time of the corresponding lyric. That is, the first lyric "la" corresponds to the second note 5 in the third measure of the second voice, and the start and end time of this note is the start and end time of the corresponding lyric; for another example, the second lyric "la" corresponds to the first note 5 in the fourth measure of the second voice, and the start and end time of this note is the start and end time of the corresponding lyric;

[0189] As Figure 15 shown in the musical score fragment of the song "Thoughts in the Silent Night", each measure of the second voice has lyrics, and the number of notes corresponding to each lyric is different. Then the start time of each lyric is the start time of the first note it corresponds to, and the end time of each lyric is the end time of the last note it corresponds to. For example, the second lyric "head" in the second measure corresponds to two notes arranged in sequence as 1 and 6. Then the start time of this lyric is the start time of the corresponding first note 1, and the end time of this lyric is the end time of the corresponding last note 6. For another example, the first lyric "moon" in the third measure corresponds to four notes arranged in sequence as 3, 5, 2, and 3. Then the start time of this lyric is the start time of the corresponding first note 3, and the end time of this lyric is the end time of the corresponding last note 3.

[0190] Embodiment 9

[0191] Please refer to Figure 2 , a spectrum-sound synchronization terminal, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements each step in any one of the spectrum-sound synchronization methods described in Embodiments 1 to 7 above.

[0192] In summary, a pitch synchronization method and a terminal provided by the present invention first determine the first start and end times of each word in the singing lyrics of the audio to be played and the third start and end times of each note in the pure note part according to the audio to be played. Then, according to the score file corresponding to the audio to be played, the first start and end times and the third start and end times, determine the second start and end times of each word in the singing lyrics corresponding to the score file and the fourth start and end times of each note in the pure note part. Next, correct the second start and end times and the fourth start and end times according to the audio to be played. Finally, determine the start and end times of each note in the score file according to the corrected second start and end times and the fourth start and end times. First, align each word in the lyrics and each note in the pure note part with each word in the singing lyrics of the audio to be played, each note in the pure note part, and each word in the singing lyrics corresponding to the score file. Then, further correct the start and end times of each aligned word and each note in the score file based on the audio to be played. Finally, determine the start and end times of each note in the score file based on the corrected start and end times of each word and each note, which can achieve the alignment of each sound of the audio to be played with each sound of its corresponding score file, improve the synchronization of the sound played during playback with the sound reached by the score, and is applicable to the playback of various songs, such as variable-speed songs, etc. Among them, the correction method can have different ways and can be freely combined, with high flexibility. And for the application scenario of multi-part score, once the score synchronization of one part score is completed, the other part scores can directly apply based on the completed one and directly perform score synchronization based on the completed part score, greatly improving the convenience and efficiency of score synchronization.

[0193] The above are only embodiments of the present invention, and do not limit the patent scope of the present invention. Any equivalent transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in the relevant technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A method for synchronizing music score and lyrics, characterized in that, it includes the steps of: S1. Determine the first start and end time of each word in the singing lyrics of the audio to be played according to the audio to be played; S2. Obtain the music score file corresponding to the audio to be played, and determine the second start and end time of each word in the singing lyrics corresponding to the music score file according to the music score file and the first start and end time, and align each word in the singing lyrics of the audio to be played with each word in the singing lyrics corresponding to the music score file; S3. Determine the start and end time of the corresponding notes in the music score file according to the second start and end time, specifically: determine the start and end time of the corresponding notes in the music score file based on the start and end time of each word in the aligned singing lyrics; The step of determining the second start and end time of each word in the singing lyrics corresponding to the music score file according to the music score file and the first start and end time includes: Unfold the singing lyrics corresponding to the music score file in chronological order according to the preset speed in the music score file; Adjust the singing lyrics corresponding to the music score file unfolded in chronological order according to the first start and end time of each word in the singing lyrics of the audio to be played to obtain the second start and end time of each word in the singing lyrics corresponding to the music score file.

2. A method for synchronizing music score and lyrics according to claim 1, characterized in that, the step S1 further includes: Determine the third start and end time of each note in the pure note part of the audio to be played according to the audio to be played; the step S3 further includes: Determine the fourth start and end time of each note in the pure note part corresponding to the music score file according to the music score file and the third start and end time.

3. A method for synchronizing music score and lyrics according to claim 2, characterized in that, before determining the start and end time of each note in the music score file in the step S3, it further includes: Correct the target start and end time according to the audio to be played, where the target start and end time includes at least one of the second start and end time, the fourth start and end time, and the start and end time of the corresponding note determined by the second start and end time.

4. A method for synchronizing music score and lyrics according to claim 3, characterized in that, the step of correcting the target start and end time according to the audio to be played includes: Convert the audio to be played into a waveform in the time domain; Determine the start point and end point of each target object of the audio to be played according to the waveform. When the target start and end time is the second start and end time, the target object is the word in the singing lyrics; when the target start and end time is the fourth start and end time, the target object is the note in the pure note part; when the target start and end time is the start and end time of the corresponding note determined by the second start and end time, the target object is the corresponding note determined by the second start and end time; Correct the target start and end time corresponding to each target object according to the start point and end point of each target object.

5. A method for synchronizing music score and lyrics according to claim 4, characterized in that, the step of determining the start point and end point of each target object of the audio to be played according to the waveform includes: Traverse each time point in the waveform in chronological order. For the traversed time point: Determine whether the amplitude of the waveform increases to more than a first preset ratio of the maximum amplitude of the waveform within a first preset time starting from the time point. If so, the time point is the starting point of a target object; If not, determine whether the amplitude of the waveform decreases to less than a second preset ratio of the maximum amplitude of the waveform within a second preset time starting from the time point. If so, the time point is the ending point of a target object; Otherwise, continue to traverse the next time point in the waveform until all time points in the waveform are traversed; Determine the starting point and ending point of each target object of the audio to be played according to the starting points and ending points of all target objects determined after the traversal is completed.

6. A spectral synchronization method according to claim 3, wherein, The correction of the target start and end time according to the audio to be played includes: Play the audio to be played, perform speech recognition or pitch recognition on the audio to be played, and determine the starting point and ending point of each target object in the singing lyrics or playing music of the audio to be played. When the target start and end time is the second start and end time, the target object is the word in the singing lyrics; when the target start and end time is the fourth start and end time, the target object is the note in the pure note part; when the target start and end time is the start and end time of the corresponding note determined by the second start and end time, the target object is the corresponding note determined by the second start and end time; Correct the target start and end time corresponding to each target object according to the starting point and ending point of each target object.

7. A spectral synchronization method according to claim 3, wherein, The correction of the target start and end time according to the audio to be played includes: Correspondingly display each target object of the audio to be played according to the target start and end time in the playing order. Adjacent target objects are distinguished by different color blocks. The start and end positions of the color blocks on the time axis represent the start and end time corresponding to each target object, and the length of the color blocks on the time axis represents the time length corresponding to each target object. When the target start and end time is the second start and end time, the target object is the word in the singing lyrics; when the target start and end time is the fourth start and end time, the target object is the note in the pure note part; when the target start and end time is the start and end time of the corresponding note determined by the second start and end time, the target object is the corresponding note determined by the second start and end time; Play the audio to be played, and synchronously display a progress bar on the color block during the playing process; Receive an adjustment instruction sent by the operating object according to the deviation between the played audio and the target object indicated by the progress bar; Correct the target start and end time according to the adjustment instruction.

8. A spectral synchronization method according to claim 7, wherein, The adjustment instruction includes the color block to be adjusted and the adjustment method; The correction of the target start and end time according to the adjustment instruction includes: Adjust the start and end time boundaries of the color block to be adjusted in the time axis direction according to the color block to be adjusted and the adjustment method; or drag the entire color block to be adjusted along the time axis direction; Determine the target start and end times according to the start and end time boundaries of the adjusted color block in the time axis direction.

9. A method for synchronizing music scores and sounds according to any one of claims 1 to 8, characterized in that the determining the start and end times of the corresponding notes in the music score file according to the second start and end times includes: determining the start and end times of the corresponding notes in the music score file according to the correspondence between the lyrics and the notes, the proportional relationship of the note durations, and the second start and end times.

10. A method for synchronizing music scores and sounds according to claim 9, characterized in that the determining the start and end times of each note in the music score file according to the correspondence between the lyrics and the notes, the proportional relationship of the note durations, and the second start and end times includes: performing the following steps for each word in the audio to be played: determining the number of notes corresponding to the word according to the correspondence between the lyrics and the notes; if the number of notes corresponding to the word is one, the start and end times of the note corresponding to the word are the second start and end times corresponding to the word; if the number of notes corresponding to the word is two or more, determining the proportional relationship of the note durations between the notes corresponding to the word; determining the start and end times corresponding to each note corresponding to the word according to the proportional relationship of the note durations and the second start and end times corresponding to the word.

11. A method for synchronizing music scores and sounds according to any one of claims 1 to 8 and 10, characterized in that the S3 further includes: judging whether the music score file contains multiple voice part music scores; if so, after determining the start and end times of each note corresponding to one of the first voice part music scores, determining the start and end times of each note corresponding to the remaining voice part music scores according to the correspondence between the remaining voice part music scores and the notes of the first voice part music score.

12. A method for synchronizing music scores and sounds according to claim 11, characterized in that the determining the start and end times of each note corresponding to the remaining voice part music scores according to the correspondence between the remaining voice part music scores and the notes of the first voice part music score includes: judging whether the relationship between the notes of the remaining voice part music scores and the notes of the first voice part music score is one-to-one and the note durations are the same within the same time period, if so, determining the start and end times of each note corresponding to the remaining voice part music scores according to the start and end times of each note of the first voice part music score within the same time period; if not, determining the start and end times of each note corresponding to the remaining voice part music scores according to the start and end times and note durations of each note of the first voice part music score within the same time period and the proportional relationship of the note durations of each note of the remaining voice part music scores.

13. A method for synchronizing music scores and sounds according to claim 11, characterized in that judging whether the music score file contains multiple voice part music scores and there is a voice part with independent lyrics except for the first voice part; If so, after determining the start and end times of each note corresponding to the eligible vocal score, according to the correspondence between each lyric of the eligible vocal score and the notes of the eligible vocal score within the same time period, and the proportional relationship between the start and end times and note durations of each note of the eligible vocal score, determine the start and end times of each lyric of the eligible vocal score.

14. A score-audio synchronization terminal, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, when the processor executes the computer program, it implements each step in a score-audio synchronization method as described in any one of claims 1 to 13.

Images