A lyrics timestamp generation method, device, equipment and medium

Abstract

The application discloses a song lyric timestamp generation method, device, equipment and medium, comprising: acquiring a plurality of target dry sound data corresponding to a same song; acquiring a target acoustic feature corresponding to each target dry sound data; determining a start time and an end time of each lyric in each target dry sound data based on the target acoustic feature, obtaining a first start time and a first end time; determining a start time and an end time of each character in each target dry sound data by using a target MFA model, obtaining a second start time and a second end time; and determining a song lyric timestamp of the song based on the first start time, the first end time, the second start time and the second end time corresponding to the plurality of target dry sound data. In this way, the finally determined song lyric timestamp fuses the time determined based on the acoustic features of the plurality of target dry sound data and the time determined by the MFA model, and the accuracy of the song lyric timestamp can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of voice processing, in particular to a lyrics timestamp generation method and device, equipment and medium. BACKGROUND

[0002] Currently, there are two methods for generating lyrics files in a song library, namely manual fine annotation and automatic generation based on original song separation dry sound. The former requires a large amount of labor cost and time cost, and the accuracy of the latter is not high, mainly because the original song separation dry sound itself has many incomplete spectrum and swallowing words, and the alignment result is difficult to guarantee in quality poor separation dry sound. In summary, in the process of realizing the present application, the inventors have found at least that the existing technology has the problem of low accuracy of generated lyrics timestamps. SUMMARY

[0003] Therefore, the purpose of the present application is to provide a lyrics timestamp generation method, device, equipment and medium, which can improve the accuracy of lyrics timestamps. The specific scheme is as follows:

[0004] In a first aspect, the present application discloses a lyrics timestamp generation method, comprising:

[0005] obtaining a plurality of target dry sound data corresponding to a same song;

[0006] obtaining a target acoustic feature corresponding to each of the target dry sound data;

[0007] determining a start time and an end time of each lyric in each of the target dry sound data based on the target acoustic feature, to obtain a first start time and a first end time;

[0008] determining a start time and an end time of each word in each of the target dry sound data by using a target MFA model, to obtain a second start time and a second end time;

[0009] determining a lyrics timestamp of the song based on the first start time, the first end time, the second start time and the second end time corresponding to a plurality of the target dry sound data.

[0010] Optionally, the obtaining a plurality of target dry sound data corresponding to a same song comprises:

[0011] obtaining a plurality of dry sound data corresponding to a same song;

[0012] determining whether each of the dry sound data meets a preset pitch standard and / or a preset timbre standard;

[0013] If yes, the corresponding dry sound data is determined as a target dry sound data.

[0014] Optionally, determining whether each of the dry sound data meets a preset pitch standard comprises:

[0015] determining a deviation between a fundamental frequency of each of the dry sound data and a melody of the song;

[0016] determining a pitch score of the dry sound data according to the deviation;

[0017] determining whether the pitch score is greater than a first preset threshold, if yes, determining that the corresponding dry sound data meets the preset pitch standard, otherwise, determining that the corresponding dry sound data does not meet the preset pitch standard.

[0018] Optionally, determining whether each of the dry sound data meets a preset timbre standard comprises:

[0019] determining a ratio between a harmonic energy and a non-harmonic energy of each of the dry sound data;

[0020] determining whether the ratio is greater than a second preset threshold, if yes, determining that the corresponding dry sound data meets the preset timbre standard, otherwise, determining that the corresponding dry sound data does not meet the preset timbre standard.

[0021] Optionally, the obtaining of the target acoustic feature corresponding to each of the target dry sound data comprises:

[0022] filtering each of the target dry sound data by using a low-pass filter to obtain a signal of each of the target dry sound data below a preset frequency;

[0023] determining an energy value of each of the signal;

[0024] Correspondingly, the determining of the start time and the end time of each of the lyrics in each of the target dry sound data based on the target acoustic feature to obtain the first start time and the first end time comprises:

[0025] determining a dry sound segment containing speech from the target dry sound data based on the energy value;

[0026] determining a start time of the dry sound segment as a start time of a corresponding lyric and determining an end time of the dry sound segment as an end time of the corresponding lyric to obtain the first start time and the first end time.

[0027] Optionally, the determining of the start time and the end time of each of the characters in each of the target dry sound data by using a target MFA model to obtain the second start time and the second end time comprises:

[0028] inputting all the dry sound segments of each of the target dry sound data into a target MFA model;

[0029] obtaining a start time and an end time of each word in each of the dry sound segments output by the target MFA model, to obtain a second start time and a second end time.

[0030] Optionally, the method further comprises:

[0031] obtaining a dry sound training data set; wherein the dry sound training data set comprises dry sound training data corresponding to an entire song;

[0032] segmenting dry sound segment data from the dry sound training data, to obtain a dry sound segment data set;

[0033] training an MFA model using the dry sound segment data set until the MFA model converges, and then determining the current MFA model as the target MFA model.

[0034] Optionally, determining the lyrics timestamp of the song based on the first start time, the first end time, the second start time and the second end time corresponding to a plurality of the target dry sound data comprises:

[0035] for any sentence of lyrics, determining a median of the first start time corresponding to the sentence of lyrics in a plurality of the target dry sound data, to obtain a first median, and determining a median of the first end time corresponding to the sentence of lyrics in a plurality of the target dry sound data, to obtain a second median;

[0036] determining the first median as the timestamp of the first word in the sentence of lyrics, and determining the second median as the timestamp of the last word in the sentence of lyrics;

[0037] determining a median of the second start time and a median of the second end time corresponding to a target word in the sentence of lyrics in a plurality of the target dry sound data, to obtain the timestamp of the target word;

[0038] wherein the target word is a word other than the first word and the last word in the sentence of lyrics.

[0039] In a third aspect, the present application discloses an electronic device, comprising:

[0040] a memory for saving a computer program;

[0041] a processor for executing the computer program to implement the lyrics timestamp generation method described above.

[0042] In a fourth aspect, the present application discloses a computer readable storage medium for storing a computer program, which, when executed by a processor, implements the aforementioned lyrics timestamp generation method.

[0043] It can be seen that the present application first acquires a plurality of target dry sound data corresponding to the same song, then acquires target acoustic features corresponding to each of the target dry sound data, then determines the start time and end time of each lyric in each of the target dry sound data based on the target acoustic features, obtains the first start time and the first end time, and determines the start time and end time of each character in each of the target dry sound data using a target MFA model, obtains the second start time and the second end time, and finally determines the lyrics timestamp of the song based on the first start time, the first end time, the second start time, and the second end time corresponding to a plurality of target dry sound data. That is, the present application determines the start time and end time of each lyric in the target dry sound data based on the acoustic features of a plurality of target dry sound data corresponding to the same song, and determines the start time and end time of each character in each of the target dry sound data using a target MFA model, and finally determines the final lyrics timestamp of the song using the time information determined by the two methods. Thus, the lyrics timestamp determined finally fuses the time determined based on the acoustic features of a plurality of target dry sound data and the time determined by the MFA model, which can improve the accuracy of the lyrics timestamp. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0045] Figure 1 A system framework schematic diagram applicable to the lyrics timestamp generation scheme provided by the present application;

[0046] Figure 2 A lyrics timestamp generation method flowchart disclosed by the present application;

[0047] Figure 3 A specific lyrics timestamp generation method flowchart disclosed by the present application;

[0048] Figure 4 A specific lyrics timestamp generation method flowchart disclosed by the present application;

[0049] Figure 5 A specific lyrics timestamp fusion schematic diagram disclosed by the present application;

[0050] Figure 6 A specific lyrics timestamp generation method flowchart disclosed in the present application;

[0051] Figure 7 A lyrics timestamp generation device structure schematic diagram provided in the present application;

[0052] Figure 8 An electronic device structure diagram provided in the present application. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0054] Currently, there are two generation methods for lyrics files in a song library, namely manual fine annotation and automatic generation based on original song separation dry sound. The former requires a large amount of labor cost and time cost, and the accuracy of the latter is not high, mainly because the original song separation dry sound itself has many incomplete spectrum and swallowing words, and the accuracy of the alignment result is difficult to guarantee on the poor quality of the separation dry sound. In summary, in the process of realizing the present application, the inventors have at least found that the existing technology has the problem of low accuracy of generated lyrics timestamp. Therefore, the embodiments of the present application disclose a lyrics timestamp generation scheme, which can improve the accuracy of lyrics timestamp.

[0055] In the lyrics timestamp generation scheme of the present application, the system framework used can be seen from Figure 1 The specific embodiments can include a background server and a plurality of user terminals in communication connection with the background server. The user terminals include but are not limited to tablet computers, notebook computers, smart phones, personal computers (PC), which are not limited here.

[0056] In the present application, the steps of the background server performing the lyrics timestamp generation method include: acquiring a plurality of target dry sound data corresponding to the same song; acquiring a target acoustic feature corresponding to each of the target dry sound data; determining the start time and end time of each lyric in each of the target dry sound data based on the target acoustic feature, obtaining a first start time and a first end time; determining the start time and end time of each character in each of the target dry sound data using a target MFA model, obtaining a second start time and a second end time; determining the lyrics timestamp of the song based on the first start time, the first end time, the second start time, and the second end time corresponding to a plurality of target dry sound data. When the user end plays the corresponding song, the background server pushes the lyrics of the song to the user end based on the lyrics timestamp of the song.

[0057] Referring to Figure 2 As shown in the drawings, the embodiments of the present application disclose a lyrics timestamp generation method, which comprises:

[0058] Step S11: acquiring a plurality of target dry sound data corresponding to the same song.

[0059] In a specific implementation, a plurality of dry sound data corresponding to the same song can be acquired; it is determined whether each of the dry sound data meets a preset pitch standard and / or a preset timbre standard; if yes, the corresponding dry sound data is determined as a target dry sound data.

[0060] The embodiments of the present application can acquire a plurality of dry sound data of the same song, for example, acquiring the dry sound data of a plurality of users of the same song through a certain karaoke APP. Since the quality of the dry sound data of the users corresponding to each song is uneven, the low-quality dry sound data has a negative effect on the final generation of the lyrics timestamp, therefore, the embodiments of the present application screen the plurality of dry sound data, which can be screened from two dimensions of timbre and / or pitch.

[0061] Among them, it is determined whether each of the dry sound data meets a preset pitch standard, which specifically includes: determining the deviation between the fundamental frequency of each of the dry sound data and the melody of the song; determining the pitch score of the dry sound data according to the deviation; determining whether the pitch score is greater than a first preset threshold, if yes, determining that the corresponding dry sound data meets the preset pitch standard, otherwise, determining that the corresponding dry sound data does not meet the preset pitch standard.

[0062] In specific embodiments, the dry sound data can be evaluated by a preset pitch scoring algorithm. Specifically, the pitch score can be determined by comparing the deviation between the fundamental frequency information of the dry sound data and the melody in the MIDI (Musical Instrument Digital Interface) file of the song. The smaller the deviation, the higher the pitch score. It can be understood that the melody in the MIDI file is a melody with standard pitch.

[0063] In addition, determining whether each dry sound data meets a preset timbre standard includes: determining the ratio between the harmonic energy and the non-harmonic energy of each dry sound data; determining whether the ratio is greater than a second preset threshold value, if yes, determining that the corresponding dry sound data meets the preset timbre standard, otherwise, determining that the corresponding dry sound data does not meet the preset timbre standard.

[0064] That is, the embodiments of the present application can evaluate the timbre of the dry sound data by analyzing the ratio of the harmonic energy and the non-harmonic energy of the dry sound data. The larger the ratio, the cleaner the dry sound data.

[0065] In this way, through the evaluation of the two dimensions of pitch and / or timbre, a number of high-quality dry sounds that meet the conditions can be finally screened out.

[0066] Step S12: Obtain the target acoustic features corresponding to each of the target dry sound data.

[0067] Step S13: Determine the start time and the end time of each lyric in each of the target dry sound data based on the target acoustic features, to obtain the first start time and the first end time.

[0068] That is, the embodiments of the present application first obtain the acoustic features of the target dry sound data, and then determine the start time and the end time of each lyric based on the acoustic features.

[0069] Step S14: Determine the start time and the end time of each character in each of the target dry sound data by using a target MFA (Montreal Forced Aligner) model, to obtain the second start time and the second end time.

[0070] Step S15: Determine the lyric timestamp of the song based on the first start time, the first end time, the second start time, and the second end time corresponding to a plurality of target dry sound data.

[0071] It should be noted that in the prior art, the original song is used for separating the dry sound, although the sound accompaniment separation technology has made good progress in the academic and industrial circles, but it still cannot avoid the problem of uneven quality of dry sound of different genres and different songs. Most of them are missing spectrum and swallowing words, so the performance of the acoustic model on the separated dry sound will be greatly reduced. The target dry sound data in the embodiment of the application is high-quality target dry sound data of multiple users after screening, the start time and end time of each lyric determined based on the acoustic characteristics of multiple high-quality target dry sound data of the same song, and the start time and end time of each word of the target dry sound data determined by the MFA model, determine the final lyric timestamp, which avoids the problem of insufficient quality of the dry sound separated by the acoustic model, affecting the accuracy of the final lyric timestamp information, and can accurately determine the lyric timestamp.

[0072] It can be seen that the embodiment of the application first acquires multiple target dry sound data corresponding to the same song, then acquires target acoustic characteristics corresponding to each of the target dry sound data, then determines the start time and end time of each lyric in each of the target dry sound data based on the target acoustic characteristics, obtains the first start time and the first end time, and determines the start time and end time of each word in each of the target dry sound data using the target MFA model, obtains the second start time and the second end time, and finally determines the lyric timestamp of the song based on the first start time, the first end time, the second start time and the second end time corresponding to multiple target dry sound data. That is, the embodiment of the application determines the start time and end time of each lyric in the target dry sound data based on the acoustic characteristics of multiple target dry sound data corresponding to the same song, and determines the start time and end time of each word in each of the target dry sound data using the target MFA model, and finally determines the final lyric timestamp of the song using the time information determined by the two methods. In this way, the final lyric timestamp fuses the time determined based on the acoustic characteristics of multiple target dry sound data and the time determined by the MFA model, which can improve the accuracy of the lyric timestamp.

[0073] Referring to Figure 3 The embodiment of the application discloses a specific lyric timestamp generation method, which comprises:

[0074] Step S21: acquiring multiple target dry sound data corresponding to the same song.

[0075] For the specific implementation process of the above step S21, please refer to the content disclosed in the foregoing embodiments, which will not be repeated here.

[0076] Step S22: filtering each of the target dry sound data by using a low-pass filter to obtain a signal of each of the target dry sound data below a preset frequency.

[0077] Step S23: determining an energy value of each of the signals.

[0078] Step S24: determining a dry sound segment containing speech from the target dry sound data based on the energy value.

[0079] It should be noted that, for a speech signal, the energy contained in the frequency band below 2KHz is higher, and the noise has higher energy in the frequency band of 2-4KHz or above 4KHz than in the frequency band of 0-2HKz. Therefore, the target dry sound data is first filtered by using a low-pass filter to filter out the signal below 2K, and then when the energy of the signal is greater than a certain threshold, it can be considered that there is speech.

[0080] In a specific embodiment, for any of the signals, an energy value of each frame data of the signal can be determined, a target signal segment is determined from the signal, wherein the target signal segment is a signal segment whose ratio of the number of frames with energy values greater than a preset energy threshold to the total number of frames of the signal segment is greater than a preset proportion threshold, and a segment in the target dry sound data corresponding to the target signal segment is determined as a dry sound segment containing speech.

[0081] It should be noted that, if the target dry sound data is data obtained by filtering according to sound quality standards, the energy intensity-based speech detection method in the embodiment has better effect for dry sound data with high signal-to-noise ratio.

[0082] Step S25: determining a start time of the dry sound segment as a start time of a corresponding sentence of lyrics, and determining an end time of the dry sound segment as an end time of the corresponding sentence of lyrics, to obtain a first start time and a first end time.

[0083] That is, the embodiment accurately determines the start time and the end time of each sentence of lyrics in the manner of VAD (Voice Activity Detection).

[0084] Step S26: determining a start time and an end time of each character in each of the target dry sound data by using a target MFA model, to obtain a second start time and a second end time.

[0085] Step S27: determining a lyrics timestamp of the song based on the first start time, the first end time, the second start time, and the second end time corresponding to a plurality of the target dry sound data.

[0086] It can be seen that the embodiment of the present application filters each target dry sound data by using a low-pass filter to obtain a signal of each target dry sound data below a preset frequency, thereby excluding the interference of noise. Then, the energy value of each signal is determined, and a dry sound segment containing speech is determined from the target dry sound data based on the energy value, so as to accurately determine the start time and end time of each sentence of lyrics, obtain a VAD result, and fuse the VAD result and the MFA result, thereby improving the accuracy of the lyrics timestamp.

[0087] Referring to Figure 4 As shown in the drawings, the embodiment of the present application discloses a specific lyrics timestamp generation method, comprising:

[0088] Step S31: obtaining dry sound training data set; wherein the dry sound training data set comprises dry sound training data corresponding to a whole song.

[0089] In a specific implementation, the dry sound training data can be data filtered by using a preset pitch standard and / or a preset timbre standard. For a specific filtering process, reference can be made to the foregoing embodiment disclosed content, which will not be repeated here.

[0090] Step S32: cutting out dry sound segment data from the dry sound training data to obtain a dry sound segment data set.

[0091] In a specific implementation, each dry sound training data can be filtered by using a low-pass filter to obtain a signal of each dry sound training data below a preset frequency; the energy value of each signal is determined; and dry sound segment data containing speech is determined from the dry sound training data based on the energy value, and the dry sound segment data containing speech is cut out from the dry sound training data to obtain the dry sound segment data set.

[0092] For a specific process of the dry sound segment data containing speech, reference can be made to the foregoing embodiment disclosed process of determining the dry sound segment containing speech from the target dry sound data.

[0093] It should be pointed out that the dry sound duration of a song is usually 3-5 minutes, and if local deviation occurs, it will cause great interference to the whole. Therefore, cutting the dry sound can improve the robustness and also reduce the model training complexity.

[0094] Step S33: training the MFA model by using the dry sound segment data set until the MFA model converges, and then determining the current MFA model as the target MFA model.

[0095] The alignment model used in the embodiments of the present application is an MFA forced alignment tool based on kaldi. First, a mapping relationship from Chinese to pinyin (for example, "Han->han, Zi->zi") and from pinyin to phonemes (han->[h,an], zi->[z,i]) is constructed, wherein the pinyin without initial consonant sound needs to be constructed by a special mark (yu->[_u,u]). When training the MFA model, the dry sound segment, the pinyin of the lyrics segment, and the mapping table from pinyin to phonemes are taken as input, and after training iterations, the MFA model converges to obtain the target MFA model.

[0096] Step S34: Obtain a plurality of target dry sound data corresponding to the same song.

[0097] Step S35: Obtain a target acoustic feature corresponding to each of the target dry sound data.

[0098] Step S36: Determine the start time and end time of each sentence of lyrics in each of the target dry sound data based on the target acoustic feature, to obtain a first start time and a first end time.

[0099] For the specific implementation process of steps S34-S36, reference can be made to the content disclosed in the foregoing embodiments, which will not be described here again.

[0100] Moreover, steps S34-S36 and steps S31-S33 do not have a sequence.

[0101] Step S37: Input all dry sound segments of each of the target dry sound data to the target MFA model.

[0102] That is, referring to the content disclosed in the foregoing embodiments, the dry sound segments containing speech are determined from the target dry sound data, and the dry sound segments containing speech are segmented from the target dry sound data and input to the target MFA model.

[0103] In the specific implementation process, all dry sound segments of each of the target dry sound data, the pinyin segments of the lyrics corresponding to the dry sound segments, and the mapping table from pinyin to phonemes are input to the target MFA model.

[0104] Step S38: Obtain the start time and end time of each character in each of the dry sound segments output by the target MFA model, to obtain a second start time and a second end time.

[0105] Step S39: Determine the lyrics timestamp of the song based on the first start time, the first end time, the second start time, and the second end time corresponding to a plurality of target dry sound data.

[0106] In a specific embodiment, for any sentence of lyrics, the median of the first start time corresponding to the sentence of lyrics in multiple target dry sound data can be determined to obtain a first median, and the median of the first end time corresponding to the sentence of lyrics in multiple target dry sound data can be determined to obtain a second median; the median of the second end time corresponding to the first word in the sentence of lyrics in multiple target dry sound data can be determined to obtain a third median, and the median of the second start time corresponding to the last word in the sentence of lyrics in multiple target dry sound data can be determined to obtain a fourth median; the first median and the third median are determined as the timestamp of the first word in the sentence of lyrics, and the second median and the fourth median are determined as the timestamp of the last word in the sentence of lyrics; the median of the second start time and the median of the second end time corresponding to the target word in the sentence of lyrics in multiple target dry sound data can be determined to obtain the timestamp of the target word; wherein, the target word is a word other than the first word and the last word in the sentence of lyrics.

[0107] It should be pointed out that the target MFA model obtained through training can analyze the timestamp information of each word in the dry voice segment, but on the one hand, the user's dry voice segment may have some problems with inaccurate rhythm, and on the other hand, the MFA model also has the problem of insufficient alignment accuracy. Therefore, information fusion is required. In the embodiment of the present application, for each line of lyrics, the start time in the timestamp of the first word and the end time in the timestamp of the last word are the median of the VAD results of all corresponding dry voice segments, and the end time of the first word, the start time of the last word, and the timestamps of other words are the median of the MFA results of all dry voice segments. In this way, high-precision lyrics timestamps can be obtained.

[0108] For example, see Figure 5 As shown, Figure 5 This is a specific diagram of lyrics timestamp fusion disclosed in an embodiment of the present application. The same line of lyrics corresponds to three dry voice segments, dry voice segment 1, dry voice segment 2, and dry voice segment 3. The result data of the three dry voice segments are median filtered. Specifically, the start time of the first word and the end time of the last word of the lyrics adopt the median of the VAD results of the three dry voice segments, and the end time of the first word, the start time of the last word, and the timestamps of other words are the median of the MFA results of the three dry voice segments.

[0109] For example, see Figure 6 As shown, Figure 6 This is a flowchart of a specific method for generating lyrics timestamps disclosed in an embodiment of the present application. In the training phase, multiple user dry voices corresponding to the same song and lyrics information are first obtained. The lyrics information is the lyrics text. Then, based on the preset pitch standard and / or preset sound quality standard, the target dry voice data is filtered out from the multiple user dry voices.Figure 6 The dry voice 1, the dry voice 2 and the dry voice 3 are included in the dry voice 1, the dry voice 2 and the dry voice 3, and in the specific implementation process, the number of dry voices is not limited to 3. The VAD algorithm analyzes the start and end times of each sentence of lyrics by using the acoustic characteristics of the dry voice data, and divides a complete dry voice into multiple dry voice segments, that is, the dry voice segments containing the voice and the start and end times of the sentence of lyrics corresponding to the dry voice segments are determined based on the preset VAD algorithm, and the dry voice segments are divided. The specific VAD algorithm can refer to the content disclosed in the foregoing embodiments. Then, the MFA model is trained by using the dry voice segments, until the model converges, and the target MFA model is obtained. In the application stage, similarly, the multiple user dry voices of the song to be generated timestamp and the lyrics information are obtained, and then the target dry voice data is selected from the multiple user dry voices based on the preset pitch standard and / or preset timbre standard, Figure 6 The dry voice 1, the dry voice 2 and the dry voice 3 are included in the dry voice 1, the dry voice 2 and the dry voice 3, and in the specific implementation process, the number of dry voices is not limited to 3. The dry voice segments containing the voice are determined based on the preset VAD algorithm, the VAD result, that is, the start and end times of the sentence of lyrics corresponding to the dry voice segments are obtained, and the dry voice segments are divided. The target MFA model trained is input, and then the VAD result and the alignment result 1, the alignment result 2 and the alignment result 3 of the MFA are fused to obtain the final lyrics timestamp.

[0110] It can be seen that the MFA model is trained by using the dry voice segments in the embodiments of the present application, the robustness of the model is improved, and the complexity of model training is reduced. Moreover, the dry voice segments containing the voice are determined from the target dry voice data, and then the start and end times of each sentence of lyrics are accurately determined to obtain the VAD result. The VAD result and the MFA result are fused, the start time of the first word and the end time of the last word of each sentence of lyrics are the median of the VAD results corresponding to multiple target dry voice data, and the median of the MFA result is used for the rest of the words, which overcomes the problem that the timestamp information obtained by a single dry voice through the MFA model is usually not accurate enough on long vowels and boundary information, thereby improving the accuracy of the lyrics timestamp.

[0111] Next, the technical solutions of the present application will be described by taking a certain music APP as an example.

[0112] Suppose that a user installs the music APP on the user terminal, opens the music APP, finds the song “Your Appearance”, clicks to sing the song, the APP plays the accompaniment of “Your Appearance”, and displays the corresponding lyrics based on the timestamp information of “Your Appearance”. The user sings the song by referring to the lyrics.

[0113] The background server of the music APP is used to generate the lyrics timestamp of Your Appearance, and the specific process includes: obtaining dry sound data of Your Appearance sung by multiple users, then screening multiple high-quality target dry sound data from the dry sound data based on preset pitch standards and preset tone standards, then filtering each target dry sound data by using a low-pass filter to obtain signals with a frequency below 2KHZ in each target dry sound data; determining the energy value of each signal, determining a dry sound segment containing speech from the target dry sound data based on the energy value, determining the starting time of the dry sound segment as the starting time of a corresponding sentence of lyrics, determining the ending time of the dry sound segment as the ending time of the corresponding sentence of lyrics, obtaining the first starting time and the first ending time, and cutting out the dry sound segment and inputting it to the target MFA model to determine the starting time and the ending time of each word in each target dry sound data by using the target MFA model, obtaining the second starting time and the second ending time, for any sentence of lyrics, determining the median of the first starting time corresponding to the sentence of lyrics in multiple target dry sound data to obtain the first median, and determining the median of the first ending time corresponding to the sentence of lyrics in multiple target dry sound data to obtain the second median; determining the median of the second ending time corresponding to the first word in the sentence of lyrics in multiple target dry sound data to obtain the third median, and determining the median of the second starting time corresponding to the last word in the sentence of lyrics in multiple target dry sound data to obtain the fourth median; determining the first word in the sentence of lyrics as the timestamp of the first word by using the first median and the third median, and determining the last word in the sentence of lyrics as the timestamp of the last word by using the second median and the fourth median; determining the median of the second starting time and the median of the second ending time corresponding to the target word in the sentence of lyrics in multiple target dry sound data to obtain the timestamp of the target word; wherein the target word is a word other than the first word and the last word in the sentence of lyrics. That is, the VAD result and the MFA are fused to obtain the final lyrics timestamp. For example, three target dry sound data are screened out, and the corresponding dry sound segments are cut out, wherein the first starting time of the three dry sound segments corresponding to the sentence of lyrics of I heard the sound of who passing by is 0 minute 13.58 seconds, 0 minute 14.01 seconds, and 0 minute 14.34 seconds, the median 0 minute 14.01 second is taken as the starting time of I, the first ending time is 0 minute 16.59 seconds, 0 minute 17.01 seconds, and 0 minute 17.34 seconds, the median 0 minute 17.01 second is taken as the ending time of tone, the ending time of I is taken as the median of the corresponding second ending time, and the starting time of tone is taken as the median of the corresponding first starting time. The timestamps of the words of listen, to, pass, come, of, who, of, and sound are the medians of the second starting time and the second ending time of the corresponding dry sound segments.

[0114] In this way, the high-precision lyrics timestamp is obtained, which can help the user to better grasp the singing rhythm, thereby improving the user experience, and can also be used as a reference standard for singing rhythm evaluation.

[0115] Referring to Figure 7 As shown in the drawings, the embodiment of the present application discloses a lyrics timestamp generation device, comprising:

[0116] The target dry sound data acquisition module 11 is configured to acquire a plurality of target dry sound data corresponding to the same song.

[0117] The target acoustic feature acquisition module 12 is configured to acquire a target acoustic feature corresponding to each of the target dry sound data.

[0118] The first time information acquisition module 13 is configured to determine a start time and an end time of each lyric in each of the target dry sound data based on the target acoustic feature, to obtain a first start time and a first end time.

[0119] The second time information acquisition module 14 is configured to determine a start time and an end time of each character in each of the target dry sound data by using a target MFA model, to obtain a second start time and a second end time.

[0120] The lyrics timestamp generation module 15 is configured to determine a lyrics timestamp of the song based on the first start time, the first end time, the second start time, and the second end time corresponding to the plurality of target dry sound data.

[0121] As can be seen, the embodiment of the present application first acquires a plurality of target dry sound data corresponding to the same song, then acquires a target acoustic feature corresponding to each of the target dry sound data, and then determines a start time and an end time of each lyric in each of the target dry sound data based on the target acoustic feature, to obtain a first start time and a first end time. Furthermore, a start time and an end time of each character in each of the target dry sound data are determined by using a target MFA model, to obtain a second start time and a second end time. Finally, a lyrics timestamp of the song is determined based on the first start time, the first end time, the second start time, and the second end time corresponding to the plurality of target dry sound data. That is, the embodiment of the present application determines a start time and an end time of each lyric in the target dry sound data based on the acoustic features of the plurality of target dry sound data corresponding to the same song, and determines a start time and an end time of each character in each of the target dry sound data by using a target MFA model. Finally, the lyrics timestamp of the song is determined based on the time information determined by the two methods. In this way, the lyrics timestamp determined finally combines the time determined based on the acoustic features of the plurality of target dry sound data and the time determined by the MFA model, which can improve the accuracy of the lyrics timestamp.

[0122] The target dry sound data acquisition module 11 specifically includes:

[0123] The dry sound data acquisition sub-module is configured to acquire a plurality of dry sound data corresponding to the same song.

[0124] The dry sound data judgment sub-module is configured to determine whether each dry sound data meets a preset pitch standard and / or a preset tone standard.

[0125] The target dry sound data determination sub-module is configured to determine the corresponding dry sound data as the target dry sound data if the dry sound data judgment sub-module result is met.

[0126] In a specific embodiment, the dry sound data judgment sub-module is specifically configured to:

[0127] determine the deviation between the fundamental frequency of each dry sound data and the melody of the song;

[0128] determine the pitch score of the dry sound data according to the deviation;

[0129] determine whether the pitch score is greater than a first preset threshold value, if yes, determine that the corresponding dry sound data meets the preset pitch standard, otherwise, determine that the corresponding dry sound data does not meet the preset pitch standard.

[0130] In a specific embodiment, the dry sound data judgment sub-module is specifically configured to:

[0131] determine the ratio between the harmonic energy and the non-harmonic energy of each dry sound data;

[0132] determine whether the ratio is greater than a second preset threshold value, if yes, determine that the corresponding dry sound data meets the preset tone standard, otherwise, determine that the corresponding dry sound data does not meet the preset tone standard.

[0133] The target acoustic feature acquisition module 12 is specifically configured to:

[0134] filter each target dry sound data using a low-pass filter to obtain a signal of each target dry sound data below a preset frequency;

[0135] determine the energy value of each signal;

[0136] Correspondingly, the first time information acquisition module 13 is specifically configured to:

[0137] determine a dry sound segment containing speech from the target dry sound data based on the energy value;

[0138] The start time of the dry sound segment is determined as the start time of the corresponding sentence of lyrics, and the end time of the dry sound segment is determined as the end time of the corresponding sentence of lyrics, to obtain a first start time and a first end time.

[0139] Further, the second time information acquisition module 14 is specifically configured to:

[0140] inputting all the dry sound segments of each of the target dry sound data into a target MFA model;

[0141] acquiring the start time and the end time of each word in each of the dry sound segments output by the target MFA model to obtain a second start time and a second end time.

[0142] In addition, the device further comprises a model training module configured to:

[0143] acquire dry sound training data sets; wherein the dry sound training data sets comprise dry sound training data corresponding to an entire song;

[0144] cutting dry sound segment data from the dry sound training data to obtain a dry sound segment data set;

[0145] training the MFA model using the dry sound segment data set until the MFA model converges, and then determining the current MFA model as the target MFA model.

[0146] In a specific embodiment, the lyrics timestamp generation module 15 is specifically configured to:

[0147] for any sentence of lyrics, determining the median of the first start time corresponding to the sentence of lyrics in a plurality of target dry sound data to obtain a first median, and determining the median of the first end time corresponding to the sentence of lyrics in a plurality of target dry sound data to obtain a second median;

[0148] determining the median of the second end time corresponding to the first word in the sentence of lyrics in a plurality of target dry sound data to obtain a third median, and determining the median of the second start time corresponding to the last word in the sentence of lyrics in a plurality of target dry sound data to obtain a fourth median;

[0149] determining the first median and the third median as the timestamp of the first word in the sentence of lyrics, and determining the second median and the fourth median as the timestamp of the last word in the sentence of lyrics;

[0150] determining the median of the second start time and the median of the second end time corresponding to the target word in the sentence of lyrics in a plurality of target dry sound data to obtain the timestamp of the target word;

[0151] The target word is a word in the lyrics of the sentence other than the first word and the last word.

[0152] Further, the embodiment of the present application further provides an electronic device. Figure 8 The electronic device 20 is shown in the structural diagram of the electronic device 20 according to an exemplary embodiment, and the content in the diagram cannot be considered as any limitation on the use range of the present application.

[0153] Figure 8 The electronic device 20 is shown in the structural diagram of the electronic device 20 according to an exemplary embodiment, and the content in the diagram cannot be considered as any limitation on the use range of the present application.

[0154] In the embodiment, the power supply 23 is configured to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 is configured to create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which is not limited here; the input and output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which is not limited here.

[0155] In addition, the memory 22 as a carrier of resource storage can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon can include an operating system 221, a computer program 222 and audio data 223, etc., and the storage mode can be temporary storage or permanent storage.

[0156] The operating system 221 is configured to manage and control each hardware device on the electronic device 20 and the computer program 222, so as to realize the operation and processing of the processor 21 on the audio data 223 in the memory 22, and the operating system 221 can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of completing the lyrics timestamp generation method executed by the electronic device 20 disclosed in any of the preceding embodiments, the computer program 222 can further include a computer program capable of completing other specific work.

[0157] Further, the application further discloses a storage medium, wherein the storage medium stores a computer program, and the computer program is loaded and executed by a processor to realize the steps of the song lyric timestamp generation method disclosed in any of the preceding embodiments.

[0158] The various embodiments are described in the present specification in progressive order, and each embodiment focuses on the difference from other embodiments. The same or similar parts among the various embodiments can be mutually referred to. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0159] Finally, it should be noted that in this document, the terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0160] The above provides a detailed introduction to the song lyric timestamp generation method, device, equipment and medium provided by the application. The principle and implementation mode of the application are described by applying specific examples in this document. The above embodiment description is only used to help understand the method and core idea of the application; at the same time, for the general technical personnel in the art, according to the idea of the application, the specific implementation mode and application range will be changed; in view of the above, the content of the specification should not be understood as the limitation of the application.

Claims

1. A method for generating lyrics timestamps, characterized in that: include: Obtain multiple target dry sound data corresponding to the same song; Obtaining target acoustic features corresponding to each target dry sound data; Determine the start time and end time of each line of lyrics in each target dry sound data based on the target acoustic feature, and obtain a first start time and a first end time; Determining the start time and end time of each word in each target dry voice data using the target MFA model to obtain a second start time and a second end time; Determine a lyrics timestamp of the song based on the first start time, the first end time, the second start time, and the second end time corresponding to the plurality of target dry sound data; Among them, the target dry sound data is the target dry sound data of multiple users that have been screened and are of high quality. The obtaining of multiple target dry sound data corresponding to the same song includes: obtaining multiple dry sound data corresponding to the same song; judging whether each of the dry sound data meets the preset pitch standard and / or preset sound quality standard; if so, determining the corresponding dry sound data as the target dry sound data.

2. The method for generating lyrics timestamp according to claim 1, wherein: Determining whether each dry sound data meets a preset pitch standard includes: determining a deviation between a fundamental frequency of each of the dry sound data and a melody of the song; determining a pitch score of the dry sound data according to the deviation; Determine whether the pitch score is greater than a first preset threshold value. If so, determine that the corresponding dry sound data meets the preset pitch standard. Otherwise, determine that the corresponding dry sound data does not meet the preset pitch standard.

3. The method for generating lyrics timestamp according to claim 1, wherein: Determining whether each dry sound data meets a preset sound quality standard includes: determining a ratio between harmonic energy and non-harmonic energy of each dry sound data; Determine whether the ratio is greater than a second preset threshold; if so, determine that the corresponding dry sound data meets the preset sound quality standard; otherwise, determine that the corresponding dry sound data does not meet the preset sound quality standard.

4. The method for generating lyrics timestamp according to claim 1, wherein: The acquiring the target acoustic feature corresponding to each target dry sound data includes: Filtering each of the target dry sound data using a low-pass filter to obtain a signal of each of the target dry sound data below a preset frequency; determining an energy value of each of said signals; Accordingly, determining the start time and end time of each line of lyrics in each target dry sound data based on the target acoustic feature to obtain a first start time and a first end time includes: determining a dry sound segment containing speech from the target dry sound data based on the energy value; The start time of the dry sound segment is determined as the start time of a corresponding line of lyrics, and the end time of the dry sound segment is determined as the end time of a corresponding line of lyrics, to obtain a first start time and a first end time.

5. The method for generating lyrics timestamp according to claim 4, characterized in that: The method of determining the start time and the end time of each word in each target dry voice data by using the target MFA model to obtain a second start time and a second end time includes: Inputting all the dry sound segments of each target dry sound data into a target MFA model; The start time and end time of each word in each dry voice segment output by the target MFA model are obtained to obtain a second start time and a second end time.

6. The method for generating lyrics timestamp according to claim 1, characterized in that: Also includes: Obtain a dry voice training data set; wherein the dry voice training data set includes dry voice training data corresponding to the entire song; Segmenting dry sound segment data from the dry sound training data to obtain a dry sound segment data set; The MFA model is trained using the dry sound segment dataset until the MFA model converges, and the current MFA model is determined as the target MFA model.

7. The method for generating lyrics timestamps according to any one of claims 1 to 6, characterized in that: The determining the lyrics timestamp of the song based on the first start time, the first end time, the second start time, and the second end time corresponding to the plurality of target dry sound data includes: For any line of lyrics, determine the median of the first start time corresponding to the line of lyrics in the plurality of target dry voice data to obtain a first median, and determine the median of the first end time corresponding to the line of lyrics in the plurality of target dry voice data to obtain a second median; Determine the median of the second end time corresponding to the first word in the sentence of lyrics in the plurality of target dry voice data to obtain a third median, and determine the median of the second start time corresponding to the last word in the sentence of lyrics in the plurality of target dry voice data to obtain a fourth median; Determining the first median and the third median as the timestamp of the first word in the sentence of lyrics, and determining the second median and the fourth median as the timestamp of the last word in the sentence of lyrics; Determine the median of the second start time and the median of the second end time corresponding to the target word in the lyrics of the sentence in the plurality of target dry voice data to obtain the timestamp of the target word; The target word is a word other than the first word and the last word in the lyrics.

8. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the method for generating lyrics timestamps as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that Used to store a computer program, which, when executed by a processor, implements the lyrics timestamp generation method according to any one of claims 1 to 7.

Images